Service/mirrors

Introduction
A content delivery network or Content Delivery Network (CDN) is a system of computers containing copies of data, placed at various points in a network so as to maximize bandwidth for access to the data from clients throughout the network. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, thereby causing a bottleneck near that server.

Goals

 * Reduce bandwidth at primary download server.
 * Improve quality of service for users.
 * Move content closer to users, thus reducing latency.

Architecture
The Sugar Labs Content Delivery Network uses MirrorBrain as a redirector. The redirector, which lives in a Sugar Labs data center, keeps track of which files are available on which mirror. When a user requests a file, the redirector points the user to the correct mirror and automatically starts the file download.

Mirrors
The current list of available mirrors is available at http://mirrors.sugarlabs.org/

Bandwidth
To run a mirror you need a lot of bandwidth! You should look at the total bandwidth used by all the mirrors.

If you have trouble with bandwidth, you should look at CloudFlare.

HDD Space
Hosting a mirror takes a lot of space. If you don't have a lot of space you can only choose to mirror some parts. For example exclude all directories but the activities (~13gb):

rsync -avzh rsync://download.sugarlabs.org/pub --exclude 'dextrose' --exclude 'hexoquinasa' --exclude 'images' --exclude 'sources' --exclude 'docs' --exclude 'packages' --exclude 'soas' /rsync/download.sugarlabs.org

For mirror administrators
All you need is a web server with enough bandwidth to serve the files. To set up a new mirror, the site administrator needs to:


 * First lets make a directory to store the data:

mkdir /rsync mkdir /rsync/download.sugarlabs.org


 * Then lets use rsync to download the data (warning: takes a long time)

rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org


 * Save the rsync command as a shell script and make it executable:

echo "rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org" > /rsync/download.sugarlabs.org/sync.sh chmod 774 /rsync/download.sugarlabs.org/sync.sh

echo "0 */2 * * * /rsync/download.sugarlabs.org/sync.sh" > asloSyncCronJob.txt crontab asloSyncCronJob.txt
 * Then lets make this to sync automatically. We can use a cron job to do that.  You could make sync every 2 hours:

If you don't want it to sync every 2 hours, have a look at a cron tutorial to change that value.


 * Publish the files via HTTP. Look at your http server documentation on how to do that.  You could set up a virtual host to serve these files: in nginx in apache


 * Setup a rsync mirror so we can view the status of your mirror. To do so, create a rsyncd.conf file and open it:

sudo nano rsyncd.conf

Then insert the following config:

log file = /rsync/log

[sugarlabs] path = /rsync/download.sugarlabs.org comment = PUT SOME INFORMATION HERE - LIKE A MOTD read only = true list = yes

Save and quit nano. Then start rsyncd so it can serve your files:

rsync --daemon --config=/etc/rsyncd.conf


 * Alert the Sugar Labs System Administrators that they would like their mirror into rotation, including the following information in the request:
 * Name and URL of the mirror operator (e.g. organization)
 * Name and email address of the administrative contact
 * ISO 3166-1 alpha-2 country code of the server location
 * HTTP base URL of the files on the mirror (typically http://mirrors.example.org/sugarlabs/)
 * rsync base URL of the files on the mirror (typically rsync://mirrors.example.org/sugarlabs/)

Please contact sysadmin AT sugarlabs DOT org if you are interested in hosting a mirror.

For Sugar Labs sysadmins
To add a new mirror to the MirrorBrain redirector:

sudo -u mirrorbrain mb new --operator-name \ --operator-url  -a -e \ -c -H  -R  -F  sudo -u mirrorbrain mb scan -e mb export --format=mirmon-apache | sudo -u mirrorbrain tee /srv/mirrorbrain/mirmon/mirrorlist-export sudo -u mirrorbrain mirmon -v -get all -c /etc/mirmon.conf
 * Choose a name for the mirror, usually the host name.
 * Register the mirror with MirrorBrain:
 * Scan and enable the mirror:
 * Export the list of mirrors for mirmon (a hourly cronjob does this, but if you don't want to wait...):
 * Finally, re-run mirmon to ensure it can check the health of the mirror (this is also done by a cronjob, but our patience is very short):