Service/mirrors
Introduction
A content delivery network or Content Delivery Network (CDN) is a system of computers containing copies of data, placed at various points in a network so as to maximize bandwidth for access to the data from clients throughout the network. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, thereby causing a bottleneck near that server.
Goals
- Reduce bandwidth at primary download server.
- Improve quality of service for users.
- Move content closer to users, thus reducing latency.
Architecture
The Sugar Labs Content Delivery Network uses MirrorBrain as a redirector. The redirector, which lives in a Sugar Labs data center, keeps track of which files are available on which mirror. When a user requests a file, the redirector points the user to the correct mirror and automatically starts the file download.
Mirrors
The current list of available mirrors is available at http://mirrors.sugarlabs.org/
Considerations
Bandwidth
To run a mirror you need a lot of bandwidth! You should look at the total bandwidth used by all the mirrors.
If you have trouble with bandwidth, you should look at CloudFlare.
HDD Space
Hosting a mirror takes a lot of space. If you don't have a lot of space you can only choose to mirror some parts. For example exclude all directories but the activities (~13gb):
rsync -avzh rsync://download.sugarlabs.org/pub --exclude 'dextrose' --exclude 'hexoquinasa' --exclude 'images' --exclude 'sources' --exclude 'docs' --exclude 'packages' --exclude 'soas' /rsync/download.sugarlabs.org
Setting up a new mirror
For mirror administrators
All you need is a web server with enough bandwidth to serve the files. To set up a new mirror, the site administrator needs to:
- First lets make a directory to store the data:
mkdir /rsync mkdir /rsync/download.sugarlabs.org
- Then lets use rsync to download the data (warning: takes a long time)
rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org
- Save the rsync command as a shell script and make it executable:
echo "rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org" > /rsync/download.sugarlabs.org/sync.sh chmod 774 /rsync/download.sugarlabs.org/sync.sh
- Then lets make this to sync automatically. We can use a cron job to do that. You could make sync every 2 hours:
echo "0 */2 * * * /rsync/download.sugarlabs.org/sync.sh" > asloSyncCronJob.txt crontab asloSyncCronJob.txt
If you don't want it to sync every 2 hours, have a look at a cron tutorial to change that value.
- Publish the files via HTTP. Look at your http server documentation on how to do that. You could set up a virtual host to serve these files: in nginx in apache
- Setup a rsync mirror so we can view the status of your mirror. To do so, create a rsyncd.conf file and open it:
sudo nano rsyncd.conf
Then insert the following config:
log file = /rsync/log
[sugarlabs]
path = /rsync/download.sugarlabs.org
comment = PUT SOME INFORMATION HERE - LIKE A MOTD
read only = true
list = yes
Save and quit nano. Then start rsyncd so it can serve your files:
rsync --daemon --config=/etc/rsyncd.conf
- Alert the Sugar Labs System Administrators that they would like their mirror into rotation, including the following information in the request:
- Name and URL of the mirror operator (e.g. organization)
- Name and email address of the administrative contact
- ISO 3166-1 alpha-2 country code of the server location
- HTTP base URL of the files on the mirror (typically http://mirrors.example.org/sugarlabs/)
- rsync base URL of the files on the mirror (typically rsync://mirrors.example.org/sugarlabs/)
Please contact sysadmin AT sugarlabs DOT org if you are interested in hosting a mirror.
For Sugar Labs sysadmins
To add a new mirror to the MirrorBrain redirector:
- Choose a name for the mirror, usually the host name.
- Register the mirror with MirrorBrain:
sudo -u mirrorbrain mb new <mirror name> --operator-name <operator name> \ --operator-url <operator URL> -a <admin name> -e <admin email> \ -c <country code> -H <base HTTP URL> -R <base rsync URL> -F <base FTP URL>
- Scan and enable the mirror:
sudo -u mirrorbrain mb scan -e <mirror name>
- Export the list of mirrors for mirmon (a hourly cronjob does this, but if you don't want to wait...):
mb export --format=mirmon-apache | sudo -u mirrorbrain tee /srv/mirrorbrain/mirmon/mirrorlist-export
- Finally, re-run mirmon to ensure it can check the health of the mirror (this is also done by a cronjob, but our patience is very short):
sudo -u mirrorbrain mirmon -v -get all -c /etc/mirmon.conf