Service/mirrors

Team Home · Join · Contacts · Resources · FAQ · Roadmap · To Do · Meetings

Introduction

A content delivery network or Content Delivery Network (CDN) is a system of computers containing copies of data, placed at various points in a network so as to maximize bandwidth for access to the data from clients throughout the network. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, thereby causing a bottleneck near that server.

Goals

Reduce bandwidth at primary download server.
Improve quality of service for users.
Move content closer to users, thus reducing latency.

Architecture

The Sugar Labs Content Delivery Network uses MirrorBrain as a redirector. The redirector, which lives in a Sugar Labs data center, keeps track of which files are available on which mirror. When a user requests a file, the redirector points the user to the correct mirror and automatically starts the file download.

Mirrors

The current list of available mirrors is available at http://mirrors.sugarlabs.org/

Considerations

Bandwidth

To run a mirror you need a lot of bandwidth! You should look at [the total bandwidth used by all the mirrors mirrors. http://stats.sugarlabs.org/download.sugarlabs.org/].

If you have trouble with bandwidth you should look at [CloudFlare http://www.cloudflare.com].

HDD Space

Hosting a mirror takes a lot of space. If you don't have a lot of space you can only choose to mirror some parts. For example exclude all directories but the activities (~13gb):

 rsync -avzh rsync://download.sugarlabs.org/pub --exclude 'dextrose' --exclude 'hexoquinasa' --exclude 'images' --exclude 'sources' --exclude 'docs' --exclude 'packages' --exclude 'soas' /rsync/download.sugarlabs.org

Setting up a new mirror

For mirror administrators

All you need is a web server with enough bandwidth to serve the files. To set up a new mirror, the site administrator needs to:

First lets make a directory to store the data:

 mkdir /rsync
 mkdir /rsync/download.sugarlabs.org

Then lets use rsync to download the data (warning: takes a long time)

 rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org

Save the rsync command as a shell script and make it executable:

 echo "rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org" > /rsync/download.sugarlabs.org/sync.sh
 chmod 774 /rsync/download.sugarlabs.org/sync.sh

Then lets make this to sync automatically. We can use a cron job to do that. You could make sync every 2 hours:

 echo "0 */2 * * * /rsync/download.sugarlabs.org/sync.sh" > asloSyncCronJob.txt
 crontab asloSyncCronJob.txt

If you don't want it to sync every 2 hours, have a look at a cron tutorial to change that value.

Publish the files via HTTP. Look at your http server documentation on how to do that. You could set up a virtual host to serve these files: in nginx in apache

Setup a rsync mirror so we can view the status of your mirror. To do so, create a rsyncd.conf file and open it:

 sudo nano rsyncd.conf

Then insert the following config:

 log file = /rsync/log

 [sugarlabs]
     path = /rsync/download.sugarlabs.org
     comment = PUT SOME INFORMATION HERE - LIKE A MOTD
     read only = true
     list = yes

Save and quit nano. Then start rsyncd so it can serve your files:

 rsync --daemon --config=/etc/rsyncd.conf

Alert the Sugar Labs System Administrators that they would like their mirror into rotation, including the following information in the request:
- Name and URL of the mirror operator (e.g. organization)
- Name and email address of the administrative contact
- ISO 3166-1 alpha-2 country code of the server location
- HTTP base URL of the files on the mirror (typically http://mirrors.example.org/sugarlabs/)
- rsync base URL of the files on the mirror (typically rsync://mirrors.example.org/sugarlabs/)

Please contact sysadmin AT sugarlabs DOT org if you are interested in hosting a mirror.

For Sugar Labs sysadmins

To add a new mirror to the MirrorBrain redirector:

Choose a name for the mirror, usually the host name.
Register the mirror with MirrorBrain:

sudo -u mirrorbrain mb new <mirror name> --operator-name <operator name> \
 --operator-url <operator URL> -a <admin name> -e <admin email> \
 -c <country code> -H <base HTTP URL> -R <base rsync URL> -F <base FTP URL>

Scan and enable the mirror:

sudo -u mirrorbrain mb scan -e <mirror name>