1,764
edits
Changes
Jump to navigation
Jump to search
Line 1:
Line 1: − + − + − + − + − Mirrorbrain, Bouncer, Fedora Mirror Manager, and Cacheboy are four possible choices for scaling up the Sugar Labs content delivery net work. Long term, Cacheboy looks like it might be the best fit for Sugar Labs, but the project must become more stable before becoming Sugar Labs primary CDN. Bouncer is currently being used as successfully by Mozilla. But development ended at the end of 2008. Mozilla is currently investigating Mirrorbrain. Mirrorbrain looks promising.+ + + + − + − *Reduce bandwith at primary download server.+ − *Improve quality of service for users. − *Move content closer to users. − + − Mirrrorbrain is used by several major project including opensuse and openoffice. It is quite stable, under activate development, and well documented, + − + − The following Recipe for installing mirrorbrain on ubuntu 9.04 is based on information at http://mirrorbrain.org/docs/ . − + − Install a standard Ubuntu LAMP server − ====Download mirrorbrain====+ − <code> − wget http://mirrorbrain.org/files/releases/mirrorbrain-2.10.0.tar.gz − tar xzf mirrorbrain-2.10.0.tar.gz − </code> − ====Install python dependencies====+ − <code> − sudo apt-get install python-sqlobject python-psycopg2 − </code> − The python cmdln module is not prepackaged for Ubutnu so it must be installed manually.+ − <code>+ − wget http://cmdln.googlecode.com/files/cmdln-1.1.2.zip − unzip cmdln-1.1.2.zip − cd cmdln-1.1.2 − sudo python setup.py install − </code> − ====Install perl dependencies for scanner====+ − The mirrorbrain scanner is written in perl and requires several dependencies. − <code>+ − sudo apt-get install libconfig-inifiles-perl libwww-perl libdbd-pg-perl libdatetime-perl libdigest-md4-perl − </code> − + − Mirrorbrain requires several Apache modules, several of which must be built manually. Apache modules are built and installed using '''apxs2'''. (APache eXtenSion tool) Apxs2 is in the apache2-threaded-dev package.+ − <code>+ − sudo apt-get install apache2-threaded-dev − </code> − =====Install and Configure mod_geoip=====+ − Mod_geoip is available as a prebuilt package.+ − <code>+ − sudo apt-get install libapache2-mod-geoip − </code> − Mod_geoip must be configured to find the the GeoIP data set.+ − <code>+ − sudo sh -c "cat > /etc/apache2/mods-available/geoip.conf << EOF − <IfModule mod_geoip.c> − GeoIPEnable On − GeoIPOutput Env − GeoIPDBFile /var/lib/GeoIP/GeoIP.dat MMapCache − </IfModule> − EOF − " − </code> − Download GeoIP data set+ − <code>+ − wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz − sudo apt-get install gzip − gunzip GeoIP.dat.gz − sudo mkdir /var/lib/GeoIP − sudo cp GeoIP.dat /var/lib/GeoIP/GeoIP.dat − </code> − Enable module.+ − <code>+ − sudo a2enmod geoip+ − </code>+ − Restart Apache+ − <code> − sudo /etc/init.d/apache2 restart − </code> − =====Install and Configure mod_form=====+ − Mod_from must be build from sctatch. − FIXME correct directory+ − Download source − <code> − wget http://apache.webthing.com/svn/apache/forms/mod_form.c − wget http://apache.webthing.com/svn/apache/forms/mod_form.h − </code> − build mod_form+ − <code> − sudo apxs2 -cia mod_form.c − </code> − create loader+ − <code> − sudo sh -c "cat > /etc/apache2/mods-available/form.load << EOF − LoadModule form_module /usr/lib/apache2/modules/mod_form.so − EOF − " − </code> − Enable module.+ − <code> − sudo a2enmod form − </code> − Restart Apache+ − <code>+ − sudo /etc/init.d/apache2 restart+ − </code>+ + + − + − Build mod_mirrorbrain − <code> − sudo apxs2 -cia mod_mirrorbrain.c − </code> − create module loader+ − <code>+ − sudo cat > /etc/apache2/mods-available/mirrorbrain.load << EOF+ − LoadModule mirror_module /usr/lib/apache2/modules/mod_mirrorbrain.so + − EOF+ − </code>+ − Enable module.+ − <code> − sudo a2enmod mirrorbrain − </code> − Restart Apache+ − <code> − sudo /etc/init.d/apache2 restart − </code> − ====Build and Install helper programs====+ − build and install geoiplookup − <code> − gcc -Wall -lGeoIP -o geoiplookup_continent geoiplookup_continent.c − sudo cp geoiplookup_contintent/usr/bin/geoiplookup_continent − </code> − Install scanner+ − <code>+ − sudo cp ../tools/scanner.pl /usr/bin/scanner+ − </code>+ − + − ====Install postgres====+ − <code>+ − + − + − + − =====Install mod_dsd=====+ − − Install − <code> − sudo apt-get install libaprutil1-dbd-pgsql − − − configure mod_dsd − <code> − sudo cat > /etc/apache2/mods-available/dbd.conf << EOF − <IfModule mod_dbd.c> − DBDriver pgsql − # note that the connection string (which is passed straight through to − # PGconnectdb in this case) looks slightly different - pass vs. password − DBDParams "host=localhost user=mb password=12345 dbname=mb_sugar connect_timeout=15" − </IfModule> − EOF − </code> − − =====Create the postgresql user account and database===== − Switch to user postgress − <code> − sudo su - postgres − </code> − − Create user − − createuser -P mb − Enter password for new role: − Enter it again: − Shall the new role be a superuser? (y/n) n − Shall the new role be allowed to create databases? (y/n) n − Shall the new role be allowed to create more new roles? (y/n) n − − − Create database − − createdb -O mb mb_sugar − createlang plpgsql mb_sugar − </code> − − Exit user postrges − − exit − </code> − − =====Edit host-based authentication===== − add line 'local mb_sugar mb trust' to the end of pg_hba.conf − − FIXME should not be trust on production machine − − sudo vim /etc/postgresql/8.4/pg_hba.conf − </code> − − Start the potgres server − <code> − sudo /etc/init.d/postgresql-8.4 restart − − − =====Import intial mirrorbrain Data===== − Import table structure, and initial data − <code> − psql -U mb -f sql/schema-postgresql.sql mb_sugar − psql -U mb -f sql/initialdata-postgresql.sql mb_sugar − − − =====Create mirrorbrain user and group===== − <code> − − sudo useradd -r -g mirrorbrain -s /bin/bash -c "MirrorBrain user" -d /home/mirrorbrain mirrorbrain − − − =====Create mirrorbrain.conf===== − <code> − sudo cat > /etc/mirrorbrain.conf << EOF − [general] − instances = samba − [samba] − dbuser = mb − dbpass = 12345 − dbdriver = postgresql − dbhost = 127.0.0.1 − # optional: dbport = ... − dbname = mb_sugar − [mirrorprobe] − mailto = dfarning@sugarlabs.org − </code> − − Set permission and priviglies − <code> − sudo chmod 0604 /etc/mirrorbrain.conf − sudo chown root:mirrorbrain /etc/mirrorbrain.conf − </code> − − =====Test mirrorbrain===== − <code> − ./mirrorbrain.py − </code> − − ====Create VirtualHost==== − <code> − sudo cat > /etc/apache2/sites-available/mirror << EOF − <VirtualHost your.host.name:80> − ServerName samba.mirrorbrain.org − ServerAdmin webmaster@mirrorbrain.org − DocumentRoot /srv/samba/pub/projects − ErrorLog /var/log/apache/samba.mirrorbrain.org/logs/error_log − CustomLog /var/log/apache/samba.mirrorbrain.org/logs/access_log combined − <Directory /srv/samba/pub/projects> − MirrorBrainEngine On − MirrorBrainDebug Off − FormGET On − MirrorBrainHandleHEADRequestLocally Off − MirrorBrainMinSize 2048 − MirrorBrainHandleDirectoryIndexLocally On − MirrorBrainExcludeUserAgent rpm/4.4.2* − MirrorBrainExcludeUserAgent *APT-HTTP* − MirrorBrainExcludeMimeType application/pgp-keys − Options FollowSymLinks Indexes − AllowOverride None − Order allow,deny − Allow from all − </Directory> − </VirtualHost> − </code> − − − − <noinclude>{{ GoogleTrans-en | es =show | bg =show | zh-CN =show | zh-TW =show | hr =show | cs =show | da =show | nl =show | fi =show | fr =show | de =show | el =show | hi =show | it =show | ja =show | ko =show | no =show | pl =show | pt =show | ro =show | ru =show | sv =show }}</noinclude> − == Subpages == − − {{Special:PrefixIndex/{{PAGENAMEE}}/}} − − − [[Category:Team]]
no edit summary
<noinclude>{{TeamHeader|Infrastructure Team}}</noinclude>
<noinclude>{{TeamHeader|Infrastructure Team}}{{TOCright}}
{{TOCright}}
[[Category:Resource]]
==Introduction==
== Introduction ==
A content delivery network or content distribution network (CDN) is a system of computers containing copies of data, placed at various points in a network so as to maximize bandwidth for access to the data from clients throughout the network. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, thereby causing a bottleneck near that server.
A content delivery network or Content Delivery Network (CDN) is a system of computers containing copies of data, placed at various points in a network so as to maximize bandwidth for access to the data from clients throughout the network. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, thereby causing a bottleneck near that server.
== Goals ==
* Reduce bandwidth at primary download server.
* Improve quality of service for users.
* Move content closer to users, thus reducing latency.
==Goals==
== Architecture ==
The Sugar Labs Content Delivery Network uses [http://www.mirrorbrain.org/ MirrorBrain] as a redirector. The redirector, which lives in a Sugar Labs data center, keeps track of which files are available on which mirror. When a user requests a file, the redirector points the user to the correct mirror and automatically starts the file download.
==[http://mirrorbrain.org/ Mirrorbrain]==
== Mirrors ==
The current list of available mirrors is available at http://mirrors.sugarlabs.org/
===Installation===
== Considerations ==
====Install LAMP Server====
=== Bandwidth ===
To run a mirror you need a lot of bandwidth! You should look at [http://stats.sugarlabs.org/download.sugarlabs.org/ the total bandwidth used by all the mirrors].
If you have trouble with bandwidth, you should look at [http://www.cloudflare.com CloudFlare].
=== HDD Space ===
Hosting a mirror takes a lot of space. If you don't have a lot of space you can only choose to mirror some parts. For example exclude all directories but the activities (~13gb):
rsync -avzh rsync://download.sugarlabs.org/pub --exclude 'dextrose' --exclude 'hexoquinasa' --exclude 'images' --exclude 'sources' --exclude 'docs' --exclude 'packages' --exclude 'soas' /rsync/download.sugarlabs.org
== Setting up a new mirror ==
====Build, Install, and Configure Apache2 modules====
=== For mirror administrators ===
All you need is a web server with enough bandwidth to serve the files. To set up a new mirror, the site administrator needs to:
* First lets make a directory to store the data:
mkdir /rsync
mkdir /rsync/download.sugarlabs.org
* Then lets use rsync to download the data (warning: takes a long time)
rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org
* Save the rsync command as a shell script and make it executable:
echo "rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org" > /rsync/download.sugarlabs.org/sync.sh
chmod 774 /rsync/download.sugarlabs.org/sync.sh
* Then lets make this to sync automatically. We can use a cron job to do that. You could make sync every 2 hours:
echo "0 */2 * * * /rsync/download.sugarlabs.org/sync.sh" > asloSyncCronJob.txt
crontab asloSyncCronJob.txt
If you don't want it to sync every 2 hours, have a look at [https://www.digitalocean.com/community/tutorials/how-to-use-cron-to-automate-tasks-on-a-vps a cron tutorial] to change that value.
* Publish the files via HTTP. Look at your http server documentation on how to do that. You could set up a virtual host to serve these files: [https://www.digitalocean.com/community/tutorials/how-to-set-up-nginx-server-blocks-virtual-hosts-on-ubuntu-14-04-lts in nginx] [https://www.digitalocean.com/community/tutorials/how-to-set-up-apache-virtual-hosts-on-ubuntu-12-04-lts in apache]
* Setup a rsync mirror so we can view the status of your mirror. To do so, create a rsyncd.conf file and open it:
sudo nano rsyncd.conf
Then insert the following config:
log file = /rsync/log
[sugarlabs]
path = /rsync/download.sugarlabs.org
comment = PUT SOME INFORMATION HERE - LIKE A MOTD
read only = true
list = yes
Save and quit nano. Then start rsyncd so it can serve your files:
====Install and configure mod_mirrorbrain====
rsync --daemon --config=/etc/rsyncd.conf
* Alert the [[Infrastructure_Team/Contacts|Sugar Labs System Administrators]] that they would like their mirror into rotation, including the following information in the request:
** Name and URL of the mirror operator (e.g. organization)
** Name and email address of the administrative contact
** [http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 ISO 3166-1 alpha-2] country code of the server location
** HTTP base URL of the files on the mirror (typically http://mirrors.example.org/sugarlabs/)
** rsync base URL of the files on the mirror (typically rsync://mirrors.example.org/sugarlabs/)
Please contact sysadmin AT sugarlabs DOT org if you are interested in hosting a mirror.
=== For Sugar Labs sysadmins ===
To add a new mirror to the MirrorBrain redirector:
* Choose a name for the mirror, usually the host name.
* Register the mirror with MirrorBrain:
sudo -u mirrorbrain mb new <mirror name> --operator-name <operator name> \
--operator-url <operator URL> -a <admin name> -e <admin email> \
-c <country code> -H <base HTTP URL> -R <base rsync URL> -F <base FTP URL>
* Scan and enable the mirror:
sudo -u mirrorbrain mb scan -e <mirror name>
sudo apt-get install postgresql-8.4
* Export the list of mirrors for mirmon (a hourly cronjob does this, but if you don't want to wait...):
</code>
mb export --format=mirmon-apache | sudo -u mirrorbrain tee /srv/mirrorbrain/mirmon/mirrorlist-export
* Finally, re-run mirmon to ensure it can check the health of the mirror (this is also done by a cronjob, but our patience is very short):
sudo -u mirrorbrain mirmon -v -get all -c /etc/mirmon.conf
</code>
<code>
</code>
<code>
<code>
<code>
</code>
</code>
sudo groupadd -r mirrorbrain
</code>