Service/mirrors

Team Home   ·   Join   ·   Contacts   ·   Resources   ·   FAQ   ·   Roadmap   ·   To Do   ·   Meetings

Introduction

A content delivery network or content distribution network (CDN) is a system of computers containing copies of data, placed at various points in a network so as to maximize bandwidth for access to the data from clients throughout the network. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, thereby causing a bottleneck near that server.

Mirrorbrain, Bouncer, Fedora Mirror Manager, and Cacheboy are four possible choices for scaling up the Sugar Labs content delivery net work. Long term, Cacheboy looks like it might be the best fit for Sugar Labs, but the project must become more stable before becoming Sugar Labs primary CDN. Bouncer is currently being used as successfully by Mozilla. But development ended at the end of 2008. Mozilla is currently investigating Mirrorbrain. Mirrorbrain looks promising.

Goals

• Reduce bandwith at primary download server.
• Improve quality of service for users.
• Move content closer to users.

Mirrorbrain

Mirrrorbrain is used by several major project including opensuse and openoffice. It is quite stable, under activate development, and well documented,

Installation

The following Recipe for installing mirrorbrain on ubuntu 9.04 is based on information at http://mirrorbrain.org/docs/ .

Install LAMP Server

Install a standard Ubuntu LAMP server

Download mirrorbrain

wget http://mirrorbrain.org/files/releases/mirrorbrain-2.10.0.tar.gz
tar xzf mirrorbrain-2.10.0.tar.gz

Install python dependencies

sudo apt-get install python-sqlobject python-psycopg2

The python cmdln module is not prepackaged for Ubutnu so it must be installed manually.

wget http://cmdln.googlecode.com/files/cmdln-1.1.2.zip
unzip cmdln-1.1.2.zip
cd cmdln-1.1.2
sudo python setup.py install

Install perl dependencies for scanner

The mirrorbrain scanner is written in perl and requires several dependencies.

sudo apt-get install libconfig-inifiles-perl libwww-perl libdbd-pg-perl libdatetime-perl libdigest-md4-perl

Build, Install, and Configure Apache2 modules

Mirrorbrain requires several Apache modules, several of which must be built manually. Apache modules are built and installed using apxs2. (APache eXtenSion tool) Apxs2 is in the apache2-threaded-dev package.

sudo apt-get install apache2-threaded-dev

Install and Configure mod_geoip

Mod_geoip is available as a prebuilt package.

sudo apt-get install libapache2-mod-geoip libgeoip-dev

Mod_geoip must be configured to find the the GeoIP data set.

sudo sh -c "cat > /etc/apache2/mods-available/geoip.conf << EOF
<IfModule mod_geoip.c>
 GeoIPEnable On
 GeoIPOutput Env
 GeoIPDBFile /var/lib/GeoIP/GeoIP.dat Standard
</IfModule>
EOF
" 

Download GeoIP data set

wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
sudo apt-get install gzip
gunzip GeoIP.dat.gz
sudo mkdir /var/lib/GeoIP
sudo cp GeoIP.dat /var/lib/GeoIP/GeoIP.dat

Enable module.

sudo a2enmod geoip

Restart Apache

sudo /etc/init.d/apache2 restart

Install and Configure mod_form

Mod_from must be build from sctatch.

FIXME correct directory Download source

wget http://apache.webthing.com/svn/apache/forms/mod_form.c
wget http://apache.webthing.com/svn/apache/forms/mod_form.h

build mod_form

sudo apxs2 -cia mod_form.c

create loader

sudo sh -c "cat > /etc/apache2/mods-available/form.load << EOF
LoadModule form_module /usr/lib/apache2/modules/mod_form.so
EOF
"

Enable module.

sudo a2enmod form

Restart Apache

sudo /etc/init.d/apache2 restart

Configure mod_dsd

configure mod_dsd

sudo sh -c "cat > /etc/apache2/mods-available/dbd.conf << EOF
 <IfModule mod_dbd.c>
    DBDriver pgsql
    # note that the connection string (which is passed straight through to
    # PGconnectdb in this case) looks slightly different - pass vs. password
    DBDParams 'host=localhost user=mb password=12345 dbname=mb_sugar connect_timeout=15'
 </IfModule>
EOF
"

Enable module.

sudo a2enmod dbd

Restart Apache

sudo /etc/init.d/apache2 restart

Install and configure mod_mirrorbrain

Build mod_mirrorbrain

sudo apxs2 -cia mod_mirrorbrain.c

create module loader

sudo sh -c "cat > /etc/apache2/mods-available/mirrorbrain.load << EOF
LoadModule mirrorbrain_module /usr/lib/apache2/modules/mod_mirrorbrain.so
EOF
"

Enable module.

sudo a2enmod mirrorbrain

Restart Apache

sudo /etc/init.d/apache2 restart

Build and Install helper programs

build and install geoiplookup

gcc -Wall -lGeoIP -o geoiplookup_continent geoiplookup_continent.c
sudo cp geoiplookup_continent /usr/bin/geoiplookup_continent

Install scanner

sudo cp ../tools/scanner.pl /usr/bin/scanner

Install postgres

sudo apt-get install postgresql-8.3

Create the postgresql user account and database

Switch to user postgress

sudo su - postgres

Create user

createuser -P mb
Enter password for new role: 
Enter it again: 
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n

Create database

createdb -O mb mb_sugar
createlang plpgsql mb_sugar

Exit user postrges

exit

Edit host-based authentication

add line 'local mb_sugar mb trust' to the end of pg_hba.conf

FIXME should not be trust on production machine

sudo vim /etc/postgresql/8.3/main/pg_hba.conf

Start the potgres server

sudo /etc/init.d/postgresql-8.3 restart

Import intial mirrorbrain Data

Import table structure, and initial data

psql -U mb -f sql/schema-postgresql.sql mb_sugar
psql -U mb -f sql/initialdata-postgresql.sql mb_sugar

Create mirrorbrain user and group

sudo groupadd -r mirrorbrain
sudo useradd -r -g mirrorbrain -s /bin/bash -c "MirrorBrain user" -d /home/mirrorbrain mirrorbrain

Create mirrorbrain.conf

sudo sh -c "cat > /etc/mirrorbrain.conf << EOF
[general]
instances = samba
[samba]
dbuser = mb
dbpass = 12345
dbdriver = postgresql
dbhost = 127.0.0.1
# optional: dbport = ...
dbname = mb_sugar
[mirrorprobe]
mailto = dfarning@sugarlabs.org
EOF
"

Set permission and priviglies

sudo chmod 0604 /etc/mirrorbrain.conf 
sudo chown root:mirrorbrain /etc/mirrorbrain.conf

Test mirrorbrain

./mirrordoctor.py

Create VirtualHost

sudo sh -c "cat > /etc/apache2/sites-available/mirrorbrain << EOF
 <VirtualHost 127.0.0.1>
   ServerName mirrors.sugarlabs.org
   ServerAdmin webmaster@mirrorbrain.org
   DocumentRoot /var/www/downloads
   ErrorLog     /var/log/apache2/mirrors.sugarlabs.org/error_log
   CustomLog    /var/log/apache2/mirrors.sugarlabs.org/access_log combined
   <Directory /var/www/downloads>
     MirrorBrainEngine On
     MirrorBrainDebug Off
     FormGET On
     MirrorBrainHandleHEADRequestLocally Off
     MirrorBrainMinSize 2048
     MirrorBrainHandleDirectoryIndexLocally On
     MirrorBrainExcludeUserAgent rpm/4.4.2*
     MirrorBrainExcludeUserAgent *APT-HTTP*
     MirrorBrainExcludeMimeType application/pgp-keys
     Options FollowSymLinks Indexes
     AllowOverride None
     Order allow,deny
     Allow from all
   </Directory>
</VirtualHost>
EOF
"

Enable site.

sudo a2ensite mirrorbrain

Restart Apache

sudo /etc/init.d/apache2 restart