Service/mirrors - Revision history

Bernie at 01:10, 10 August 2014

2014-08-10T01:10:38Z

← Older revision		Revision as of 21:10, 9 August 2014
Line 96:		Line 96:
	* Scan and enable the mirror:		* Scan and enable the mirror:
	sudo -u mirrorbrain mb scan -e <mirror name>		sudo -u mirrorbrain mb scan -e <mirror name>
			* Export the list of mirrors for mirmon (a hourly cronjob does this, but if you don't want to wait...):
			mb export --format=mirmon-apache \| sudo -u mirrorbrain tee /srv/mirrorbrain/mirmon/mirrorlist-export
			* Finally, re-run mirmon to ensure it can check the health of the mirror (this is also done by a cronjob, but our patience is very short):
			sudo -u mirrorbrain mirmon -v -get all -c /etc/mirmon.conf

FGrose: /* Bandwidth */

2014-08-10T00:28:12Z

Bandwidth

← Older revision		Revision as of 20:28, 9 August 2014
Line 20:		Line 20:
	=== Bandwidth ===		=== Bandwidth ===

	To run a mirror you need a lot of bandwidth! You should look at [~~the total bandwidth used by all the mirrors mirrors.~~ http://stats.sugarlabs.org/download.sugarlabs.org/].		To run a mirror you need a lot of bandwidth! You should look at [http://stats.sugarlabs.org/download.sugarlabs.org/ the total bandwidth used by all the mirrors].

	If you have trouble with bandwidth you should look at [~~CloudFlare~~ http://www.cloudflare.com].		If you have trouble with bandwidth, you should look at [http://www.cloudflare.com CloudFlare].

	=== HDD Space ===		=== HDD Space ===

SAMdroid: Add considerations

2014-08-10T00:17:52Z

Add considerations

← Older revision		Revision as of 20:17, 9 August 2014
Line 16:		Line 16:
	The current list of available mirrors is available at http://mirrors.sugarlabs.org/		The current list of available mirrors is available at http://mirrors.sugarlabs.org/

			== Considerations ==

			=== Bandwidth ===

			To run a mirror you need a lot of bandwidth! You should look at [the total bandwidth used by all the mirrors mirrors. http://stats.sugarlabs.org/download.sugarlabs.org/].

			If you have trouble with bandwidth you should look at [CloudFlare http://www.cloudflare.com].

			=== HDD Space ===

			Hosting a mirror takes a lot of space. If you don't have a lot of space you can only choose to mirror some parts. For example exclude all directories but the activities (~13gb):

			rsync -avzh rsync://download.sugarlabs.org/pub --exclude 'dextrose' --exclude 'hexoquinasa' --exclude 'images' --exclude 'sources' --exclude 'docs' --exclude 'packages' --exclude 'soas' /rsync/download.sugarlabs.org

	== Setting up a new mirror ==		== Setting up a new mirror ==

SAMdroid: Added a tutorial

2014-08-09T23:29:10Z

Added a tutorial

← Older revision		Revision as of 19:29, 9 August 2014
Line 22:		Line 22:
	All you need is a web server with enough bandwidth to serve the files. To set up a new mirror, the site administrator needs to:		All you need is a web server with enough bandwidth to serve the files. To set up a new mirror, the site administrator needs to:

	* ~~Set up rsync job~~ to ~~synchronize a copy of~~ the ~~files on http~~://download.sugarlabs.org ~~onto their local system. For example copy it into ''/~~rsync/download~~.suarlabs.org/''~~:		* First lets make a directory to store the data:

			mkdir /rsync
			mkdir /rsync/download.sugarlabs.org

			* Then lets use rsync to download the data (warning: takes a long time)

	rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org		rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org

	* ~~You can schedule~~ it ~~daily or more frequently, but please don't~~ sync ~~more often than once per hour~~. ~~For example, use cron to re-~~sync ~~every 2 hours (on linux):~~		* Save the rsync command as a shell script and make it executable:

			echo "rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org" > /rsync/download.sugarlabs.org/sync.sh
			chmod 774 /rsync/download.sugarlabs.org/sync.sh

	echo "0 /2 * * rsync ~~-avzh rsync:/~~/download.sugarlabs.org/~~pub /rsync/download.sugarlabs~~.~~org~~" > asloSyncCronJob.txt		* Then lets make this to sync automatically. We can use a cron job to do that. You could make sync every 2 hours:

			echo "0 /2 * * /rsync/download.sugarlabs.org/sync.sh" > asloSyncCronJob.txt
	crontab asloSyncCronJob.txt		crontab asloSyncCronJob.txt

	* Publish the files via HTTP ~~and~~ -- ~~if possible~~ -- ~~rsync. Rsync allows for efficient~~ [~~http~~://www.~~mirrorbrain~~.~~org~~/~~features~~/#mirror~~-monitoring~~ mirror ~~monitoring~~] ~~(scanning)~~. ~~FTP is less efficient, but still preferred over an HTTP~~-only ~~mirror~~.		If you don't want it to sync every 2 hours, have a look at [https://www.digitalocean.com/community/tutorials/how-to-use-cron-to-automate-tasks-on-a-vps a cron tutorial] to change that value.

			* Publish the files via HTTP. Look at your http server documentation on how to do that. You could set up a virtual host to serve these files: [https://www.digitalocean.com/community/tutorials/how-to-set-up-nginx-server-blocks-virtual-hosts-on-ubuntu-14-04-lts in nginx] [https://www.digitalocean.com/community/tutorials/how-to-set-up-apache-virtual-hosts-on-ubuntu-12-04-lts in apache]

			* Setup a rsync mirror so we can view the status of your mirror. To do so, create a rsyncd.conf file and open it:

			sudo nano rsyncd.conf

			Then insert the following config:

			log file = /rsync/log

			[sugarlabs]
			path = /rsync/download.sugarlabs.org
			comment = PUT SOME INFORMATION HERE - LIKE A MOTD
			read only = true
			list = yes

			Save and quit nano. Then start rsyncd so it can serve your files:

			rsync --daemon --config=/etc/rsyncd.conf

	* Alert the [[Infrastructure_Team/Contacts\|Sugar Labs System Administrators]] that they would like their mirror into rotation, including the following information in the request:		* Alert the [[Infrastructure_Team/Contacts\|Sugar Labs System Administrators]] that they would like their mirror into rotation, including the following information in the request:
	** Name and URL of the mirror operator (e.g. organization)		** Name and URL of the mirror operator (e.g. organization)
	** Name and email address of the administrative contact		** Name and email address of the administrative contact
	** [http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 ISO 3166-1 alpha-2] country code of the server location		** [http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 ISO 3166-1 alpha-2] country code of the server location
	** HTTP base URL of the files on the mirror (typically http://mirrors.example.org/sugarlabs/)		** HTTP base URL of the files on the mirror (typically http://mirrors.example.org/sugarlabs/)
	** rsync base URL of the files on the mirror (typically rsync://mirrors.example.org/sugarlabs/)		** rsync base URL of the files on the mirror (typically rsync://mirrors.example.org/sugarlabs/)

SAMdroid: /* For mirror administrators */ Add tutorial commands

2014-08-07T12:04:46Z

For mirror administrators: Add tutorial commands

← Older revision		Revision as of 08:04, 7 August 2014
Line 22:		Line 22:
	All you need is a web server with enough bandwidth to serve the files. To set up a new mirror, the site administrator needs to:		All you need is a web server with enough bandwidth to serve the files. To set up a new mirror, the site administrator needs to:

	* Set up rsync job to synchronize a copy of the files on http://~~downloads~~.sugarlabs.org onto their local system. You can schedule it daily or more frequently, but please don't sync more often than once per hour.		* Set up rsync job to synchronize a copy of the files on http://download.sugarlabs.org onto their local system. For example copy it into ''/rsync/download.suarlabs.org/'':

			rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org

			* You can schedule it daily or more frequently, but please don't sync more often than once per hour. For example, use cron to re-sync every 2 hours (on linux):

			echo "0 /2 * * rsync -avzh rsync://download.sugarlabs.org/pub /rsync/download.sugarlabs.org" > asloSyncCronJob.txt
			crontab asloSyncCronJob.txt

	* Publish the files via HTTP and -- if possible -- rsync. Rsync allows for efficient [http://www.mirrorbrain.org/features/#mirror-monitoring mirror monitoring] (scanning). FTP is less efficient, but still preferred over an HTTP-only mirror.		* Publish the files via HTTP and -- if possible -- rsync. Rsync allows for efficient [http://www.mirrorbrain.org/features/#mirror-monitoring mirror monitoring] (scanning). FTP is less efficient, but still preferred over an HTTP-only mirror.
	* Alert the [[Infrastructure_Team/Contacts\|Sugar Labs System Administrators]] that they would like their mirror into rotation, including the following information in the request:		* Alert the [[Infrastructure_Team/Contacts\|Sugar Labs System Administrators]] that they would like their mirror into rotation, including the following information in the request:

Bernie: Bernie moved page Sysadmin/Content Delivery Network to Service/mirrors

2014-08-06T21:35:50Z

Bernie moved page Sysadmin/Content Delivery Network to Service/mirrors

← Older revision	Revision as of 17:35, 6 August 2014
(No difference)

Bernie at 00:02, 2 May 2012

2012-05-02T00:02:25Z

← Older revision		Revision as of 20:02, 1 May 2012
Line 2:		Line 2:
	[[Category:Resource]]		[[Category:Resource]]

	==Introduction==		== Introduction ==
	A content delivery network or Content Delivery Network (CDN) is a system of computers containing copies of data, placed at various points in a network so as to maximize bandwidth for access to the data from clients throughout the network. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, thereby causing a bottleneck near that server.		A content delivery network or Content Delivery Network (CDN) is a system of computers containing copies of data, placed at various points in a network so as to maximize bandwidth for access to the data from clients throughout the network. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, thereby causing a bottleneck near that server.

	==Goals==		== Goals ==
	* Reduce bandwidth at primary download server.		* Reduce bandwidth at primary download server.
	* Improve quality of service for users.		* Improve quality of service for users.
	* Move content closer to users, thus reducing latency.		* Move content closer to users, thus reducing latency.

	==Mirrors==		== Architecture ==
			The Sugar Labs Content Delivery Network uses [http://www.mirrorbrain.org/ MirrorBrain] as a redirector. The redirector, which lives in a Sugar Labs data center, keeps track of which files are available on which mirror. When a user requests a file, the redirector points the user to the correct mirror and automatically starts the file download.

			== Mirrors ==
	The current list of available mirrors is available at http://mirrors.sugarlabs.org/		The current list of available mirrors is available at http://mirrors.sugarlabs.org/

	~~==Setting up a Mirror==~~
	~~Setting up a mirror is very easy. All you need is a web server which is available via http, and (if possible) rsync.~~

	~~Please contact sysadmin AT sugarlabs DOT org if you are interested in hosting~~ a mirror.		== Setting up a new mirror ==

	== ~~System Summary~~ ==		=== For mirror administrators ===
	~~The Sugar Labs Content Delivery Network uses [http://www.mirrorbrain.org/ MirrorBrain] as~~ a ~~redirector~~. ~~The redirector, which lives in~~ a ~~Sugar Labs data center, keeps track of which files are available on which~~ mirror~~. When a user requests a file~~, the ~~redirector points the user~~ to ~~the correct mirror and automatically starts the file download.~~		All you need is a web server with enough bandwidth to serve the files. To set up a new mirror, the site administrator needs to:

	~~=== Mirrors ===~~		* Set up rsync job to synchronize a copy of the files on http://downloads.sugarlabs.org onto their local system. You can schedule it daily or more frequently, but please don't sync more often than once per hour.
	~~[http://mirrors.sugarlabs.org Mirrors] are completely passive. To set up a mirror, a mirror system administrator needs to:~~
	* Set up rsync job to synchronize a copy of the files on http://downloads.sugarlabs.org onto their local system.
	* Publish the files via HTTP and -- if possible -- rsync. Rsync allows for efficient [http://www.mirrorbrain.org/features/#mirror-monitoring mirror monitoring] (scanning). FTP is less efficient, but still preferred over an HTTP-only mirror.		* Publish the files via HTTP and -- if possible -- rsync. Rsync allows for efficient [http://www.mirrorbrain.org/features/#mirror-monitoring mirror monitoring] (scanning). FTP is less efficient, but still preferred over an HTTP-only mirror.
	* Alert the Sugar Labs ~~system administrators~~ that they would like their mirror into rotation, including the following information in the request:		* Alert the [[Infrastructure_Team/Contacts\|Sugar Labs System Administrators]] that they would like their mirror into rotation, including the following information in the request:
	** Name and URL of the mirror operator (e.g. ~~organisation~~)		** Name and URL of the mirror operator (e.g. organization)
	** Name and email address of the administrative contact		** Name and email address of the administrative contact
	** [http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 ISO 3166-1 alpha-2] country code of the server location		** [http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 ISO 3166-1 alpha-2] country code of the server location
	** HTTP base URL of the files on the mirror		** HTTP base URL of the files on the mirror (typically http://mirrors.example.org/sugarlabs/)
	** rsync base URL of the files on the mirror		** rsync base URL of the files on the mirror (typically rsync://mirrors.example.org/sugarlabs/)

			Please contact sysadmin AT sugarlabs DOT org if you are interested in hosting a mirror.

			=== For Sugar Labs sysadmins ===

	~~=== Adding~~ a new mirror to the ~~Mirror Brain~~ redirector ~~===~~		To add a new mirror to the MirrorBrain redirector:

	* Choose a name for the mirror, usually the host name.		* Choose a name for the mirror, usually the host name.

Bernie at 23:48, 1 May 2012

2012-05-01T23:48:46Z

← Older revision		Revision as of 19:48, 1 May 2012
Line 3:		Line 3:

	==Introduction==		==Introduction==
	A content delivery network or ~~content distribution network~~ (CDN) is a system of computers containing copies of data, placed at various points in a network so as to maximize bandwidth for access to the data from clients throughout the network. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, thereby causing a bottleneck near that server.		A content delivery network or Content Delivery Network (CDN) is a system of computers containing copies of data, placed at various points in a network so as to maximize bandwidth for access to the data from clients throughout the network. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, thereby causing a bottleneck near that server.

	~~==Mirrors==~~
	~~The current list of available mirrors is available at http://mirrors.sugarlabs.org/~~

	==Goals==		==Goals==
	* Reduce bandwidth at primary download server.		* Reduce bandwidth at primary download server.
	* Improve quality of service for users.		* Improve quality of service for users.
	* Move content closer to users.		* Move content closer to users, thus reducing latency.

			==Mirrors==
			The current list of available mirrors is available at http://mirrors.sugarlabs.org/

	==Setting up a Mirror==		==Setting up a Mirror==
	Setting up a mirror is very easy. All you need is a web server which is available via http, and (if possible) rsync.		Setting up a mirror is very easy. All you need is a web server which is available via http, and (if possible) rsync.

	Please contact ~~systems@~~sugarlabs.org if you are interested.		Please contact sysadmin AT sugarlabs DOT org if you are interested in hosting a mirror.

	== System Summary ==		== System Summary ==
Line 23:		Line 23:
	=== Mirrors ===		=== Mirrors ===
	[http://mirrors.sugarlabs.org Mirrors] are completely passive. To set up a mirror, a mirror system administrator needs to:		[http://mirrors.sugarlabs.org Mirrors] are completely passive. To set up a mirror, a mirror system administrator needs to:
	* Set up rsync job to ~~synchronise~~ a copy of the files on downloads.sugarlabs.org onto their local system.		* Set up rsync job to synchronize a copy of the files on http://downloads.sugarlabs.org onto their local system.
	* Publish the files via HTTP and -- if possible -- rsync. Rsync allows for efficient [http://www.mirrorbrain.org/features/#mirror-monitoring mirror monitoring] (scanning). FTP is less efficient, but still preferred over an HTTP-only mirror.		* Publish the files via HTTP and -- if possible -- rsync. Rsync allows for efficient [http://www.mirrorbrain.org/features/#mirror-monitoring mirror monitoring] (scanning). FTP is less efficient, but still preferred over an HTTP-only mirror.
	* Alert the Sugar Labs system administrators that they would like their mirror into rotation, including the following information in the request:		* Alert the Sugar Labs system administrators that they would like their mirror into rotation, including the following information in the request:
Line 32:		Line 32:
	** rsync base URL of the files on the mirror		** rsync base URL of the files on the mirror

	=== ~~Redirector~~ ===		=== Adding a new mirror to the Mirror Brain redirector ===

	* Choose a name for the mirror, usually the host name.		* Choose a name for the mirror, usually the host name.

Sascha silbe: Instructions for adding a mirror, minor changes

2011-07-24T09:10:19Z

Instructions for adding a mirror, minor changes

← Older revision		Revision as of 05:10, 24 July 2011
Line 14:		Line 14:

	==Setting up a Mirror==		==Setting up a Mirror==
	Setting up a mirror is very easy. All you need is a web server which is available via http, and (if possible) rsync.		Setting up a mirror is very easy. All you need is a web server which is available via http, and (if possible) rsync.

	Please contact systems@sugarlabs.org if you are interested.		Please contact systems@sugarlabs.org if you are interested.

	== System Summary ==		== System Summary ==
	The Sugar Labs Content Delivery Network uses MirrorBrain as a redirector. The redirector, which lives in a Sugar Labs data center, keeps track of which files are available on which mirror. When a a user requests a file, the redirector points the user to the correct mirror and automatically starts the file download.		The Sugar Labs Content Delivery Network uses [http://www.mirrorbrain.org/ MirrorBrain] as a redirector. The redirector, which lives in a Sugar Labs data center, keeps track of which files are available on which mirror. When a user requests a file, the redirector points the user to the correct mirror and automatically starts the file download.

	=== Mirrors ===		=== Mirrors ===
	[http://mirrors.sugarlabs.org Mirrors] are completely passive. To set up a mirror, an mirror system administrator needs to:		[http://mirrors.sugarlabs.org Mirrors] are completely passive. To set up a mirror, a mirror system administrator needs to:
	* Set up rsync job to synchronise a copy of the files on downloads.sugarlabs.org onto their local system.		* Set up rsync job to synchronise a copy of the files on downloads.sugarlabs.org onto their local system.
	* Alert the Sugar Labs system administrators that they would like their mirror into rotation.		* Publish the files via HTTP and -- if possible -- rsync. Rsync allows for efficient [http://www.mirrorbrain.org/features/#mirror-monitoring mirror monitoring] (scanning). FTP is less efficient, but still preferred over an HTTP-only mirror.
			* Alert the Sugar Labs system administrators that they would like their mirror into rotation, including the following information in the request:
			** Name and URL of the mirror operator (e.g. organisation)
			** Name and email address of the administrative contact
			** [http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 ISO 3166-1 alpha-2] country code of the server location
			** HTTP base URL of the files on the mirror
			** rsync base URL of the files on the mirror

	=== Redirector ===		=== Redirector ===

	* Register mirror with mirrorbrain.		* Choose a name for the mirror, usually the host name.
	* mirror ~~probe~~		* Register the mirror with MirrorBrain:
			sudo -u mirrorbrain mb new <mirror name> --operator-name <operator name> \
			--operator-url <operator URL> -a <admin name> -e <admin email> \
			-c <country code> -H <base HTTP URL> -R <base rsync URL> -F <base FTP URL>
			* Scan and enable the mirror:
			sudo -u mirrorbrain mb scan -e <mirror name>

RafaelOrtiz: /* Mirrors */

2010-11-09T14:32:28Z

Mirrors

← Older revision		Revision as of 10:32, 9 November 2010
Line 22:		Line 22:

	=== Mirrors ===		=== Mirrors ===
	[http://mirrors.sugarlabs.org \| Mirrors] are completely passive. To set up a mirror, an mirror system administrator needs to:		[http://mirrors.sugarlabs.org Mirrors] are completely passive. To set up a mirror, an mirror system administrator needs to:
	* Set up rsync job to synchronise a copy of the files on downloads.sugarlabs.org onto their local system.		* Set up rsync job to synchronise a copy of the files on downloads.sugarlabs.org onto their local system.
	* Alert the Sugar Labs system administrators that they would like their mirror into rotation.		* Alert the Sugar Labs system administrators that they would like their mirror into rotation.