Platform Team/Server Kit/Guide/Usage Statistics: Difference between revisions

(4 intermediate revisions by one other user not shown)

Line 10:

==Factors affecting network bandwidth==

The ~~two primary factors affecting~~ the network bandwidth ~~are:~~

The only factor that really affects the network bandwidth usage is the resolution at which the data is being stored. Currently, the default step-size is 60 seconds.

* the ~~size of~~ the ~~RRDs~~ being ~~synced~~

* and the ~~sync frequency~~.

==~~=RRD~~ size===

==Factors affecting database size on the client side==

* This is primarily controlled through RRA (parameters). I will upload more notes on this as I get to know RRD's better. (Anish)

~~===Sync frequency===~~

The factor that affects the database size on the client is the RRAs (Round Robin Archives) being cached on the Client XO.

* This is ~~controlled by~~ the ~~<tt>update-timeout</tt> parameter in~~ the ~~sugar-client configuration~~.

<~~span style~~="~~color~~: ~~red~~;">'''~~Caution:~~''' The ~~default RRD parameters and~~ the sync frequency ~~don~~'t ~~quite match up~~. ~~The RRDs track usage statistics~~ for ~~around 8~~-~~10 days~~ and the ~~sync frequency~~ is ~~one hour!~~ This ~~needs~~ to be ~~tweaked appropriately while deploying~~ in ~~a production environment to save~~ on ~~bandwidth~~.</~~span~~>

The default RRA configuration on the client side reads something like

RRA:AVERAGE:0.5:1:4320

RRA:AVERAGE:0.5:5:2016

This means that there is one RRA (Round robin archive) within the DB that stores one value every minute for 4320 samples. This makes the total length of this 1 minute resolution RRA to be 4320 minutes = 3 days

The other RRA stores an average of 5 samples, for a total of 2016 samples. Thus its total duration is 5x2016 min= 10080 min= 7 days.

Thus the default duration of high resolution samples is 3 days and lower resolution samples is 7 days. Tweaking these values will affect both the database size on the client size and the network usage.

For locations with very intermittent connectivity, having bigger-duration databases might be advisable. sample RRA's could be

RRA:AVERAGE:0.5:1:43200 (30 days)

RRA:AVERAGE:0.5:5:20160 (70 days)

The above sample values will not be having any impact on network usage, since we're not increasing their resolution, frequency or adding/removing more RRA's

==Notes on reading/interpreting the RRD databases==

===Dumping all information===

* To dump all the information contained in an rrd file, one could use 'rrdtool dump' command.

* To dump the same info in a more pretty print format (tabular), you may use [http://people.sugarlabs.org/anish/dsreport dsreport]. You'll need to install rrdtool-perl to make the script work.

==Notes on usage of the actual server/client system==

Update: The following version numbers have been tested to work. Previous versions were buggy. Subsequent versions may be good. These are the upstream [http://download.sugarlabs.org/packages/Server:/Factory/Fedora-14/ Fedora], [http://download.sugarlabs.org/packages/Server:/Factory/Ubuntu-10.04/ Ubuntu] and [http://download.sugarlabs.org/packages/Server:/Factory/Debian-6.0/ Debian] package repository paths. [http://git.sugarlabs.org/server This] is the upstream git repository. The Dextrose-3 operating system will ship the latest stable sugar-client side packages.

===Server side packages===

* active-document-0.1-3.1

* restful-document-0.1-4.1

* sugar-client-1.2-16.1

* sugar-stats-client-0.1-6.1

* sugar-stats-server-0.1-6.1

* gevent

===Client side packages===

* sugar-client-1.2-16.1

* sugar-stats-client-0.1-6.1

Note: as you may have noticed, the package sugar-stats-client is needed to be installed on both the Client and the Server. The name is a bit misleading as it is more of a library that provides common functionality. [http://git.sugarlabs.org/server/client/commit/dcaed79104cb0c5f36558a8518ef9e5866d2c2a9 This name has/will be changed in future releases].

===The configuration files===

* There should be configuration files both on the client and the server. Here are the sample configurations:

====Client side config file: /etc/sugar-client.conf====

<pre>

[main]

# url to connect to school server's RESTfull API

api-url = http://schoolserver:8000

# process auto-registeration on a school server on startup; multiple argument,

# if mentioned more than once and previously registered server's IP is different

# from "schoolserver"'s one, process re-registrtion

auto-register = 0

# if, after registration, there are backups to restore on the server, process

# restore automatically

auto-restore = False

# if more than 0, do regular backups with specified delay in seconds

backup-timeout = 0

# rsync url to backup/restore Journal

backup-url =

# debug logging level; multiple argument

debug = 0

# jabber server for Sugar collaboraiton

jabber-url =

# specify machine's serial number instead of detecting it

machine-sn =

# specify machine's UUID instead of detecting it

machine-uuid =

# specify user's nickname instead of detecting it

nickname =

# school server's FQDN, will be auto-set after registration

schoolserver =

# CA certificates file to get access to --stats-url via HTTPS

stats-ca-certs = /home/anish/Documents/openssl/server.crt

# if more than 0 and --stats-url specified, ping stats server with specified

# delay in seconds to coordinate stats gathering work

stats-timeout = 60

# stats server's http(s) url to interact with; if omited, any stats related

# functionality is disabled

stats-url = https://192.168.1.55:8000

# current user's unique identity; needs to be set only from profile

# configuration file; will be auto-created on initial registration

uid =

# optional command-line arguments to pass to a packager while updating the

# system

update-args =

# if more than 0, do regular unattended updates with specified delay in seconds

update-timeout = 0

</pre>

'''Important paramerters to configure are'''

* <code>stats-ca-certs</code>: Path of the SSL certificate in case SSL auth is being used.

* <code>stats-url</code>: The url of the server. This needs to be prefixed with <code>http://</code> or <code>https://</code> to work without/with SSL.

* <code>stats-timeout</code>: The sync frequency of the collected statistics between the client and the server. Note, changing this wouldn't typically affect bandwidth consumption, which is actually determined by the RRAs.

* <code>machine-sn, machine-uuid, nickname</code>: Specify these values explicitly if you want them to be automatically determined. Might be useful in making stats anonymous by setting common machine-sn and/or nickname.

====Server side config file: /etc/sugar-stats.conf====

<pre>

[stats]

# enable stats collecting

stats = True

# path to SSL certificate to serve stats server via HTTPS

stats-certfile = /home/anish/Documents/openssl/server.crt

# space separated list of RRAs for RRD databases on client side

stats-client-rras = RRA:AVERAGE:0.5:1:4320 RRA:AVERAGE:0.5:5:2016

# path to SSL certificate keyfile to serve stats server via HTTPS

stats-keyfile = /home/anish/Documents/openssl/server.key

# path to the root directory to place stats

stats-root = /var/lib/sugar-stats/rrd

# space separated list of RRAs for RRD databases on a server side

stats-server-rras = RRA:AVERAGE:0.5:1:10080 RRA:AVERAGE:0.5:5:8928 RRA:AVERAGE:0.5:15:35040

# step interval in seconds for RRD databases

stats-step = 60

[active-document]

# path to the root directory to place documents' data and indexes

data-root = /var/lib/sugar-stats/users

# limit the resulting list for search requests

find-limit = 32

# flush index every specified changes

index-flush-threshold = 32

# flush index index after specified seconds since the last change

index-flush-timeout = 60

# for concurent access, run index writer in separate thread; this option

# specifies the writer's queue size; 0 means not threading the writer

index-write-queue = 256

[main]

# debug logging level; multiple argument

debug = 0

# hostname to listen incomming connections

host = 192.168.1.55

# path to the directory to place log files

logdir = /var/log/sugar-stats

# port number to listen incomming connections

port = 8000

# path to the directory to place pid files

rundir = /var/run/sugar-stats

# switch off user credentials check; disabling this option will require

# OpenSSH-5.6 or later.

trust-users = True

</pre>

'''Important paramerters to configure are'''

* <code>stats</code>: Must be true for stats collection to be enabled

* <code>stats-certfile</code>: Path of the SSL certificate on the server (if SSL auth is being used)

* <code>stats-keyfile</code>: Path of the SSL private key on the server (if SSL auth is being used)

* <code>stats-client-rras</code>: Configuration of Round Robin Archives on the client. See [[#Factors affecting database size on the client side]]

* <code>stats-server-rras</code>: Configuration of Round Robin Archives on the server. See [[#Factors affecting database size on the client side]]. This will '''NOT''' affect network bandwidth, but merely space consumed on the server.

* <code>stats-step</code>: Step size of '''ALL''' RRAs. This '''WILL''' affect bandwidth consumption. See [[#Factors affecting database size on the client side]]

* <code>trust-users</code>: Set this to True if using openSSH version lower than 5.6

===Notes on using SSL auth===

The steps broadly involved are:

* [http://www.akadia.com/services/ssh_test_certificate.html Generating private key, and self-signed certificates (for test env).] The certificate should be without a passphrase.

* Copying the certificate on the client, and configure <code>sugar-client</code> accordingly.

* Copying the certificate and private key on the server, and configure <code>sugar-stats-server</code> accordingly.

'''Notes on debugging the SSL auth system'''

* Check the <code>stats-url</code> in <code>sugar-client</code> configuration is <code>https://</code> and not <code>http://</code>.

* Check that the certificates are valid or have not expired.

* To check the SSL side of things, following commands maybe run on the client and the server.

** Server: <code>openssl s_server -cert server.crt -key server.key -accept 8000 </code>

** Client: <code>openssl s_client -connect 192.168.1.55:8000 -CAfile server.crt </code> (Replace the ipaddress with the correct server-machine url or IP address.

** If the process runs smoothly, you should see <code>Verify return code: 0 (ok)</code> at the end on the client side terminal. Fails will accompany an error code which maybe looked up [https://www.openssl.org/docs/apps/verify.html here]

@@ Line 10: / Line 10: @@
 ==Factors affecting network bandwidth==
-The two primary factors affecting the network bandwidth are:
+The only factor that really affects the network bandwidth usage is the resolution at which the data is being stored. Currently, the default step-size is 60 seconds.
-* the size of the RRDs being synced
-* and the sync frequency.
-===RRD size===
+==Factors affecting database size on the client side==
-* This is primarily controlled through RRA (parameters). I will upload more notes on this as I get to know RRD's better. (Anish)
-===Sync frequency===
+The factor that affects the database size on the client is the RRAs (Round Robin Archives) being cached on the Client XO.
-* This is controlled by the <tt>update-timeout</tt> parameter in the sugar-client configuration.
-<span style="color: red;">'''Caution:''' The default RRD parameters and the sync frequency don't quite match up. The RRDs track usage statistics for around 8-10 days and the sync frequency is one hour! This needs to be tweaked appropriately while deploying in a production environment to save on bandwidth.</span>
+The default RRA configuration on the client side reads something like
+RRA:AVERAGE:0.5:1:4320
+RRA:AVERAGE:0.5:5:2016
+This means that there is one RRA (Round robin archive) within the DB that stores one value every minute for 4320 samples. This makes the total length of this 1 minute resolution RRA to be 4320 minutes = 3 days
+The other RRA stores an average of 5 samples, for a total of 2016 samples. Thus its total duration is 5x2016 min= 10080 min= 7 days.
+Thus the default duration of high resolution samples is 3 days and lower resolution samples is 7 days. Tweaking these values will affect both the database size on the client size and the network usage.
+For locations with very intermittent connectivity, having bigger-duration databases might be advisable. sample RRA's could be
+RRA:AVERAGE:0.5:1:43200 (30 days)
+RRA:AVERAGE:0.5:5:20160 (70 days)
+The above sample values will not be having any impact on network usage, since we're not increasing their resolution, frequency or adding/removing more RRA's
+==Notes on reading/interpreting the RRD databases==
+===Dumping all information===
+* To dump all the information contained in an rrd file, one could use 'rrdtool dump' command.
+* To dump the same info in a more pretty print format (tabular), you may use [http://people.sugarlabs.org/anish/dsreport dsreport]. You'll need to install rrdtool-perl to make the script work.
+==Notes on usage of the actual server/client system==
+Update: The following version numbers have been tested to work. Previous versions were buggy. Subsequent versions may be good. These are the upstream [http://download.sugarlabs.org/packages/Server:/Factory/Fedora-14/ Fedora], [http://download.sugarlabs.org/packages/Server:/Factory/Ubuntu-10.04/ Ubuntu] and [http://download.sugarlabs.org/packages/Server:/Factory/Debian-6.0/ Debian] package repository paths. [http://git.sugarlabs.org/server This] is the upstream git repository. The Dextrose-3 operating system will ship the latest stable sugar-client side packages.
+===Server side packages===
+* active-document-0.1-3.1
+* restful-document-0.1-4.1
+* sugar-client-1.2-16.1
+* sugar-stats-client-0.1-6.1
+* sugar-stats-server-0.1-6.1
+* gevent
+===Client side packages===
+* sugar-client-1.2-16.1
+* sugar-stats-client-0.1-6.1
+Note: as you may have noticed, the package sugar-stats-client is needed to be installed on both the Client and the Server. The name is a bit misleading as it is more of a library that provides common functionality. [http://git.sugarlabs.org/server/client/commit/dcaed79104cb0c5f36558a8518ef9e5866d2c2a9 This name has/will be changed in future releases].
+===The configuration files===
+* There should be configuration files both on the client and the server. Here are the sample configurations:
+====Client side config file: /etc/sugar-client.conf====
+<pre>
+[main]
+# url to connect to school server's RESTfull API
+api-url = http://schoolserver:8000
+# process auto-registeration on a school server on startup; multiple argument,
+# if mentioned more than once and previously registered server's IP is different
+# from "schoolserver"'s one, process re-registrtion
+auto-register = 0
+# if, after registration, there are backups to restore on the server, process
+# restore automatically
+auto-restore = False
+# if more than 0, do regular backups with specified delay in seconds
+backup-timeout = 0
+# rsync url to backup/restore Journal
+backup-url =
+# debug logging level; multiple argument
+debug = 0
+# jabber server for Sugar collaboraiton
+jabber-url =
+# specify machine's serial number instead of detecting it
+machine-sn =
+# specify machine's UUID instead of detecting it
+machine-uuid =
+# specify user's nickname instead of detecting it
+nickname =
+# school server's FQDN, will be auto-set after registration
+schoolserver =
+# CA certificates file to get access to --stats-url via HTTPS
+stats-ca-certs = /home/anish/Documents/openssl/server.crt
+# if more than 0 and --stats-url specified, ping stats server with specified
+# delay in seconds to coordinate stats gathering work
+stats-timeout = 60
+# stats server's http(s) url to interact with; if omited, any stats related
+# functionality is disabled
+stats-url = https://192.168.1.55:8000
+# current user's unique identity; needs to be set only from profile
+# configuration file; will be auto-created on initial registration
+uid =
+# optional command-line arguments to pass to a packager while updating the
+# system
+update-args =
+# if more than 0, do regular unattended updates with specified delay in seconds
+update-timeout = 0
+</pre>
+'''Important paramerters to configure are'''
+* <code>stats-ca-certs</code>: Path of the SSL certificate in case SSL auth is being used.
+* <code>stats-url</code>: The url of the server. This needs to be prefixed with <code>http://</code> or <code>https://</code> to work without/with SSL.
+* <code>stats-timeout</code>: The sync frequency of the collected statistics between the client and the server. Note, changing this wouldn't typically affect bandwidth consumption, which is actually determined by the RRAs.
+* <code>machine-sn, machine-uuid, nickname</code>: Specify these values explicitly if you want them to be automatically determined. Might be useful in making stats anonymous by setting common machine-sn and/or nickname.
+====Server side config file: /etc/sugar-stats.conf====
+<pre>
+[stats]
+# enable stats collecting
+stats = True
+# path to SSL certificate to serve stats server via HTTPS
+stats-certfile = /home/anish/Documents/openssl/server.crt
+# space separated list of RRAs for RRD databases on client side
+stats-client-rras = RRA:AVERAGE:0.5:1:4320 RRA:AVERAGE:0.5:5:2016
+# path to SSL certificate keyfile to serve stats server via HTTPS
+stats-keyfile = /home/anish/Documents/openssl/server.key
+# path to the root directory to place stats
+stats-root = /var/lib/sugar-stats/rrd
+# space separated list of RRAs for RRD databases on a server side
+stats-server-rras = RRA:AVERAGE:0.5:1:10080 RRA:AVERAGE:0.5:5:8928 RRA:AVERAGE:0.5:15:35040
+# step interval in seconds for RRD databases
+stats-step = 60
+[active-document]
+# path to the root directory to place documents' data and indexes
+data-root = /var/lib/sugar-stats/users
+# limit the resulting list for search requests
+find-limit = 32
+# flush index every specified changes
+index-flush-threshold = 32
+# flush index index after specified seconds since the last change
+index-flush-timeout = 60
+# for concurent access, run index writer in separate thread; this option
+# specifies the writer's queue size; 0 means not threading the writer
+index-write-queue = 256
+[main]
+# debug logging level; multiple argument
+debug = 0
+# hostname to listen incomming connections
+host = 192.168.1.55
+# path to the directory to place log files
+logdir = /var/log/sugar-stats
+# port number to listen incomming connections
+port = 8000
+# path to the directory to place pid files
+rundir = /var/run/sugar-stats
+# switch off user credentials check; disabling this option will require
+# OpenSSH-5.6 or later.
+trust-users = True
+</pre>
+'''Important paramerters to configure are'''
+* <code>stats</code>: Must be true for stats collection to be enabled
+* <code>stats-certfile</code>: Path of the SSL certificate on the server (if SSL auth is being used)
+* <code>stats-keyfile</code>: Path of the SSL private key on the server (if SSL auth is being used)
+* <code>stats-client-rras</code>: Configuration of Round Robin Archives on the client. See [[#Factors affecting database size on the client side]]
+* <code>stats-server-rras</code>: Configuration of Round Robin Archives on the server. See [[#Factors affecting database size on the client side]]. This will '''NOT''' affect network bandwidth, but merely space consumed on the server.
+* <code>stats-step</code>: Step size of '''ALL''' RRAs. This '''WILL''' affect bandwidth consumption. See [[#Factors affecting database size on the client side]]
+* <code>trust-users</code>: Set this to True if using openSSH version lower than 5.6
+===Notes on using SSL auth===
+The steps broadly involved are:
+* [http://www.akadia.com/services/ssh_test_certificate.html Generating private key, and self-signed certificates (for test env).] The certificate should be without a passphrase.
+* Copying the certificate on the client, and configure <code>sugar-client</code> accordingly.
+* Copying the certificate and private key on the server, and configure <code>sugar-stats-server</code> accordingly.
+'''Notes on debugging the SSL auth system'''
+* Check the <code>stats-url</code> in <code>sugar-client</code> configuration is <code>https://</code> and not <code>http://</code>.
+* Check that the certificates are valid or have not expired.
+* To check the SSL side of things, following commands maybe run on the client and the server.
+** Server: <code>openssl s_server -cert server.crt -key server.key -accept 8000 </code>
+** Client: <code>openssl s_client -connect 192.168.1.55:8000 -CAfile server.crt </code> (Replace the ipaddress with the correct server-machine url or IP address.
+** If the process runs smoothly, you should see <code>Verify return code: 0 (ok)</code> at the end on the client side terminal. Fails will accompany an error code which maybe looked up [https://www.openssl.org/docs/apps/verify.html here]