572
edits
Changes
→Network traffic
=== Concepts ===
=== Concepts ===
[[File:Harvest-diagram.png]]
[[File:Harvest-diagram-4.png]]
* Activities refers to the sugar applications that are being used.
* Activities refers to the sugar applications that are being used.
* Learners refers to the sugar users.
* Learners refers to the sugar users.
* Instances refers to the different sessions an particular activity, owned by one learner.
* Instances refers to the different entries of an particular activity, owned by one learner.
* Launches refers to the different times the same session is started.
* Launches refers to the metadata associated with each time the instance is started.
* Counters refers to the network traffic measurements.
=== Metadata ===
=== Metadata ===
|'''Type'''
|'''Type'''
|-
|-
|rowspan="3"|Learners
|rowspan="4"|Learners
|serial_number
|serial_number
|Hashed laptop identifier
|Hashed laptop identifier
|-
|-
|birthdate
|birthdate
|Approximate date of birth of the user (See [[#Custom_Groups|Custom Groups]])
|Approximate date of birth of the user
|Unix time
|Unix time
|-
|-
|gender
|gender
|Gender of the user
|Gender of the user (either male, female, or unspecified by the user)
|String
|-
|grouping
|Custom group associated with his learner (see [[#Custom_Groups|Custom Groups]])
|String
|String
|-
|-
|String
|String
|-
|-
|rowspan="10"|Instances
|rowspan="9"|Instances
|object_id
|object_id
|Entry identifier
|Entry identifier
|buddies
|buddies
|Number of user's associated to the entry
|Number of user's associated to the entry
|Integer
|Integer
|-
|-
|String
|String
|-
|-
|rowspan="1"|Launches
|rowspan="2"|Launches
|timestamp
|timestamp
|Launch time for an particular entry
|Launch time for an particular entry
|Unix time
|Unix time
|-
|spent_time
|seconds that the activity was opened (see [[#Spent_time|Spent time]])
|Integer
|-
|rowspan="3"|Counters
|timestamp
|Time stamp for the beginning of the day (see [[#Network_traffic|Network traffic]])
|Unix time
|-
|download
|Bytes downloaded during that day
|Integer
|-
|upload
|Bytes uploaded during that day
|Integer
|}
|}
'''''Observation:''' All the metadata names, matches the original names of the journal metadata.''
'''''Observation:''' All the metadata names, matches the original names of the journal metadata.''
== How does it work? ==
The project comprises two pieces of software: a harvest server that can be localed anywhere in the cloud, and a harvest client that runs in the learners machine. The harvest server exposes a service, accessible from the Internet, for metadata storage. The harvest clients collect metadata from the Journal and sends it to server.
== When does it collect? ==
* Data is collected when Sugar starts and when Sugar successfully connects to a network.
* Once it has successfully collected data, it won't sent another report until the next collecting period, weekly or monthly.
* In order to avoid service peaks, Harvest applies a random chance for executing the collection process.
* Also, if the server is unresponsive, it won't retry for couple hours.
== What are the advantages? ==
* Learners data are never copied nor transferred out of their machines.
* The collection is being done continuously over time. This means that its sampling is very fine grained.
* It is very lightweight. It can be deployed in a central server.
* Does not require OS customization. The client is based on Sugar's web service framework, and it can be installed on any existing Sugar 0.100+ distribution.
== What is implemented so far? ==
Pretty much everything as it concerns for metadata collection.
=== Harvest server ===
* Back-end service for storage.
* SSL data encryption.
* API Key authorization.
* Control scripts based on systemd.
* DB migrations and continuous integration support.
* RPM packaging.
=== Harvest client ===
* Journal metadata collection.
* Web service extension.
* Extension controls from the web service control panel.
* Random selection.
* Exclusive log for debugging.
* Hashed serial numbers.
* Restricted retry policy.
* RPM packaging.
== Code ==
* https://github.com/tchx84/harvest-client
* https://github.com/tchx84/harvest-server
== External dependencies ==
=== Custom Groups ===
=== Custom Groups ===
However, not every deployment may want to group users by age, e.g., in Australia, they group students by grade.
However, not every deployment may want to group users by age, e.g., in Australia, they group students by grade.
Using a configuration file (/usr/share/sugar/data/group-stats.defaults), it is possible to configure the selection into groups
Using a configuration file (<code>/usr/share/sugar/data/group-labels.defaults</code>), it is possible to configure the selection into groups specific to the needs of a deployment. The configuration file is a JSON-encoded Python dictionary that maps ages to labels and icons associated with those labels. There is also an environment variable <code>$SUGAR_GROUP_LABELS</code> defined in <code>sugar.in</code>.
specific to the needs of a deployment. The configuration file is a JSON-encoded Python dictionary that maps ages to labels and icons associated with those labels.
The file that configures the default Sugar behavior is shown here (with added CRs for readability in the wiki):
The file that configures the default Sugar behavior is shown here (with added CRs for readability in the wiki):
]}
]}
== How does it work? ==
=== Network traffic ===
Harvest-monitor is a lightweight daemon which uses custom iptables counters to do measurements on network traffic. These counters are then accumulated in a SQLite database, where each row presents a day. This is an optional feature. If available, harvest-client will collect these measurements and report it to the server.
*The source code can be found at: https://github.com/tchx84/harvest-monitor
*The RPM package can be downloaded from: http://www.sugarlabs.org/~tch/repos/f18/harvest-monitor-0.2.0-2.noarch.rpm
* The collection is being done continuously over time. This means that its sampling is very fine grained.
* Does not require OS customization. The client is based on Sugar's web service framework, and it can be installed on any existing Sugar 0.100+ distribution.
== What is implemented so far? ==
=== Spent time ===
This is based on downstream sugar-toolkit and sugar-toolkit-gtk3 patches by Manuel Quiñones and Martin Abente. These patches allow the activities to count the time (in seconds) for when it was opened. Only takes into account the time for when activity is the main screen. This is also a optional feature.
* sugar-toolkit downstream pathches: https://github.com/manuq/sugar-toolkit/tree/spent-time
* Back-end service for storage.
* sugar-toolkit-gtk3 downstream patches: https://github.com/manuq/sugar-toolkit-gtk3/tree/spent-time-3
* SSL data encryption.
The precision of the time tracking can be improved by taking into account power management events and other sugar UI events. In order to do so, harvest-tracker must be installed.
* harvest-tracker source code: https://github.com/tchx84/harvest-tracker
* https://github.com/tchx84/harvest-client
* harvest-tracker RPM package: http://www.sugarlabs.org/~tch/repos/f18/harvest-tracker-0.3.0-1.noarch.rpm
* https://github.com/tchx84/harvest-server
== RPMs ==
== RPMs ==