Difference between revisions of "Platform Team/packagekit-backend-presolve"
Jump to navigation
Jump to search
m (→Usage examples) |
|||
| Line 1: | Line 1: | ||
== Summary == | == Summary == | ||
| − | This is a [http://www.packagekit.org/ PackageKit] backend which is intended to be used on restricted systems like XO-1 laptops where regular package management routines are too heavy for limited | + | This is a [http://www.packagekit.org/ PackageKit] backend which is intended to be used on restricted systems, like XO-1 laptops, where regular package management routines are too heavy for limited amounts of memory or CPU resources. |
== The reason == | == The reason == | ||
| Line 8: | Line 8: | ||
* [http://wiki.laptop.org/go/Olpc-update olpc-update]<br>[[wikipedia:Rsync|Rsync]] the full system. | * [http://wiki.laptop.org/go/Olpc-update olpc-update]<br>[[wikipedia:Rsync|Rsync]] the full system. | ||
| − | * [[Platform_Team/Server_Kit/sugar-client|sugar-client]]'s [[Platform_Team/Server_Kit/sugar-client#Unattended_system_update|unattended update]] in [[Dextrose]]<br>Regular update but only for particular, much smaller than main ones, repository(ies). | + | * [[Platform_Team/Server_Kit/sugar-client|sugar-client]]'s [[Platform_Team/Server_Kit/sugar-client#Unattended_system_update|unattended update]] in [[Dextrose]]<br>Regular update, but only for particular, much smaller than main ones, repository(ies). |
| − | Both | + | Both methods are useful for the situations they were created for. But there is a situation when they both become less useful. Having [[Activity Library]] (or its distributed version on regional or school server), XO users might download activities directly from the Internet (or a distributed server). So, it is becoming hard to use: |
| − | * olpc-update,<br>[[Activity Library]] might have non-standard dependencies and it is either hard to keep olpc-update server up-to-date, or, | + | * olpc-update,<br>[[Activity Library]] might have non-standard dependencies, and it is either hard to keep the olpc-update server up-to-date, or, impractical to install all possible dependencies on every XO. |
* sugar-client,<br>activity dependencies are, in most cases, from official repositories when sugar-client's 3rd party repository are useless. | * sugar-client,<br>activity dependencies are, in most cases, from official repositories when sugar-client's 3rd party repository are useless. | ||
| − | In other words, there is a need to install particular packages from official Fedora repository and using existing methods is either impractical (olpc-update) or impossible (sugar-client). | + | In other words, there is a need to install particular packages from an official Fedora repository and using existing methods is either impractical (olpc-update) or impossible (sugar-client). The fallback to use regular methods (like yum) is also not useful, because they take too much memory (at the beginning) and CPU resources. This is a special problem on XO-1 laptops. |
| − | The packagekit-backend-presolve is intended to solve this issue by providing standard way (it is PackageKit backend, i.e., sugar launching code will be common for all platforms, it will call PackageKit | + | The packagekit-backend-presolve method is intended to solve this issue by providing a standard way (it is PackageKit backend, i.e., sugar launching code will be common for all platforms, it will always call PackageKit) to install activity dependencies and consumes as little as possible resources on XO laptops. The backend was designed to support reliable launching from services like [[Activity Library]] and/or [[Sugar Network]] when there are activities that might have non-standard dependencies. This backend might be treated as a temporary solution, i.e., when either the XO is powerful enough, or regular installation procedures take fewer resources, it will be possible to switch to the default PackageKit backend (without making changes in activity launching code). |
== Concept == | == Concept == | ||
| − | The general idea is to pre-resolve dependencies for a limited set of top-level packages and store this information on a server. Clients need to download only small amount of information (compared with entire list of available packages) to reuse it as-is (and avoid reading database into memory during solving process). | + | The general idea is to pre-resolve dependencies for a limited set of top-level packages and store this information on a server. Clients need to download only a small amount of information (compared with the entire list of available packages) to reuse it as-is (and avoid reading a database into memory during the solving process). |
| − | Another important | + | Another important feature is that pre-resolved dependency graphs don't include packages from the base system. In other words, a particular dependency tree doesn't end with the {{Code|glibc}} package. Instead, it ends with the {{Code|sugar}} package. This expedient should drastically reduce the size of trees. |
Finally, missing packages will be installed directly by using the {{Code|rpm -i}} command. | Finally, missing packages will be installed directly by using the {{Code|rpm -i}} command. | ||
| Line 30: | Line 30: | ||
=== Upsides === | === Upsides === | ||
| − | * Users download metadata on a package by package basis, thus avoiding the need to download the full packages database; | + | * Users download metadata on a package-by-package basis, thus avoiding the need to download the full packages database; |
| − | * After downloading, metadata will be reused as-is, i.e., after checking what packages were already installed, system will download only | + | * After downloading, metadata will be reused as-is, i.e., after checking what packages were already installed, the system will download only missing RPMs, and install them directly by calling the {{Code|rpm -i}} command; |
* Activities on resources like [[Activity Library]] or [[Sugar Network]] will be distro-agnostic. That is, they will work unmodified on all platforms that provide PackageKit (which means all mainstream GNU/Linux distributions). | * Activities on resources like [[Activity Library]] or [[Sugar Network]] will be distro-agnostic. That is, they will work unmodified on all platforms that provide PackageKit (which means all mainstream GNU/Linux distributions). | ||
=== Downsides === | === Downsides === | ||
| − | * The project is intended to install only limited number of top-level packages, i.e., packages that have pre-resolved dependency trees on a server; for example, it is not | + | * The project is intended to install only a limited number of top-level packages, i.e., packages that have pre-resolved dependency trees on a server; for example, it is not suitable for a full system update; |
* Need to support server-side generation of dependency trees. | * Need to support server-side generation of dependency trees. | ||
== Implementation == | == Implementation == | ||
| − | === | + | === Premises === |
| − | The implementation is following | + | The implementation is following these premises: |
| − | * Being targeted only to XO laptops (assuming that other platforms are more powerful | + | * Being targeted only to XO laptops (assuming that other platforms are more powerful, thus the standard PackageKit backend can be used); |
| − | * Keep less work on XO side and more work on server side to make installation process on client side as fast and reliable as possible; | + | * Keep less work on the XO side and more work on the server side to make the installation process on a client's side as fast and reliable as possible; |
| − | * Implement only | + | * Implement only priority functionality, and provide more complex features as required, e.g., the current implementation doesn't support updating already installed packages. Leave that to hard (and rare, e.g., yearly) full system updates (re-flashes or calling olpc-update) and incremental regular updates (sugar-client). |
=== Overview === | === Overview === | ||
* Clients keep a list of packages installed by the back-end; | * Clients keep a list of packages installed by the back-end; | ||
| − | * After getting install request, the back-end will check if the requested package is already installed; | + | * After getting an install request, the back-end will check if the requested package is already installed; |
| − | * If not, it downloads a file from | + | * If not, it downloads a file from a known HTTP location with the requested package's name; |
| − | * Downloaded file contains RPM url for requested package and a list of package-name/url pairs for all dependencies (starting from some known point); | + | * Downloaded file contains the RPM url for the requested package and a list of package-name/url pairs for all dependencies (starting from some known point); |
* All missing packages will be downloaded and installed by directly calling {{Code|rpm -i}}; | * All missing packages will be downloaded and installed by directly calling {{Code|rpm -i}}; | ||
* Installed packages list will be updated correspondingly. | * Installed packages list will be updated correspondingly. | ||
| Line 60: | Line 60: | ||
== Usage examples == | == Usage examples == | ||
| − | [[Sugar Network]], in attempt to provide GNU/Linux distribution agnostic launches of activities it provides | + | [[Sugar Network]], in its attempt to provide GNU/Linux-distribution-agnostic launches of activities, it provides and supports packagekit-backend-presolve users. |
API server provides the following kinds of urls: | API server provides the following kinds of urls: | ||
* {{Code|<nowiki>http://api-devel.network.sugarlabs.org/packages</nowiki>}}<br>list all supported platforms; | * {{Code|<nowiki>http://api-devel.network.sugarlabs.org/packages</nowiki>}}<br>list all supported platforms; | ||
| − | * {{Code|<nowiki>http://api-devel.network.sugarlabs.org/packages/</nowiki><PLATFORM>}}<br>list all supported packages for particular platform; | + | * {{Code|<nowiki>http://api-devel.network.sugarlabs.org/packages/</nowiki><PLATFORM>}}<br>list all supported packages for a particular platform; |
| − | * {{Code|<nowiki>http://api-devel.network.sugarlabs.org/packages/</nowiki><PLATFORM>/<PACKAGE>}}<br>pre-solved dependency graph for particular package. | + | * {{Code|<nowiki>http://api-devel.network.sugarlabs.org/packages/</nowiki><PLATFORM>/<PACKAGE>}}<br>pre-solved dependency graph for a particular package. |
Where: | Where: | ||
| Line 73: | Line 73: | ||
* {{Code|PACKAGE}}<br>a name of the particular package. | * {{Code|PACKAGE}}<br>a name of the particular package. | ||
| − | In order to support distribution agnostic launches, [[Sugar Network]] supports metadata ''database'' for all packages that are being used as dependencies for activities it provides (see ''Packages'' project using one of existing [[Sugar_Network#Try_it|Sugar Network clients]]). Every entry in the ''database'' contains a map of native package names for particular GNU/Linux | + | In order to support distribution-agnostic launches, [[Sugar Network]] supports a metadata ''database'' for all packages that are being used as dependencies for activities it provides (see ''Packages'' project using one of the existing [[Sugar_Network#Try_it|Sugar Network clients]]). Every entry in the ''database'' contains a map of native package names for a particular GNU/Linux distribution. So, activities mention only the ''database'' entry name as a dependency. [[Sugar Network]] will automatically generate dependency graphs for packagekit-backend-presolve using the ''metadata'' database. Afterwards, on the launch side, the ''database'' entry name will be resolved to a package name according to the local distribution and it will be passed to the PackageKit (on XO laptops, packagekit-backend-presolve will be used; on regular platforms, the default PackageKit back-end is used). |
== Sources == | == Sources == | ||
Revision as of 22:53, 14 September 2012
Summary
This is a PackageKit backend which is intended to be used on restricted systems, like XO-1 laptops, where regular package management routines are too heavy for limited amounts of memory or CPU resources.
The reason
The known, for the author of these lines, current practice is:
- olpc-update
Rsync the full system. - sugar-client's unattended update in Dextrose
Regular update, but only for particular, much smaller than main ones, repository(ies).
Both methods are useful for the situations they were created for. But there is a situation when they both become less useful. Having Activity Library (or its distributed version on regional or school server), XO users might download activities directly from the Internet (or a distributed server). So, it is becoming hard to use:
- olpc-update,
Activity Library might have non-standard dependencies, and it is either hard to keep the olpc-update server up-to-date, or, impractical to install all possible dependencies on every XO.
- sugar-client,
activity dependencies are, in most cases, from official repositories when sugar-client's 3rd party repository are useless.
In other words, there is a need to install particular packages from an official Fedora repository and using existing methods is either impractical (olpc-update) or impossible (sugar-client). The fallback to use regular methods (like yum) is also not useful, because they take too much memory (at the beginning) and CPU resources. This is a special problem on XO-1 laptops.
The packagekit-backend-presolve method is intended to solve this issue by providing a standard way (it is PackageKit backend, i.e., sugar launching code will be common for all platforms, it will always call PackageKit) to install activity dependencies and consumes as little as possible resources on XO laptops. The backend was designed to support reliable launching from services like Activity Library and/or Sugar Network when there are activities that might have non-standard dependencies. This backend might be treated as a temporary solution, i.e., when either the XO is powerful enough, or regular installation procedures take fewer resources, it will be possible to switch to the default PackageKit backend (without making changes in activity launching code).
Concept
The general idea is to pre-resolve dependencies for a limited set of top-level packages and store this information on a server. Clients need to download only a small amount of information (compared with the entire list of available packages) to reuse it as-is (and avoid reading a database into memory during the solving process).
Another important feature is that pre-resolved dependency graphs don't include packages from the base system. In other words, a particular dependency tree doesn't end with the glibc package. Instead, it ends with the sugar package. This expedient should drastically reduce the size of trees.
Finally, missing packages will be installed directly by using the rpm -i command.
Upsides
- Users download metadata on a package-by-package basis, thus avoiding the need to download the full packages database;
- After downloading, metadata will be reused as-is, i.e., after checking what packages were already installed, the system will download only missing RPMs, and install them directly by calling the
rpm -icommand; - Activities on resources like Activity Library or Sugar Network will be distro-agnostic. That is, they will work unmodified on all platforms that provide PackageKit (which means all mainstream GNU/Linux distributions).
Downsides
- The project is intended to install only a limited number of top-level packages, i.e., packages that have pre-resolved dependency trees on a server; for example, it is not suitable for a full system update;
- Need to support server-side generation of dependency trees.
Implementation
Premises
The implementation is following these premises:
- Being targeted only to XO laptops (assuming that other platforms are more powerful, thus the standard PackageKit backend can be used);
- Keep less work on the XO side and more work on the server side to make the installation process on a client's side as fast and reliable as possible;
- Implement only priority functionality, and provide more complex features as required, e.g., the current implementation doesn't support updating already installed packages. Leave that to hard (and rare, e.g., yearly) full system updates (re-flashes or calling olpc-update) and incremental regular updates (sugar-client).
Overview
- Clients keep a list of packages installed by the back-end;
- After getting an install request, the back-end will check if the requested package is already installed;
- If not, it downloads a file from a known HTTP location with the requested package's name;
- Downloaded file contains the RPM url for the requested package and a list of package-name/url pairs for all dependencies (starting from some known point);
- All missing packages will be downloaded and installed by directly calling
rpm -i; - Installed packages list will be updated correspondingly.
Usage examples
Sugar Network, in its attempt to provide GNU/Linux-distribution-agnostic launches of activities, it provides and supports packagekit-backend-presolve users.
API server provides the following kinds of urls:
http://api-devel.network.sugarlabs.org/packages
list all supported platforms;http://api-devel.network.sugarlabs.org/packages/<PLATFORM>
list all supported packages for a particular platform;http://api-devel.network.sugarlabs.org/packages/<PLATFORM>/<PACKAGE>
pre-solved dependency graph for a particular package.
Where:
PLATFORM
one of supported packagekit-backend-presolve platforms, e.g., OLPC-13.0.1;PACKAGE
a name of the particular package.
In order to support distribution-agnostic launches, Sugar Network supports a metadata database for all packages that are being used as dependencies for activities it provides (see Packages project using one of the existing Sugar Network clients). Every entry in the database contains a map of native package names for a particular GNU/Linux distribution. So, activities mention only the database entry name as a dependency. Sugar Network will automatically generate dependency graphs for packagekit-backend-presolve using the metadata database. Afterwards, on the launch side, the database entry name will be resolved to a package name according to the local distribution and it will be passed to the PackageKit (on XO laptops, packagekit-backend-presolve will be used; on regular platforms, the default PackageKit back-end is used).
Sources
Clone source repository from Gitorious project:
git clone git://git.sugarlabs.org/desktop/packagekit-backend-presolve.git
Getting involved
- Subscribe to the sugar-devel mailing list and email with the subject prefixed with [PLATFORM].
- Ask your question on IRC channels, #sugar (not logged) or #sugar-newbies (logged).