Features/GTK3
Summary
Sugar needs to rebase itself on new generations of its key underlying technologies: GTK+ 3 and PyGObject Introspection. This page aims to summarise options and community opinions around this challenging shift, and to help formulate a plan of how it will be executed. In other words, it tries to take community input, answer all the unanswered questions, and present a logical path forward that can be adopted by the developers.
Owner
- This plan/proposal maintained by Daniel Drake (other editors welcome!)
- A number of developers will be needed at each stage to successfully execute this; Daniel offers his assistance for a coordination/oversight role if that would be useful.
Current status
- Targeted release: (SUGAR_VERSION)
- Last updated: (DATE)
- Percentage of completion: XX%
Detailed Description
Two major changes have recently occurred in Sugar's underlying technologies. Firstly, GTK+ 2 has been obsoleted by GTK+ 3, and GNOME is now based on GTK+ 3. Secondly, PyGTK, the underlying Python library that Sugar uses to call into GTK+, has been deprecated in favour of PyGObject Introspection (hereafter "PyGI").
In order to remain innovative and current, and to take advantage of the latest developments, Sugar needs to follow suit and move to GTK3 and PyGI. Lagging behind on this conversion is already bringing negative consequences to Sugar; notably 2 of our most important activities (Read and Browse) are already broken and without a future until we catch up again.
Unfortunately, this is an API-incompatible change. As confirmed by the PyGI developers, PyGTK and PyGI cannot be mixed in the same process. This means that the conversion process can't be done on a fine, incremental basis, and if sugar-toolkit were to be simply replaced with a PyGI/GTK3 port, this means that all existing activities would stop working until they themselves have also been ported - all activities will need modifications as part of this feature. The community has expressed desire for old activities to continue working (many are unmaintained); unfortunately this is not realistic in the long term. As a compromise, this feature discussion includes the requirement that for a certain period of time, both PyGTK/GTK2 and PyGI/GTK3 activities must be able to function alongside each other. As detailed below, this can be pulled off quite easily and in a way that will not drain resources.
This project will involve modifications to almost every file within Sugar and its activities, making it a huge task, even though the vast majority of code is trivial to port. This feature discussion attempts to identify ways that this porting process can be done in distinct stages, where Sugar remains functional at the completion of each stage, making this more manageable.
For the most part, the port from PyGTK to PyGI only involves changing the names that are used to access methods and constants. PyGI uses a slightly different and more consistent naming scheme to wrap GTK+.
import gtk gtk.MessageDialog(None, 0, gtk.MESSAGE_INFO, gtk.BUTTONS_CLOSE, "Hello World").run()
The above PyGTK code rewritten as PyGI is:
from gi.repository import Gtk Gtk.MessageDialog(None, 0, Gtk.MessageType.INFO, Gtk.ButtonsType.CLOSE, "Hello World").run()
PyGI provides a script which can be used to automate much of the conversion.
The move from GTK2 to GTK3 is also expected to be unproblematic, because the vast majority of code does not need any changes - most GTK3 API/ABI is identical to GTK2. The cases which do require some intervention to solve will take some time, but the changes are well documented and the number of cases encountered should be low.
Benefit to Sugar
- PyGI is technologically better than PyGTK. It is a nicer way of calling into GObject-style libraries from Python that means less maintenance is needed upstream (PyGObject automates the creation of bindings to a degree much higher than PyGTK ever could, and automatically achieves more complete coverage).
- PyGTK is no longer maintained; PyGI is actively maintained.
- The move to GTK3 allows us to keep up with our GNOME neighbours, as they improve and refine the base technologies that we share.
- The move to PyGI is expected to result in lower memory usage and faster startup.
- Browse has no future under GTK2, it needs to move to WebKit and that move is dependent on Sugar moving to PyGI/GTK3.
- Similarly, Read has a grim future under GTK2 due to static evince bindings no longer being maintained and libevince itself moved to GTK3; we need to switch to PyGI/GTK3 to be able to keep calling into evince and let the Read activity live on.
Implementation Plan
Sugar is divided into different components, many of which run in different processes. This means that we are able to divide up the required work on a process-by-process basis. While this work is being conducted, some Sugar processes might be based on PyGI/GTK3 and others based on PyGTK/GTK2, but the platform would keep functioning at each stage. There would be some system resource overhead during this transitional time (as the system would need to have PyGTK, GTK2, PyGI and GTK3 all in memory) but this feature implementation would end with the whole sugar ecosystem using PyGI/GTK3.
HippoCanvas removal
A prerequisite to porting a Sugar process, component or activity is to remove all its usage of hippocanvas. This library is unmaintained, would be painful to port to GTK3, and can be done better with standard GTK+ code at the Python level. Most users of HippoCanvas can switch to custom GtkContainer widgets.
One complication is that hippo usage is not limited to Sugar's core; some activities use hippo, or they pull on sugar-toolkit classes which are implemented with hippo. These are: Chat (please list others here).
Backwards-compatibility for activities
As mentioned above, as mixing PyGTK/GTK2 and PyGI/GTK3 is impossible, a straightforward port of sugar-toolkit to PyGI/GTK3 would break all activities. However, having backwards compatibility for activities for a limited time is considered a requirement, and having a "flag day" where all activities stop working until ported is not seen as an option - there must be a transition period.
As the conversion process from PyGTK to PyGI makes minor changes to almost every single line of code that involves GTK/glib, maintaining both PyGI and PyGTK codepaths in the same files is not realistic (there would be an unrealistic amount of if conditions). We are therefore required to have two copies of sugar-toolkit during the transition period: one for ported PyGI/GTK3 activities, and another for PyGTK/GTK2 activities that have not yet been ported.
As we do not have plentiful developer resources, I propose that the PyGTK/GTK2 version of sugar-toolkit is frozen as soon as this feature development is underway. No bugfixes or improvements, just a copy of the code at that point.
Selecting a sugar-toolkit version
With two versions of sugar-toolkit available, activities need a mechanism for choosing which toolkit they will use. As we require existing activities to continue working without modifications, this selection mechanism must be opt-in - ported activities must somehow mark themselves as being GTK3-ported, and hence will receive the GTK3 version of sugar-toolkit, and all other activities will receive the GTK2 version.
One easy way to achieve this would be to give the GTK3 version of sugar-toolkit a different name. The unported GTK2 activities could continue to import sugar.foo and ported GTK3 activities would import sugar1.foo. "sugar1" is chosen as an example to indicate Sugar-1.0, but other naming schemes would obviously work equally well.
If the community has strong desires, it would be possible maintain the sugar module name through a Python trick which could be deployed during the transition period, and then removed once the GTK2 version is dropped. Here is how it could be done:
- The new GTK3 version of sugar toolkit would be installed with name sugar, and the old GTK2 version would be installed with name sugar_gtk2.
- Before sugar-activity imports sugar.activity (or any other sugar-toolkit class), it would look for an empty file called "GTK3_PORTED" in the activity's directory. If present, it would run a little trick:
import os
import sys
if not os.path.exists("GTK3_PORTED"):
import sugar_gtk2
sys.modules["sugar"] = sugar_gtk2
- The result is that all unported/unmodified activities (without the GTK3_PORTED file) would import sugar.foo as before, but the above trick that modifies Python's module table would result in them (transparently, magically, automatically) being redirected to sugar_gtk2.foo.
- At the end of the transition period, sugar_gtk2 would be deleted in entirity, the above addition to sugar-activity would be dropped, and activities could drop the GTK3_PORTED files at their leisure.
Further consideration must be given to sugar-base. This package installs some classes which are then used by sugar-toolkit, so they would need to be duplicated into sugar_gtk2. This may be a good opportunity to consider merging sugar-base with sugar-toolkit.
Proposed plan of action
The steps below prioritise the porting of sugar-toolkit, as this is where Sugar would see the most immediate benefit: the revival of Browse and Read. The steps that follow can largely be parallelised.
- Remove hippocanvas from sugar-toolkit
- Port sugar-toolkit to GTK3 while keeping backwards compatibility
- Rescue Browse and Read, and allow independent activity porting efforts to begin
- Remove hippocanvas from all other parts of Sugar, including activities
- Port Sugar core to GTK3
- Some time later, remove sugar-toolkit's GTK2 compatibility.
- I propose that this happens 1 year after release of the first sugar-toolkit GTK3 version
How to port
I plan to start a page at Features/GTK3/Porting that details the porting process, that could be provided as documentation to anyone involved in these efforts (including those working on porting Sugar core, but also those working on activities). The content covered will be:
- How to remove hippo and what to replace it with (links to commits that have done this for other activities, etc)
- How to select the GTK3 version of sugar-toolkit
- How to handle each of the sugar-toolkit API changes (detailed in the following section)
- How to port from PyGTK/GTK2 to PyGI/GTK3.
- This basically involves reading the PyGObject porting documentation, using pygi-convert.sh, then checking the result.
- The Migrating from GTK+ 2.x to GTK+ 3 should also be read.
Version number considerations
It goes without saying that such a migration would be the basis of a new major release. When this topic has been discussed before, people have toyed with the idea of calling the GTK3 version of Sugar "Sugar-1.0".
There are arguments for:
- This is undoubtedly a paradigm shift for Sugar, so a major bump to the version number is called for.
- It would make the change really obvious
And there are arguments against:
- Some people might interpret the number 1.0 as indicating a higher level of maturity than what the developers feel
- Some people want a much longer lead-up time to Sugar-1.0 so that the API can be refined/reworked/perfected
Tomeu, Simon and Marco agreed that the 1.0 version number could be used here, and communicated in a slightly different sense: we are really still in the first iteration and 1.0 will be when that first iteration reaches maturity, without big changes in the API. After 1.0 we can start working on what will be one day 2.0 which should be the second iteration of Sugar, hopefully using what we have learned during these years.
The migration of sugar-toolkit, sugar, datastore, etc, is likely to take more than 1 release cycle, but the above scheme can still apply. When each component is ported to GTK3, it would then pick up the 1.0 tag. For example, the first major release that includes any GTK3 may well include sugar-toolkit-1.0 (GTK3 ported) alongside sugar-datastore-0.96 (not yet ported).
API changes
The fact that almost all Sugar components and activities require sweeping changes as part of this shift presents an interesting opportunity for us to make API changes in sugar-toolkit. The importance here should still be placed on the technology shift, rather than on the opportunity to produce a perfect API (which we could spend all eternity designing and discussing), but this is an opportunity that we should not miss.
I propose that once sugar-toolkit is ported and mostly operational, we run a 30-day window where API changes that have seen some kind of planning and discussion below can be made and committed.
Plain text as the default for palettes
Sugar's palette classes currently accept strings, but then they pass those strings to GTK as markup (always, unconditionally). However, markup is only used in a handful of places, and this means that all users of palettes that draw in translations or strings from other sources which might contain characters such as < or > must escape their arguments, leading to big patches like this.
As agreed here, it would make more sense for palettes to pass those strings as standard text by default, and to have different functions to opt-in to receive markup processing. Then the excessive escaping would go, and the only users would have to escape would be those who use markup.
Removal of keep button
Sugar-0.94 proposes the removal of the Keep button, but the old KeepButton classes would be kept around for activities that directly use it. Porting sugar-toolkit to GTK3 would be a good time to remove these classes.
Removal of old toolbars
Sugar-0.86 redesigned activity toolbars, with the new toolbars implemented by new classes, and the old classes being kept around so that old activities are not immediately broken. This would be a good opportunity to remove the long-deprecated old activity toolbar classes.
Dependencies
Sugar would start depending on PyGObject built with gobject introspection support, GTK3, and (at the end of the transition period) would drop its dependencies on GTK2 and PyGTK.
Contingency plan
Hopefully, this page has been convincing in that this change is of necessity. Either way, we should consider our options for the case where this migration is found to be more difficult than predicted.
As this migration would take over the course of a major release (or perhaps 2-3 of them, as components are ported individually), our users have the option to remain on older releases in the face of stability issues, and as developers we have the option of delaying major releases until things are ready.
If new PyGI/GTK3-based major releases are found to be unstable, we also have the option of allowing interested community members to pick up maintenance of older GTK2-based release branches (e.g. 0.94) with a "master first" commit policy. Personally, I would suggest that it would be a better use of resources to have those people help fix any problems with the GTK3 version, but as an open source community this is something we must be open to.
In the event of problems in the GTK3 version of sugar-toolkit, activity authors have the (default) choice of staying with GTK2. The transition period before the GTK2 version is removed would obviously be extended in the face of significant problems with GTK3 version.