Difference between revisions of "Features/Touch/Development"

 
(78 intermediate revisions by 2 users not shown)
Line 1: Line 1:
==Todo==
+
This page is about the development of the touch feature for the Sugar interface. Each item has a detailed implementation plan. The design decisions are based on these related pages:
  
=== Triage touch features ===
+
: [[Design_Team/Sugar_Shell_Touch_Input|Sugar Shell Touch Input]]
  
==== Essential ====
+
: [[Talk:Design_Team/Sugar_Shell_Touch_Input|Sugar Shell Touch Input Talk]]
  
* OSK
+
: [[Design_Team/Proposals/Touchscreen/On-screen_Keyboard|On Screen Keyboard]]
** Canvas auto panning based on input focus
+
 
** Show Frame or OSK, never both
+
: [[Design_Team/Activity_Touch_Input|Activity Touch Input]]
** Hide OSK key on OSK
+
 
** Show OSK button on device Frame edge
+
 
** Sugar theme
+
=Todo (essential)=
** Internationalisation
+
 
*** e.g. layout covers our accent needs
+
==On screen keybord (OSK)==
** Multilingual
+
==== Canvas auto panning based on input focus====
*** e.g. ability to switching between at least two languages
+
* [http://gitorious.org/maliit/maliit-framework/commit/fbf52a206f14ce57b6ec9eba309d8dd67592d882/diffs maliit-framework: Emit GtkIMContext::clear-area on VKB relocations]
** Landscape/portrait
+
* [http://git.gnome.org/browse/gtk+/log/?h=wip/im-osk-position GTK+ branch from Carlos]
** Reveal on text input focus, auto **dismiss** on loss of focus
+
* [http://dev.laptop.org/~erikos/touch/0001-Window-Handle-request-unset-clear-area-to-keep-focus.patch patch for the sugar-toolkit-gtk3] to listen in the 'request-clear-area' and 'unset-clear-area' signal
** Don't auto reveal in laptop mode, only ebook mode
+
 
* Text selection
+
==== Show Frame or OSK, never both====
** Selection handles
+
When the OSK is up and the frame is revealed the OSK should go away, same when the Frame is up and the OSK is revealed (manually or automatically) the Frame should hide.
** Cut(?)/Copy/Paste/Speak selection palette
+
* [http://gitorious.org/maliit/maliit-framework/blobs/master/maliit-glib/maliitinputmethod.h#line74 Maliit-glib API "maliit_input_method_hide"].
* Drag scrolling support
+
 
* Hide/show mouse cursor based on input method
+
==== Hide OSK key on OSK====
* Palette behaviours
+
There should be a key on the OSK to manually hide the keyboard. Currently you can already hide the Maliit keyboard by a vertical stroke but that is not as discoverable and does not work as reliable (as of today).
** Tapping icons with no primary action should instantly open full palette
+
 
*** e.g. Right Frame edge buddy icons, central home buddy icon
+
The [http://gitorious.org/maliit/maliit-plugins/blobs/master/maliit-keyboard/data/styles/olpc-xo/main.ini#line29 functionality is already there] to add it to the style. We just have to add the key in the layout [http://gitorious.org/maliit/maliit-plugins/blobs/master/maliit-keyboard/data/languages/de.xml (per language)].
*** e.g. Access Point device icon
+
 
* Up/down menu/combo overflow widgets need to be larger
+
==== Show OSK button on device Frame edge====
* Frame
+
There should be a key on the Frame in the device section to manually reveal the OSK.
** Expose/hide gesture!!!
+
* [http://gitorious.org/maliit/maliit-framework/blobs/master/maliit-glib/maliitinputmethod.h#line71 Maliit-glib API "maliit_input_method_show"], <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0009-Add-a-toolbutton-in-the-device-section-to-reveal-the.patch patch that adds a device button, clicking on it shows the OSK (works only when we ported the shell to dynamic bindings]
** Button for top left (fallback)
+
 
* Home view
+
==== Sugar theme====
** Disable Activity icon dragging in fixed layouts
+
The OSK should have a Sugar theme and good quality font.
*** e.g. allows easier access to long hold of Activity icon to access palette
+
 
* Neighbourhood view
+
==== Internationalisation====
** Remove primary action from AP icons
+
* e.g. layout covers our accent needs
** Remove primary action from ad-hoc icons
+
 
** Remove primary action from shared activity icons
+
==== Multilingual====
* Journal
+
* there should be the ability to switch between at least two languages, at the moment this can be done by swiping horizontally, this is not as discoverable and hard to do, would be nice to have a button for that (see iOS)
** Touch & hold on an entry should invoke full palette
+
* add languages, could be done in a control panel section
** …Dragging and entry needs to work but not prevent touch * hold palette if finger moves a little
+
 
*** e.g. to copy it to another volume
+
==== Landscape/portrait====
* Combobox
+
Currently not working anymore, [https://bugs.maliit.org/show_bug.cgi?id=183 upstream Maliit bug].
** Single touch of a combo should lock open it's palette and allow drag scrolling
+
 
** Should not show blank space when previous selection was near the bottom of the list
+
==== Reveal on text input focus, auto **dismiss** on loss of focus====
* Touch & hold == right click equivalent
+
* Reveal the OSK when clicking e.g. on a GTK entry (working quite well already)
* Visual feedback for button presses
+
* Dismiss the OSK when defocus: e.g. in Browse the OSK comes up when clicking in the url entry, clicking on the canvas defocus the entry and hide the OSK (is already working fine), in other places e.g. in the Neighborhood View when the search field is focused we have to see how we can do better, we can experiment with a) not putting the default focus in the search widget; b) on physical keyboard typing auto focus into the search widget. For b) we would need to check there is not some other text input widget in focus, e.g. in Journal view you might be trying to retitle an entry and would not want the search to grab input at that time, <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0007-Unfocus-the-search-entry-in-the-Home-and-Neighborhoo.patch patch to defocus when clicking on the canvas] and [http://dev.laptop.org/~erikos/touch/0008-Set-the-GTK_IM_MODULE-env-variable-to-Maliit.patch patch to set the env variable for Maliit]
** Generic solution - use a mouse HW cursor with alpha white 'glow' marking all primary touch events?  
+
 
** Existing GTK button down highlights work well in toolbars
+
====Don't auto reveal in laptop mode, only ebook mode====
** Shell canvas area views
+
For devices which have a hardware keyboard and a touchscreen the OSK should not be auto revealed in the "laptop mode". You will be able to manually reveal it with the button in the Frame.
*** Use grey round rect outline for presses
+
 
**** AP icons
+
== Text selection ==
**** Activity icons
+
* Selection handles
**** Buddy icons
+
* Cut(?)/Copy/Paste/Speak selection palette
**** Ad-hoc icons
+
* [http://wiki.sugarlabs.org/go/Design_Team/Sugar_Shell_Touch_Input#Text_editing design discussions]
*** Only really needed where there is no palette and/or primary action is too slow
+
* [http://git.gnome.org/browse/gtk+/log/?h=touch-text-selection GTK+ branch from Carlos]
* Visual feedback for touch and hold
+
 
** Animated circle timer?
+
== Drag scrolling support==
** Blinking grey round rect outline
+
For example scrolling in the Journal or the activity list view.
* Activities
+
 
** Write
+
== Hide/show mouse cursor based on input method==
** Browse
+
When in Ebook mode, or when using the touch screen, the cursor should be hidden. When using the trackpad the cursor should be revealed. Idealy fade the cursor in and out of visibility using alpha if possible.
** Paint
+
 
** Read
+
== Palette behaviours ==
** Calculate
+
* Tapping/clicking icons with no primary action should instantly open full palette
** Turtle Art
+
** Frame: buddy icons
 +
** central home buddy icon, <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0001-Make-the-palette-come-up-on-left-click-for-the-buddy.patch patch]
 +
** Frame: network device icons and the battery icon, <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0006-Frame-Device-left-click-on-network-icons-or-battery-.patch patch]
 +
* Tapping/clicking toolbar icons with no primary action that are open should close the palette
 +
** e.g. Journal: sort icon
 +
* Home View activity icons: leave behavior as is or always show palette, ideally we have the intermediate screen
 +
* Journal: auto-resume is the primary action when clicking on an activity entry, current behavior is correct here
 +
 
 +
== Frame ==
 +
* Expose/hide gesture: reveal/hide the Frame by a swipe in/out gesture, revealing would be a gesture over the frame area into the center and the other way around to hide, will collide when you sketch at the edge of the Paint activity if the gesture is not carefully tuned (a swipe, not slow drag, of short length perpendicular to frame edge, diagonal at corners).
 +
* Button for top left (fallback)
 +
* Use the square game button on the XO as a Frame button, <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0003-Use-the-square-game-button-on-the-XO-as-a-Frame-butt.patch patch]
 +
* do not hide frame on mouse out, do hide frame when going to a hot corner (and frame is visible), <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0005-New-frame-interaction.patch patch]
 +
* hide frame when an activity is selected (todo: hide as well when resume in the palette is clicked), <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0005-New-frame-interaction.patch patch] (the problem is when you want to get to the secondary options - use touch & hold ?, the resume option should be more visible as primary option - hint at separator)
 +
* hide frame when a zoom level is selected, <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0005-New-frame-interaction.patch patch]
 +
 
 +
==Switch between Views==
 +
* use Frame (see Frame section)
 +
* discuss: use Gesture (pinch/zoom, or horizontal swipe)
 +
 
 +
==Switch between activities==
 +
* use Frame (see Frame section)
 +
* discuss: use Gesture (horizontal swipe)
 +
 
 +
==Gestures==
 +
* we want the following gestures to be supported in Activities: touch-and-hold, swipes (left/right/up/down), and pinch-to-zoom
 +
* we need global gestures as well (e.g. Frame)
 +
* http://git.gnome.org/browse/gtk+/commit/?h=gestures
 +
 
 +
== Home view==
 +
* Disable Activity icon dragging in fixed layouts
 +
** e.g. allows easier access to long hold of Activity icon to access palette
 +
 
 +
== Neighbourhood view==
 +
* Remove primary action from AP icons (raise palette), <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0002-Make-the-AP-icon-the-Ad-hoc-and-the-Mesh-network-ico.patch patch]
 +
* Remove primary action from ad-hoc/mesh icons (raise palette), <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0002-Make-the-AP-icon-the-Ad-hoc-and-the-Mesh-network-ico.patch patch]
 +
* Remove primary action from shared activity icons (raise palette), <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0004-Neighbourhood-view-remove-primary-action-from-shared.patch patch]
 +
* Remove primary action from buddy icons (raise palette), <span style="color:#f99506;">testing</span> [http://dev.laptop.org/~erikos/touch/0001-Make-the-palette-come-up-on-left-click-for-the-buddy.patch patch]
 +
 
 +
== Journal ==
 +
* Touch & hold on an entry should invoke full palette
 +
* Dragging an entry needs to work but not prevent the touch and hold palette (if finger moves a little)
 +
** e.g. to copy it to another volume
 +
** see how it is handled in the home screen in iOS for touch & hold and the swipe gesture
 +
* add copy option for the DOCUMENTS folder in the entry Palette and Details View Copy toolbar button
 +
* Detail/List View: Erase (and Duplicate) tools should both raise confirmation alerts
 +
 
 +
==Combobox==
 +
* Single touch of a combo should lock open it's palette and allow drag scrolling, <span style="color:#2ce713;"> Fix from: Carlos Garnacho </span>[http://git.gnome.org/browse/gtk+/commit/?id=bd3ca2b30efc534f8b7c18dfd8a9f072592044c7 landed in GTK+ 3.5.10]
 +
* Should not show blank space when previous selection was near the bottom of the list (not as bad when the scrolling works, [https://bugzilla.gnome.org/show_bug.cgi?id=678113 discussion why technically this is really hard to do])
 +
* Up/down menu/combo overflow widgets need to be larger (could probably be done in the theme)
 +
* [http://dev.laptop.org/~erikos/touch/activity.py activity.py] for HelloWorld activity that has a ComboBox to test with
 +
 
 +
== Touch & hold '==' right click equivalent==
 +
The right click is the equivalent to touch and hold, e.g. bringing up the Palette immediately.
 +
 
 +
== Visual feedback for button presses==
 +
* Generic solution - use a mouse HW cursor with alpha white 'glow' marking all primary touch events?  
 +
* Existing GTK button down highlights work well in toolbars
 +
* Shell canvas area views
 +
** Use grey round rect outline for presses
 +
*** AP icons
 +
*** Activity icons
 +
*** Buddy icons
 +
*** Ad-hoc icons
 +
** Only really needed where there is no palette and/or primary action is too slow
 +
 
 +
== Visual feedback for touch and hold ==
 +
* Animated circle timer? Carlos did some work on that in GTK+, see in [http://people.gnome.org/~carlosg/oggs/multitouch.ogg this video], at ~30 seconds.
 +
* Blinking grey round rect outline
  
==== Ideal ====
+
= Todo (ideal) =
  
 
* Palette behaviours
 
* Palette behaviours
Line 91: Line 160:
 
* Priorities UI redraw over other processes
 
* Priorities UI redraw over other processes
 
** e.g. don't stall the UI thread
 
** e.g. don't stall the UI thread
* Activities
 
** Physics
 
** Clock
 
** Memorize
 
** Image Viewer
 
** Abacus
 
** FotoToon
 
** Implode
 
** Labyrinth
 
** Chat
 
** Help
 
*** e.g. pick up web kit features
 
** WikipediaEN/ES
 
*** e.g. pick up web kit features
 
** Maze
 
  
==== Polish ====
+
 
 +
= Todo (polish) =
  
 
* Smooth transitions/animations
 
* Smooth transitions/animations
Line 119: Line 174:
 
* Pre-light hover effects (remove for touch screen input)
 
* Pre-light hover effects (remove for touch screen input)
 
* hspan/vspan separators, make sure they are large enough for finger interaction
 
* hspan/vspan separators, make sure they are large enough for finger interaction
* Activities
 
** Record
 
** Distance
 
** Get Books
 
** InfoSlicer
 
** Jukebox
 
** Log
 
** Measure
 
** Moon
 
** Portfolio
 
** Pippy
 
** Ruler
 
** Speak
 
** StopWatch
 
** Terminal
 
** Typing Turtle
 
** Words
 
 
* OSK
 
* OSK
 
** Subtle key click feedback
 
** Subtle key click feedback
Line 147: Line 185:
 
** Palette options presented in fullscreen UI, before activity pulse loader
 
** Palette options presented in fullscreen UI, before activity pulse loader
  
==== Not in this cycle ====
 
  
* Activities
+
=Done=
** TamTamEdit
+
 
** TamTamJam
+
=Test=
** TamTamMini
+
==Apply patches to Sugar==
** TamTamSynthLab
+
* yum install gettext GConf2-devel gtk2-devel intltool perl-XML-Parser pkgconfig python pygtk2-devel make gcc
** Etoys
+
* git clone git://git.sugarlabs.org/sugar/mainline.git shell
** Scratch
+
* cd shell
 +
* git checkout 6fbc4188249a6f59cc55381824cf1ec1f90dc5e8 (master has changed already since the patches have been made)
 +
* ./autogen.sh --prefix=/usr
 +
* wget http://dev.laptop.org/~erikos/touch/0001-Make-the-palette-come-up-on-left-click-for-the-buddy.patch
 +
* git am 0001-Make-the-palette-come-up-on-left-click-for-the-buddy.patch
 +
* make
 +
* sudo make install
 +
 
 +
==Patches==
 +
The patches are in [http://dev.laptop.org/~erikos/touch/ my public home folder]. They are as well linked for the individual item they fix above. Most of them should be independent from each other, to be sure to not run into any dep issues you need to apply them in order.
  
==Done==
+
==Restart Sugar==
 +
If you want to be able to restart Sugar with the ctrl+alt+bksp command on the XO you can set the Option "DontZap" to "false" in the "ServerFlags" section in the xorg config file (/etc/X11/xorg.conf.d/xo1.75.conf).

Latest revision as of 07:02, 22 August 2012

This page is about the development of the touch feature for the Sugar interface. Each item has a detailed implementation plan. The design decisions are based on these related pages:

Sugar Shell Touch Input
Sugar Shell Touch Input Talk
On Screen Keyboard
Activity Touch Input


Todo (essential)

On screen keybord (OSK)

Canvas auto panning based on input focus

Show Frame or OSK, never both

When the OSK is up and the frame is revealed the OSK should go away, same when the Frame is up and the OSK is revealed (manually or automatically) the Frame should hide.

Hide OSK key on OSK

There should be a key on the OSK to manually hide the keyboard. Currently you can already hide the Maliit keyboard by a vertical stroke but that is not as discoverable and does not work as reliable (as of today).

The functionality is already there to add it to the style. We just have to add the key in the layout (per language).

Show OSK button on device Frame edge

There should be a key on the Frame in the device section to manually reveal the OSK.

Sugar theme

The OSK should have a Sugar theme and good quality font.

Internationalisation

  • e.g. layout covers our accent needs

Multilingual

  • there should be the ability to switch between at least two languages, at the moment this can be done by swiping horizontally, this is not as discoverable and hard to do, would be nice to have a button for that (see iOS)
  • add languages, could be done in a control panel section

Landscape/portrait

Currently not working anymore, upstream Maliit bug.

Reveal on text input focus, auto **dismiss** on loss of focus

  • Reveal the OSK when clicking e.g. on a GTK entry (working quite well already)
  • Dismiss the OSK when defocus: e.g. in Browse the OSK comes up when clicking in the url entry, clicking on the canvas defocus the entry and hide the OSK (is already working fine), in other places e.g. in the Neighborhood View when the search field is focused we have to see how we can do better, we can experiment with a) not putting the default focus in the search widget; b) on physical keyboard typing auto focus into the search widget. For b) we would need to check there is not some other text input widget in focus, e.g. in Journal view you might be trying to retitle an entry and would not want the search to grab input at that time, testing patch to defocus when clicking on the canvas and patch to set the env variable for Maliit

Don't auto reveal in laptop mode, only ebook mode

For devices which have a hardware keyboard and a touchscreen the OSK should not be auto revealed in the "laptop mode". You will be able to manually reveal it with the button in the Frame.

Text selection

Drag scrolling support

For example scrolling in the Journal or the activity list view.

Hide/show mouse cursor based on input method

When in Ebook mode, or when using the touch screen, the cursor should be hidden. When using the trackpad the cursor should be revealed. Idealy fade the cursor in and out of visibility using alpha if possible.

Palette behaviours

  • Tapping/clicking icons with no primary action should instantly open full palette
    • Frame: buddy icons
    • central home buddy icon, testing patch
    • Frame: network device icons and the battery icon, testing patch
  • Tapping/clicking toolbar icons with no primary action that are open should close the palette
    • e.g. Journal: sort icon
  • Home View activity icons: leave behavior as is or always show palette, ideally we have the intermediate screen
  • Journal: auto-resume is the primary action when clicking on an activity entry, current behavior is correct here

Frame

  • Expose/hide gesture: reveal/hide the Frame by a swipe in/out gesture, revealing would be a gesture over the frame area into the center and the other way around to hide, will collide when you sketch at the edge of the Paint activity if the gesture is not carefully tuned (a swipe, not slow drag, of short length perpendicular to frame edge, diagonal at corners).
  • Button for top left (fallback)
  • Use the square game button on the XO as a Frame button, testing patch
  • do not hide frame on mouse out, do hide frame when going to a hot corner (and frame is visible), testing patch
  • hide frame when an activity is selected (todo: hide as well when resume in the palette is clicked), testing patch (the problem is when you want to get to the secondary options - use touch & hold ?, the resume option should be more visible as primary option - hint at separator)
  • hide frame when a zoom level is selected, testing patch

Switch between Views

  • use Frame (see Frame section)
  • discuss: use Gesture (pinch/zoom, or horizontal swipe)

Switch between activities

  • use Frame (see Frame section)
  • discuss: use Gesture (horizontal swipe)

Gestures

Home view

  • Disable Activity icon dragging in fixed layouts
    • e.g. allows easier access to long hold of Activity icon to access palette

Neighbourhood view

  • Remove primary action from AP icons (raise palette), testing patch
  • Remove primary action from ad-hoc/mesh icons (raise palette), testing patch
  • Remove primary action from shared activity icons (raise palette), testing patch
  • Remove primary action from buddy icons (raise palette), testing patch

Journal

  • Touch & hold on an entry should invoke full palette
  • Dragging an entry needs to work but not prevent the touch and hold palette (if finger moves a little)
    • e.g. to copy it to another volume
    • see how it is handled in the home screen in iOS for touch & hold and the swipe gesture
  • add copy option for the DOCUMENTS folder in the entry Palette and Details View Copy toolbar button
  • Detail/List View: Erase (and Duplicate) tools should both raise confirmation alerts

Combobox

  • Single touch of a combo should lock open it's palette and allow drag scrolling, Fix from: Carlos Garnacho landed in GTK+ 3.5.10
  • Should not show blank space when previous selection was near the bottom of the list (not as bad when the scrolling works, discussion why technically this is really hard to do)
  • Up/down menu/combo overflow widgets need to be larger (could probably be done in the theme)
  • activity.py for HelloWorld activity that has a ComboBox to test with

Touch & hold '==' right click equivalent

The right click is the equivalent to touch and hold, e.g. bringing up the Palette immediately.

Visual feedback for button presses

  • Generic solution - use a mouse HW cursor with alpha white 'glow' marking all primary touch events?
  • Existing GTK button down highlights work well in toolbars
  • Shell canvas area views
    • Use grey round rect outline for presses
      • AP icons
      • Activity icons
      • Buddy icons
      • Ad-hoc icons
    • Only really needed where there is no palette and/or primary action is too slow

Visual feedback for touch and hold

  • Animated circle timer? Carlos did some work on that in GTK+, see in this video, at ~30 seconds.
  • Blinking grey round rect outline

Todo (ideal)

  • Palette behaviours
    • Single touch should never trigger hover content
    • Touch and hold should raise hover content
    • Touching the icon of an open palette should close the palette
      • e.g. Journal list sort palette
  • Toolbar overflow widget should be same size as Sugar toolbar button icons
  • Zoom to content behaviour
    • e.g. Browse
  • Frame
    • Activity Zoom icon (and physical key) should act like Alt-Tab
  • Home view
    • Improve layouts
      • e.g. larger Activity icon usage where possible
      • e.g. Increase separation between icons where possible
  • Details view
    • Alert dialogue needed for Erase toolbar button
    • Alert dialogue needed for Duplicate toolbar button
  • Accelerometer auto rotate based on rotation when laptop is in ebook mode
  • Drag scrolling
    • Smooth!!
  • Priorities UI redraw over other processes
    • e.g. don't stall the UI thread


Todo (polish)

  • Smooth transitions/animations
  • Text selection
    • Feature extensible selection palette
      • e.g. Activity specific formatting
  • Home view
    • New grid layout option
  • Make sure Activities and shell works well in portrait
    • e.g. toolbar button overflow in some Activities
  • Pre-light hover effects (remove for touch screen input)
  • hspan/vspan separators, make sure they are large enough for finger interaction
  • OSK
    • Subtle key click feedback
    • Custom context
    • Custom key strips/extras based on Activity
    • Predictive text
      • Context sensitive
      • Locale sensitive
  • Activity start/resume screen
    • Remove start/resume palette
    • Palette options presented in fullscreen UI, before activity pulse loader


Done

Test

Apply patches to Sugar

Patches

The patches are in my public home folder. They are as well linked for the individual item they fix above. Most of them should be independent from each other, to be sure to not run into any dep issues you need to apply them in order.

Restart Sugar

If you want to be able to restart Sugar with the ctrl+alt+bksp command on the XO you can set the Option "DontZap" to "false" in the "ServerFlags" section in the xorg config file (/etc/X11/xorg.conf.d/xo1.75.conf).