Difference between revisions of "User:Alsroot/trash/Object Bundles"

From Sugar Labs
Jump to navigation Jump to search
 
(26 intermediate revisions by 5 users not shown)
Line 10: Line 10:
 
* package Journal objects to bundles to preserve sugar related metadata
 
* package Journal objects to bundles to preserve sugar related metadata
 
* support composite Journal objects e.g. in case of library bundles instead of having .xol in Journal and unpacked files in ~/Library, all library files will be represented by one Journal entry, could be opened in Browse(''[Metadata]/mime_type'' should be text/html for that purpose) and could be transformed back to .xo on demand(e.g. for uploading library in Browse's pick-file field).
 
* support composite Journal objects e.g. in case of library bundles instead of having .xol in Journal and unpacked files in ~/Library, all library files will be represented by one Journal entry, could be opened in Browse(''[Metadata]/mime_type'' should be text/html for that purpose) and could be transformed back to .xo on demand(e.g. for uploading library in Browse's pick-file field).
 +
* Object Bundles exist only out of sugar (after uploading to the Journal .xo will be removed)
  
 
== Owner ==
 
== Owner ==
Line 18: Line 19:
 
== Current status ==
 
== Current status ==
  
* Targeted release: 0.86
+
* Targeted release:
* Last updated: Tue Jul 21 02:18:17 UTC 2009
+
* Last updated: Mon Jul 27 05:31:42 UTC 2009
* Percentage of completion: 0%
+
* Percentage of completion: 75%
  
 
== Detailed Description ==
 
== Detailed Description ==
Line 26: Line 27:
 
This feature is a first approach to unified format for all types of bundles(in 0.86 it will support only Journal entries and new library bundles).
 
This feature is a first approach to unified format for all types of bundles(in 0.86 it will support only Journal entries and new library bundles).
  
=== Bundle files hierarchy ===
+
Object bundle should have ''METADATA'' file in the top directory of .xo bundle.
  
* ''MANIFEST'' EOL-terminalted list of files inside bundle
+
=== METADATA file ===
* ''METADATA'' file in [http://docs.python.org/library/configparser.html INI] format which describes bundle
 
  
=== METADATA file ===
+
This file is in [http://docs.python.org/library/configparser.html INI] format which describes how to setup bundle.
  
Object bundle can be installed to Journal in two forms:
+
Object bundle can be installed to Journal in two forms.
* files from bundle will be unpacked and installing as separate files, bundle itself will be removed(similar to .xoj)<br>''METADATA'' should not contain ''[Entry]'' section
 
* bundles will be installed as a composite object i.e. as a directory of packaged to the bundle files<br>''METADATA'' should contain ''[Entry]'' section
 
  
Any field in METADATA file can have ''_file'' suffix, in that case content of this field(substring w/o ''_file'' suffix) will be fetched from file inside of the bundle.
+
* '''Each file from bundle as separate Journal entry'''<br>files from bundle will be unpacked and installed as separate Journal entries, bundle itself will be removed(similar to .xoj)<br>final Journal entries can have ''activity'' field, so they are regular Journal entries that imported to Journal(w/o ''activity'' field) or created by activities(with ''activity'' field)<br>''METADATA'' should contain ''[Entry]'' sections for each Journal entry<br>if bundle has more then one entry, sections should be parted by different suffixes e.g. ''[Entry2]'', ''[Entry.additional]'' etc.
  
==== [Metadata] section ====
+
* '''Composite Journal entry'''<br>bundle will be installed as a composite object i.e. as a directory of packaged to the bundle files that are represented by one Journal entry<br>final Journal entry can't have ''activity'' field<br>''METADATA'' should contain only one ''[Bundle]'' section
  
This section is mandatory. It describes Datastore metadata fields of final entry in Journal. If bundle is composite(section ''[Entry]'' presents) this section defines Journal entry to represent composite object, otherwise Journal entry for ''[Metadata]/entry'' file.
+
METADATA file can have one or several sections(depends on installation method) that describe metadata fields of final entry(ies) in Journal.
  
 
{| border=1 cellpadding=3 style="border: 1px solid white; border-collapse: collapse; background: #e3e4e5;"
 
{| border=1 cellpadding=3 style="border: 1px solid white; border-collapse: collapse; background: #e3e4e5;"
Line 51: Line 49:
 
| ''entry''
 
| ''entry''
 
| mandatory
 
| mandatory
| if bundle is composite, ''entry'' defines access point to composite object (e.g. index.html for library bundles); otherwise it defines file which will be installed to Journal
+
| if bundle is composite, ''entry'' defines access point to composite object (e.g. index.html for library bundles);<br>otherwise it defines file which will be installed to Journal
 
|-
 
|-
 
| ''mime_type''
 
| ''mime_type''
 
| mandatory
 
| mandatory
| define metadata for a file which will be stored in Journal
+
| define metadata for final Journal entry
 +
|-
 +
| ''uid''
 +
| ignored
 
|-
 
|-
 
| ''*''
 
| ''*''
Line 63: Line 64:
 
|}
 
|}
  
METADATA file could have several ''[Manifest]'' sections, in that case they should be parted by different suffixes e.g. ''[Manifest2]'', ''[Manifest.additional]'' etc. Multi-object bundles could be utilized in >0.86 for collections of objects or actions.
+
Any field in ''METADATA'' file can have ''_file'' suffix, in that case content of this field(substring w/o ''_file'' suffix) will be fetched from file inside of the bundle.
 
 
==== [Entry] section ====
 
 
 
If this section exists bundle will be installed as a composite object i.e. all bundle files will be represented by one Journal entry(could be useful in case of libraries).
 
 
 
{| border=1 cellpadding=3 style="border: 1px solid white; border-collapse: collapse; background: #e3e4e5;"
 
|-style="background:#787878; color: white;"
 
! Field
 
! Flags
 
! Notes
 
|-
 
| ''exec''
 
| optional
 
| how to activate Journal entry, overrides ''[Metadata]'' fields(like ''activity'' and ''mime_type'') of ''[Metadata]/entry'' file
 
|-
 
|}
 
 
 
=== MANIFEST file ===
 
 
 
File which contains a new-line-terminated list of file names inside bundle.
 
  
 
== Benefit to Sugar ==
 
== Benefit to Sugar ==
Line 103: Line 84:
 
** journal entries, former .xoj bundles (0.86)
 
** journal entries, former .xoj bundles (0.86)
 
* make Browse upload object bundles when the server says so
 
* make Browse upload object bundles when the server says so
* in an ideal implementation, composite feature should come from Datastore, but since DS has been extensively hacked in 0.86-0.88 cycles, for the first time, it could be part of shell code
+
* in an ideal implementation, composite feature should come from Datastore, but since DS has been extensively hacked in 0.86-0.88 cycles, for the first time, it could be a part of shell code
 +
 
 +
==== Implementation ====
 +
 
 +
* [http://git.sugarlabs.org/projects/sugar/repos/objectbundle sugar]
 +
* [http://git.sugarlabs.org/projects/sugar-toolkit/repos/objectbundle sugar-toolkit]
 +
* [http://git.sugarlabs.org/projects/browse/repos/objectbundle Browse]
 +
 
 +
Example of PHP code which could be used on server side to unpack object bundles to get metadata (like title, description, preview etc.)
 +
 
 +
require_once('Archive/Zip.php');
 +
 +
function rmtree($dir) {
 +
    $dir = "$dir";
 +
 +
    if ($dh = opendir($dir)) {
 +
        while (FALSE !== ($item = readdir($dh))) {
 +
            if ($item != '.' && $item != '..') {
 +
                $subdir = $dir . '/' . "$item";
 +
                if (is_dir($subdir))
 +
                    rmtree($subdir);
 +
                else
 +
                    unlink($subdir);
 +
            }
 +
        }
 +
        closedir($dh);
 +
        rmdir($dir);
 +
    }
 +
}
 +
 +
function unbundle($bundle) {
 +
    $zip = new Archive_Zip($bundle);
 +
 +
    $tmpdir = tempnam('/tmp', 'sugar');
 +
    if (file_exists($tmpdir))
 +
        unlink($tmpdir);
 +
    if (!mkdir($tmpdir)) {
 +
        return null;
 +
    }
 +
 +
    $out = $zip->extract(array('add_path' => $tmpdir, 'by_name' => array('METADATA')));
 +
    if (empty($out)) {
 +
        rmtree($tmpdir);
 +
        return null;
 +
    }
 +
 +
    $metadata = parse_ini_file($out[0]['filename'], true);
 +
    if (!$metadata) {
 +
        rmtree($tmpdir);
 +
        return null;
 +
    }
 +
 +
    $result = array();
 +
 +
    foreach ($metadata as $section => $items) {
 +
        if (!(substr($section, 0, 5) == 'Entry' || $section == 'Bundle'))
 +
            continue;
 +
        foreach ($items as $name => $value) {
 +
            if (substr($name, -5) == '_file') {
 +
                $name = substr($name, 0, -5);
 +
 +
                $out = $zip->extract(array('add_path' => $tmpdir, 'by_name' => array($value)));
 +
                if (empty($out)) {
 +
                    rmtree($tmpdir);
 +
                    return null;
 +
                }
 +
 +
                $value = file_get_contents($tmpdir.'/'.$value);
 +
            }
 +
            $result[$name] = $value;
 +
        }
 +
        break;
 +
    }
 +
 +
    rmtree($tmpdir);
 +
    return $result;
 +
}
 +
 +
// example usage of unbundle() function
 +
 +
$bundle = 'foo.xo';
 +
$metadata = unbundle($bundle);
 +
 +
if (!$metadata)
 +
    echo 'Can not open '.$bundle."\n";
 +
else
 +
    foreach ($metadata as $name => $value)
 +
        echo $name.'='.$value."\n";
  
 
== How To Test ==
 
== How To Test ==
Line 113: Line 181:
 
# Resume it from Journal
 
# Resume it from Journal
  
How to test composite object:
+
How to test composite object (unfortunately [http://dev.sugarlabs.org/ticket/1101 #1101] prevents uploading bundles with library to wiki):
 
# Download any .xol bundle
 
# Download any .xol bundle
 
# Activate it(it should be opened in Browse)
 
# Activate it(it should be opened in Browse)
Line 121: Line 189:
 
== User Experience ==
 
== User Experience ==
  
Except that [http://wiki.laptop.org/go/Journal_entry_bundles .xoj] has:
+
We could have HTTP servers(see [[Features/Server Objects Sharing]]) that request .xo files, in that case Browse will provide object bundle for chosen(in ObjectChooser) Journal entry otherwise Browse will upload raw Journal entries.
* users can create object bundles outside of sugar(they do not have to follow JSON rules)
 
  
 
== Dependencies ==
 
== Dependencies ==
Line 134: Line 201:
 
== Documentation ==
 
== Documentation ==
  
 +
* [http://www.mail-archive.com/sugar-devel@lists.sugarlabs.org/msg06874.html Email thread]
 
* [[Unified Bundles]]
 
* [[Unified Bundles]]
 +
* [[User:Alsroot/trash/Unified Objects | Unified Objects]]
 
* [http://wiki.laptop.org/go/Journal_entry_bundles Journal entry bundles]
 
* [http://wiki.laptop.org/go/Journal_entry_bundles Journal entry bundles]
 
* [http://wiki.laptop.org/go/Bundle_concepts Bundle concepts]
 
* [http://wiki.laptop.org/go/Bundle_concepts Bundle concepts]
 +
* [http://wiki.laptop.org/go/Manifest_Specification#Contents_Manifests MANIFEST specification]
  
 
== Release Notes ==
 
== Release Notes ==
Line 143: Line 213:
 
== Comments and Discussion ==
 
== Comments and Discussion ==
 
* See [[{{TALKPAGENAME}}|discussion tab for this feature]] <!-- This adds a link to the "discussion" tab associated with your page.  This provides the ability to have ongoing comments or conversation without bogging down the main feature page -->
 
* See [[{{TALKPAGENAME}}|discussion tab for this feature]] <!-- This adds a link to the "discussion" tab associated with your page.  This provides the ability to have ongoing comments or conversation without bogging down the main feature page -->
 
 
[[Category:Feature Page Incomplete]]
 
[[Category:Feature]]
 

Latest revision as of 14:27, 1 March 2010


Summary

  • package Journal objects to bundles to preserve sugar related metadata
  • support composite Journal objects e.g. in case of library bundles instead of having .xol in Journal and unpacked files in ~/Library, all library files will be represented by one Journal entry, could be opened in Browse([Metadata]/mime_type should be text/html for that purpose) and could be transformed back to .xo on demand(e.g. for uploading library in Browse's pick-file field).
  • Object Bundles exist only out of sugar (after uploading to the Journal .xo will be removed)

Owner

Current status

  • Targeted release:
  • Last updated: Mon Jul 27 05:31:42 UTC 2009
  • Percentage of completion: 75%

Detailed Description

This feature is a first approach to unified format for all types of bundles(in 0.86 it will support only Journal entries and new library bundles).

Object bundle should have METADATA file in the top directory of .xo bundle.

METADATA file

This file is in INI format which describes how to setup bundle.

Object bundle can be installed to Journal in two forms.

  • Each file from bundle as separate Journal entry
    files from bundle will be unpacked and installed as separate Journal entries, bundle itself will be removed(similar to .xoj)
    final Journal entries can have activity field, so they are regular Journal entries that imported to Journal(w/o activity field) or created by activities(with activity field)
    METADATA should contain [Entry] sections for each Journal entry
    if bundle has more then one entry, sections should be parted by different suffixes e.g. [Entry2], [Entry.additional] etc.
  • Composite Journal entry
    bundle will be installed as a composite object i.e. as a directory of packaged to the bundle files that are represented by one Journal entry
    final Journal entry can't have activity field
    METADATA should contain only one [Bundle] section

METADATA file can have one or several sections(depends on installation method) that describe metadata fields of final entry(ies) in Journal.

Field Flags Notes
entry mandatory if bundle is composite, entry defines access point to composite object (e.g. index.html for library bundles);
otherwise it defines file which will be installed to Journal
mime_type mandatory define metadata for final Journal entry
uid ignored
* optional any system, users predefined and arbitrary Datastore field

Any field in METADATA file can have _file suffix, in that case content of this field(substring w/o _file suffix) will be fetched from file inside of the bundle.

Benefit to Sugar

This feature is a first approach to unified format for all types of bundles.

Current implementation is similar to .xoj bundles except that:

  • it uses INI format instead of json(to make it more user-editing friendly)
  • any field's value could fetched from file inside of the bundle
  • supports multi-object bundles

Scope

  • deprecate .xol bundles
  • deprecate .xoj bundles
  • provide unified format for metadata file in .xo bundles which should support
    • activities, former .xo bundles (0.88)
    • libraries, former .xol bundles (0.86)
    • journal entries, former .xoj bundles (0.86)
  • make Browse upload object bundles when the server says so
  • in an ideal implementation, composite feature should come from Datastore, but since DS has been extensively hacked in 0.86-0.88 cycles, for the first time, it could be a part of shell code

Implementation

Example of PHP code which could be used on server side to unpack object bundles to get metadata (like title, description, preview etc.)

require_once('Archive/Zip.php');

function rmtree($dir) {
   $dir = "$dir";

   if ($dh = opendir($dir)) {
       while (FALSE !== ($item = readdir($dh))) {
           if ($item != '.' && $item != '..') {
               $subdir = $dir . '/' . "$item";
               if (is_dir($subdir))
                   rmtree($subdir);
               else
                   unlink($subdir);
           }
       }
       closedir($dh);
       rmdir($dir);
   }
}

function unbundle($bundle) {
   $zip = new Archive_Zip($bundle);

   $tmpdir = tempnam('/tmp', 'sugar');
   if (file_exists($tmpdir))
       unlink($tmpdir);
   if (!mkdir($tmpdir)) {
       return null;
   }

   $out = $zip->extract(array('add_path' => $tmpdir, 'by_name' => array('METADATA')));
   if (empty($out)) {
       rmtree($tmpdir);
       return null;
   }

   $metadata = parse_ini_file($out[0]['filename'], true);
   if (!$metadata) {
       rmtree($tmpdir);
       return null;
   }

   $result = array();

   foreach ($metadata as $section => $items) {
       if (!(substr($section, 0, 5) == 'Entry' || $section == 'Bundle'))
           continue;
       foreach ($items as $name => $value) {
           if (substr($name, -5) == '_file') {
               $name = substr($name, 0, -5);

               $out = $zip->extract(array('add_path' => $tmpdir, 'by_name' => array($value)));
               if (empty($out)) {
                   rmtree($tmpdir);
                   return null;
               }

               $value = file_get_contents($tmpdir.'/'.$value);
           }
           $result[$name] = $value;
       }
       break;
   }

   rmtree($tmpdir);
   return $result;
}

// example usage of unbundle() function

$bundle = 'foo.xo';
$metadata = unbundle($bundle);

if (!$metadata)
   echo 'Can not open '.$bundle."\n";
else
   foreach ($metadata as $name => $value)
       echo $name.'='.$value."\n";

How To Test

How to test non-composite objects:

  1. Create a new TurtleArt activity
  2. Upload the new entry to the SL wiki using Browse
  3. Use Browse to download the entry back to Journal
  4. Resume it from Journal

How to test composite object (unfortunately #1101 prevents uploading bundles with library to wiki):

  1. Download any .xol bundle
  2. Activate it(it should be opened in Browse)
  3. Upload Journal entry with library to SL wiki using Browse
  4. Download .xo with library and open it

User Experience

We could have HTTP servers(see Features/Server Objects Sharing) that request .xo files, in that case Browse will provide object bundle for chosen(in ObjectChooser) Journal entry otherwise Browse will upload raw Journal entries.

Dependencies

Fructose dependencies.

Contingency Plan

None necessary, revert to previous release behaviour.

Documentation

Release Notes

The Sugar Release Notes inform end-users about what is new in the release. An Example is 0.84/Notes. The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns. If there are any such changes involved in this feature, indicate them here. You can also link to upstream documentation if it satisfies this need. This information forms the basis of the release notes edited by the release team and shipped with the release.

Comments and Discussion