Sugar Labs

Development Team/Datastore Rewrite: Difference between revisions

← Older edit
Tomeu (talk | contribs)
647 edits
 
(13 intermediate revisions by 6 users not shown)
Line 1: Line 1:
<noinclude>{{ GoogleTrans-en | es =show | bg =show | zh-CN =show | zh-TW =show | hr =show | cs =show | da =show | nl =show | fi =show | fr =show | de =show | el =show | hi =show | it =show | ja =show | ko =show | no =show | pl =show | pt =show | ro =show | ru =show | sv =show }}</noinclude>
{{TOCright}}
== Introduction ==
This page describes the design of a new DataStore implementation that was shipped first in the [[0.84]] (April 2009) Sugar release.
== Goals ==
== Goals ==


Line 38: Line 45:


Metadata for each entry is stored in several files, one per property. In this way, if corruption happened on one those properties, the rest of the entry (and the other entries in the DS) would be unaffected.
Metadata for each entry is stored in several files, one per property. In this way, if corruption happened on one those properties, the rest of the entry (and the other entries in the DS) would be unaffected.
: See [[olpc:Low-level_Activity_API#Meta_Data]].


=== Queries are accelerated with a disposable database ===
=== Queries are accelerated with a disposable database ===
Line 57: Line 65:
             data
             data
             metadata
             metadata
                  uid
                  checksum
                   activity_id
                   activity_id
                   mime_type
                   mime_type
Line 66: Line 76:
       2b90597c-0912-4e7f-8eeb-71a0f004490d
       2b90597c-0912-4e7f-8eeb-71a0f004490d
             data
             data
            checksum ~> checksums/464493d8d929436b6152e868867ed451
             metadata
             metadata
                  uid
                  checksum
                   activity_id
                   activity_id
                   mime_type
                   mime_type
Line 77: Line 88:
       3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a
       3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a
             data -> 2b/2b90597c-0912-4e7f-8eeb-71a0f004490d/data
             data -> 2b/2b90597c-0912-4e7f-8eeb-71a0f004490d/data
            checksum ~> checksums/464493d8d929436b6152e868867ed451
             metadata
             metadata
                  uid
                  checksum
                   activity_id
                   activity_id
                   mime_type
                   mime_type
Line 89: Line 101:
             data
             data
             metadata
             metadata
                  uid
                  checksum
                   activity_id
                   activity_id
                   mime_type
                   mime_type
Line 98: Line 112:
             data
             data
             metadata
             metadata
                  uid
                  checksum
                   activity_id
                   activity_id
                   mime_type
                   mime_type
Line 106: Line 122:
  checksums
  checksums
       464493d8d929436b6152e868867ed451
       464493d8d929436b6152e868867ed451
             2b90597c-0912-4e7f-8eeb-71a0f004490d ~> 2b90597c-0912-4e7f-8eeb-71a0f004490d/data
             2b90597c-0912-4e7f-8eeb-71a0f004490d
             3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a ~> 3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a/data
             3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a
  index
  index
       flintlock
       flintlock
Line 123: Line 139:
       value.baseB
       value.baseB
       value.DB
       value.DB
index_updated
version


'''1a''': directory holding entries, it's only function is to avoid having too many directories in a single directory, as this is considered specially harmful on jffs2.
'''1a''': directory holding entries, it's only function is to avoid having too many directories in a single directory, as this is considered specially harmful on jffs2.
Line 135: Line 153:


'''3c/3cdf5f0e-...-cbedfcfe1c4a/data''': hard link to the same file in the entry '''2b90597c-...-71a0f004490d'''
'''3c/3cdf5f0e-...-cbedfcfe1c4a/data''': hard link to the same file in the entry '''2b90597c-...-71a0f004490d'''
'''3c/3cdf5f0e-...-cbedfcfe1c4a/checksum ~> checksums/464493d8d929436b6152e868867ed451''': symbolic link to the file in '''checksums'''. Used to get the checksum of the entry without having to recalculate it nor read it from the metadata file


'''checksums''': directory containing a directory per each file contained in the DS, named by its md5 checksum
'''checksums''': directory containing a directory per each file contained in the DS, named by its md5 checksum


'''checksums/464493d8d929436b6152e868867ed451''': directory containing links to all the entries that contain a file with this checksum
'''checksums/464493d8d929436b6152e868867ed451''': directory containing files named by each entry that contain a file with this checksum


'''checksums/464493d8d929436b6152e868867ed451/2b90597c-...-71a0f004490d''' symbolic link to a file in an entry with this checksum.
'''checksums/464493d8d929436b6152e868867ed451/2b90597c-...-71a0f004490d''' file named by the uid of an entry the file of which has this checksum.


'''index''': directory containing all files that belong to the search database. Can be deleted and recreated from the rest of the DS if needed without incurring in data loss.
'''index''': directory containing all files that belong to the search database. Can be deleted and recreated from the rest of the DS if needed without incurring in data loss.
'''index_updated''': When this file is not present, is because the xapian index is being rebuilt because it cannot be opened. Queries should fall back to display all the files on disk until the index is rebuilt.
'''version''': file that contains the version of the file layout. Currently we are at 1. Is updated when an earlier version is migrated to a newer one.


== Source code ==
== Source code ==


http://dev.laptop.org/git?p=users/tomeu/datastore;a=summary
http://git.sugarlabs.org/projects/sugar-datastore
Retrieved from "https://wiki.sugarlabs.org/go/Development_Team/Datastore_Rewrite"