Sugar Labs

Development Team/Datastore Rewrite: Difference between revisions

← Older edit
Tomeu (talk | contribs)
647 edits
 
(17 intermediate revisions by 6 users not shown)
Line 1: Line 1:
<noinclude>{{ GoogleTrans-en | es =show | bg =show | zh-CN =show | zh-TW =show | hr =show | cs =show | da =show | nl =show | fi =show | fr =show | de =show | el =show | hi =show | it =show | ja =show | ko =show | no =show | pl =show | pt =show | ro =show | ru =show | sv =show }}</noinclude>
{{TOCright}}
== Introduction ==
This page describes the design of a new DataStore implementation that was shipped first in the [[0.84]] (April 2009) Sugar release.
== Goals ==
== Goals ==


Line 35: Line 42:
By examining the directory structure, we know where is localized the data related to each entry. We don't depend any more on a binary structure that could become corrupted and unusable as a whole.
By examining the directory structure, we know where is localized the data related to each entry. We don't depend any more on a binary structure that could become corrupted and unusable as a whole.


=== Metadata is stored as a single file for each entry ===
=== Metadata is stored in a single file per property ===


Each entry has its metadata stored in a single file, so that if corruption happened on one of those, the rest of the entries would be unaffected. The format in which metadata is stored even allows to recover from a malformed property by just dropping it.
Metadata for each entry is stored in several files, one per property. In this way, if corruption happened on one those properties, the rest of the entry (and the other entries in the DS) would be unaffected.
: See [[olpc:Low-level_Activity_API#Meta_Data]].


=== Queries are accelerated with a disposable database ===
=== Queries are accelerated with a disposable database ===
Line 55: Line 63:
  1a
  1a
       1ab88287-766a-4d98-a7c0-4233dc48647a
       1ab88287-766a-4d98-a7c0-4233dc48647a
             1ab88287-766a-4d98-a7c0-4233dc48647a
             data
             metadata
             metadata
                  uid
                  checksum
                  activity_id
                  mime_type
                  preview
                  share-scope
                  timestamp
                  title
  2b
  2b
       2b90597c-0912-4e7f-8eeb-71a0f004490d
       2b90597c-0912-4e7f-8eeb-71a0f004490d
             2b90597c-0912-4e7f-8eeb-71a0f004490d
             data
             checksum ~> checksums/464493d8d929436b6152e868867ed451
             metadata
            extra_metadata
                  uid
                  checksum
                  activity_id
                  mime_type
                   preview
                   preview
            metadata
                  share-scope
                  timestamp
                  title
  3c
  3c
       3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a
       3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a
             3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a -> 2b/2b90597c-0912-4e7f-8eeb-71a0f004490d/2b90597c-0912-4e7f-8eeb-71a0f004490d
             data -> 2b/2b90597c-0912-4e7f-8eeb-71a0f004490d/data
             checksum ~> checksums/464493d8d929436b6152e868867ed451
             metadata
            extra_metadata
                  uid
                  checksum
                  activity_id
                  mime_type
                   preview
                   preview
            metadata
                  share-scope
                  timestamp
                  title
  4d
  4d
       4db11d29-2f07-4452-bd8e-22a6a483ac19
       4db11d29-2f07-4452-bd8e-22a6a483ac19
             4db11d29-2f07-4452-bd8e-22a6a483ac19
             data
             metadata
             metadata
            extra_metadata
                  uid
                  checksum
                  activity_id
                  mime_type
                   preview
                   preview
                  share-scope
                  timestamp
                  title
       4d9f2027-b41e-4015-a848-6b3972193eb8
       4d9f2027-b41e-4015-a848-6b3972193eb8
             4d9f2027-b41e-4015-a848-6b3972193eb8
             data
             metadata
             metadata
            extra_metadata
                  uid
                  checksum
                  activity_id
                  mime_type
                   preview
                   preview
                  share-scope
                  timestamp
                  title
  checksums
  checksums
       464493d8d929436b6152e868867ed451
       464493d8d929436b6152e868867ed451
             2b90597c-0912-4e7f-8eeb-71a0f004490d ~> 2b90597c-0912-4e7f-8eeb-71a0f004490d/2b90597c-0912-4e7f-8eeb-71a0f004490d
             2b90597c-0912-4e7f-8eeb-71a0f004490d
             3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a ~> 3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a/3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a
             3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a
  index
  index
       flintlock
       flintlock
Line 101: Line 139:
       value.baseB
       value.baseB
       value.DB
       value.DB
index_updated
version


'''1a''': directory holding entries, it's only function is to avoid having too many directories in a single directory, as this is considered specially harmful on jffs2.
'''1a''': directory holding entries, it's only function is to avoid having too many directories in a single directory, as this is considered specially harmful on jffs2.


'''1a/1ab88287-766a-4d98-a7c0-4233dc48647a''': directory holding the files related to one entry
'''1a/1ab88287-...-4233dc48647a''': directory holding the files related to one entry


'''1a/1ab88287-766a-4d98-a7c0-4233dc48647a/1ab88287-766a-4d98-a7c0-4233dc48647a''': "data" file related to an entry
'''1a/1ab88287-...-4233dc48647a/data''': file related to an entry


'''1a/1ab88287-766a-4d98-a7c0-4233dc48647a/metadata''': file containing the metadata properties of an entry
'''1a/1ab88287-...-4233dc48647a/metadata''': directory containing a file for each metadata property of an entry


'''2b/2b90597c-0912-4e7f-8eeb-71a0f004490d/extra_metadata''': directory containing those metadata properties not contained in '''metadata''' for being either binary data or too big
'''2b/2b90597c-...-71a0f004490d/metadata/activity_id''': file containing the value of the '''activity_id''' property


'''2b/2b90597c-0912-4e7f-8eeb-71a0f004490d/extra_metadata/preview''': file containing the value of the '''preview''' property
'''3c/3cdf5f0e-...-cbedfcfe1c4a/data''': hard link to the same file in the entry '''2b90597c-...-71a0f004490d'''


'''3c/3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a/3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a''': hard link to the same file in the entry '''2b90597c-0912-4e7f-8eeb-71a0f004490d'''
'''checksums''': directory containing a directory per each file contained in the DS, named by its md5 checksum


'''3c/3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a/checksum ~> checksums/464493d8d929436b6152e868867ed451''': symbolic link to the file in '''checksums'''. Used to get the checksum of the entry without having to recalculate it nor read it from the metadata file
'''checksums/464493d8d929436b6152e868867ed451''': directory containing files named by each entry that contain a file with this checksum


'''checksums''': directory containing a directory per each file contained in the DS, named by its md5 checksum
'''checksums/464493d8d929436b6152e868867ed451/2b90597c-...-71a0f004490d''' file named by the uid of an entry the file of which has this checksum.
 
'''index''': directory containing all files that belong to the search database. Can be deleted and recreated from the rest of the DS if needed without incurring in data loss.
 
'''index_updated''': When this file is not present, is because the xapian index is being rebuilt because it cannot be opened. Queries should fall back to display all the files on disk until the index is rebuilt.


'''checksums/464493d8d929436b6152e868867ed451''': directory containing links to all the
'''version''': file that contains the version of the file layout. Currently we are at 1. Is updated when an earlier version is migrated to a newer one.


== Source code ==
== Source code ==


http://dev.laptop.org/git?p=users/tomeu/datastore;a=summary
http://git.sugarlabs.org/projects/sugar-datastore
Retrieved from "https://wiki.sugarlabs.org/go/Development_Team/Datastore_Rewrite"