Development Team/Datastore Rewrite: Difference between revisions
m moved Walter is a wanker 12/Datastore Rewrite to Development Team/Datastore Rewrite over redirect: revert |
|||
| (15 intermediate revisions by 6 users not shown) | |||
| Line 1: | Line 1: | ||
<noinclude>{{ GoogleTrans-en | es =show | bg =show | zh-CN =show | zh-TW =show | hr =show | cs =show | da =show | nl =show | fi =show | fr =show | de =show | el =show | hi =show | it =show | ja =show | ko =show | no =show | pl =show | pt =show | ro =show | ru =show | sv =show }}</noinclude> | |||
{{TOCright}} | |||
== Introduction == | |||
This page describes the design of a new DataStore implementation that was shipped first in the [[0.84]] (April 2009) Sugar release. | |||
== Goals == | == Goals == | ||
| Line 35: | Line 42: | ||
By examining the directory structure, we know where is localized the data related to each entry. We don't depend any more on a binary structure that could become corrupted and unusable as a whole. | By examining the directory structure, we know where is localized the data related to each entry. We don't depend any more on a binary structure that could become corrupted and unusable as a whole. | ||
=== Metadata is stored | === Metadata is stored in a single file per property === | ||
Metadata for each entry is stored in several files, one per property. In this way, if corruption happened on one those properties, the rest of the entry (and the other entries in the DS) would be unaffected. | |||
: See [[olpc:Low-level_Activity_API#Meta_Data]]. | |||
=== Queries are accelerated with a disposable database === | === Queries are accelerated with a disposable database === | ||
| Line 55: | Line 63: | ||
1a | 1a | ||
1ab88287-766a-4d98-a7c0-4233dc48647a | 1ab88287-766a-4d98-a7c0-4233dc48647a | ||
data | |||
metadata | metadata | ||
uid | |||
checksum | |||
activity_id | activity_id | ||
mime_type | mime_type | ||
| Line 65: | Line 75: | ||
2b | 2b | ||
2b90597c-0912-4e7f-8eeb-71a0f004490d | 2b90597c-0912-4e7f-8eeb-71a0f004490d | ||
data | |||
metadata | metadata | ||
uid | |||
checksum | |||
activity_id | activity_id | ||
mime_type | mime_type | ||
| Line 76: | Line 87: | ||
3c | 3c | ||
3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a | 3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a | ||
data -> 2b/2b90597c-0912-4e7f-8eeb-71a0f004490d/data | |||
metadata | metadata | ||
uid | |||
checksum | |||
activity_id | activity_id | ||
mime_type | mime_type | ||
| Line 87: | Line 99: | ||
4d | 4d | ||
4db11d29-2f07-4452-bd8e-22a6a483ac19 | 4db11d29-2f07-4452-bd8e-22a6a483ac19 | ||
data | |||
metadata | metadata | ||
uid | |||
checksum | |||
activity_id | activity_id | ||
mime_type | mime_type | ||
| Line 96: | Line 110: | ||
title | title | ||
4d9f2027-b41e-4015-a848-6b3972193eb8 | 4d9f2027-b41e-4015-a848-6b3972193eb8 | ||
data | |||
metadata | metadata | ||
uid | |||
checksum | |||
activity_id | activity_id | ||
mime_type | mime_type | ||
| Line 106: | Line 122: | ||
checksums | checksums | ||
464493d8d929436b6152e868867ed451 | 464493d8d929436b6152e868867ed451 | ||
2b90597c-0912-4e7f-8eeb-71a0f004490d | |||
3cdf5f0e-7595-4166-b1f9-cbedfcfe1c4a | |||
index | index | ||
flintlock | flintlock | ||
| Line 123: | Line 139: | ||
value.baseB | value.baseB | ||
value.DB | value.DB | ||
index_updated | |||
version | |||
'''1a''': directory holding entries, it's only function is to avoid having too many directories in a single directory, as this is considered specially harmful on jffs2. | '''1a''': directory holding entries, it's only function is to avoid having too many directories in a single directory, as this is considered specially harmful on jffs2. | ||
| Line 128: | Line 146: | ||
'''1a/1ab88287-...-4233dc48647a''': directory holding the files related to one entry | '''1a/1ab88287-...-4233dc48647a''': directory holding the files related to one entry | ||
'''1a/1ab88287-...-4233dc48647a/ | '''1a/1ab88287-...-4233dc48647a/data''': file related to an entry | ||
'''1a/1ab88287-...-4233dc48647a/metadata''': directory containing a file for each metadata property of an entry | '''1a/1ab88287-...-4233dc48647a/metadata''': directory containing a file for each metadata property of an entry | ||
| Line 134: | Line 152: | ||
'''2b/2b90597c-...-71a0f004490d/metadata/activity_id''': file containing the value of the '''activity_id''' property | '''2b/2b90597c-...-71a0f004490d/metadata/activity_id''': file containing the value of the '''activity_id''' property | ||
'''3c/3cdf5f0e-...-cbedfcfe1c4a/ | '''3c/3cdf5f0e-...-cbedfcfe1c4a/data''': hard link to the same file in the entry '''2b90597c-...-71a0f004490d''' | ||
'''checksums''': directory containing a directory per each file contained in the DS, named by its md5 checksum | '''checksums''': directory containing a directory per each file contained in the DS, named by its md5 checksum | ||
'''checksums/464493d8d929436b6152e868867ed451''': directory containing | '''checksums/464493d8d929436b6152e868867ed451''': directory containing files named by each entry that contain a file with this checksum | ||
'''checksums/464493d8d929436b6152e868867ed451/2b90597c-...-71a0f004490d''' | '''checksums/464493d8d929436b6152e868867ed451/2b90597c-...-71a0f004490d''' file named by the uid of an entry the file of which has this checksum. | ||
'''index''': directory containing all files that belong to the search database. Can be deleted and recreated from the rest of the DS if needed without incurring in data loss. | '''index''': directory containing all files that belong to the search database. Can be deleted and recreated from the rest of the DS if needed without incurring in data loss. | ||
'''index_updated''': When this file is not present, is because the xapian index is being rebuilt because it cannot be opened. Queries should fall back to display all the files on disk until the index is rebuilt. | |||
'''version''': file that contains the version of the file layout. Currently we are at 1. Is updated when an earlier version is migrated to a newer one. | |||
== Source code == | == Source code == | ||
http:// | http://git.sugarlabs.org/projects/sugar-datastore | ||