[Gramps-devel] [Fwd: request for large data files]

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[Gramps-devel] [Fwd: request for large data files]

Don Allingham
Alex is having some email issues, so I an forwarding this for him.

Don

Hi everybody,

As a part of development effort towards 2.2 release
of gramps, we are moving to the true atomic transactions
in the grdb format. In short, this means that any
change will either be completely written into the database
or not written at all. This insures data integrity and
avoids database breakage when some pieces have already
been written, but some other did not due to some error, and
then the database is in the inconsistent (corrupted) state.

The Berkeley DB transactions require many changes that are
almost complete in the development branch. One of these
changes is that the record locking is performed for
transactions. This is a low-level DB stuff and should not
be the concern of the user. However, in practice huge
transactions such as large data imports may require a lot
of locks. If the number of required locks exceeds the
database setting, the import will fail.

In order to get a good idea about the good settings
for the database locks, it would be nice to have many
large datasets. We can generate some dummy sets, but
it is also useful to have an idea about real data.
The number of needed locks depends on the data structure.

We would like to ask for contributions of large datafiles
(5 thousand people and more), in gedcom or gramps XML format,
in order to gather information about the database lock settings.
If you are willing to contribute your data, please feel free
to FTP it to ftp.gramps-project.org into the "incoming"
directory. The username is "anonymous", the password is
any string.

Only Don and myself can get files from this site, and we will
not make your data public. Your data will be deleted from
the server once either of us downloads it to his copmuter.
We will also destroy your data from our computers when we
are finished with analyzing the statistics. However, if you
feel like protecting vital data, please don't hesitate to
do that since the exact details of the data are not important
for us. What's important is the statistics of the real
data (how many records are linked to how many other records,
etc).

If you decide to contribute your data, please email
either don or shura at gramps-project dot org domain
that you have done so (or if you have a problem).
Contributions are much appreciated, and will eventually
benefit all the users after 2.2 is out.

Thanks,
Alex
--
Alex Roitman  http://gramps-project.org

Reply | Threaded
Open this post in threaded view
|

[Gramps-devel] Re: [Fwd: request for large data files]

Don Allingham

A correction to Alex's message:

The username for anonymous ftp on ftp.gramps-project.org is not
"anonymous", but "[hidden email]"

Don

Don Allingham wrote:

> Alex is having some email issues, so I an forwarding this for him.
>
> Don
>
> ------------------------------------------------------------------------
>
> Subject:
> request for lare data files
> From:
> "Alex Roitman" <[hidden email]>
> Date:
> Fri, 13 Jan 2006 13:33:07 -0500
> To:
> [hidden email]
>
> To:
> [hidden email]
>
>
> Hi everybody,
>
> As a part of development effort towards 2.2 release
> of gramps, we are moving to the true atomic transactions
> in the grdb format. In short, this means that any
> change will either be completely written into the database
> or not written at all. This insures data integrity and
> avoids database breakage when some pieces have already
> been written, but some other did not due to some error, and
> then the database is in the inconsistent (corrupted) state.
>
> The Berkeley DB transactions require many changes that are
> almost complete in the development branch. One of these
> changes is that the record locking is performed for
> transactions. This is a low-level DB stuff and should not
> be the concern of the user. However, in practice huge
> transactions such as large data imports may require a lot
> of locks. If the number of required locks exceeds the
> database setting, the import will fail.
>
> In order to get a good idea about the good settings
> for the database locks, it would be nice to have many
> large datasets. We can generate some dummy sets, but
> it is also useful to have an idea about real data.
> The number of needed locks depends on the data structure.
>
> We would like to ask for contributions of large datafiles
> (5 thousand people and more), in gedcom or gramps XML format,
> in order to gather information about the database lock settings.
> If you are willing to contribute your data, please feel free
> to FTP it to ftp.gramps-project.org into the "incoming"
> directory. The username is "anonymous", the password is
> any string.
>
> Only Don and myself can get files from this site, and we will
> not make your data public. Your data will be deleted from
> the server once either of us downloads it to his copmuter.
> We will also destroy your data from our computers when we
> are finished with analyzing the statistics. However, if you
> feel like protecting vital data, please don't hesitate to
> do that since the exact details of the data are not important
> for us. What's important is the statistics of the real
> data (how many records are linked to how many other records,
> etc).
>
> If you decide to contribute your data, please email
> either don or shura at gramps-project dot org domain
> that you have done so (or if you have a problem).
> Contributions are much appreciated, and will eventually
> benefit all the users after 2.2 is out.
>
> Thanks,
> Alex



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel