upgrade in HEAD

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

upgrade in HEAD

Alex Roitman
Richard,

I was done with the database upgrade path from stable version
today. Unfortunately, the reference_map breaks the upgrade in
a few ways:

1. I need to go over existing objects and modify them, then
   re-commit. The database is first opened, and at that time
   all the tables are opened/setup. Then we run upgrade.
   The very first commit tries to _update_references()
   and that fails because the table is empty and nothing
   is found.

2. Even if I did not have the older data format, we need
   some kind of initialization for the existing objects.
   I may have 100000 people with lots of references,
   and empty reference_map. Any edit will try updating
   the ref map and fail because it is empty. So we need
   an upgrade path from no_ref_map to ref_map state.

3. Back to the first issue -- we will have this problem
   even after number 2 is corrected. Should we be tolerant
   to the failure of the ref_map updates? Or better yet,
   should there be a way to turn it off and then go
   and update refs for a series of commits (or not)?

I realize that it's too early and the stuff is very
unfinished. Just wanted to start thinking about it,
as I don't see a good solution just yet.

Alex

--
Alexander Roitman   http://www.gramps-project.org

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: upgrade in HEAD

Richard Taylor-2
Alex

On Friday 16 December 2005 04:21, Alex Roitman wrote:

> Richard,
>
> I was done with the database upgrade path from stable version
> today. Unfortunately, the reference_map breaks the upgrade in
> a few ways:
>
> 1. I need to go over existing objects and modify them, then
>    re-commit. The database is first opened, and at that time
>    all the tables are opened/setup. Then we run upgrade.
>    The very first commit tries to _update_references()
>    and that fails because the table is empty and nothing
>    is found.
>

I am not sure what is going on here. The _update_reference_map()
method should be able to handle the situation of an existing object being
modified. Can you send me a traceback please or better yet write a testcase
(see below).

> 2. Even if I did not have the older data format, we need
>    some kind of initialization for the existing objects.
>    I may have 100000 people with lots of references,
>    and empty reference_map. Any edit will try updating
>    the ref map and fail because it is empty. So we need
>    an upgrade path from no_ref_map to ref_map state.
>

reindex_reference_map method added to GrampsBSDDB. I am not
sure where to hook it into the upgrade process.

> 3. Back to the first issue -- we will have this problem
>    even after number 2 is corrected. Should we be tolerant
>    to the failure of the ref_map updates? Or better yet,
>    should there be a way to turn it off and then go
>    and update refs for a series of commits (or not)?
>

Turning off the updates and batching them afterwards would not save any time.
The _update_reference_map()  method should do the right thing and add any
references for a the object it is passed whether or not it is already in the
reference_map. If it is not doing this then there is a bug.

I have added a unittest for the reference_map functionality in the test/
directory. You can run it from the test/ directory with:

        python RunAllTests.py

or if you want some debug information:

        python RunAllTests.py -v 2

If it is failing then I would like to add test cases to the unittest.

Richard

--
Jabber: [hidden email]
PGPKey: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xA7DA9FD9
Key fingerprint = D051 A121 E7C3 485F 3C0E  1593 ED9E D868 A7DA 9FD9


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: upgrade in HEAD

Alex Roitman
Richard,

On Fri, 2005-12-16 at 12:02 +0000, Richard Taylor wrote:

> >
> > I was done with the database upgrade path from stable version
> > today. Unfortunately, the reference_map breaks the upgrade in
> > a few ways:
> >
> > 1. I need to go over existing objects and modify them, then
> >    re-commit. The database is first opened, and at that time
> >    all the tables are opened/setup. Then we run upgrade.
> >    The very first commit tries to _update_references()
> >    and that fails because the table is empty and nothing
> >    is found.
> >
>
> I am not sure what is going on here. The _update_reference_map()
> method should be able to handle the situation of an existing object being
> modified. Can you send me a traceback please or better yet write a testcase
> (see below).
Sure, here's an easy testcase: take a grdb file created by 2.0.9
and try opening it with HEAD. It fails with the "could not open"
dialog, but this is only because of the exception caught by line
656 in src/ViewManager.py that handles any DB error. Commenting
this exception out reveals the traceback to _update_reference_map
that fails with "no key/data pair found" DB error. I am in a hurry
now, so I'll send you the exact traceback if you cannot reproduce
it for some reason.

> > 2. Even if I did not have the older data format, we need
> >    some kind of initialization for the existing objects.
> >    I may have 100000 people with lots of references,
> >    and empty reference_map. Any edit will try updating
> >    the ref map and fail because it is empty. So we need
> >    an upgrade path from no_ref_map to ref_map state.
> >
>
> reindex_reference_map method added to GrampsBSDDB. I am not
> sure where to hook it into the upgrade process.
This should either go into gramps_upgrade_9 or into a new
upgrade step, gramps_upgrade_10 to be added. But we should
resolve the first issue before this, I think.

> > 3. Back to the first issue -- we will have this problem
> >    even after number 2 is corrected. Should we be tolerant
> >    to the failure of the ref_map updates? Or better yet,
> >    should there be a way to turn it off and then go
> >    and update refs for a series of commits (or not)?
> >
>
> Turning off the updates and batching them afterwards would not save any time.

This is not a matter of time, it's a matter of being able to
upgrade the other changes in the objects without bumping into
the reference_map issue. Slow upgrade is OK.

> The _update_reference_map()  method should do the right thing and add any
> references for a the object it is passed whether or not it is already in the
> reference_map. If it is not doing this then there is a bug.

OK, so then fixing it should remove my concerns :-)

> I have added a unittest for the reference_map functionality in the test/
> directory. You can run it from the test/ directory with:
>
> python RunAllTests.py
>
> or if you want some debug information:
>
> python RunAllTests.py -v 2
>
> If it is failing then I would like to add test cases to the unittest.
I will try that later today, thanks!

Alex

--
Alexander Roitman   http://www.gramps-project.org

signature.asc (196 bytes) Download Attachment