Cannot export/backup: note.db corrupt?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Cannot export/backup: note.db corrupt?

Jeff Silver
When I try to export to Gramps XML, I get this:
161186: ERROR: _exportassistant.py: line 600: Error exporting your Family Tree
Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/gramps/gui/plug/export/_exportassistant.py", line 596, in save
     self.option_box_instance)
   File "/usr/lib/python3/dist-packages/gramps/plugins/export/exportxml.py", line 1334, in export_data
     return g.write(filename)
   File "/usr/lib/python3/dist-packages/gramps/plugins/export/exportxml.py", line 1357, in write
     ret = GrampsXmlWriter.write(self, filename)
   File "/usr/lib/python3/dist-packages/gramps/plugins/export/exportxml.py", line 176, in write
     self.write_xml_data()
   File "/usr/lib/python3/dist-packages/gramps/plugins/export/exportxml.py", line 336, in write_xml_data
     note = self.db.get_note_from_handle(key)
   File "/usr/lib/python3/dist-packages/gramps/gen/db/read.py", line 787, in get_note_from_handle
     return self.get_from_handle(handle, Note, self.note_map)
   File "/usr/lib/python3/dist-packages/gramps/gen/db/write.py", line 2149, in get_from_handle
     newobj.unserialize(data)
   File "/usr/lib/python3/dist-packages/gramps/gen/lib/note.py", line 152, in unserialize
     the_type, self.change, tag_list, self.private) = data
ValueError: too many values to unpack (expected 8)

I get the same error on trying to create a backup.

I hacked in a diagnostic print and found that the record that note.py is trying to unpack is not the expected 8-tuple,
but a binary string, part of which is the text of a (recently added) note. That note appears elsewhere in the data, in a
tuple as expected.
db_verify on note.db gives this error:
db_verify: BDB0677 Page 60: overflow page of invalid type 0
db_verify: note.db: BDB0090 DB_VERIFY_BAD: Database verification failed
BDB5105 Verification of note.db failed.
But, oddly, if I do "db_verify *.db", all files are reported to be OK.
Running db_recover made no difference.

Gramps 4.2.2, Python 3.5.2, BSDDB: 6.1.0 (5, 3, 28)

I could upgrade Gramps, but I'm wary of doing that on potentially corrupted data.

Any ideas?

Jeff

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: Cannot export/backup: note.db corrupt?

Ron Johnson
On 01/27/2018 07:29 AM, Jeff Silver wrote:

> When I try to export to Gramps XML, I get this:
> 161186: ERROR: _exportassistant.py: line 600: Error exporting your Family
> Tree
> Traceback (most recent call last):
>   File
> "/usr/lib/python3/dist-packages/gramps/gui/plug/export/_exportassistant.py",
> line 596, in save
>     self.option_box_instance)
>   File
> "/usr/lib/python3/dist-packages/gramps/plugins/export/exportxml.py", line
> 1334, in export_data
>     return g.write(filename)
>   File
> "/usr/lib/python3/dist-packages/gramps/plugins/export/exportxml.py", line
> 1357, in write
>     ret = GrampsXmlWriter.write(self, filename)
>   File
> "/usr/lib/python3/dist-packages/gramps/plugins/export/exportxml.py", line
> 176, in write
>     self.write_xml_data()
>   File
> "/usr/lib/python3/dist-packages/gramps/plugins/export/exportxml.py", line
> 336, in write_xml_data
>     note = self.db.get_note_from_handle(key)
>   File "/usr/lib/python3/dist-packages/gramps/gen/db/read.py", line 787,
> in get_note_from_handle
>     return self.get_from_handle(handle, Note, self.note_map)
>   File "/usr/lib/python3/dist-packages/gramps/gen/db/write.py", line 2149,
> in get_from_handle
>     newobj.unserialize(data)
>   File "/usr/lib/python3/dist-packages/gramps/gen/lib/note.py", line 152,
> in unserialize
>     the_type, self.change, tag_list, self.private) = data
> ValueError: too many values to unpack (expected 8)
>
> I get the same error on trying to create a backup.

Backups *are* exports (just without the ability to filter or choose the
output type).

>
> I hacked in a diagnostic print and found that the record that note.py is
> trying to unpack is not the expected 8-tuple, but a binary string, part of
> which is the text of a (recently added) note. That note appears elsewhere
> in the data, in a tuple as expected.
> db_verify on note.db gives this error:
> db_verify: BDB0677 Page 60: overflow page of invalid type 0
> db_verify: note.db: BDB0090 DB_VERIFY_BAD: Database verification failed
> BDB5105 Verification of note.db failed.
> But, oddly, if I do "db_verify *.db", all files are reported to be OK.
> Running db_recover made no difference.
>
> Gramps 4.2.2, Python 3.5.2, BSDDB: 6.1.0 (5, 3, 28)
>
> I could upgrade Gramps, but I'm wary of doing that on potentially
> corrupted data.
>
> Any ideas?

First, (after shutting down gramps, of course) "cp -ra" the relevant
.gramps/grampsdb/xxxxxxxx directory to Somewhere Safe.

Then I'd delete that note and try again to backup the database.


--
World Peace Through Nuclear Pacification


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: Cannot export/backup: note.db corrupt?

Jeff Silver

>> I get the same error on trying to create a backup.
> Backups *are* exports (just without the ability to filter or choose the output type).
OK. Thanks for the clarification.
>
> First, (after shutting down gramps, of course) "cp -ra" the relevant .gramps/grampsdb/xxxxxxxx directory to Somewhere
> Safe.
>
> Then I'd delete that note and try again to backup the database.
I took a tar archive for safety (but I never object to being reminded to do that!).
I had already tried editing the note and deleting the text. That didn't help - the text was still in the database, both
cleanly and in the bad record.
I went to "Notes" to delete it completely, but that also gave an error, presumably on reading the bad record, but I
haven't checked that:
7713: ERROR: grampsapp.py: line 107: Unhandled exception
Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/gramps/gui/viewmanager.py", line 1015, in view_changed
     self.__change_page(page_num)
   File "/usr/lib/python3/dist-packages/gramps/gui/viewmanager.py", line 1028, in __change_page
     self.active_page.set_active()
   File "/usr/lib/python3/dist-packages/gramps/gui/views/listview.py", line 293, in set_active
     NavigationView.set_active(self)
   File "/usr/lib/python3/dist-packages/gramps/gui/views/navigationview.py", line 154, in set_active
     PageView.set_active(self)
   File "/usr/lib/python3/dist-packages/gramps/gui/views/pageview.py", line 304, in set_active
     self.build_tree()
   File "/usr/lib/python3/dist-packages/gramps/gui/views/listview.py", line 324, in build_tree
     sort_map=self.column_order())
   File "/usr/lib/python3/dist-packages/gramps/gui/views/treemodels/notemodel.py", line 79, in __init__
     sort_map=sort_map)
   File "/usr/lib/python3/dist-packages/gramps/gui/views/treemodels/flatbasemodel.py", line 488, in __init__
     self.rebuild_data()
   File "/usr/lib/python3/dist-packages/gramps/gui/views/treemodels/flatbasemodel.py", line 614, in _rebuild_filter
     allkeys = self.sort_keys()
   File "/usr/lib/python3/dist-packages/gramps/gui/views/treemodels/flatbasemodel.py", line 572, in sort_keys
     for key, data in cursor]
   File "/usr/lib/python3/dist-packages/gramps/gui/views/treemodels/flatbasemodel.py", line 571, in <listcomp>
     srt_keys=[(self.sort_func(data), key.decode('utf8'))
   File "/usr/lib/python3/dist-packages/gramps/gen/db/cursor.py", line 100, in __iter__
     data = _n()
   File "/usr/lib/python3/dist-packages/gramps/gen/db/cursor.py", line 115, in get
     return (data[0], loads(data[1])) if data else None
EOFError: Ran out of input



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: Cannot export/backup: note.db corrupt?

Ron Johnson
In reply to this post by Ron Johnson
On 01/27/2018 11:53 AM, Jeff Silver wrote:
[snip]

>> First, (after shutting down gramps, of course) "cp -ra" the relevant
>> .gramps/grampsdb/xxxxxxxx directory to Somewhere Safe.
>>
>> Then I'd delete that note and try again to backup the database.
> I took a tar archive for safety (but I never object to being reminded to
> do that!).
> I had already tried editing the note and deleting the text. That didn't
> help - the text was still in the database, both cleanly and in the bad
> record.
> I went to "Notes" to delete it completely, but that also gave an error,
> presumably on reading the bad record, but I haven't checked that:
> 7713: ERROR: grampsapp.py: line 107: Unhandled exception
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/gramps/gui/viewmanager.py", line
> 1015, in view_changed
>     self.__change_page(page_num)
>   File "/usr/lib/python3/dist-packages/gramps/gui/viewmanager.py", line
> 1028, in __change_page
>     self.active_page.set_active()
>   File "/usr/lib/python3/dist-packages/gramps/gui/views/listview.py", line
> 293, in set_active
>     NavigationView.set_active(self)
>   File
> "/usr/lib/python3/dist-packages/gramps/gui/views/navigationview.py", line
> 154, in set_active
>     PageView.set_active(self)
>   File "/usr/lib/python3/dist-packages/gramps/gui/views/pageview.py", line
> 304, in set_active
>     self.build_tree()
>   File "/usr/lib/python3/dist-packages/gramps/gui/views/listview.py", line
> 324, in build_tree
>     sort_map=self.column_order())
>   File
> "/usr/lib/python3/dist-packages/gramps/gui/views/treemodels/notemodel.py",
> line 79, in __init__
>     sort_map=sort_map)
>   File
> "/usr/lib/python3/dist-packages/gramps/gui/views/treemodels/flatbasemodel.py",
> line 488, in __init__
>     self.rebuild_data()
>   File
> "/usr/lib/python3/dist-packages/gramps/gui/views/treemodels/flatbasemodel.py",
> line 614, in _rebuild_filter
>     allkeys = self.sort_keys()
>   File
> "/usr/lib/python3/dist-packages/gramps/gui/views/treemodels/flatbasemodel.py",
> line 572, in sort_keys
>     for key, data in cursor]
>   File
> "/usr/lib/python3/dist-packages/gramps/gui/views/treemodels/flatbasemodel.py",
> line 571, in <listcomp>
>     srt_keys=[(self.sort_func(data), key.decode('utf8'))
>   File "/usr/lib/python3/dist-packages/gramps/gen/db/cursor.py", line 100,
> in __iter__
>     data = _n()
>   File "/usr/lib/python3/dist-packages/gramps/gen/db/cursor.py", line 115,
> in get
>     return (data[0], loads(data[1])) if data else None
> EOFError: Ran out of input

Time to file a bug report!!!  :(

--
World Peace Through Nuclear Pacification


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: Cannot export/backup: note.db corrupt?

Jeff Silver
The underlying problem seems to be a truncated record in note.db, so it doesn't unpickle.
I found this by hacking in a print statement here in gen/db/cursor.py:

>         def get(self, flags=0, **kwargs):
>             """
>             Issue DBCursor get call (with DB_RMW flag if update requested)
>             Return results to caller
>             """
>             data = self.cursor.get(
>                         _flags | flags | (db.DB_RMW if self._update else 0),
>                         **kwargs)
>
>             print('data: %r' % (data,))     # <---------- hacked line
>             try:
>                 return (data[0], loads(data[1])) if data else None
>             except UnicodeDecodeError:
>                 #we need to assume we opened data in python3 saved in python2
>                 return (data[0], loads(data[1], encoding='utf-8')) if data else None

(My apologies for the python2 programming style!)
Then I opened the Family Tree and went to the Notes page, and looked at the last one that was printed. To be sure that
this was the problem, I pasted the data from an earlier record into a little unpickle script and it worked fine, then
the one I had identified got the same unpickling error that gramps itself got.

I know who the offending note relates to, but can't find the note using the GUI. Is there some way I can remove that
record directly from the database? And if so, can it be done without breaking something else?

If I can't do it by direct database access, do you think it would be OK to extend the try/recover block to replace the
second data item with some fixed text, that I could then edit in the GUI?

--
Jeff

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: Cannot export/backup: note.db corrupt?

Tim Lyons
Administrator




--
Sent from: http://gramps.1791082.n4.nabble.com/GRAMPS-User-f1807095.html

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: Cannot export/backup: note.db corrupt?

Tim Lyons
Administrator
In reply to this post by Jeff Silver
Jeff Silver wrote
> If I can't do it by direct database access, do you think it would be OK to
> extend the try/recover block to replace the
> second data item with some fixed text, that I could then edit in the GUI?

The ideal thing would be to provide a fix in gramps/plugins/tool/check.py.
There was a problem that sounds similar see ckeckmediasourceref at line
1933: "This repairs a problem with database upgrade from database schema
version 15 to 16. Mediarefs on source primary objects can contain
sourcerefs, and these were not converted to citations." Maybe the note
contains something unexpected.

Meanwhile, it is a good idea to make the code resilient to the problem, and
if possible log a warning about the problem see
https://www.gramps-project.org/wiki/index.php?title=Gramps_4.2_Wiki_Manual_-_Error_and_Warning_Reference#Warnings
with as much context information as possible.

regards,
Tim.



--
Sent from: http://gramps.1791082.n4.nabble.com/GRAMPS-User-f1807095.html

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: Cannot export/backup: note.db corrupt?

Jeff Silver
Paul Culley kindly pointed me at gramps/plugins/export/exportxml.py where I could catch the Exception on export. That
enabled me to do a successful export, and the corresponding import of that file cleaned up outstanding references to 3
notes that had got corrupted in the database.

This is probably one for the devs rather than users: Aside from handling the Exceptions, would it make sense to have the
IDs of notes in plain text in the database, rather than inside the pickle? Then, someone with a corrupt pickle could
track down references.

Anyway, my problem is now solved.
Thanks to all contributors.

Jeff

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org