pickle compatibility(?), Python3 vs. Python2

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

pickle compatibility(?), Python3 vs. Python2

Paul Franklin-5
I just read this, and don't claim to understand it fully,
so I am alerting you (older and wiser heads) about it.

https://gramps-project.org/bugs/view.php?id=8134#c38862

------------------------------------------------------------------------------
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Paul Franklin-5
On 10/24/14, Paul Franklin <[hidden email]> wrote:
> I just read this, and don't claim to understand it fully,
> so I am alerting you (older and wiser heads) about it.
>
> https://gramps-project.org/bugs/view.php?id=8134#c38862

We have another one, on Fedora 21 this time.
https://gramps-project.org/bugs/view.php?id=8258

I sure wish somebody who understands the Berkeley
DB would look into this (possibly UTF-8) pickle.loads problem.

Thanks.

------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Paul Franklin-5
On 12/23/14, Paul Franklin <[hidden email]> wrote:

> On 10/24/14, Paul Franklin <[hidden email]> wrote:
>> I just read this, and don't claim to understand it fully,
>> so I am alerting you (older and wiser heads) about it.
>>
>> https://gramps-project.org/bugs/view.php?id=8134#c38862
>
> We have another one, on Fedora 21 this time.
> https://gramps-project.org/bugs/view.php?id=8258
>
> I sure wish somebody who understands the Berkeley
> DB would look into this (possibly UTF-8) pickle.loads problem.
>
> Thanks.
>

Any chance of somebody looking into this before 4.1.2?

Thanks.

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Benny Malengier
These are people going from python 2 to python 3 on same database (no pass over xml?) it seems.

Question is how cPickle was storing the strings.

See https://docs.python.org/3.4/library/pickle.html#pickle.loads

As this defaults to read python2 as ascii, that would not be ok and give these crashes. Question is, do we need encoding='bytes' or encoding='utf-8'.
Should be one of the two.
I would keep errors='strict' as then we know at least something is wrong.

As there is already a fix for such an error here:
http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/gen/db/write.py#l938

I would expect the same thing to be needed for other loads. Try to find who did above fix, and find the commit.
The same fix via try block is here:
http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/gen/db/write.py#l2141

Benny

2015-01-27 19:47 GMT+01:00 Paul Franklin <[hidden email]>:
On 12/23/14, Paul Franklin <[hidden email]> wrote:
> On 10/24/14, Paul Franklin <[hidden email]> wrote:
>> I just read this, and don't claim to understand it fully,
>> so I am alerting you (older and wiser heads) about it.
>>
>> https://gramps-project.org/bugs/view.php?id=8134#c38862
>
> We have another one, on Fedora 21 this time.
> https://gramps-project.org/bugs/view.php?id=8258
>
> I sure wish somebody who understands the Berkeley
> DB would look into this (possibly UTF-8) pickle.loads problem.
>
> Thanks.
>

Any chance of somebody looking into this before 4.1.2?

Thanks.

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel


------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Paul Franklin-5
On 1/28/15, Benny Malengier <[hidden email]> wrote:

> These are people going from python 2 to python 3 on same database (no pass
> over xml?) it seems.
>
> Question is how cPickle was storing the strings.
>
> See https://docs.python.org/3.4/library/pickle.html#pickle.loads
>
> As this defaults to read python2 as ascii, that would not be ok and give
> these crashes. Question is, do we need encoding='bytes' or
> encoding='utf-8'.
> Should be one of the two.
> I would keep errors='strict' as then we know at least something is wrong.
>
> As there is already a fix for such an error here:
> http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/gen/db/write.py#l938
>
> I would expect the same thing to be needed for other loads. Try to find who
> did above fix, and find the commit.
> The same fix via try block is here:
> http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/gen/db/write.py#l2141
>
> Benny
>
> 2015-01-27 19:47 GMT+01:00 Paul Franklin <[hidden email]>:
>
>> On 12/23/14, Paul Franklin <[hidden email]> wrote:
>> > On 10/24/14, Paul Franklin <[hidden email]> wrote:
>> >> I just read this, and don't claim to understand it fully,
>> >> so I am alerting you (older and wiser heads) about it.
>> >>
>> >> https://gramps-project.org/bugs/view.php?id=8134#c38862
>> >
>> > We have another one, on Fedora 21 this time.
>> > https://gramps-project.org/bugs/view.php?id=8258
>> >
>> > I sure wish somebody who understands the Berkeley
>> > DB would look into this (possibly UTF-8) pickle.loads problem.
>> >
>> > Thanks.
>> >
>>
>> Any chance of somebody looking into this before 4.1.2?
>>
>> Thanks.
Hi Benny,

According to a "git blame" you committed both those lines
(attached, lines 944 and 2144).

(I originally tried to attach the whole "git blame" output,
which of course shows all the other pickle.loads too, but
it was rejected by the SourceForge mailing list software,
and our list moderator, for being over 50k chars.)

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

write.py=blame-pruned.txt (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Benny Malengier
In reply to this post by Benny Malengier


2015-01-28 19:25 GMT+01:00 Paul Franklin <[hidden email]>:
On 1/28/15, Benny Malengier <[hidden email]> wrote:
> These are people going from python 2 to python 3 on same database (no pass
> over xml?) it seems.
>
> Question is how cPickle was storing the strings.
>
> See https://docs.python.org/3.4/library/pickle.html#pickle.loads
>
> As this defaults to read python2 as ascii, that would not be ok and give
> these crashes. Question is, do we need encoding='bytes' or
> encoding='utf-8'.
> Should be one of the two.
> I would keep errors='strict' as then we know at least something is wrong.
>
> As there is already a fix for such an error here:
> http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/gen/db/write.py#l938
>
> I would expect the same thing to be needed for other loads. Try to find who
> did above fix, and find the commit.
> The same fix via try block is here:
> http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/gen/db/write.py#l2141
>
> Benny
>
> 2015-01-27 19:47 GMT+01:00 Paul Franklin <[hidden email]>:
>
>> On 12/23/14, Paul Franklin <[hidden email]> wrote:
>> > On 10/24/14, Paul Franklin <[hidden email]> wrote:
>> >> I just read this, and don't claim to understand it fully,
>> >> so I am alerting you (older and wiser heads) about it.
>> >>
>> >> https://gramps-project.org/bugs/view.php?id=8134#c38862
>> >
>> > We have another one, on Fedora 21 this time.
>> > https://gramps-project.org/bugs/view.php?id=8258
>> >
>> > I sure wish somebody who understands the Berkeley
>> > DB would look into this (possibly UTF-8) pickle.loads problem.
>> >
>> > Thanks.
>> >
>>
>> Any chance of somebody looking into this before 4.1.2?
>>
>> Thanks.
>>
>>

Hi Benny,

According to a "git blame" you committed both those lines
(attached, lines 944 and 2144).

Haha, you want to chase me away again Paul? March 2013, I don't remember starting python3 conversion so far back.

So, I would say, apply same approach on the line that gives the UnicodeDecodeError. So a try, except block.
I can't think of a reason to do this in try/except, but doing it like that is in line with those approaches. My guess would be that staying in python2 or 3 works in the try block, so the except is for people who mixed (and have non-ascii data). However, it does mean retrieving data will be quite a bit slower for them. I assume in 2013 it made sense so as to stay in the old code path for default code ...

Anyway, that fix needs general testing for some time by more than the person who runs it now.
Other uses of pickle.load need to be looked at too. Note that for handle, sec_tables and ref_tables the utf-8 argument is not needed. Only text data.

Benny

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

manzi.sam
In reply to this post by Benny Malengier
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Benny Malengier
In reply to this post by Benny Malengier
Paul,

his dbshelve error later is worrying though. Writing away the new data fails because dbshelve loads the old data.I added a not the bug ticket on how to handle that.

2015-01-28 22:27 GMT+01:00 Benny Malengier <[hidden email]>:


2015-01-28 19:25 GMT+01:00 Paul Franklin <[hidden email]>:
On 1/28/15, Benny Malengier <[hidden email]> wrote:
> These are people going from python 2 to python 3 on same database (no pass
> over xml?) it seems.
>
> Question is how cPickle was storing the strings.
>
> See https://docs.python.org/3.4/library/pickle.html#pickle.loads
>
> As this defaults to read python2 as ascii, that would not be ok and give
> these crashes. Question is, do we need encoding='bytes' or
> encoding='utf-8'.
> Should be one of the two.
> I would keep errors='strict' as then we know at least something is wrong.
>
> As there is already a fix for such an error here:
> http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/gen/db/write.py#l938
>
> I would expect the same thing to be needed for other loads. Try to find who
> did above fix, and find the commit.
> The same fix via try block is here:
> http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/gen/db/write.py#l2141
>
> Benny
>
> 2015-01-27 19:47 GMT+01:00 Paul Franklin <[hidden email]>:
>
>> On 12/23/14, Paul Franklin <[hidden email]> wrote:
>> > On 10/24/14, Paul Franklin <[hidden email]> wrote:
>> >> I just read this, and don't claim to understand it fully,
>> >> so I am alerting you (older and wiser heads) about it.
>> >>
>> >> https://gramps-project.org/bugs/view.php?id=8134#c38862
>> >
>> > We have another one, on Fedora 21 this time.
>> > https://gramps-project.org/bugs/view.php?id=8258
>> >
>> > I sure wish somebody who understands the Berkeley
>> > DB would look into this (possibly UTF-8) pickle.loads problem.
>> >
>> > Thanks.
>> >
>>
>> Any chance of somebody looking into this before 4.1.2?
>>
>> Thanks.
>>
>>

Hi Benny,

According to a "git blame" you committed both those lines
(attached, lines 944 and 2144).

Haha, you want to chase me away again Paul? March 2013, I don't remember starting python3 conversion so far back.

So, I would say, apply same approach on the line that gives the UnicodeDecodeError. So a try, except block.
I can't think of a reason to do this in try/except, but doing it like that is in line with those approaches. My guess would be that staying in python2 or 3 works in the try block, so the except is for people who mixed (and have non-ascii data). However, it does mean retrieving data will be quite a bit slower for them. I assume in 2013 it made sense so as to stay in the old code path for default code ...

Anyway, that fix needs general testing for some time by more than the person who runs it now.
Other uses of pickle.load need to be looked at too. Note that for handle, sec_tables and ref_tables the utf-8 argument is not needed. Only text data.

Benny


------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Benny Malengier
In reply to this post by manzi.sam


2015-01-28 22:58 GMT+01:00 Sam Manzi <[hidden email]>:
It does look like (as far as I can tell) it may have something to do with python 2 using the default "Protocol version 0 ... ASCII protocol..."[1] and python 3 using the default  "Protocol version 3 ... It has explicit support for bytes objects and cannot be unpickled by Python 2.x. ..."[1]  and as such Gramps needs to be explict in choosing which protocol to use and setting it via pickle.HIGHEST_PROTOCOL" and or similar[3].

[1] https://docs.python.org/2/library/pickle.html#data-stream-format
[2] https://docs.python.org/3/library/pickle.html#data-stream-format
[3] http://stackoverflow.com/questions/23582489/python-pickle-protocol-choice


Yes, point is, do we support just moving from python2 to 3? It seems so, as that is what people are doing and hitting those places where pickle.load is done on the python2 saved data.

So, we would need everywhere the paradigm that is already present in some places. Try to load via dbshelve, if UnicodeDecodeError, assume it is python2 data openend in python3 (we could add test on python3 if we want), and use map.db.get to obtain raw data, then unpickle ourselves passing utf-8.

Benny
 

On 28 January 2015 at 20:41, Benny Malengier <[hidden email]> wrote:
These are people going from python 2 to python 3 on same database (no pass over xml?) it seems.

Question is how cPickle was storing the strings.

See https://docs.python.org/3.4/library/pickle.html#pickle.loads

As this defaults to read python2 as ascii, that would not be ok and give these crashes. Question is, do we need encoding='bytes' or encoding='utf-8'.
Should be one of the two.



2015-01-27 19:47 GMT+01:00 Paul Franklin <[hidden email]>:
On 12/23/14, Paul Franklin <[hidden email]> wrote:
> On 10/24/14, Paul Franklin <[hidden email]> wrote:
>> I just read this, and don't claim to understand it fully,
>> so I am alerting you (older and wiser heads) about it.
>>
>> https://gramps-project.org/bugs/view.php?id=8134#c38862
>
> We have another one, on Fedora 21 this time.
> https://gramps-project.org/bugs/view.php?id=8258
>
> I sure wish somebody who understands the Berkeley
> DB would look into this (possibly UTF-8) pickle.loads problem.
>
> Thanks.
>

Any chance of somebody looking into this before 4.1.2?




------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Tim Lyons
Administrator
Benny Malengier wrote
Yes, point is, do we support just moving from python2 to 3? It seems so, as
that is what people are doing and hitting those places where pickle.load is
done on the python2 saved data.

So, we would need everywhere the paradigm that is already present in some
places. Try to load via dbshelve, if UnicodeDecodeError, assume it is
python2 data openend in python3 (we could add test on python3 if we want),
and use map.db.get to obtain raw data, then unpickle ourselves passing
utf-8.
There is a 'pythonversion.txt' file in the database directory, which indicates which version of python was used to create the database. If the file is not present, you know that the database was created before gramps was upgraded to support python3, and hence the database must have been created by python2.

Tim.
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Paul Franklin-5
In reply to this post by Benny Malengier
On 1/28/15, Benny Malengier <[hidden email]> wrote:
> These are people going from python 2 to python 3 on same database (no pass
> over xml?) it seems.

My belief is that these are people whose distro gramps
is being upgraded and in the process the distro changed
gramps to use Python3 instead.  So the user is unaware
of the change, in general, and just knows their gramps is
being upgraded (from 4.0.x to 4.1.y I think).

The bug tracker is starting to get various Python3 errors
now -- including GUI ones I do not have the knowledge to
take care of -- which I believe is resulting from that.

I haven't been tracking it but I think Debian and Ubuntu
and Fedora are in that category.

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Benny Malengier


2015-01-29 2:15 GMT+01:00 Paul Franklin <[hidden email]>:
On 1/28/15, Benny Malengier <[hidden email]> wrote:
> These are people going from python 2 to python 3 on same database (no pass
> over xml?) it seems.

My belief is that these are people whose distro gramps
is being upgraded and in the process the distro changed
gramps to use Python3 instead.  So the user is unaware
of the change, in general, and just knows their gramps is
being upgraded (from 4.0.x to 4.1.y I think).

The bug tracker is starting to get various Python3 errors
now -- including GUI ones I do not have the knowledge to
take care of -- which I believe is resulting from that.

I haven't been tracking it but I think Debian and Ubuntu
and Fedora are in that category.

Ok, this makes sense.

We could in the repair database utility add a utility that:
1/obtains data with pickle.loads( ... .utf-8)
2/immediately saves again with pickle dump.

That should convert all to the new pickle format, instead of try, except blocks everywhere to counter bugs.

Benny


------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Ross Gammon
In reply to this post by Paul Franklin-5
On 01/29/2015 02:15 AM, Paul Franklin wrote:
> My belief is that these are people whose distro gramps
> is being upgraded and in the process the distro changed
> gramps to use Python3 instead.  So the user is unaware
> of the change, in general, and just knows their gramps is
> being upgraded (from 4.0.x to 4.1.y I think).

[...]

> I haven't been tracking it but I think Debian and Ubuntu
> and Fedora are in that category.

Paul is correct - Debian (& therefore Ubuntu) switched to Python 3 with
the release of 4.1.0.

Ross


------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Paul Franklin-5
In reply to this post by Benny Malengier
On 1/28/15, Benny Malengier <[hidden email]> wrote:

> So, I would say, apply same approach on the line that gives the
> UnicodeDecodeError. So a try, except block.

> Anyway, that fix needs general testing for some time by more than the
> person who runs it now.

I don't know how to trigger that bug.  And I am not
willing to make any change unless I can run a test.

I tried creating a database in gramps41, gramps40,
gramps34, and gramps33, in all cases using Python2
and the (native) example.gramps database (since I
believe it has utf-8 data in it), and then running gramps41
with Python3 on that database.  No crashes.

Is there some special kind of data being read in at
that point, whose types are all ascii in example.gramps?
Etc.

Thanks.

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Benny Malengier


2015-02-02 1:42 GMT+01:00 Paul Franklin <[hidden email]>:
On 1/28/15, Benny Malengier <[hidden email]> wrote:

> So, I would say, apply same approach on the line that gives the
> UnicodeDecodeError. So a try, except block.

> Anyway, that fix needs general testing for some time by more than the
> person who runs it now.

I don't know how to trigger that bug.  And I am not
willing to make any change unless I can run a test.

I tried creating a database in gramps41, gramps40,
gramps34, and gramps33, in all cases using Python2
and the (native) example.gramps database (since I
believe it has utf-8 data in it), and then running gramps41
with Python3 on that database.  No crashes.

Is there some special kind of data being read in at
that point, whose types are all ascii in example.gramps?

Yes, when doing the python2 to 3 tests, I also used example.gramps. Clearly it did not trigger the bug, or we would have  obtained it ourself.

utf-8 data is mostly limited to names and family names. We would need to store something in example.gramps which only has e.g. chinese in every field possible. That would stress the system much more. As error was in get_person_from_gramps_id

creating a person and pasting everywhere utf-8 char with all subobjects and links too, seems like the first step in trying to reproduce this.

Benny




Etc.

Thanks.


------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Paul Franklin-5
On 2/2/15, Benny Malengier <[hidden email]> wrote:
> creating a person and pasting everywhere utf-8 char with all subobjects and
> links too, seems like the first step in trying to reproduce this.

The reporter for 8258 has supplied a zipped copy of a
gramps DB directory, so Enno and I are investigating,
since that database crashes things.

I added your patch in 8134 to commit_base but I am
still getting crashes.  As Enno said in 8258, even by
using the 8258 user's patch an error shows up deeper
down in check.py, for media.

Can you suggest an approach (maybe like your 8134
patch?) to process maps similarly, e.g. tag_map and
media_map (and maybe others for all I know)?

Thanks.

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Paul Franklin-5
In reply to this post by Paul Franklin-5
On 12/23/14, Paul Franklin <[hidden email]> wrote:

> On 10/24/14, Paul Franklin <[hidden email]> wrote:
>> I just read this, and don't claim to understand it fully,
>> so I am alerting you (older and wiser heads) about it.
>>
>> https://gramps-project.org/bugs/view.php?id=8134#c38862
>
> We have another one, on Fedora 21 this time.
> https://gramps-project.org/bugs/view.php?id=8258
>
> I sure wish somebody who understands the Berkeley
> DB would look into this (possibly UTF-8) pickle.loads problem.
>
> Thanks.
>

We have another brand-new one.  As with some of the
others, the check tool is attempting to access a "map"
directly:

info = self.db.event_map[bhandle]

https://gramps-project.org/bugs/view.php?id=8357

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Tim Lyons
Administrator
Paul Franklin-5 wrote
On 12/23/14, Paul Franklin <[hidden email]> wrote:
> On 10/24/14, Paul Franklin <[hidden email]> wrote:
>> I just read this, and don't claim to understand it fully,
>> so I am alerting you (older and wiser heads) about it.
>>
>> https://gramps-project.org/bugs/view.php?id=8134#c38862
>
> We have another one, on Fedora 21 this time.
> https://gramps-project.org/bugs/view.php?id=8258
>
> I sure wish somebody who understands the Berkeley
> DB would look into this (possibly UTF-8) pickle.loads problem.
>
> Thanks.
>

We have another brand-new one.  As with some of the
others, the check tool is attempting to access a "map"
directly:

info = self.db.event_map[bhandle]

https://gramps-project.org/bugs/view.php?id=8357
Please note that because the Check tool uses one single transaction out of which an exception has been thrown, it is quite likely that the DB is corrupted.

Regards,
Tim.
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

Paul Franklin-5
I would really like to get this/these fixed for 4.1.2, for
all our users who run Debian/Ubuntu/Fedora/etc. who
don't make an XML file before they upgrade their gramps
(from Python2 to Python3).

Benny suggested a fix for when we do the pickle.loads
ourselves but most of the filed bugs involve a read of various
maps (tag_map, event_map, citation_map, etc.), and I
don't know how to fix that/those (or possibly where).

We have a user's zipped database directory which exhibits
(at least) one of the places it crashes, if any of you feel
like investigating this.  It repeats every time, just by starting
gramps (without running the check tool, etc.), if that helps.

Thanks.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: pickle compatibility(?), Python3 vs. Python2

enno
Hi Paul,

> I would really like to get this/these fixed for 4.1.2, for
> all our users who run Debian/Ubuntu/Fedora/etc. who
> don't make an XML file before they upgrade their gramps
> (from Python2 to Python3).
>
> Benny suggested a fix for when we do the pickle.loads
> ourselves but most of the filed bugs involve a read of various
> maps (tag_map, event_map, citation_map, etc.), and I
> don't know how to fix that/those (or possibly where).
>
> We have a user's zipped database directory which exhibits
> (at least) one of the places it crashes, if any of you feel
> like investigating this.  It repeats every time, just by starting
> gramps (without running the check tool, etc.), if that helps.
Are you referring to https://gramps-project.org/bugs/view.php?id=8258 ?

I downloaded that ZIP again, and hoped to find something by comparing
XMLs exported from the database as downloaded (Python 2), and the
converted one (Python 3), using the get_raw_tag_data patch that I posted
to that report.

With check crashing, and person and location view crashing too, I
figured that I'd see some character encoding differences comparing
Python 2 and 3 exported XMLs, but this is not the case. They are
identical in cmp and diff!

Now, when I import either XML (being the same) under Python 3, all views
work OK, and check too, and the exported XML, is identical again!

Is there anything that we can learn from this? If the Python 3 converted
database triggers crashes in all sorts of views, and check, and the
Python 3 imported XML database does not, it proves that database
contents are different. In fact, there's two places where things go wrong:

   File
"/home/enno/gramps-source/gramps/gui/filters/sidebar/_sidebarfilter.py",
line 213, in _tag_rebuild
     for handle in self.dbstate.db.get_tag_handles(sort_handles=True):
   File "/home/enno/gramps-source/gramps/gen/db/read.py", line 1195, in
get_tag_handles
     handle_list.sort(key=self.__sortbytag_key)
   File "/home/enno/gramps-source/gramps/gen/db/read.py", line 1855, in
__sortbytag_key
     tag = self.tag_map[key][1]
   File "/usr/lib/python3/dist-packages/bsddb3/dbshelve.py", line 152,
in __getitem__
     return pickle.loads(data)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 1:
ordinal not in range(128)

   File "/home/enno/gramps-source/gramps/plugins/tool/check.py", line
174, in __init__
     checker.fix_encoding()
   File "/home/enno/gramps-source/gramps/plugins/tool/check.py", line
351, in fix_encoding
     data = self.db.media_map[bhandle]
   File "/usr/lib/python3/dist-packages/bsddb3/dbshelve.py", line 152,
in __getitem__
     return pickle.loads(data)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4:
ordinal not in range(128)

This suggest that both tag_map and media_map can't handle the UTF-8
stored in there. The 0xc5 in the tag_map is the start character for the
s with reversed circumflex that I can see in the user's tag when I use
the database imported from Gramps XML.

Apparently, the XML code does all the necessary things, but other parts
do not.

regards,

Enno


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
12