May Check and Repair tool ask user to backup and import his data?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

May Check and Repair tool ask user to backup and import his data?

Michiel Nauta
Hi,

I am working on bug 5140. It is about import of an XML file with
identical handles for different primary objects. Previous versions of
Gramps, before 3.3.0, just accepted these files, version 3.3.0 raises an
error. Files shouldn't have such duplicate handles, handles are supposed
to be unique.

Fixing the import is not the problem, it is the Check and Repair tool
that is. The Check and Repair tool needs to look for such cross table
duplicate handles (as I call them) and fix them. I haven't got the code
for the Check and Repair tool yet, but I fear this could be quite
complicated because it needs code at the database level. It is far
easier to ask the user to make a backup of his database and import that
into an empty family tree. So that is what I do in the patch attached to
the bug. My question is, is this solution acceptable?

Michiel

------------------------------------------------------------------------------
FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
Subversion made easy with a complete admin console. Easy
to use, easy to manage, easy to install, easy to extend.
Get a Free download of the new open ALM Subversion platform now.
http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re : May Check and Repair tool ask user to backup and import his data?

jerome
Hi,


This could be an acceptable solution because I suppose this is a very special case, maybe related to use of handle.bash some months ago ???

http://gramps-project.org/wiki/index.php?title=Handle.bash

Note, there is also others issues where to create an new database and to import content is sometimes a solution.

http://www.gramps-project.org/bugs/view.php?id=5092
http://www.gramps-project.org/bugs/view.php?id=4515#c16956



Jérôme


--- En date de : Ven 12.8.11, Michiel Nauta <[hidden email]> a écrit :

> De: Michiel Nauta <[hidden email]>
> Objet: [Gramps-devel] May Check and Repair tool ask user to backup and import his data?
> À: "Gramps Development List" <[hidden email]>
> Date: Vendredi 12 août 2011, 23h14
> Hi,
>
> I am working on bug 5140. It is about import of an XML file
> with
> identical handles for different primary objects. Previous
> versions of
> Gramps, before 3.3.0, just accepted these files, version
> 3.3.0 raises an
> error. Files shouldn't have such duplicate handles, handles
> are supposed
> to be unique.
>
> Fixing the import is not the problem, it is the Check and
> Repair tool
> that is. The Check and Repair tool needs to look for such
> cross table
> duplicate handles (as I call them) and fix them. I haven't
> got the code
> for the Check and Repair tool yet, but I fear this could be
> quite
> complicated because it needs code at the database level. It
> is far
> easier to ask the user to make a backup of his database and
> import that
> into an empty family tree. So that is what I do in the
> patch attached to
> the bug. My question is, is this solution acceptable?
>
> Michiel
>
> ------------------------------------------------------------------------------
> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
> Subversion made easy with a complete admin console. Easy
> to use, easy to manage, easy to install, easy to extend.
> Get a Free download of the new open ALM Subversion platform
> now.
> http://p.sf.net/sfu/wandisco-dev2dev
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>

------------------------------------------------------------------------------
FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
Subversion made easy with a complete admin console. Easy
to use, easy to manage, easy to install, easy to extend.
Get a Free download of the new open ALM Subversion platform now.
http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re : May Check and Repair tool ask user to backup and import his data?

Julio Sánchez-2

Are we talking files that contain themselves duplicates or files containing handles that duplicate those already in the database?

If the same handle appears in two databases is because one is derived from the other.  That is, two entries in different databases that have the same handle, represent the same person, family, whatever.

If handles are unique and a handle appears on import equal to one already in the database, the best option is to merge them.  It is never an error.

An XML file that contains itself duplicate handles has ambiguous references and cannot be imported reliably.

Regards,

Julio


2011/8/13 jerome <[hidden email]>
Hi,


This could be an acceptable solution because I suppose this is a very special case, maybe related to use of handle.bash some months ago ???

http://gramps-project.org/wiki/index.php?title=Handle.bash

Note, there is also others issues where to create an new database and to import content is sometimes a solution.

http://www.gramps-project.org/bugs/view.php?id=5092
http://www.gramps-project.org/bugs/view.php?id=4515#c16956



Jérôme


--- En date de : Ven 12.8.11, Michiel Nauta <[hidden email]> a écrit :

> De: Michiel Nauta <[hidden email]>
> Objet: [Gramps-devel] May Check and Repair tool ask user to backup and import his data?
> À: "Gramps Development List" <[hidden email]>
> Date: Vendredi 12 août 2011, 23h14
> Hi,
>
> I am working on bug 5140. It is about import of an XML file
> with
> identical handles for different primary objects. Previous
> versions of
> Gramps, before 3.3.0, just accepted these files, version
> 3.3.0 raises an
> error. Files shouldn't have such duplicate handles, handles
> are supposed
> to be unique.
>
> Fixing the import is not the problem, it is the Check and
> Repair tool
> that is. The Check and Repair tool needs to look for such
> cross table
> duplicate handles (as I call them) and fix them. I haven't
> got the code
> for the Check and Repair tool yet, but I fear this could be
> quite
> complicated because it needs code at the database level. It
> is far
> easier to ask the user to make a backup of his database and
> import that
> into an empty family tree. So that is what I do in the
> patch attached to
> the bug. My question is, is this solution acceptable?
>
> Michiel
>
> ------------------------------------------------------------------------------
> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
> Subversion made easy with a complete admin console. Easy
> to use, easy to manage, easy to install, easy to extend.
> Get a Free download of the new open ALM Subversion platform
> now.
> http://p.sf.net/sfu/wandisco-dev2dev
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>

------------------------------------------------------------------------------
FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
Subversion made easy with a complete admin console. Easy
to use, easy to manage, easy to install, easy to extend.
Get a Free download of the new open ALM Subversion platform now.
http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel



--
Julio Sánchez Fernández, CISM, CISA, CRISC



------------------------------------------------------------------------------
FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
Subversion made easy with a complete admin console. Easy
to use, easy to manage, easy to install, easy to extend.
Get a Free download of the new open ALM Subversion platform now.
http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re : May Check and Repair tool ask user to backup and import his data?

Michiel Nauta
Hi Julio,

We are talking about one XML file with a person, a place and a source
that all three have the same handle.

You can reliably import such a file because with each reference you know
what kind of primary object that reference points to:
a sourceref with hlink="123456" point to the source while a childref
with hlink="123456" points to the person.

Michiel

On 08/13/2011 03:44 PM, Julio Sánchez wrote:

> Are we talking files that contain themselves duplicates or files containing
> handles that duplicate those already in the database?
>
> If the same handle appears in two databases is because one is derived from
> the other.  That is, two entries in different databases that have the same
> handle, represent the same person, family, whatever.
>
> If handles are unique and a handle appears on import equal to one already in
> the database, the best option is to merge them.  It is never an error.
>
> An XML file that contains itself duplicate handles has ambiguous references
> and cannot be imported reliably.
>
> Regards,
>
> Julio
>
>
> 2011/8/13 jerome<[hidden email]>
>
>> Hi,
>>
>>
>> This could be an acceptable solution because I suppose this is a very
>> special case, maybe related to use of handle.bash some months ago ???
>>
>> http://gramps-project.org/wiki/index.php?title=Handle.bash
>>
>> Note, there is also others issues where to create an new database and to
>> import content is sometimes a solution.
>>
>> http://www.gramps-project.org/bugs/view.php?id=5092
>> http://www.gramps-project.org/bugs/view.php?id=4515#c16956
>>
>>
>>
>> Jérôme
>>
>>
>> --- En date de : Ven 12.8.11, Michiel Nauta<[hidden email]>  a
>> écrit :
>>
>>> De: Michiel Nauta<[hidden email]>
>>> Objet: [Gramps-devel] May Check and Repair tool ask user to backup and
>> import his data?
>>> À: "Gramps Development List"<[hidden email]>
>>> Date: Vendredi 12 août 2011, 23h14
>>> Hi,
>>>
>>> I am working on bug 5140. It is about import of an XML file
>>> with
>>> identical handles for different primary objects. Previous
>>> versions of
>>> Gramps, before 3.3.0, just accepted these files, version
>>> 3.3.0 raises an
>>> error. Files shouldn't have such duplicate handles, handles
>>> are supposed
>>> to be unique.
>>>
>>> Fixing the import is not the problem, it is the Check and
>>> Repair tool
>>> that is. The Check and Repair tool needs to look for such
>>> cross table
>>> duplicate handles (as I call them) and fix them. I haven't
>>> got the code
>>> for the Check and Repair tool yet, but I fear this could be
>>> quite
>>> complicated because it needs code at the database level. It
>>> is far
>>> easier to ask the user to make a backup of his database and
>>> import that
>>> into an empty family tree. So that is what I do in the
>>> patch attached to
>>> the bug. My question is, is this solution acceptable?
>>>
>>> Michiel
>>>
>>>
>> ------------------------------------------------------------------------------
>>> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
>>> Subversion made easy with a complete admin console. Easy
>>> to use, easy to manage, easy to install, easy to extend.
>>> Get a Free download of the new open ALM Subversion platform
>>> now.
>>> http://p.sf.net/sfu/wandisco-dev2dev
>>> _______________________________________________
>>> Gramps-devel mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
>> Subversion made easy with a complete admin console. Easy
>> to use, easy to manage, easy to install, easy to extend.
>> Get a Free download of the new open ALM Subversion platform now.
>> http://p.sf.net/sfu/wandisco-dev2dev
>> _______________________________________________
>> Gramps-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>
>
>
>

------------------------------------------------------------------------------
FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
Subversion made easy with a complete admin console. Easy
to use, easy to manage, easy to install, easy to extend.
Get a Free download of the new open ALM Subversion platform now.
http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re : May Check and Repair tool ask user to backup and import his data?

DS Blank
On Sun, Aug 14, 2011 at 4:24 AM, Michiel Nauta <[hidden email]> wrote:

> Hi Julio,
>
> We are talking about one XML file with a person, a place and a source
> that all three have the same handle.
>
> You can reliably import such a file because with each reference you know
> what kind of primary object that reference points to:
> a sourceref with hlink="123456" point to the source while a childref
> with hlink="123456" points to the person.
>
> Michiel
>
> On 08/13/2011 03:44 PM, Julio Sánchez wrote:
>> Are we talking files that contain themselves duplicates or files containing
>> handles that duplicate those already in the database?
>>
>> If the same handle appears in two databases is because one is derived from
>> the other.  That is, two entries in different databases that have the same
>> handle, represent the same person, family, whatever.
>>
>> If handles are unique and a handle appears on import equal to one already in
>> the database, the best option is to merge them.  It is never an error.
>>
>> An XML file that contains itself duplicate handles has ambiguous references
>> and cannot be imported reliably.
>>
>> Regards,
>>
>> Julio
>>
>>
>> 2011/8/13 jerome<[hidden email]>
>>
>>> Hi,
>>>
>>>
>>> This could be an acceptable solution because I suppose this is a very
>>> special case, maybe related to use of handle.bash some months ago ???
>>>
>>> http://gramps-project.org/wiki/index.php?title=Handle.bash
>>>
>>> Note, there is also others issues where to create an new database and to
>>> import content is sometimes a solution.
>>>
>>> http://www.gramps-project.org/bugs/view.php?id=5092
>>> http://www.gramps-project.org/bugs/view.php?id=4515#c16956
>>>
>>>
>>>
>>> Jérôme
>>>
>>>
>>> --- En date de : Ven 12.8.11, Michiel Nauta<[hidden email]>  a
>>> écrit :
>>>
>>>> De: Michiel Nauta<[hidden email]>
>>>> Objet: [Gramps-devel] May Check and Repair tool ask user to backup and
>>> import his data?
>>>> À: "Gramps Development List"<[hidden email]>
>>>> Date: Vendredi 12 août 2011, 23h14
>>>> Hi,
>>>>
>>>> I am working on bug 5140. It is about import of an XML file
>>>> with
>>>> identical handles for different primary objects. Previous
>>>> versions of
>>>> Gramps, before 3.3.0, just accepted these files, version
>>>> 3.3.0 raises an
>>>> error. Files shouldn't have such duplicate handles, handles
>>>> are supposed
>>>> to be unique.
>>>>
>>>> Fixing the import is not the problem, it is the Check and
>>>> Repair tool
>>>> that is. The Check and Repair tool needs to look for such
>>>> cross table
>>>> duplicate handles (as I call them) and fix them. I haven't
>>>> got the code
>>>> for the Check and Repair tool yet, but I fear this could be
>>>> quite
>>>> complicated because it needs code at the database level. It
>>>> is far
>>>> easier to ask the user to make a backup of his database and
>>>> import that
>>>> into an empty family tree. So that is what I do in the
>>>> patch attached to
>>>> the bug. My question is, is this solution acceptable?
>>>>
>>>> Michiel

Michiel,

Thanks for working on this issue.

Perhaps now is the time to seriously look at turning handles into
GUIDs (global unique identifiers). If we do that, perhaps then we can
already ensure (with near 100% probability) that handles are unique
(not just between objects in a tree, but with all other objects on the
planet) and that any collisions "must" refer to the same object. We
can then turn the old, short handles into an attribute that is used in
NarrWeb addresses, if one wished (for backwards compatibility).

I guess I'd rather spend time towards some really interesting features
than trying to fix this issue that shouldn't be possible anyway.

There are also some interesting points to discuss with GUIDs: if a
person can be uniquely identified in one tree, does that pose privacy
issues for that individual when they are in another? I think that
there are ways to work around this (for example, private objects
should always get a new GUID when generating data for general viewing
(eg, NarWeb)). But we won't have these discussions until we begin to
move in that direction.

-Doug

------------------------------------------------------------------------------
FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
Subversion made easy with a complete admin console. Easy
to use, easy to manage, easy to install, easy to extend.
Get a Free download of the new open ALM Subversion platform now.
http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re : May Check and Repair tool ask user to backup and import his data?

Doug
In reply to this post by jerome
Hi,

I'd be unhappy to see the patch taken as a full resolution
of this and related bugs just yet.
Some sorts of editing need the database to be exported as
csv (or xml) for editing and re-imported back onto the
original: I've not been able so far to be sure if merging
always takes place perfectly or if it leaves duplicates
remain that need identifying.

For this case, requiring the import to create a new family
tree wouldn't be helpful.

Doug



On 13/08/11 05:58, jerome wrote:

> Hi,
>
>
> This could be an acceptable solution because I suppose this is a very special case, maybe related to use of handle.bash some months ago ???
>
> http://gramps-project.org/wiki/index.php?title=Handle.bash
>
> Note, there is also others issues where to create an new database and to import content is sometimes a solution.
>
> http://www.gramps-project.org/bugs/view.php?id=5092
> http://www.gramps-project.org/bugs/view.php?id=4515#c16956
>
>
>
> Jérôme
>
>
> --- En date de : Ven 12.8.11, Michiel Nauta<[hidden email]>  a écrit :
>
>> De: Michiel Nauta<[hidden email]>
>> Objet: [Gramps-devel] May Check and Repair tool ask user to backup and import his data?
>> À: "Gramps Development List"<[hidden email]>
>> Date: Vendredi 12 août 2011, 23h14
>> Hi,
>>
>> I am working on bug 5140. It is about import of an XML file
>> with
>> identical handles for different primary objects. Previous
>> versions of
>> Gramps, before 3.3.0, just accepted these files, version
>> 3.3.0 raises an
>> error. Files shouldn't have such duplicate handles, handles
>> are supposed
>> to be unique.
>>
>> Fixing the import is not the problem, it is the Check and
>> Repair tool
>> that is. The Check and Repair tool needs to look for such
>> cross table
>> duplicate handles (as I call them) and fix them. I haven't
>> got the code
>> for the Check and Repair tool yet, but I fear this could be
>> quite
>> complicated because it needs code at the database level. It
>> is far
>> easier to ask the user to make a backup of his database and
>> import that
>> into an empty family tree. So that is what I do in the
>> patch attached to
>> the bug. My question is, is this solution acceptable?
>>
>> Michiel
>>
>> ------------------------------------------------------------------------------
>> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
>> Subversion made easy with a complete admin console. Easy
>> to use, easy to manage, easy to install, easy to extend.
>> Get a Free download of the new open ALM Subversion platform
>> now.
>> http://p.sf.net/sfu/wandisco-dev2dev
>> _______________________________________________
>> Gramps-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>
>
> ------------------------------------------------------------------------------
> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
> Subversion made easy with a complete admin console. Easy
> to use, easy to manage, easy to install, easy to extend.
> Get a Free download of the new open ALM Subversion platform now.
> http://p.sf.net/sfu/wandisco-dev2dev
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>


------------------------------------------------------------------------------
FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
Subversion made easy with a complete admin console. Easy
to use, easy to manage, easy to install, easy to extend.
Get a Free download of the new open ALM Subversion platform now.
http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re : May Check and Repair tool ask user to backup and import his data?

Michiel Nauta
In reply to this post by DS Blank


On 08/14/2011 02:34 PM, Doug Blank wrote:

>
> Michiel,
>
> Thanks for working on this issue.
>
> Perhaps now is the time to seriously look at turning handles into
> GUIDs (global unique identifiers). If we do that, perhaps then we can
> already ensure (with near 100% probability) that handles are unique
> (not just between objects in a tree, but with all other objects on the
> planet) and that any collisions "must" refer to the same object. We
> can then turn the old, short handles into an attribute that is used in
> NarrWeb addresses, if one wished (for backwards compatibility).
>
> I guess I'd rather spend time towards some really interesting features
> than trying to fix this issue that shouldn't be possible anyway.
>
> There are also some interesting points to discuss with GUIDs: if a
> person can be uniquely identified in one tree, does that pose privacy
> issues for that individual when they are in another? I think that
> there are ways to work around this (for example, private objects
> should always get a new GUID when generating data for general viewing
> (eg, NarWeb)). But we won't have these discussions until we begin to
> move in that direction.
>
> -Doug
>
Doug,

What I need now is a solution for the coming 3.3.1 release. What you are
suggesting is a solution that would only be acceptable for a new major
release. Note also that to introduce GUIDs, this file needs to be
cleaned up anyhow before you can convert the handle into a GUID.

Interesting point you bring forward with respect to GUIDs and privacy.

Michiel

------------------------------------------------------------------------------
FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
Subversion made easy with a complete admin console. Easy
to use, easy to manage, easy to install, easy to extend.
Get a Free download of the new open ALM Subversion platform now.
http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re : May Check and Repair tool ask user to backup and import his data?

Michiel Nauta
In reply to this post by Doug
Hi Doug,

I am not sure I understand your objection. The patch is very specific
for the case where there is a handle value that is in use by two primary
objects of different kind such as a person and a source. I don't see how
export, editing and reimport could create such a situation. If you have
the know how to use this way of working, then you should also know that
what is indicated by the DTD to be an "id" that that should be unique.

Michiel

On 08/14/2011 04:17 PM, doug wrote:

> Hi,
>
> I'd be unhappy to see the patch taken as a full resolution
> of this and related bugs just yet.
> Some sorts of editing need the database to be exported as
> csv (or xml) for editing and re-imported back onto the
> original: I've not been able so far to be sure if merging
> always takes place perfectly or if it leaves duplicates
> remain that need identifying.
>
> For this case, requiring the import to create a new family
> tree wouldn't be helpful.
>
> Doug
>
>
>
> On 13/08/11 05:58, jerome wrote:
>> Hi,
>>
>>
>> This could be an acceptable solution because I suppose this is a very special case, maybe related to use of handle.bash some months ago ???
>>
>> http://gramps-project.org/wiki/index.php?title=Handle.bash
>>
>> Note, there is also others issues where to create an new database and to import content is sometimes a solution.
>>
>> http://www.gramps-project.org/bugs/view.php?id=5092
>> http://www.gramps-project.org/bugs/view.php?id=4515#c16956
>>
>>
>>
>> Jérôme
>>
>>
>> --- En date de : Ven 12.8.11, Michiel Nauta<[hidden email]>   a écrit :
>>
>>> De: Michiel Nauta<[hidden email]>
>>> Objet: [Gramps-devel] May Check and Repair tool ask user to backup and import his data?
>>> À: "Gramps Development List"<[hidden email]>
>>> Date: Vendredi 12 août 2011, 23h14
>>> Hi,
>>>
>>> I am working on bug 5140. It is about import of an XML file
>>> with
>>> identical handles for different primary objects. Previous
>>> versions of
>>> Gramps, before 3.3.0, just accepted these files, version
>>> 3.3.0 raises an
>>> error. Files shouldn't have such duplicate handles, handles
>>> are supposed
>>> to be unique.
>>>
>>> Fixing the import is not the problem, it is the Check and
>>> Repair tool
>>> that is. The Check and Repair tool needs to look for such
>>> cross table
>>> duplicate handles (as I call them) and fix them. I haven't
>>> got the code
>>> for the Check and Repair tool yet, but I fear this could be
>>> quite
>>> complicated because it needs code at the database level. It
>>> is far
>>> easier to ask the user to make a backup of his database and
>>> import that
>>> into an empty family tree. So that is what I do in the
>>> patch attached to
>>> the bug. My question is, is this solution acceptable?
>>>
>>> Michiel
>>>
>>> ------------------------------------------------------------------------------
>>> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
>>> Subversion made easy with a complete admin console. Easy
>>> to use, easy to manage, easy to install, easy to extend.
>>> Get a Free download of the new open ALM Subversion platform
>>> now.
>>> http://p.sf.net/sfu/wandisco-dev2dev
>>> _______________________________________________
>>> Gramps-devel mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>>
>>
>> ------------------------------------------------------------------------------
>> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
>> Subversion made easy with a complete admin console. Easy
>> to use, easy to manage, easy to install, easy to extend.
>> Get a Free download of the new open ALM Subversion platform now.
>> http://p.sf.net/sfu/wandisco-dev2dev
>> _______________________________________________
>> Gramps-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>
>
>
> ------------------------------------------------------------------------------
> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
> Subversion made easy with a complete admin console. Easy
> to use, easy to manage, easy to install, easy to extend.
> Get a Free download of the new open ALM Subversion platform now.
> http://p.sf.net/sfu/wandisco-dev2dev
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>

------------------------------------------------------------------------------
FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
Subversion made easy with a complete admin console. Easy
to use, easy to manage, easy to install, easy to extend.
Get a Free download of the new open ALM Subversion platform now.
http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: May Check and Repair tool ask user to backup and import his data?

Nick Hall-6
In reply to this post by Michiel Nauta
Michiel,

Ideally, we should update the Check and Repair tool to fix this
problem.  If the database can become corrupt then we should be able to
detect and repair the problem.

How did the database and XML export get these duplicate handles in the
first place?

Does a duplicate handle in a different table actually cause a problem?

If we allow these duplicate handles, then we should be able to import
them (as with v3.2).  If duplicate handles cause badly formatted XML
should we actually try to import it?  Perhaps we should just output and
error and abort the import.

Nick.


On 12/08/11 22:14, Michiel Nauta wrote:

> Hi,
>
> I am working on bug 5140. It is about import of an XML file with
> identical handles for different primary objects. Previous versions of
> Gramps, before 3.3.0, just accepted these files, version 3.3.0 raises an
> error. Files shouldn't have such duplicate handles, handles are supposed
> to be unique.
>
> Fixing the import is not the problem, it is the Check and Repair tool
> that is. The Check and Repair tool needs to look for such cross table
> duplicate handles (as I call them) and fix them. I haven't got the code
> for the Check and Repair tool yet, but I fear this could be quite
> complicated because it needs code at the database level. It is far
> easier to ask the user to make a backup of his database and import that
> into an empty family tree. So that is what I do in the patch attached to
> the bug. My question is, is this solution acceptable?
>
> Michiel
>
> ------------------------------------------------------------------------------
> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
> Subversion made easy with a complete admin console. Easy
> to use, easy to manage, easy to install, easy to extend.
> Get a Free download of the new open ALM Subversion platform now.
> http://p.sf.net/sfu/wandisco-dev2dev
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>
>

------------------------------------------------------------------------------
FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
Subversion made easy with a complete admin console. Easy
to use, easy to manage, easy to install, easy to extend.
Get a Free download of the new open ALM Subversion platform now.
http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re : May Check and Repair tool ask user to backup and import his data?

Doug
In reply to this post by Michiel Nauta
Hi Michiel,

My concern was not with the patch as such.
It's that various bugs related to Importing, Merging and
Finding Duplicates seem to have got tied together - "related
to ....", "same as...", and so forth.
I was afraid this association might end up with persisting
problems - in particular the Find Possible Duplicates -
getting marked as resolved in the bug tracker and then shelved.


Doug

On 14/08/11 22:14, Michiel Nauta wrote:

> Hi Doug,
>
> I am not sure I understand your objection. The patch is very
> specific for the case where there is a handle value that is
> in use by two primary objects of different kind such as a
> person and a source. I don't see how export, editing and
> reimport could create such a situation. If you have the know
> how to use this way of working, then you should also know
> that what is indicated by the DTD to be an "id" that that
> should be unique.
>
> Michiel
>
> On 08/14/2011 04:17 PM, doug wrote:
>> Hi,
>>
>> I'd be unhappy to see the patch taken as a full resolution
>> of this and related bugs just yet.
>> Some sorts of editing need the database to be exported as
>> csv (or xml) for editing and re-imported back onto the
>> original: I've not been able so far to be sure if merging
>> always takes place perfectly or if it leaves duplicates
>> remain that need identifying.
>>
>> For this case, requiring the import to create a new family
>> tree wouldn't be helpful.
>>
>> Doug
>>
>>
>>
>> On 13/08/11 05:58, jerome wrote:
>>> Hi,
>>>
>>>
>>> This could be an acceptable solution because I suppose
>>> this is a very special case, maybe related to use of
>>> handle.bash some months ago ???
>>>
>>> http://gramps-project.org/wiki/index.php?title=Handle.bash
>>>
>>> Note, there is also others issues where to create an new
>>> database and to import content is sometimes a solution.
>>>
>>> http://www.gramps-project.org/bugs/view.php?id=5092
>>> http://www.gramps-project.org/bugs/view.php?id=4515#c16956
>>>
>>>
>>>
>>> Jérôme
>>>
>>>
>>> --- En date de : Ven 12.8.11, Michiel
>>> Nauta<[hidden email]> a écrit :
>>>
>>>> De: Michiel Nauta<[hidden email]>
>>>> Objet: [Gramps-devel] May Check and Repair tool ask user
>>>> to backup and import his data?
>>>> À: "Gramps Development
>>>> List"<[hidden email]>
>>>> Date: Vendredi 12 août 2011, 23h14
>>>> Hi,
>>>>
>>>> I am working on bug 5140. It is about import of an XML file
>>>> with
>>>> identical handles for different primary objects. Previous
>>>> versions of
>>>> Gramps, before 3.3.0, just accepted these files, version
>>>> 3.3.0 raises an
>>>> error. Files shouldn't have such duplicate handles, handles
>>>> are supposed
>>>> to be unique.
>>>>
>>>> Fixing the import is not the problem, it is the Check and
>>>> Repair tool
>>>> that is. The Check and Repair tool needs to look for such
>>>> cross table
>>>> duplicate handles (as I call them) and fix them. I haven't
>>>> got the code
>>>> for the Check and Repair tool yet, but I fear this could be
>>>> quite
>>>> complicated because it needs code at the database level. It
>>>> is far
>>>> easier to ask the user to make a backup of his database and
>>>> import that
>>>> into an empty family tree. So that is what I do in the
>>>> patch attached to
>>>> the bug. My question is, is this solution acceptable?
>>>>
>>>> Michiel
>>>>
>>>> ------------------------------------------------------------------------------
>>>>
>>>> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
>>>> Subversion made easy with a complete admin console. Easy
>>>> to use, easy to manage, easy to install, easy to extend.
>>>> Get a Free download of the new open ALM Subversion platform
>>>> now.
>>>> http://p.sf.net/sfu/wandisco-dev2dev
>>>> _______________________________________________
>>>> Gramps-devel mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>>>
>>>
>>> ------------------------------------------------------------------------------
>>>
>>> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
>>> Subversion made easy with a complete admin console. Easy
>>> to use, easy to manage, easy to install, easy to extend.
>>> Get a Free download of the new open ALM Subversion
>>> platform now.
>>> http://p.sf.net/sfu/wandisco-dev2dev
>>> _______________________________________________
>>> Gramps-devel mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>>
>>
>>
>> ------------------------------------------------------------------------------
>>
>> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
>> Subversion made easy with a complete admin console. Easy
>> to use, easy to manage, easy to install, easy to extend.
>> Get a Free download of the new open ALM Subversion
>> platform now.
>> http://p.sf.net/sfu/wandisco-dev2dev
>> _______________________________________________
>> Gramps-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>
>


------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at:  http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: May Check and Repair tool ask user to backup and import his data?

Michiel Nauta
In reply to this post by Nick Hall-6
Hi Nick,

Answers in between.

On 08/15/2011 01:14 AM, Nick Hall wrote:
> Michiel,
>
> Ideally, we should update the Check and Repair tool to fix this
> problem.  If the database can become corrupt then we should be able to
> detect and repair the problem.
>
Ideally I agree, but changing the database handles of objects that are
already stored in the database is not easy. We would need to write some
new code at the database level to make this possible. Is it worth it for
something that is not supposed to happen and in practice is probably
very rare?

> How did the database and XML export get these duplicate handles in the
> first place?
This is not known. The duplicate handles are in the database for several
years. The reporter is an old time user of Gramps and can't remember
anything special about the data. I see nothing suspicious when look at
the XML: the handle is a good random number in agreement with other
handles and the times data was changed points to an import in perhaps
Gramps version 2.2.10, but that doesn't mean that the error was
introduced by that import, perhaps the imported data already contained
the duplicate handles.

>
> Does a duplicate handle in a different table actually cause a problem?
>
The reference tab contains false information. I am worried that the
merge routines might cause trouble. Basically anything that uses the
find_backlink_handles method will get erroneous information.

> If we allow these duplicate handles, then we should be able to import
> them (as with v3.2).  If duplicate handles cause badly formatted XML
> should we actually try to import it?  Perhaps we should just output and
> error and abort the import.
>
Duplicate handles cause invalid XML. But it is not the import that is
the problem, that is easy to fix. It is the Check and Repair tool that
is so difficult. So that is why I ask if it is OK to tell the user to
backup his data and re-import it. This is kind of new behavior of the
Check and Repair tool because thus far any error found by the Check can
be repaired by the Repair.

Michiel

> Nick.
>
>
> On 12/08/11 22:14, Michiel Nauta wrote:
>> Hi,
>>
>> I am working on bug 5140. It is about import of an XML file with
>> identical handles for different primary objects. Previous versions of
>> Gramps, before 3.3.0, just accepted these files, version 3.3.0 raises an
>> error. Files shouldn't have such duplicate handles, handles are supposed
>> to be unique.
>>
>> Fixing the import is not the problem, it is the Check and Repair tool
>> that is. The Check and Repair tool needs to look for such cross table
>> duplicate handles (as I call them) and fix them. I haven't got the code
>> for the Check and Repair tool yet, but I fear this could be quite
>> complicated because it needs code at the database level. It is far
>> easier to ask the user to make a backup of his database and import that
>> into an empty family tree. So that is what I do in the patch attached to
>> the bug. My question is, is this solution acceptable?
>>
>> Michiel
>>
>> ------------------------------------------------------------------------------
>> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
>> Subversion made easy with a complete admin console. Easy
>> to use, easy to manage, easy to install, easy to extend.
>> Get a Free download of the new open ALM Subversion platform now.
>> http://p.sf.net/sfu/wandisco-dev2dev
>> _______________________________________________
>> Gramps-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>
>>
>
> ------------------------------------------------------------------------------
> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
> Subversion made easy with a complete admin console. Easy
> to use, easy to manage, easy to install, easy to extend.
> Get a Free download of the new open ALM Subversion platform now.
> http://p.sf.net/sfu/wandisco-dev2dev
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>

------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at:  http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: May Check and Repair tool ask user to backup and import his data?

Nick Hall-6
Michiel,

Thanks for your reply.

So the main problem is that the key to the reference table is a handle
which is not qualified by an object class.

Changing handles would involve database level code, which would need
lots of testing.  Fixing the references for the changed handles would
also be difficult without reliable back references.

Your proposed solution is probably the best idea.  The Check and Repair
tool should report the error but not fix it.  The XML import should be
modified to cope with the duplicate handles.  This will give us a way to
fix a problem which is hopefully very rare.

Nick.


On 15/08/11 22:27, Michiel Nauta wrote:

> Hi Nick,
>
> Answers in between.
>
> On 08/15/2011 01:14 AM, Nick Hall wrote:
>> Michiel,
>>
>> Ideally, we should update the Check and Repair tool to fix this
>> problem.  If the database can become corrupt then we should be able to
>> detect and repair the problem.
>>
> Ideally I agree, but changing the database handles of objects that are
> already stored in the database is not easy. We would need to write
> some new code at the database level to make this possible. Is it worth
> it for something that is not supposed to happen and in practice is
> probably very rare?
>
>> How did the database and XML export get these duplicate handles in the
>> first place?
> This is not known. The duplicate handles are in the database for
> several years. The reporter is an old time user of Gramps and can't
> remember anything special about the data. I see nothing suspicious
> when look at the XML: the handle is a good random number in agreement
> with other handles and the times data was changed points to an import
> in perhaps Gramps version 2.2.10, but that doesn't mean that the error
> was introduced by that import, perhaps the imported data already
> contained the duplicate handles.
>
>>
>> Does a duplicate handle in a different table actually cause a problem?
>>
> The reference tab contains false information. I am worried that the
> merge routines might cause trouble. Basically anything that uses the
> find_backlink_handles method will get erroneous information.
>
>> If we allow these duplicate handles, then we should be able to import
>> them (as with v3.2).  If duplicate handles cause badly formatted XML
>> should we actually try to import it?  Perhaps we should just output and
>> error and abort the import.
>>
> Duplicate handles cause invalid XML. But it is not the import that is
> the problem, that is easy to fix. It is the Check and Repair tool that
> is so difficult. So that is why I ask if it is OK to tell the user to
> backup his data and re-import it. This is kind of new behavior of the
> Check and Repair tool because thus far any error found by the Check
> can be repaired by the Repair.
>
> Michiel
>
>> Nick.
>>
>>
>> On 12/08/11 22:14, Michiel Nauta wrote:
>>> Hi,
>>>
>>> I am working on bug 5140. It is about import of an XML file with
>>> identical handles for different primary objects. Previous versions of
>>> Gramps, before 3.3.0, just accepted these files, version 3.3.0
>>> raises an
>>> error. Files shouldn't have such duplicate handles, handles are
>>> supposed
>>> to be unique.
>>>
>>> Fixing the import is not the problem, it is the Check and Repair tool
>>> that is. The Check and Repair tool needs to look for such cross table
>>> duplicate handles (as I call them) and fix them. I haven't got the code
>>> for the Check and Repair tool yet, but I fear this could be quite
>>> complicated because it needs code at the database level. It is far
>>> easier to ask the user to make a backup of his database and import that
>>> into an empty family tree. So that is what I do in the patch
>>> attached to
>>> the bug. My question is, is this solution acceptable?
>>>
>>> Michiel
>>>
>>> ------------------------------------------------------------------------------
>>>
>>> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
>>> Subversion made easy with a complete admin console. Easy
>>> to use, easy to manage, easy to install, easy to extend.
>>> Get a Free download of the new open ALM Subversion platform now.
>>> http://p.sf.net/sfu/wandisco-dev2dev
>>> _______________________________________________
>>> Gramps-devel mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>>
>>>
>>
>> ------------------------------------------------------------------------------
>>
>> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
>> Subversion made easy with a complete admin console. Easy
>> to use, easy to manage, easy to install, easy to extend.
>> Get a Free download of the new open ALM Subversion platform now.
>> http://p.sf.net/sfu/wandisco-dev2dev
>> _______________________________________________
>> Gramps-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>
>
>

------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at:  http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel