Quantcast

Help with importing a backup into existing database

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Help with importing a backup into existing database

Ken B.
Hi All,

    I have been using Gramps for a long time, and get very frustrated when I import data (in xml) from other people.

    Most items, people and events import and merge correctly. The main problem I have is with sources and media. These are duplicated, and take a lot of work to correct.

    I have a friend who understands programing and is willing to help me try to fix this. Is there anyone who can point me in the direction of the files to start with ? Or able to provide advice to someone willing to learn, if I want to try this myself ?

Thank you,
Ken.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with importing a backup into existing database

Benny Malengier
You should check the mail archives about UID and import/merge
Eg: http://www.nabble.com/External-Links-to-second-GRAMPS-Report-pages-td20304109.html

GRAMPS does _not_ support import/merge now, whatever happens is pure side effect of me and others trying to make it not too much broken, however there is no system, and the next release might do things differently.

Some more info here: http://gramps-project.org/wiki/index.php?title=GEPS_009:_Import_Export_Merge

So, some tasks to start:

1/add the UID creation tool based on Julio's patch
2/review the merge code as activated now on merging objects. This is the code that the merge on import must call too. Here Julio has also provided a patch for his case
3/stick to XML import for the merge scenario, and implement the merge correctly. My guess is it should become a two pass process:
  a/ go over the xml, see if UID's present, see if same UID is present in the existing database. This is quick for handle's, but a problem for double UID's in the attribute tag. So a new table needed? For non persons and family's this can be done based on handle, so it is fast. For your use case, import of xml, you can work also based on handle, so you can skip this
  b/ import the entire data, using different keys for the one's that must be merged
  c/after import, start the merging process based on the collected data, using same procedures as used by  the normal merge routine of the objects.

Above are just ideas for now, after research other solutions might be opted for. The idea to use only one set of merge routine should reduce code complexity. The fact that import stores without merge would be for the same reason.

Benny

2008/11/23 Ken B <[hidden email]>
Hi All,

    I have been using Gramps for a long time, and get very frustrated when I import data (in xml) from other people.

    Most items, people and events import and merge correctly. The main problem I have is with sources and media. These are duplicated, and take a lot of work to correct.

    I have a friend who understands programing and is willing to help me try to fix this. Is there anyone who can point me in the direction of the files to start with ? Or able to provide advice to someone willing to learn, if I want to try this myself ?

Thank you,
Ken.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with importing a backup into existing database

Ken B.
Thank you for your reply, Benny.
That is great information to start with. Hopefully we will be able to produce something useful over time, that we can send to you.

Ken.

Benny Malengier wrote:
You should check the mail archives about UID and import/merge
Eg: http://www.nabble.com/External-Links-to-second-GRAMPS-Report-pages-td20304109.html

GRAMPS does _not_ support import/merge now, whatever happens is pure side effect of me and others trying to make it not too much broken, however there is no system, and the next release might do things differently.

Some more info here: http://gramps-project.org/wiki/index.php?title=GEPS_009:_Import_Export_Merge

So, some tasks to start:

1/add the UID creation tool based on Julio's patch
2/review the merge code as activated now on merging objects. This is the code that the merge on import must call too. Here Julio has also provided a patch for his case
3/stick to XML import for the merge scenario, and implement the merge correctly. My guess is it should become a two pass process:
  a/ go over the xml, see if UID's present, see if same UID is present in the existing database. This is quick for handle's, but a problem for double UID's in the attribute tag. So a new table needed? For non persons and family's this can be done based on handle, so it is fast. For your use case, import of xml, you can work also based on handle, so you can skip this
  b/ import the entire data, using different keys for the one's that must be merged
  c/after import, start the merging process based on the collected data, using same procedures as used by  the normal merge routine of the objects.

Above are just ideas for now, after research other solutions might be opted for. The idea to use only one set of merge routine should reduce code complexity. The fact that import stores without merge would be for the same reason.

Benny

2008/11/23 Ken B <[hidden email]>
Hi All,

    I have been using Gramps for a long time, and get very frustrated when I import data (in xml) from other people.

    Most items, people and events import and merge correctly. The main problem I have is with sources and media. These are duplicated, and take a lot of work to correct.

    I have a friend who understands programing and is willing to help me try to fix this. Is there anyone who can point me in the direction of the files to start with ? Or able to provide advice to someone willing to learn, if I want to try this myself ?

Thank you,
Ken.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with importing a backup into existing database

Ken B.
In reply to this post by Benny Malengier
Hi All,

    My friend has been discussing this with his class, and a student made the suggestion "Because the duplicates (events, sources, attributes etc) are also duplicte Gramps ID would it not make more sense to add a search for these to the 'Check and Repair Database' tool ?
    What do you all think ? Is it worth making this a feature request ?
Ken.

Benny Malengier wrote:
You should check the mail archives about UID and import/merge
Eg: http://www.nabble.com/External-Links-to-second-GRAMPS-Report-pages-td20304109.html

GRAMPS does _not_ support import/merge now, whatever happens is pure side effect of me and others trying to make it not too much broken, however there is no system, and the next release might do things differently.

Some more info here: http://gramps-project.org/wiki/index.php?title=GEPS_009:_Import_Export_Merge

So, some tasks to start:

1/add the UID creation tool based on Julio's patch
2/review the merge code as activated now on merging objects. This is the code that the merge on import must call too. Here Julio has also provided a patch for his case
3/stick to XML import for the merge scenario, and implement the merge correctly. My guess is it should become a two pass process:
  a/ go over the xml, see if UID's present, see if same UID is present in the existing database. This is quick for handle's, but a problem for double UID's in the attribute tag. So a new table needed? For non persons and family's this can be done based on handle, so it is fast. For your use case, import of xml, you can work also based on handle, so you can skip this
  b/ import the entire data, using different keys for the one's that must be merged
  c/after import, start the merging process based on the collected data, using same procedures as used by  the normal merge routine of the objects.

Above are just ideas for now, after research other solutions might be opted for. The idea to use only one set of merge routine should reduce code complexity. The fact that import stores without merge would be for the same reason.

Benny

2008/11/23 Ken B <[hidden email]>
Hi All,

    I have been using Gramps for a long time, and get very frustrated when I import data (in xml) from other people.

    Most items, people and events import and merge correctly. The main problem I have is with sources and media. These are duplicated, and take a lot of work to correct.

    I have a friend who understands programing and is willing to help me try to fix this. Is there anyone who can point me in the direction of the files to start with ? Or able to provide advice to someone willing to learn, if I want to try this myself ?

Thank you,
Ken.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with importing a backup into existing database

Benny Malengier
Gramps should not allow the creation of duplicate ID's.
Anything that can create it is a bug.
Check and repair could check it, but then, it can check a lot of things extra. All impacts the speed of the tool.
If a user sees duplicate ID's which should not be there because different people, the renumber tool can fix it, or he can do it manually. If it is the same ID because the record has been duplicated due to a bug, then the duplicate search finds it.

So, if you think your suggestion adds something, do the request, but from your mail I don't see what is the use case is.

Benny

2008/12/2 Ken B <[hidden email]>
Hi All,

    My friend has been discussing this with his class, and a student made the suggestion "Because the duplicates (events, sources, attributes etc) are also duplicte Gramps ID would it not make more sense to add a search for these to the 'Check and Repair Database' tool ?
    What do you all think ? Is it worth making this a feature request ?
Ken.

Benny Malengier wrote:
You should check the mail archives about UID and import/merge
Eg: http://www.nabble.com/External-Links-to-second-GRAMPS-Report-pages-td20304109.html

GRAMPS does _not_ support import/merge now, whatever happens is pure side effect of me and others trying to make it not too much broken, however there is no system, and the next release might do things differently.

Some more info here: http://gramps-project.org/wiki/index.php?title=GEPS_009:_Import_Export_Merge

So, some tasks to start:

1/add the UID creation tool based on Julio's patch
2/review the merge code as activated now on merging objects. This is the code that the merge on import must call too. Here Julio has also provided a patch for his case
3/stick to XML import for the merge scenario, and implement the merge correctly. My guess is it should become a two pass process:
  a/ go over the xml, see if UID's present, see if same UID is present in the existing database. This is quick for handle's, but a problem for double UID's in the attribute tag. So a new table needed? For non persons and family's this can be done based on handle, so it is fast. For your use case, import of xml, you can work also based on handle, so you can skip this
  b/ import the entire data, using different keys for the one's that must be merged
  c/after import, start the merging process based on the collected data, using same procedures as used by  the normal merge routine of the objects.

Above are just ideas for now, after research other solutions might be opted for. The idea to use only one set of merge routine should reduce code complexity. The fact that import stores without merge would be for the same reason.

Benny

2008/11/23 Ken B <[hidden email]>
Hi All,

    I have been using Gramps for a long time, and get very frustrated when I import data (in xml) from other people.

    Most items, people and events import and merge correctly. The main problem I have is with sources and media. These are duplicated, and take a lot of work to correct.

    I have a friend who understands programing and is willing to help me try to fix this. Is there anyone who can point me in the direction of the files to start with ? Or able to provide advice to someone willing to learn, if I want to try this myself ?

Thank you,
Ken.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with importing a backup into existing database

jerome
In reply to this post by Ken B.
Ken,

> "Because the duplicates (events, sources, attributes etc) are also duplicte Gramps ID would it not make more sense to add a search for these to the 'Check and Repair Database' tool ?

The tool "Reorder GRAMPS ID" (under Menu: Tools-->Database
Processing-->) already deals with duplicated ID, no ?
http://www.gramps-project.org/wiki/index.php?title=Gramps_3.0_Wiki_Manual_-_Tools#Reorder_GRAMPS_ID


Jérôme


Ken B a écrit :

>    Hi All,
>
>     My friend has been discussing this with his class, and a student
> made the suggestion "Because the duplicates (events, sources, attributes
> etc) are also duplicte Gramps ID would it not make more sense to add a
> search for these to the 'Check and Repair Database' tool ?
>     What do you all think ? Is it worth making this a feature request ?
> Ken.
>
> Benny Malengier wrote:
>> You should check the mail archives about UID and import/merge
>> Eg:
>> http://www.nabble.com/External-Links-to-second-GRAMPS-Report-pages-td20304109.html
>>
>> GRAMPS does _not_ support import/merge now, whatever happens is pure
>> side effect of me and others trying to make it not too much broken,
>> however there is no system, and the next release might do things
>> differently.
>>
>> Some more info here:
>> http://gramps-project.org/wiki/index.php?title=GEPS_009:_Import_Export_Merge
>>
>> So, some tasks to start:
>>
>> 1/add the UID creation tool based on Julio's patch
>> 2/review the merge code as activated now on merging objects. This is
>> the code that the merge on import must call too. Here Julio has also
>> provided a patch for his case
>> 3/stick to XML import for the merge scenario, and implement the merge
>> correctly. My guess is it should become a two pass process:
>>   a/ go over the xml, see if UID's present, see if same UID is present
>> in the existing database. This is quick for handle's, but a problem
>> for double UID's in the attribute tag. So a new table needed? For non
>> persons and family's this can be done based on handle, so it is fast.
>> For your use case, import of xml, you can work also based on handle,
>> so you can skip this
>>   b/ import the entire data, using different keys for the one's that
>> must be merged
>>   c/after import, start the merging process based on the collected
>> data, using same procedures as used by  the normal merge routine of
>> the objects.
>>
>> Above are just ideas for now, after research other solutions might be
>> opted for. The idea to use only one set of merge routine should reduce
>> code complexity. The fact that import stores without merge would be
>> for the same reason.
>>
>> Benny
>>
>> 2008/11/23 Ken B <[hidden email] <mailto:[hidden email]>>
>>
>>     Hi All,
>>
>>         I have been using Gramps for a long time, and get very
>>     frustrated when I import data (in xml) from other people.
>>
>>         Most items, people and events import and merge correctly. The
>>     main problem I have is with sources and media. These are
>>     duplicated, and take a lot of work to correct.
>>
>>         I have a friend who understands programing and is willing to
>>     help me try to fix this. Is there anyone who can point me in the
>>     direction of the files to start with ? Or able to provide advice
>>     to someone willing to learn, if I want to try this myself ?
>>
>>     Thank you,
>>     Ken.
>>
>>     -------------------------------------------------------------------------
>>     This SF.Net email is sponsored by the Moblin Your Move Developer's
>>     challenge
>>     Build the coolest Linux based applications with Moblin SDK & win
>>     great prizes
>>     Grand prize is a trip for two to an Open Source event anywhere in
>>     the world
>>     http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>     <http://moblin-contest.org/redirect.php?banner_id=100&url=/>
>>     _______________________________________________
>>     Gramps-devel mailing list
>>     [hidden email]
>>     <mailto:[hidden email]>
>>     https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>
>>
>
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Loading...