GEDCOM support enhancements

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

GEDCOM support enhancements

prculley
Gentlemen:
As a few of you know, I have been addressing GEDCOM related bugs from the bug tracker for a while now, and I have dealt with nearly all of them that I am aware of.
However, there are still some things I would like to address.  They mostly fall into the category of 'enhancements' rather than bugs although some involve better handling of valid GEDCOM spec constructs.

My question is, do I continue to propose PRs for 'enhancements' within our standard GEDCOM import/export modules, or do I create or use some kind of plugin that does everything standard GEDCOM importer does but adds support for non-standard GEDCOM from various programs?

I note that even our standard importer has support for some non-standard constructs, so there is some precedent for them.

My goals in this effort would be to;
  • Maintain compatibility with GEDCOM spec version 5.5.1 and where feasible, 5.5.
  • Not break anything...
  • Reduce data loss for users when importing/exporting GEDCOM.  Not to imply that data would always appear where expected, but it would not be lost.  So some items might end up in notes, or events, where they existed somewhere else in the originating program.
  • Support non-standard GEDCOM constructs from some of the major programs as identified by requests in bug tracker or elsewhere.
  • Improve our own Gramps GEDCOM export and import back, so that data goes back where it came from where feasible.
  • Submit code changes in relatively small bites, so that testing will be easier.  Downside is more PRs to evaluate by other developers.

P.S. I am aware of the GedcomExtensions Addon, that seems to do some tweaks to the GEDCOM exporter.  It appears to use Class inheritance tricks to change a very few of the exporter methods.  I don't think this would be as easy as that for the importer; but I could be wrong.  In any event it would be more difficult than just fixing the standard code...

P.P.S.  I expect that I will get a request for more/better test code if I do this; I'm already thinking about a differences based testing method for exporters...  It looks like my importer testing code may be accepted soon...

Paul Culley


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: GEDCOM support enhancements

Eric Doutreleau

Hi Paul

I have written a "non officieal" plugin based on GedcomExtensions for exporting gedcom specific to the geneanet site.

I guess the problems for non standard gedcom is that often we have to make a choice.

for exemple for geneanet for an even there s only two role

principal

witness

then to not lose other participant to an event i have to convert it to witness.

i can imagine it wouldn't be the right choice for other programs that have other roles defined for events.


Le 01/06/2016 à 18:47, Paul Culley a écrit :
Gentlemen:
As a few of you know, I have been addressing GEDCOM related bugs from the bug tracker for a while now, and I have dealt with nearly all of them that I am aware of.
However, there are still some things I would like to address.  They mostly fall into the category of 'enhancements' rather than bugs although some involve better handling of valid GEDCOM spec constructs.

My question is, do I continue to propose PRs for 'enhancements' within our standard GEDCOM import/export modules, or do I create or use some kind of plugin that does everything standard GEDCOM importer does but adds support for non-standard GEDCOM from various programs?

I note that even our standard importer has support for some non-standard constructs, so there is some precedent for them.

My goals in this effort would be to;
  • Maintain compatibility with GEDCOM spec version 5.5.1 and where feasible, 5.5.
  • Not break anything...
  • Reduce data loss for users when importing/exporting GEDCOM.  Not to imply that data would always appear where expected, but it would not be lost.  So some items might end up in notes, or events, where they existed somewhere else in the originating program.
  • Support non-standard GEDCOM constructs from some of the major programs as identified by requests in bug tracker or elsewhere.
  • Improve our own Gramps GEDCOM export and import back, so that data goes back where it came from where feasible.
  • Submit code changes in relatively small bites, so that testing will be easier.  Downside is more PRs to evaluate by other developers.

P.S. I am aware of the GedcomExtensions Addon, that seems to do some tweaks to the GEDCOM exporter.  It appears to use Class inheritance tricks to change a very few of the exporter methods.  I don't think this would be as easy as that for the importer; but I could be wrong.  In any event it would be more difficult than just fixing the standard code...

P.P.S.  I expect that I will get a request for more/better test code if I do this; I'm already thinking about a differences based testing method for exporters...  It looks like my importer testing code may be accepted soon...

Paul Culley



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e


_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: GEDCOM support enhancements

Tim Lyons
Administrator
In reply to this post by prculley
prculley wrote
Gentlemen:
As a few of you know, I have been addressing GEDCOM related bugs from the
bug tracker for a while now, and I have dealt with nearly all of them that
I am aware of.
However, there are still some things I would like to address.  They mostly
fall into the category of 'enhancements' rather than bugs although some
involve better handling of valid GEDCOM spec constructs.

My question is, do I continue to propose PRs for 'enhancements' within our
standard GEDCOM import/export modules, or do I create or use some kind of
plugin that does everything standard GEDCOM importer does but adds support
for non-standard GEDCOM from various programs?

I note that even our standard importer has support for *some* non-standard
constructs, so there is some precedent for them.

My goals in this effort would be to;

   - Maintain compatibility with GEDCOM spec version 5.5.1 and where
   feasible, 5.5.
   - Not break anything...
   - Reduce data loss for users when importing/exporting GEDCOM.  Not to
   imply that data would always appear where expected, but it would not be
   lost.  So some items might end up in notes, or events, where they existed
   somewhere else in the originating program.
   - Support non-standard GEDCOM constructs from some of the major programs
   as identified by requests in bug tracker or elsewhere.
   - Improve our own Gramps GEDCOM export and import back, so that data
   goes back where it came from where feasible.
   - Submit code changes in relatively small bites, so that testing will be
   easier.  Downside is more PRs to evaluate by other developers.
Sounds excellent. I have been trying to do just that for GEDCOM import (and export) for some time (but haven't been so active just recently).

I don't think better handling of valid GEDCOM constructs is an enhancement, rather a bug fix.

People who try to import GEDCOMs produced by other programs (even if those programs use GEDCOM extensions) expect the GEDCOM to be imported correctly, so they would probably regard not supporting their GEDCOM as a bug as well

As far as whether to support the non-standard GEDCOM from various programs in core Gramps, or extensions:

(1) I think that there was some discussion about GEDCOM extensions (for both import and export) some time ago, and it was decided that it was rather difficult to support and maintain the various program specific extensions in the core Gramps code, and as a result, Doug was kind enough to implement the plug-in architecture for Gramps export.

Unfortunately, he did not at the time implement the same thing for import. I think doing so would be an excellent idea. I have asked him if he could do so (http://gramps.1791082.n4.nabble.com/Geneanet-Gramps-importing-woes-tp4673883p4673965.html), but unfortunately he has been busy with the database change, so not able to do so yet. Doug, any chance you could find the time to do so now?

(2) It would be nice if the import extensions could be done as a plug-in, but unless and until that architecture is implemented, the best thing is to enable the extensions in the core Gramps code subject to a test as to the source of the import (as is done in some cases at present). Besides not confusing the existing code, this should make it easier to identify the extensions in the future so they can be removed to a plug-in.

(3) The only downside to a plug-in is that the file to be imported would need to have a different file name extension so that it can be recognized as needing a different import plug-in. This would make things a bit difficult for Aunt Martha, but is probably still a better solution. At least the user could be warned when the source of the GEDCOM is parsed that there may be an addon that does a better job.

(Or maybe, the first import pass could detect the source of the GEDCOM, and invoke a specific addon if there was one.


prculley wrote
P.S. I am aware of the GedcomExtensions Addon, that seems to do some tweaks
to the GEDCOM exporter.  It appears to use Class inheritance tricks to
change a very few of the exporter methods.  I don't think this would be as
easy as that for the importer; but I could be wrong.  In any event it would
be more difficult than just fixing the standard code...

P.P.S.  I expect that I will get a request for more/better test code if I
do this; I'm already thinking about a differences based testing method for
exporters...  It looks like my importer testing code may be accepted soon...

Paul Culley
I don't think changing the architecture for addons/plugins for import would be any harder than for exports. I don't know how to do it.

Actually writing the code for the addon is very simple, I wrote some changes for my own purposes (for export to be readable by an iPad app) and it was very simple and neat.


(In the above, addon and plug-in mean exactly teh same thing)
Reply | Threaded
Open this post in threaded view
|

Re: GEDCOM support enhancements

Paul Franklin-5
On 6/4/16, Tim Lyons <[hidden email]> wrote:

> I don't think better handling of valid GEDCOM constructs is an enhancement,
> rather a bug fix.

I agree.  That statement seems incontrovertible to me.

> People who try to import GEDCOMs produced by other programs (even if those
> programs use GEDCOM extensions) expect the GEDCOM to be imported correctly,
> so they would probably regard not supporting their GEDCOM as a bug as well

I think it is important that we keep such perspective
in mind.  We are producing a genealogy program and
as everybody knows GEDCOM is the interchange
standard -- whether we like it or not, whether it's ancient
or poorly constructed or anything else.  It's what our
users use (and some of us although not me), and we
should support it and continue to support it.  Which
includes fixing any problems we have with it, or coping
at least.

> As far as whether to support the non-standard GEDCOM from various programs
> in core Gramps, or extensions:

> (2) It would be nice if the import extensions could be done as a plug-in,
> but unless and until that architecture is implemented, the best thing is to
> enable the extensions in the core Gramps code ...

I agree with that also.  I think it should continue to be
done in the core gramps code, as such non-standard
extensions have been for some time now.  We need to
support such non-standard GEDCOM imports somehow
and that is the only place which exists right now.

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel