Testing Proposal

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Testing Proposal

prculley
Gentlemen;
I've been trying to submit patches to gramps to improve GEDCOM import.  Doug Blank has suggested that I smust also create automate-able tests for each patch, suitable for regression testing and to show the before/after effects of the patch.

I've looked over what testing tools we have now in https://gramps-project.org/wiki/index.php?title=Testing_Gramps, and what I could find in the code.
I could not find any obvious GEDCOM import tests, at least that tried to verify that the import actually works.  At least one test imports the sample.ged to use as source material for other tests.  But no obvious validation of the data.  It probably checks for issues that might crash the program to some degree.

I would like to propose an automated test method that would work fairly well for import tests, and may help in other circumstances.

1) start with the import files (ex. sample.ged for GEDCOM), additional files as bugs are reported.
2) during development or after bugs are patched, import the file, check that it worked correctly (manually) and create an "expected.gramps" xml output file corresponding to the import file.
3) Write a routine which compares the expected xml with the an xml generated during automated testing.
4) Write a routine which scans a directory of importable files, imports each in turn, compares an exported xml with the associated 'expected' xml in the same directory, and does the usual automated 'assert' on any errors.

Assumptions:
1) Gramps xml contains ALL useful data in a family tree. 
2) Gramps xml is sufficiently stable in layout so as to not require constant careful rebuilding of 'expected' files.
3) using import files for test cases is easier than writing unit tests for the many methods in the importers.  May not be as thorough, but it would take a long daunting time to do a reasonable job on a method by method basis.
4) most import bugs end up with sample import files in the issue tracker to aid in debugging, these can be easily modified or used as is as the test files.  This is certainly true in the bugs I have worked on already.

Notes:
* The importvcard_test.py by Michiel D. Nauta, contains one promising method of comparing xml files; I would try to adapt this code into a more general xml equivalence comparison routine.
* Changes to the structure of data, as represented in the xml files would require re-creation of all the 'expected' files. 

Please respond with any comments.  I may have missed some interesting test methods, or you may not like this approach for one reason or the other.

Paul Culley

P.S. I'm obviously still pretty new at all this so bear with me as I try to learn and contribute.  The more feedback I get the more I learn, hopefully increasing my value to the project.


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing Proposal

DS Blank
Devs,

On Sun, May 8, 2016 at 3:07 PM, Paul Culley <[hidden email]> wrote:
Gentlemen;
I've been trying to submit patches to gramps to improve GEDCOM import.  Doug Blank has suggested that I smust also create automate-able tests for each patch, suitable for regression testing and to show the before/after effects of the patch.

I wouldn't say that it is a requirement to have a test on all PRs of course, but it is a really good idea, especially on these type of PRs where it is almost impossible to tell by looking what is correct. Some comments below:
 

I've looked over what testing tools we have now in https://gramps-project.org/wiki/index.php?title=Testing_Gramps, and what I could find in the code.
I could not find any obvious GEDCOM import tests, at least that tried to verify that the import actually works.  At least one test imports the sample.ged to use as source material for other tests.  But no obvious validation of the data.  It probably checks for issues that might crash the program to some degree.

I would like to propose an automated test method that would work fairly well for import tests, and may help in other circumstances.

1) start with the import files (ex. sample.ged for GEDCOM), additional files as bugs are reported.
2) during development or after bugs are patched, import the file, check that it worked correctly (manually) and create an "expected.gramps" xml output file corresponding to the import file.
3) Write a routine which compares the expected xml with the an xml generated during automated testing.
4) Write a routine which scans a directory of importable files, imports each in turn, compares an exported xml with the associated 'expected' xml in the same directory, and does the usual automated 'assert' on any errors.

Assumptions:
1) Gramps xml contains ALL useful data in a family tree. 
2) Gramps xml is sufficiently stable in layout so as to not require constant careful rebuilding of 'expected' files.
3) using import files for test cases is easier than writing unit tests for the many methods in the importers.  May not be as thorough, but it would take a long daunting time to do a reasonable job on a method by method basis.
4) most import bugs end up with sample import files in the issue tracker to aid in debugging, these can be easily modified or used as is as the test files.  This is certainly true in the bugs I have worked on already.

Notes:
* The importvcard_test.py by Michiel D. Nauta, contains one promising method of comparing xml files; I would try to adapt this code into a more general xml equivalence comparison routine.
* Changes to the structure of data, as represented in the xml files would require re-creation of all the 'expected' files. 

Please respond with any comments.  I may have missed some interesting test methods, or you may not like this approach for one reason or the other.

Paul Culley

P.S. I'm obviously still pretty new at all this so bear with me as I try to learn and contribute.  The more feedback I get the more I learn, hopefully increasing my value to the project.

The master branch contains a renewed commitment to testing. As part of that commitment, every push and PR gets run against Travis CI, continuation integration testing system. You can see the exact commands that are run on Travis in .travis.yml. The last few lines show the command run via nose.

importvcard_test.py may not be the best example, as that is one file that is ignored by the test system, and hasn't been converted to work with it yet. That is a lot of code to write for testing, and we have perhaps better tools for that now.

It would be great to have a fine-grained test run after an entire import. That will exercise a lot of code all at once. We are only at 35.61% of our code being covered via Travis:


and even the current test code isn't really testing all that is run. But at least it makes sure that it is running with each change. This has caught many issues already. But we need to turn those into specific tests. To me, this is one of the single most import types of improvement that we can make to Gramps.

When we do have a difference between what went in and what came out with a holistic test after import, then we still have to hunt down what went wrong. For that reason, it is also good to have very specific tests on each method, as I tried to show here:


As to your proposal, in general, it is a good idea. Some notes: 

Our XML does change, and objects don't know how to represent (or compare themselves) in XML. However, each object can represent itself via a JSON-like representation, called generally a "struct". Every object has to_struct() and from_struct() methods. Those can be much more programmatically compared.

There is some code that might be useful:

gramps/plugins/test/test_reports.py - Runs the importer code, including a GEDCOM file. Currently, only tests to make sure that it succeeds by looking at stdout and stderr. It could also check all of the objects, as you suggest, such as:

gramps/gen/merge/diff.py - actually does a precise difference of two databases. See use of it here:


We might be able to turn that into the type of tests you suggest.

Hope this is useful! We very much appreciate all contributions, and hope to turn those PRs into ones that can easily be seen as correct. Thanks again, and welcome to Gramps development!

-Doug
 


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel



------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing Proposal

enno
Op 09-05-16 om 12:56 schreef Doug Blank:

> gramps/gen/merge/diff.py - actually does a precise difference of two
> databases. See use of it here:
>
> https://github.com/gramps-project/addons-source/blob/master/Differences/differences.py 
>
>
> We might be able to turn that into the type of tests you suggest.

I just ran a test importing the same GEDCOM into 2 empty trees, and
because diff.py is so precise, it reported that all elements in one were
missing in the other one, and vice versa. And the cause of that is what
you implied, but didn't say, and that's handles.

When I compare uncompressed .gramps backups with Diffuse, I can see that
the handles are different, but the IDs are the same. And since every top
level object has a handle and a unique ID, you can improve the diff by
translating every handle to the corresponding ID. That will require an
extra pass before running the actual diff, but it should work, because
the assignment of IDs during import is predictable.

Ignoring handles would be easier, but is not a good idea, because it
will ignore changes in joins. What needs to be ignored however is the
change tag. That's because some imported objects have change="0", while
others have a real time stamp, which will of course be different each time.

regards,

Enno


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing Proposal

DS Blank
On Mon, May 9, 2016 at 8:55 AM, Enno Borgsteede <[hidden email]> wrote:
Op 09-05-16 om 12:56 schreef Doug Blank:

> gramps/gen/merge/diff.py - actually does a precise difference of two
> databases. See use of it here:
>
> https://github.com/gramps-project/addons-source/blob/master/Differences/differences.py
>
>
> We might be able to turn that into the type of tests you suggest.

I just ran a test importing the same GEDCOM into 2 empty trees, and
because diff.py is so precise, it reported that all elements in one were
missing in the other one, and vice versa. And the cause of that is what
you implied, but didn't say, and that's handles.

When I compare uncompressed .gramps backups with Diffuse, I can see that
the handles are different, but the IDs are the same. And since every top
level object has a handle and a unique ID, you can improve the diff by
translating every handle to the corresponding ID. That will require an
extra pass before running the actual diff, but it should work, because
the assignment of IDs during import is predictable.

Very clever! That could be an option on the importing of any file for testing. It doesn't even have to be the gramps_id, but could be any predictable function (eg, a counter). But the gramps_id replacement gives additional information, and would allow comparisons with other imports (csv, for example) that might import data in a different order.
 

Ignoring handles would be easier, but is not a good idea, because it
will ignore changes in joins. What needs to be ignored however is the
change tag. That's because some imported objects have change="0", while
others have a real time stamp, which will of course be different each time.

But only ignore change when appropriate. There is a bug right now on that very issue [1] for sources, and only on BSDDB. Weird.

-Doug

 

regards,

Enno


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing Proposal

prculley
The xml comparison code in importvcard_test also dealt with changing handles and and the 'change' attrib; the former by converting handles to gramps_id, and the latter by ignoring them.  Since we are dealing with creation of database via import, I would think that the change tags should be quite consistent; I'll test this to see what I get.  Then it might be possible to equate a change time in 'expected' to a change time in 'tested'.  That is why I thought this code offered a starting place for comparisons.

Anyway, I'm not wedded to the idea of xml diffs, I just thought that it might be MORE stable (and complete) than some other form.  How would a database compare work?  Would we still use xml as the source of data for 'expected', import it then run the (modified for our use) diff.py? 

Regarding the unit tests vs. functional like this proposal, I agree that unit tests would be great.  But I dread the idea of programmatically setting up all the preconditions and dealing with call-outs to most of the methods present in the libgedcom.py module (for instance).  It seems to me that if we keep the functional tests small (lots of little imports, rather than one big one), and provide comments (perhaps in the imported notes) then debugging would not be too bad.

I appreciate the responses, and I'm more than willing to spend time on this, I'm retired, so I have the time but probably pretty slow compared to professionals, both because this is still new to me, and my memory for details like syntax isn't that great.  But I slog along.

Paul Culley

On Mon, May 9, 2016 at 8:26 AM, Doug Blank <[hidden email]> wrote:
On Mon, May 9, 2016 at 8:55 AM, Enno Borgsteede <[hidden email]> wrote:
Op 09-05-16 om 12:56 schreef Doug Blank:

> gramps/gen/merge/diff.py - actually does a precise difference of two
> databases. See use of it here:
>
> https://github.com/gramps-project/addons-source/blob/master/Differences/differences.py
>
>
> We might be able to turn that into the type of tests you suggest.

I just ran a test importing the same GEDCOM into 2 empty trees, and
because diff.py is so precise, it reported that all elements in one were
missing in the other one, and vice versa. And the cause of that is what
you implied, but didn't say, and that's handles.

When I compare uncompressed .gramps backups with Diffuse, I can see that
the handles are different, but the IDs are the same. And since every top
level object has a handle and a unique ID, you can improve the diff by
translating every handle to the corresponding ID. That will require an
extra pass before running the actual diff, but it should work, because
the assignment of IDs during import is predictable.

Very clever! That could be an option on the importing of any file for testing. It doesn't even have to be the gramps_id, but could be any predictable function (eg, a counter). But the gramps_id replacement gives additional information, and would allow comparisons with other imports (csv, for example) that might import data in a different order.
 

Ignoring handles would be easier, but is not a good idea, because it
will ignore changes in joins. What needs to be ignored however is the
change tag. That's because some imported objects have change="0", while
others have a real time stamp, which will of course be different each time.

But only ignore change when appropriate. There is a bug right now on that very issue [1] for sources, and only on BSDDB. Weird.

-Doug

 

regards,

Enno


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel



------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing Proposal

enno
Op 09-05-16 om 16:16 schreef Paul Culley:
> The xml comparison code in importvcard_test also dealt with changing
> handles and and the 'change' attrib; the former by converting handles
> to gramps_id, and the latter by ignoring them.  Since we are dealing
> with creation of database via import, I would think that the change
> tags should be quite consistent; I'll test this to see what I get.  
> Then it might be possible to equate a change time in 'expected' to a
> change time in 'tested'.  That is why I thought this code offered a
> starting place for comparisons.

Good thinking. I knew that my idea could hardly be original. :-)

> Anyway, I'm not wedded to the idea of xml diffs, I just thought that
> it might be MORE stable (and complete) than some other form.  How
> would a database compare work?  Would we still use xml as the source
> of data for 'expected', import it then run the (modified for our use)
> diff.py?

In a database, no matter whether it's BSDDB or DB-API, most of the
handles that reference other objects sit in blobs, so there's no way to
replace those with IDs. In an empty database, handles imported from XML
are preserved, but since new ones are generated during GEDCOM import,
database compare will fail anyway.

Comparing XMLs assumes that XML export is complete, and testing that
assumption is another thing. One could take a database created from a
GEDCOM, or one with some history of editing, export that, and import
that in a new database, and then compare all columns in that. There
should be no difference on those, if handles and time stamps are
preserved, and if XML export is not complete, it may show, depending on
the original contents. This is not easy to test, because you can't seed
the database with an XML import, because if that comes from an export
that was incomplete, there will still be no diff.

regards,

Enno


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing Proposal

prculley
In reply to this post by prculley
I wonder if the database comparison issue could be made easier if we made the handle creation deterministic, for testing purposes only.

I see that currently (according to wiki), handles are created from a timestamp, and two random numbers.  I haven't found it in the code yet, but if we added a command line option to go into a testing mode, which forced handle creation into a fixed seeded pseudo random number with no timestamp, and possibly also forced the 'change' clock to zero or some fixed number, the diff function would probably work as it is coded now.
Food for thought.

Paul Culley

On Mon, May 9, 2016 at 9:16 AM, Paul Culley <[hidden email]> wrote:
The xml comparison code in importvcard_test also dealt with changing handles and and the 'change' attrib; the former by converting handles to gramps_id, and the latter by ignoring them.  Since we are dealing with creation of database via import, I would think that the change tags should be quite consistent; I'll test this to see what I get.  Then it might be possible to equate a change time in 'expected' to a change time in 'tested'.  That is why I thought this code offered a starting place for comparisons.

Anyway, I'm not wedded to the idea of xml diffs, I just thought that it might be MORE stable (and complete) than some other form.  How would a database compare work?  Would we still use xml as the source of data for 'expected', import it then run the (modified for our use) diff.py? 

Regarding the unit tests vs. functional like this proposal, I agree that unit tests would be great.  But I dread the idea of programmatically setting up all the preconditions and dealing with call-outs to most of the methods present in the libgedcom.py module (for instance).  It seems to me that if we keep the functional tests small (lots of little imports, rather than one big one), and provide comments (perhaps in the imported notes) then debugging would not be too bad.

I appreciate the responses, and I'm more than willing to spend time on this, I'm retired, so I have the time but probably pretty slow compared to professionals, both because this is still new to me, and my memory for details like syntax isn't that great.  But I slog along.

Paul Culley

On Mon, May 9, 2016 at 8:26 AM, Doug Blank <[hidden email]> wrote:
On Mon, May 9, 2016 at 8:55 AM, Enno Borgsteede <[hidden email]> wrote:
Op 09-05-16 om 12:56 schreef Doug Blank:

> gramps/gen/merge/diff.py - actually does a precise difference of two
> databases. See use of it here:
>
> https://github.com/gramps-project/addons-source/blob/master/Differences/differences.py
>
>
> We might be able to turn that into the type of tests you suggest.

I just ran a test importing the same GEDCOM into 2 empty trees, and
because diff.py is so precise, it reported that all elements in one were
missing in the other one, and vice versa. And the cause of that is what
you implied, but didn't say, and that's handles.

When I compare uncompressed .gramps backups with Diffuse, I can see that
the handles are different, but the IDs are the same. And since every top
level object has a handle and a unique ID, you can improve the diff by
translating every handle to the corresponding ID. That will require an
extra pass before running the actual diff, but it should work, because
the assignment of IDs during import is predictable.

Very clever! That could be an option on the importing of any file for testing. It doesn't even have to be the gramps_id, but could be any predictable function (eg, a counter). But the gramps_id replacement gives additional information, and would allow comparisons with other imports (csv, for example) that might import data in a different order.
 

Ignoring handles would be easier, but is not a good idea, because it
will ignore changes in joins. What needs to be ignored however is the
change tag. That's because some imported objects have change="0", while
others have a real time stamp, which will of course be different each time.

But only ignore change when appropriate. There is a bug right now on that very issue [1] for sources, and only on BSDDB. Weird.

-Doug

 

regards,

Enno


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel




------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing Proposal

DS Blank
On Mon, May 9, 2016 at 12:11 PM, Paul Culley <[hidden email]> wrote:
I wonder if the database comparison issue could be made easier if we made the handle creation deterministic, for testing purposes only.

I see that currently (according to wiki), handles are created from a timestamp, and two random numbers.  I haven't found it in the code yet, but if we added a command line option to go into a testing mode, which forced handle creation into a fixed seeded pseudo random number with no timestamp, and possibly also forced the 'change' clock to zero or some fixed number, the diff function would probably work as it is coded now.
Food for thought.

That is what I meant:

> Very clever! That could be an option on the importing of any file for testing. It doesn't even have 
> to be the gramps_id, but could be any predictable function (eg, a counter). But the gramps_id 
> replacement gives additional information, and would allow comparisons with other imports 
> (csv, for example) that might import data in a different order.

The problem is when it could be imported in a different order, and thus change the deterministic order. In that situation, the gramps_id makes even more sense.

-Doug
 


Paul Culley

On Mon, May 9, 2016 at 9:16 AM, Paul Culley <[hidden email]> wrote:
The xml comparison code in importvcard_test also dealt with changing handles and and the 'change' attrib; the former by converting handles to gramps_id, and the latter by ignoring them.  Since we are dealing with creation of database via import, I would think that the change tags should be quite consistent; I'll test this to see what I get.  Then it might be possible to equate a change time in 'expected' to a change time in 'tested'.  That is why I thought this code offered a starting place for comparisons.

Anyway, I'm not wedded to the idea of xml diffs, I just thought that it might be MORE stable (and complete) than some other form.  How would a database compare work?  Would we still use xml as the source of data for 'expected', import it then run the (modified for our use) diff.py? 

Regarding the unit tests vs. functional like this proposal, I agree that unit tests would be great.  But I dread the idea of programmatically setting up all the preconditions and dealing with call-outs to most of the methods present in the libgedcom.py module (for instance).  It seems to me that if we keep the functional tests small (lots of little imports, rather than one big one), and provide comments (perhaps in the imported notes) then debugging would not be too bad.

I appreciate the responses, and I'm more than willing to spend time on this, I'm retired, so I have the time but probably pretty slow compared to professionals, both because this is still new to me, and my memory for details like syntax isn't that great.  But I slog along.

Paul Culley

On Mon, May 9, 2016 at 8:26 AM, Doug Blank <[hidden email]> wrote:
On Mon, May 9, 2016 at 8:55 AM, Enno Borgsteede <[hidden email]> wrote:
Op 09-05-16 om 12:56 schreef Doug Blank:

> gramps/gen/merge/diff.py - actually does a precise difference of two
> databases. See use of it here:
>
> https://github.com/gramps-project/addons-source/blob/master/Differences/differences.py
>
>
> We might be able to turn that into the type of tests you suggest.

I just ran a test importing the same GEDCOM into 2 empty trees, and
because diff.py is so precise, it reported that all elements in one were
missing in the other one, and vice versa. And the cause of that is what
you implied, but didn't say, and that's handles.

When I compare uncompressed .gramps backups with Diffuse, I can see that
the handles are different, but the IDs are the same. And since every top
level object has a handle and a unique ID, you can improve the diff by
translating every handle to the corresponding ID. That will require an
extra pass before running the actual diff, but it should work, because
the assignment of IDs during import is predictable.

Very clever! That could be an option on the importing of any file for testing. It doesn't even have to be the gramps_id, but could be any predictable function (eg, a counter). But the gramps_id replacement gives additional information, and would allow comparisons with other imports (csv, for example) that might import data in a different order.
 

Ignoring handles would be easier, but is not a good idea, because it
will ignore changes in joins. What needs to be ignored however is the
change tag. That's because some imported objects have change="0", while
others have a real time stamp, which will of course be different each time.

But only ignore change when appropriate. There is a bug right now on that very issue [1] for sources, and only on BSDDB. Weird.

-Doug

 

regards,

Enno


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel




------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel



------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing Proposal

prculley
So I made a quick test of Doug's differences plug-in, with the create_id function (used for handles etc.) simplified and deterministic.
It was able to compare the same data base imported twice and find them equal.  There was a bug in the xml exporter that messed me up (bug report 9445), but once I got passed that the comparison between databases was successful.
The differences plugin uses a gramps function diff_dbs, already existing in gen\merge\diff.py.  That code already ignores 'changed' times, so that is not an issue.

So I can create a comparison tool most easily if we decide to support the deterministic create_id CLI option.  Or I can work harder and figure out a way to deal with handles.

Thoughts anyone?

Paul Culley

On Mon, May 9, 2016 at 11:44 AM, Doug Blank <[hidden email]> wrote:
On Mon, May 9, 2016 at 12:11 PM, Paul Culley <[hidden email]> wrote:
I wonder if the database comparison issue could be made easier if we made the handle creation deterministic, for testing purposes only.

I see that currently (according to wiki), handles are created from a timestamp, and two random numbers.  I haven't found it in the code yet, but if we added a command line option to go into a testing mode, which forced handle creation into a fixed seeded pseudo random number with no timestamp, and possibly also forced the 'change' clock to zero or some fixed number, the diff function would probably work as it is coded now.
Food for thought.

That is what I meant:

> Very clever! That could be an option on the importing of any file for testing. It doesn't even have 
> to be the gramps_id, but could be any predictable function (eg, a counter). But the gramps_id 
> replacement gives additional information, and would allow comparisons with other imports 
> (csv, for example) that might import data in a different order.

The problem is when it could be imported in a different order, and thus change the deterministic order. In that situation, the gramps_id makes even more sense.

-Doug
 


Paul Culley

On Mon, May 9, 2016 at 9:16 AM, Paul Culley <[hidden email]> wrote:
The xml comparison code in importvcard_test also dealt with changing handles and and the 'change' attrib; the former by converting handles to gramps_id, and the latter by ignoring them.  Since we are dealing with creation of database via import, I would think that the change tags should be quite consistent; I'll test this to see what I get.  Then it might be possible to equate a change time in 'expected' to a change time in 'tested'.  That is why I thought this code offered a starting place for comparisons.

Anyway, I'm not wedded to the idea of xml diffs, I just thought that it might be MORE stable (and complete) than some other form.  How would a database compare work?  Would we still use xml as the source of data for 'expected', import it then run the (modified for our use) diff.py? 

Regarding the unit tests vs. functional like this proposal, I agree that unit tests would be great.  But I dread the idea of programmatically setting up all the preconditions and dealing with call-outs to most of the methods present in the libgedcom.py module (for instance).  It seems to me that if we keep the functional tests small (lots of little imports, rather than one big one), and provide comments (perhaps in the imported notes) then debugging would not be too bad.

I appreciate the responses, and I'm more than willing to spend time on this, I'm retired, so I have the time but probably pretty slow compared to professionals, both because this is still new to me, and my memory for details like syntax isn't that great.  But I slog along.

Paul Culley

On Mon, May 9, 2016 at 8:26 AM, Doug Blank <[hidden email]> wrote:
On Mon, May 9, 2016 at 8:55 AM, Enno Borgsteede <[hidden email]> wrote:
Op 09-05-16 om 12:56 schreef Doug Blank:

> gramps/gen/merge/diff.py - actually does a precise difference of two
> databases. See use of it here:
>
> https://github.com/gramps-project/addons-source/blob/master/Differences/differences.py
>
>
> We might be able to turn that into the type of tests you suggest.

I just ran a test importing the same GEDCOM into 2 empty trees, and
because diff.py is so precise, it reported that all elements in one were
missing in the other one, and vice versa. And the cause of that is what
you implied, but didn't say, and that's handles.

When I compare uncompressed .gramps backups with Diffuse, I can see that
the handles are different, but the IDs are the same. And since every top
level object has a handle and a unique ID, you can improve the diff by
translating every handle to the corresponding ID. That will require an
extra pass before running the actual diff, but it should work, because
the assignment of IDs during import is predictable.

Very clever! That could be an option on the importing of any file for testing. It doesn't even have to be the gramps_id, but could be any predictable function (eg, a counter). But the gramps_id replacement gives additional information, and would allow comparisons with other imports (csv, for example) that might import data in a different order.
 

Ignoring handles would be easier, but is not a good idea, because it
will ignore changes in joins. What needs to be ignored however is the
change tag. That's because some imported objects have change="0", while
others have a real time stamp, which will of course be different each time.

But only ignore change when appropriate. There is a bug right now on that very issue [1] for sources, and only on BSDDB. Weird.

-Doug

 

regards,

Enno


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel


------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel




------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel




------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel