Fwd: unittests

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: unittests

Paul Franklin-5
I am transferring this to the list, as I want to discuss
unittests and example.gramps publically.

---------- Forwarded message ----------
From: Tim Lyons <[hidden email]>
Date: Tue, 5 Jul 2016 13:44:28 +0100
Subject: unittests
To: Paul Franklin <[hidden email]>

Unfortunately, the gramps/gen/filters/rules/test tests all depend on
the exact contents of example.gramps. so commit

https://github.com/gramps-project/gramps/commit/a60ae79ca06daba8da17d5177e806e70c9777d4d

causes travis to fail:

https://travis-ci.org/gramps-project/gramps/builds/141945865

I suggest that for the time being, all the gramps/gen/filters/rules/
test tests should be disabled, just so that the unittests gets back on
track, then in slower time, the filter tests might be
modified...somehow.

Sorry, but I don't know how to do that.

regards,
Tim.

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: unittests

Paul Franklin-5
On 7/5/16, Paul Franklin <[hidden email]> wrote:

> I am transferring this to the list, as I want to discuss
> unittests and example.gramps publically.
>
> ---------- Forwarded message ----------
> From: Tim Lyons <[hidden email]>
> Date: Tue, 5 Jul 2016 13:44:28 +0100
> Subject: unittests
> To: Paul Franklin <[hidden email]>
>
> Unfortunately, the gramps/gen/filters/rules/test tests all depend on
> the exact contents of example.gramps. so commit
>
> https://github.com/gramps-project/gramps/commit/a60ae79ca06daba8da17d5177e806e70c9777d4d
>
> causes travis to fail:
>
> https://travis-ci.org/gramps-project/gramps/builds/141945865
>
> I suggest that for the time being, all the gramps/gen/filters/rules/
> test tests should be disabled, just so that the unittests gets back on
> track, then in slower time, the filter tests might be
> modified...somehow.
>
> Sorry, but I don't know how to do that.
>
> regards,
> Tim.
>

I am philosophically opposed to unittests using example.gramps.

I think if any unittest needs a family tree, the unittest should
generate it itself, the way the vcard tests MD Nauta wrote do.

However I understand that is more work and that while some
people are enthusiasts about unit testing, they don't want to
do things that way.  A "quick and dirty" approach is being taken,
running a report rather than testing individual things, using
the (gigantic) example.gramps file when they need test cases,
rather than a one-person test-family tree, or whatever.

Instead of using example.gramps (or data.gramps, which Nick
directed (to the list a few weeks ago) should be the only files
in the "example/gramps" folder), any unittests which need a
family tree should be rewritten to use files in the data/tests
directory which Paul Culley set up, if they are not going to be
rewritten to generate their own tree.

This is what I thought an earlier discussion (on the list) said.

Any family tree files (.gramps, .ged, .csv, and so on) in the
data/tests directory are clearly meant for testing.  I assume
that nobody will touch them without also touching their unit
test.

Unfortunately, I have never written a unit test in my life.  I
therefore leave the changing -- or disabling -- of any gramps
unit tests to somebody who understands them.

I already said, in that earlier thread, that I would leave the
moving of such files as datetest.gramps to Paul, after he'd
said that he will need to modify something in his tests to
cope with that move.

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: unittests

Paul Franklin-5
On 7/5/16, Paul Franklin <[hidden email]> wrote:

> I am philosophically opposed to unittests using example.gramps.

> Instead of using example.gramps (or data.gramps, which Nick
> directed (to the list a few weeks ago) should be the only files
> in the "example/gramps" folder), any unittests which need a
> family tree should be rewritten to use files in the data/tests
> directory which Paul Culley set up, if they are not going to be
> rewritten to generate their own tree.
>
> This is what I thought an earlier discussion (on the list) said.

If the information in example/gramps/example.gramps (or the
similar example/gramps/data.gramps) is needed for unittests,
then the information should somehow be put/moved/copied to
the data/tests directory.

If files are chosen to be copied, for expediency, then I suggest
any in data/tests should have different names than the two
classic example files (example.gramps and data.gramps),
to try to minimize confusion in the future.  Remember, we are
all volunteers; we come, we go; there will be new ones just
as there were old ones.  Anything done should be as intuitive
as possible, as self-documenting as possible.  IMHO.

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: unittests

Paul Franklin-5
Speaking of unittests, I would like to see a discussion
and then a policy decision as to where and how unittests
will be or can be or should be run.

For instance, I propose that all unittests should be runnable
via the "setup.py test" command.  That is not the case now.

Specifically, I propose that no unittests should be accessible
or able to be run by any user.  No unit test code should be
included in any gramps-generated tar or zip file for instance.
If this cannot be done on github then I propose we never put
any zip file on github and only have tar files on SourceForge,
which we control.

As a corollary, no unit tests or test-data files should ever be
included in any gramps-prepared package, such as the AIO
or the Debian or Mac packages we make.  I specifically
mean the data/tests directory, besides all the *.py files.

As a corollary to that, all mentions of test-related software
packages should either be removed from the README or
be written to explicitly state that they are only for gramps
development and are highly discouraged from inclusion
in any user-level package.

I would include in that statement such development-related
software as "Meta" -- which was recently put into the README.
I don't think anything related to future development should
be in any user-level package.  Any user who is sophisticated
enough to do development will know where to get things.

Furthermore, I propose that all unittests be runnable by any
developer, easily.  That means that no unittests should be
written to require unix-only software, since we have Windows
and Mac developers also.  Don't take a parochial view.

How can "nosetests" things be run on Windows for instance?
If it's not possible it shouldn't be in our test suite.

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: unittests

prculley
In reply to this post by Paul Franklin-5
Just a note for this email history.  I updated the gramps/gen/filters/rules/test tests so they would work with the current, updated example/gramps/example.gramps file.  I have not yet submitted a PR on copying this to the data/test directory.

Paul Culley

On Tue, Jul 5, 2016 at 11:51 AM, Paul Franklin <[hidden email]> wrote:
I am transferring this to the list, as I want to discuss
unittests and example.gramps publically.

---------- Forwarded message ----------
From: Tim Lyons <[hidden email]>
Date: Tue, 5 Jul 2016 13:44:28 +0100
Subject: unittests
To: Paul Franklin <[hidden email]>

Unfortunately, the gramps/gen/filters/rules/test tests all depend on
the exact contents of example.gramps. so commit

https://github.com/gramps-project/gramps/commit/a60ae79ca06daba8da17d5177e806e70c9777d4d

causes travis to fail:

https://travis-ci.org/gramps-project/gramps/builds/141945865

I suggest that for the time being, all the gramps/gen/filters/rules/
test tests should be disabled, just so that the unittests gets back on
track, then in slower time, the filter tests might be
modified...somehow.

Sorry, but I don't know how to do that.

regards,
Tim.

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel


------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: unittests

Ross Gammon
In reply to this post by Paul Franklin-5
Hi Paul,

On 07/05/2016 11:42 PM, Paul Franklin wrote:
> Specifically, I propose that no unittests should be accessible
> or able to be run by any user.  No unit test code should be
> included in any gramps-generated tar or zip file for instance.
> If this cannot be done on github then I propose we never put
> any zip file on github and only have tar files on SourceForge,
> which we control.

Please keep the unit test code in the tarball. In Debian we run the
testsuites during the build process, and periodically over time. This
helps to make sure no incompatibilities with the underlying libraries
(version of python etc.) go without being detected, that might cause
bugs for Debian users.

I don't see any harm in having the tests installed with the other code.
But if larger datafiles needed for the test are in a special directory,
these could also be excluded from "setup.py install" for those concerned
about the size of the installation.

Cheers,

Ross


------------------------------------------------------------------------------

_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: unittests

Paul Franklin-5
On 7/30/16, Ross Gammon <[hidden email]> wrote:

> On 07/05/2016 11:42 PM, Paul Franklin wrote:
>> Specifically, I propose that no unittests should be accessible
>> or able to be run by any user.  No unit test code should be
>> included in any gramps-generated tar or zip file for instance.
>> If this cannot be done on github then I propose we never put
>> any zip file on github and only have tar files on SourceForge,
>> which we control.
>
> Please keep the unit test code in the tarball. In Debian we run the
> testsuites during the build process, and periodically over time. This
> helps to make sure no incompatibilities with the underlying libraries
> (version of python etc.) go without being detected, that might cause
> bugs for Debian users.

Are you telling me that every single unittest works for you,
every last one of them, with no patches or changes?

------------------------------------------------------------------------------
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: unittests

Ross Gammon
On 07/30/2016 06:06 PM, Paul Franklin wrote:

> On 7/30/16, Ross Gammon <[hidden email]> wrote:
>> On 07/05/2016 11:42 PM, Paul Franklin wrote:
>>> Specifically, I propose that no unittests should be accessible
>>> or able to be run by any user.  No unit test code should be
>>> included in any gramps-generated tar or zip file for instance.
>>> If this cannot be done on github then I propose we never put
>>> any zip file on github and only have tar files on SourceForge,
>>> which we control.
>>
>> Please keep the unit test code in the tarball. In Debian we run the
>> testsuites during the build process, and periodically over time. This
>> helps to make sure no incompatibilities with the underlying libraries
>> (version of python etc.) go without being detected, that might cause
>> bugs for Debian users.
>
> Are you telling me that every single unittest works for you,
> every last one of them, with no patches or changes?
>
No. Some of the unit tests have never passed (since I enabled the
testsuite), and I exclude them.

Currently (4.2.3):
override_dh_auto_test:
        HOME=$(CURDIR)/build \
        DJANGO_SETTINGS_MODULE=gramps.webapp.settings nosetests3 -vv \
        --exclude=TestcaseGenerator \
        --exclude=vcard \
        --exclude=merge_ref_test \
        --exclude=test3b_delete_tree_constraint \
        --exclude=test_consistent_with_DISPLAY_env \
        --exclude=test4_arbitrary_uncode_path \
        --exclude-dir=gramps/webapp

I remember this list was pretty close to the exclusion list on master in
travis.yml
(https://github.com/gramps-project/gramps/blob/master/.travis.yml)
around the time 4.x was released. I can see that the exclusion list on
master is much smaller now, so more tests must be passing. I look
forward to enabling more tests when 5.0 is released!

Note: I can see that the builds are not reproducible on i386 & armhf
architectures. I need to look into that:
https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/gramps.html


------------------------------------------------------------------------------

_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (836 bytes) Download Attachment