Quantcast

unit testing -- review and recommendations

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

unit testing -- review and recommendations

jgsack
Summary (but please read additional text beyond the summary)
*******

==> Point (1) below requires consensus and a decision. I believe I could
manage to do and/or lead the effort[*] on the remaining points. The
conversion would take a few days; the documentation probably longer.

  [*] except for attending to build changes that may be required.


1) We should create a subdir in each source directory to hold unit
tests. The tests would look in their immediate parent directory for
their target module. Test modules should follow the existing practice of
appending a "-_Test" suffix to the name of the module under test.

Example:
  .../src/GrampsLogger/t
  .../src/GrampsLogger/t/_RotateHandler_Test.py

The subdir might be ".../tests", or ".../test" if ".../t" is thought too
cryptic. (TBD)

NB: The distribution build process will need to be examined, and perhaps
adjusted to skip the test subdirs and contents.

2) Move the few tests we have into their respective new ".../t"
locations, fix their names as necessary, and update per current codebase
so that they run and succeed.

3) Revise the existing regression test driver RunAllTests.py to find
tests in the source tree rather than the parallel directory (as now).
Extend or improve options as required, and document usage.

4) Create wiki page for developer guidelines, tips, etc

5) Start writing unit test modules and developing regression testing
practices.


Background and Further Explanation
**********************************

Gramps presently has 11 unit test modules (in 3 subdirs under .../test).
The unit tests are run as a regression test batch via the script
.../test/RunAllTests.py.

Individual tests can be run from the command line in their respective
subdirectories.

There is also one regular module (under .../src) containing an embedded
unit test, that is, a unittest-based test -- run when executed as a
program (via if __name__ == "__main__"). This should be extracted to an
external test module.

   In addition to the tests based on the python unittest module, there
 are several other test scripts in the .../test directory. At least one
 of these (dates.sh) uses supporting test code in .../src.
   While these may be very useful, they deviate from the unit test
 philosophy of clearly indicating a success or fail result such as is
 desirable for automated testing. For now, I recommend lumping them
 into a "miscellanea" category. They can remain in the .../test dir.

We have about 600 total python code modules, so the coverage isn't very
significant. In addition, some of those 11 existing test files seem to
have gotten a bit stale.

Writing standalone unit tests is fairly easy, but making a serious
regression test _framework_ requires a bit more planning, which I
discuss in the two more sections below. Further discussion invited!

Code Organization
=================

In my opinion, the existing deployment of test code sources in a
parallel directory tree partially explains the underuse of unit testing.

There are several ways of organizing test modules. The most popular ones
seem to be

 a. put test modules in a parallel tree (as now)
 b. mingle test modules with the production modules
 c. create a test subdirectory in each production directory

General note: Both (b) and (c) _may_ make the distribution-building
process more complicated by requiring explicit ignores (or includes). I
suspect it is not a big difficulty, though

Style (a) has the major disadvantage of effectively hiding test modules
from developer view, making it harder to discover existing test modules
or to quickly implement a new one because of the directory navigation
required in the editing environment. In addition, the test module has a
somewhat cumbersome import operation for the module-under-test.

Style (b) is preferred by many (but not me) and is the easiest for
discovery and creation of test modules.  Any "clutter" objection _might_
be countered by the argument that test code is as important as the
working code.

Style (c) is _my_ preference because it segregates test modules from
working (or production) modules, and also gives a convenient place to
put test data and maybe other files without cluttering the production
code. Python itself uses this test file organization. But, consider:
 - test modules are not as discoverable as in (b), but it is much easier
than for (a) -- one need only peek into the immediate test subdir
 - test modules need a "sys.path.append('..')" to be able to import the
module-under-test. This would not be needed with (b), but is similar to
what is presently used in (a).
- The test subdir name might be "./test" or "./tests", or perhaps "./t",
the latter being the subdir name commonly used in perl source trees.
Although it would be slightly cryptic, I have a slight preference for
using a single letter subdir name "t", on the basis that it makes it
less painful for developers to simultaneously edit source and test
code. I would like to hear other opinions on this.

==> Recommendation: We should change to style (c), although I would
respect community preferences for (b). We should do this before adding
new tests. Then we should publish some guidelines encouraging unit test
writing, and illustrating techniques and required framework interfaces
(which are minimal, as discussed below). Objections to naming the subdir
".../t" are invited.

Batch Testing
=============

  Disclaimer: Although I am somewhat of an evangelist for unit testing
  and while I have  written lots of standalone python unit tests, I am
  not an expert on managing whole trees of tests as is needed for a nice
  automated regression test capability.

Recently, however, I have looked into the magic in our own
.../test/RunAllTests.py and in python's test.test_support and
test.regrtest modules. I have finally even got around to studying the
unittest module in order to figure out how all this magic works, :-)
It's not really as complex as a first reading of the unittest
documentation suggests. If it is thought useful, I could probably throw
together a wiki page describing the python unittest facility for
developers who may be as confused as I was a while ago.

Our existing RunAllTests.py is a reasonable starting point for a new
utility after converting to organization style (c). The existing
convention of naming tests with a suffix "_Test" seems reasonable given
the abundance of modules with a leading underscore.  The more common
convention of using a prefix "test_" seems ugly and error prone when
modules already have a leading underscore.

  Some of the existing tests have names other than module_Test.py, but I
  believe it is useful to stick to convention of naming the test after
  the module-under-test wherever possible.

==> Recommendation: Rewrite RunAllTests.py to search in the source tree
for test modules *_Test.py. Modify to allow an optional target (subtree
or single file or list?) rather than always testing the entire Gramps
tree. Document usage. There are a couple of options needing further
investigation -- logging and possible performance test integration.

Regards,
..jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

Stéphane Charette-2
Question:  I'm assuming you are speaking about 3.0, and not 2.2.x?  I
don't object to either, I ask simply because it seems like a lot of
work for a branch that will be terminated once 3.0 is stabilized.

> There are several ways of organizing test modules. The most popular ones
> seem to be
>
>  a. put test modules in a parallel tree (as now)
>  b. mingle test modules with the production modules
>  c. create a test subdirectory in each production directory
[...]
> Style (c) is _my_ preference because it segregates test modules from
> working (or production) modules, and also gives a convenient place to
> put test data and maybe other files without cluttering the production
> code. [...]

You've put much more thought into this than I have!  I support your
preference for style "C".

Stéphane

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

jgsack
Stéphane Charette wrote:
> Question:  I'm assuming you are speaking about 3.0, and not 2.2.x?  I
> don't object to either, I ask simply because it seems like a lot of
> work for a branch that will be terminated once 3.0 is stabilized.

On first glance, there would not seem to be any need to develop unit
tests for 2.2.x. However, when fixing bugs there might be a benefit from
making a small test that demonstrates the bug, then fix the bug, and
demonstrate that it's fixed. But other than this kind of thing, I would
not see any need to write tests for old code.


>> There are several ways of organizing test modules. The most popular ones
>> seem to be
>>
>>  a. put test modules in a parallel tree (as now)
>>  b. mingle test modules with the production modules
>>  c. create a test subdirectory in each production directory
> [...]
>> Style (c) is _my_ preference because it segregates test modules from
>> working (or production) modules, and also gives a convenient place to
>> put test data and maybe other files without cluttering the production
>> code. [...]
>
> You've put much more thought into this than I have!  I support your
> preference for style "C".

There is the question in your area of responsibilities regarding
excluding the test subdirs and their content. Do you see any headaches
there?

Thanks for the feedback,
..jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

Alex Roitman
On Mon, 2007-10-29 at 20:01 -0700, James G. Sack (jim) wrote:
> >
> > You've put much more thought into this than I have!  I support your
> > preference for style "C".
>
> There is the question in your area of responsibilities regarding
> excluding the test subdirs and their content. Do you see any headaches
> there?

That should not be a problem. If the dir is not listed in the parent
dir's Makefile, it will be ignored by build tools.

If we want to, we can ship test things in the tarballs.
The autotools are flexible and should not pose problems.

Alex

--
Alexander Roitman   http://gramps-project.org

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

Douglas S. Blank
In reply to this post by jgsack
Jim,

Thanks for leading this charge! Good stuff you've done already.

I am completely naive about testing, but a couple of comments:

1) Could the test files have the same names as what they are testing if
they are in a subdir? Such as:  plugins/Calendar.py and
plugins/t/Calendar.py

2) I would have guessed that """b. mingle test modules with the production
modules""" would be the most popular among Python developers. I like
having the tests very close to what they're testing. But are there
load-time issues? Is it considered bad form to have test code in the live
module?

3) Related to the two above: if we do have tests and plugins, how will
those be packaged? Will we need to download two files if we want a
third-party's plugin? Or maybe plugins should have the tests inside the
file regardless of what we do throughout the rest of the code?

Thanks again!

-Doug

On Mon, October 29, 2007 7:57 pm, James G. Sack (jim) wrote:

> Summary (but please read additional text beyond the summary)
> *******
>
> ==> Point (1) below requires consensus and a decision. I believe I could
> manage to do and/or lead the effort[*] on the remaining points. The
> conversion would take a few days; the documentation probably longer.
>
>   [*] except for attending to build changes that may be required.
>
>
> 1) We should create a subdir in each source directory to hold unit
> tests. The tests would look in their immediate parent directory for
> their target module. Test modules should follow the existing practice of
> appending a "-_Test" suffix to the name of the module under test.
>
> Example:
>   .../src/GrampsLogger/t
>   .../src/GrampsLogger/t/_RotateHandler_Test.py
>
> The subdir might be ".../tests", or ".../test" if ".../t" is thought too
> cryptic. (TBD)
>
> NB: The distribution build process will need to be examined, and perhaps
> adjusted to skip the test subdirs and contents.
>
> 2) Move the few tests we have into their respective new ".../t"
> locations, fix their names as necessary, and update per current codebase
> so that they run and succeed.
>
> 3) Revise the existing regression test driver RunAllTests.py to find
> tests in the source tree rather than the parallel directory (as now).
> Extend or improve options as required, and document usage.
>
> 4) Create wiki page for developer guidelines, tips, etc
>
> 5) Start writing unit test modules and developing regression testing
> practices.
>
>
> Background and Further Explanation
> **********************************
>
> Gramps presently has 11 unit test modules (in 3 subdirs under .../test).
> The unit tests are run as a regression test batch via the script
> .../test/RunAllTests.py.
>
> Individual tests can be run from the command line in their respective
> subdirectories.
>
> There is also one regular module (under .../src) containing an embedded
> unit test, that is, a unittest-based test -- run when executed as a
> program (via if __name__ == "__main__"). This should be extracted to an
> external test module.
>
>    In addition to the tests based on the python unittest module, there
>  are several other test scripts in the .../test directory. At least one
>  of these (dates.sh) uses supporting test code in .../src.
>    While these may be very useful, they deviate from the unit test
>  philosophy of clearly indicating a success or fail result such as is
>  desirable for automated testing. For now, I recommend lumping them
>  into a "miscellanea" category. They can remain in the .../test dir.
>
> We have about 600 total python code modules, so the coverage isn't very
> significant. In addition, some of those 11 existing test files seem to
> have gotten a bit stale.
>
> Writing standalone unit tests is fairly easy, but making a serious
> regression test _framework_ requires a bit more planning, which I
> discuss in the two more sections below. Further discussion invited!
>
> Code Organization
> =================
>
> In my opinion, the existing deployment of test code sources in a
> parallel directory tree partially explains the underuse of unit testing.
>
> There are several ways of organizing test modules. The most popular ones
> seem to be
>
>  a. put test modules in a parallel tree (as now)
>  b. mingle test modules with the production modules
>  c. create a test subdirectory in each production directory
>
> General note: Both (b) and (c) _may_ make the distribution-building
> process more complicated by requiring explicit ignores (or includes). I
> suspect it is not a big difficulty, though
>
> Style (a) has the major disadvantage of effectively hiding test modules
> from developer view, making it harder to discover existing test modules
> or to quickly implement a new one because of the directory navigation
> required in the editing environment. In addition, the test module has a
> somewhat cumbersome import operation for the module-under-test.
>
> Style (b) is preferred by many (but not me) and is the easiest for
> discovery and creation of test modules.  Any "clutter" objection _might_
> be countered by the argument that test code is as important as the
> working code.
>
> Style (c) is _my_ preference because it segregates test modules from
> working (or production) modules, and also gives a convenient place to
> put test data and maybe other files without cluttering the production
> code. Python itself uses this test file organization. But, consider:
>  - test modules are not as discoverable as in (b), but it is much easier
> than for (a) -- one need only peek into the immediate test subdir
>  - test modules need a "sys.path.append('..')" to be able to import the
> module-under-test. This would not be needed with (b), but is similar to
> what is presently used in (a).
> - The test subdir name might be "./test" or "./tests", or perhaps "./t",
> the latter being the subdir name commonly used in perl source trees.
> Although it would be slightly cryptic, I have a slight preference for
> using a single letter subdir name "t", on the basis that it makes it
> less painful for developers to simultaneously edit source and test
> code. I would like to hear other opinions on this.
>
> ==> Recommendation: We should change to style (c), although I would
> respect community preferences for (b). We should do this before adding
> new tests. Then we should publish some guidelines encouraging unit test
> writing, and illustrating techniques and required framework interfaces
> (which are minimal, as discussed below). Objections to naming the subdir
> ".../t" are invited.
>
> Batch Testing
> =============
>
>   Disclaimer: Although I am somewhat of an evangelist for unit testing
>   and while I have  written lots of standalone python unit tests, I am
>   not an expert on managing whole trees of tests as is needed for a nice
>   automated regression test capability.
>
> Recently, however, I have looked into the magic in our own
> .../test/RunAllTests.py and in python's test.test_support and
> test.regrtest modules. I have finally even got around to studying the
> unittest module in order to figure out how all this magic works, :-)
> It's not really as complex as a first reading of the unittest
> documentation suggests. If it is thought useful, I could probably throw
> together a wiki page describing the python unittest facility for
> developers who may be as confused as I was a while ago.
>
> Our existing RunAllTests.py is a reasonable starting point for a new
> utility after converting to organization style (c). The existing
> convention of naming tests with a suffix "_Test" seems reasonable given
> the abundance of modules with a leading underscore.  The more common
> convention of using a prefix "test_" seems ugly and error prone when
> modules already have a leading underscore.
>
>   Some of the existing tests have names other than module_Test.py, but I
>   believe it is useful to stick to convention of naming the test after
>   the module-under-test wherever possible.
>
> ==> Recommendation: Rewrite RunAllTests.py to search in the source tree
> for test modules *_Test.py. Modify to allow an optional target (subtree
> or single file or list?) rather than always testing the entire Gramps
> tree. Document usage. There are a couple of options needing further
> investigation -- logging and possible performance test integration.
>
> Regards,
> ..jim


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

Brian Matherly
In reply to this post by jgsack
Jim,

Thanks for stepping up for this task. You are certainly welcome to tackle it.

I have a few questions/concerns:

1) What exactly is your end goal for unit tests?

2) What things do you hope to prove with automated unit tests?

3) What code should have unit tests?

4) What code should not have unit tests?

5) I presume you don't intend to one day have a unit test for every line of code in Gramps.

6) Who will maintain the unit tests as the code changes?

7) I would hate to make unit testing a policy as it might discourage new developers. I can see the response now: "Dear Joe, your patch for the new xyz feature is great. But I can't commit it because it doesn't have a unit test. Please write a unit test and resubmit your patch."

All in all, I personally prefer option "a" because it is the least obtrusive and the least commitment. We don't have strict policies around here about unit test procedures. And I'm not sure that we are interested in implementing any. If the tests become stale, someone could pick them up in the future.

Option "c" is tolerable for me. But I will be quite annoyed if you litter the source with subdirectories named "t" and then get tired of Gramps and move on. The contents of those directories will one day grow stale and we'll have to remove them by hand.

Thanks again for your willingness to contribute. Please don't be discouraged by my questions. I like unit tests as much as the next guy. But in my experience, everyone is quick to say "yeah, we should do unit tests", but are slow in actually writing and maintaining them (myself included). I hope you are prepared to run the testing project solo if people don't put their code where their mouth is.

~Brian


----- Original Message ----
From: James G. Sack (jim) <[hidden email]>
To: gramps-devel <[hidden email]>
Sent: Monday, October 29, 2007 6:57:55 PM
Subject: [Gramps-devel] unit testing -- review and recommendations


Summary (but please read additional text beyond the summary)
*******

==> Point (1) below requires consensus and a decision. I believe I
 could
manage to do and/or lead the effort[*] on the remaining points. The
conversion would take a few days; the documentation probably longer.

  [*] except for attending to build changes that may be required.


1) We should create a subdir in each source directory to hold unit
tests. The tests would look in their immediate parent directory for
their target module. Test modules should follow the existing practice
 of
appending a "-_Test" suffix to the name of the module under test.

Example:
  .../src/GrampsLogger/t
  .../src/GrampsLogger/t/_RotateHandler_Test.py

The subdir might be ".../tests", or ".../test" if ".../t" is thought
 too
cryptic. (TBD)

NB: The distribution build process will need to be examined, and
 perhaps
adjusted to skip the test subdirs and contents.

2) Move the few tests we have into their respective new ".../t"
locations, fix their names as necessary, and update per current
 codebase
so that they run and succeed.

3) Revise the existing regression test driver RunAllTests.py to find
tests in the source tree rather than the parallel directory (as now).
Extend or improve options as required, and document usage.

4) Create wiki page for developer guidelines, tips, etc

5) Start writing unit test modules and developing regression testing
practices.


Background and Further Explanation
**********************************

Gramps presently has 11 unit test modules (in 3 subdirs under
 .../test).
The unit tests are run as a regression test batch via the script
.../test/RunAllTests.py.

Individual tests can be run from the command line in their respective
subdirectories.

There is also one regular module (under .../src) containing an embedded
unit test, that is, a unittest-based test -- run when executed as a
program (via if __name__ == "__main__"). This should be extracted to an
external test module.

   In addition to the tests based on the python unittest module, there
 are several other test scripts in the .../test directory. At least one
 of these (dates.sh) uses supporting test code in .../src.
   While these may be very useful, they deviate from the unit test
 philosophy of clearly indicating a success or fail result such as is
 desirable for automated testing. For now, I recommend lumping them
 into a "miscellanea" category. They can remain in the .../test dir.

We have about 600 total python code modules, so the coverage isn't very
significant. In addition, some of those 11 existing test files seem to
have gotten a bit stale.

Writing standalone unit tests is fairly easy, but making a serious
regression test _framework_ requires a bit more planning, which I
discuss in the two more sections below. Further discussion invited!

Code Organization
=================

In my opinion, the existing deployment of test code sources in a
parallel directory tree partially explains the underuse of unit
 testing.

There are several ways of organizing test modules. The most popular
 ones
seem to be

 a. put test modules in a parallel tree (as now)
 b. mingle test modules with the production modules
 c. create a test subdirectory in each production directory

General note: Both (b) and (c) _may_ make the distribution-building
process more complicated by requiring explicit ignores (or includes). I
suspect it is not a big difficulty, though

Style (a) has the major disadvantage of effectively hiding test modules
from developer view, making it harder to discover existing test modules
or to quickly implement a new one because of the directory navigation
required in the editing environment. In addition, the test module has a
somewhat cumbersome import operation for the module-under-test.

Style (b) is preferred by many (but not me) and is the easiest for
discovery and creation of test modules.  Any "clutter" objection
 _might_
be countered by the argument that test code is as important as the
working code.

Style (c) is _my_ preference because it segregates test modules from
working (or production) modules, and also gives a convenient place to
put test data and maybe other files without cluttering the production
code. Python itself uses this test file organization. But, consider:
 - test modules are not as discoverable as in (b), but it is much
 easier
than for (a) -- one need only peek into the immediate test subdir
 - test modules need a "sys.path.append('..')" to be able to import the
module-under-test. This would not be needed with (b), but is similar to
what is presently used in (a).
- The test subdir name might be "./test" or "./tests", or perhaps
 "./t",
the latter being the subdir name commonly used in perl source trees.
Although it would be slightly cryptic, I have a slight preference for
using a single letter subdir name "t", on the basis that it makes it
less painful for developers to simultaneously edit source and test
code. I would like to hear other opinions on this.

==> Recommendation: We should change to style (c), although I would
respect community preferences for (b). We should do this before adding
new tests. Then we should publish some guidelines encouraging unit test
writing, and illustrating techniques and required framework interfaces
(which are minimal, as discussed below). Objections to naming the
 subdir
".../t" are invited.

Batch Testing
=============

  Disclaimer: Although I am somewhat of an evangelist for unit testing
  and while I have  written lots of standalone python unit tests, I am
  not an expert on managing whole trees of tests as is needed for a
 nice
  automated regression test capability.

Recently, however, I have looked into the magic in our own
.../test/RunAllTests.py and in python's test.test_support and
test.regrtest modules. I have finally even got around to studying the
unittest module in order to figure out how all this magic works, :-)
It's not really as complex as a first reading of the unittest
documentation suggests. If it is thought useful, I could probably throw
together a wiki page describing the python unittest facility for
developers who may be as confused as I was a while ago.

Our existing RunAllTests.py is a reasonable starting point for a new
utility after converting to organization style (c). The existing
convention of naming tests with a suffix "_Test" seems reasonable given
the abundance of modules with a leading underscore.  The more common
convention of using a prefix "test_" seems ugly and error prone when
modules already have a leading underscore.

  Some of the existing tests have names other than module_Test.py, but
 I
  believe it is useful to stick to convention of naming the test after
  the module-under-test wherever possible.

==> Recommendation: Rewrite RunAllTests.py to search in the source tree
for test modules *_Test.py. Modify to allow an optional target (subtree
or single file or list?) rather than always testing the entire Gramps
tree. Document usage. There are a couple of options needing further
investigation -- logging and possible performance test integration.

Regards,
..jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel





-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

jgsack
In reply to this post by Douglas S. Blank
Douglas S. Blank wrote:
> Jim,
>
> Thanks for leading this charge! Good stuff you've done already.
>
> I am completely naive about testing, but a couple of comments:
>
> 1) Could the test files have the same names as what they are testing if
> they are in a subdir? Such as:  plugins/Calendar.py and
> plugins/t/Calendar.py

Technically yes, I think that should be possible but I would argue that
sooner or later somebody would get confused between them. Additionally,
there might actually be minor technical problems ensuring the MUT
(module under test) gets imported into the test module instead of a
recursive import! All-in-all I don't see any benefit that would override
the possible confusion and import worries.

> 2) I would have guessed that """b. mingle test modules with the production
> modules""" would be the most popular among Python developers. I like
> having the tests very close to what they're testing. But are there
> load-time issues? Is it considered bad form to have test code in the live
> module?

No load issues -- if the names are different, of course. ;-)
No question but mingling puts the test right next to the MUT in a sorted
listing, whereas in the subdir case, the developer has to know and want
to look in the subdir.
"Bad form" may in fact be the real question, but I think it is quite
subjective. I have heard some people say it is strongly preferred (for
visibility and edit-ease), whereas my own preference (and GvR's
apparently) is to have subdirs. My preference is based on wishing to see
working source code uncluttered by other files. Admittedly I already
have to overlook .pyc files.

>
> 3) Related to the two above: if we do have tests and plugins, how will
> those be packaged? Will we need to download two files if we want a
> third-party's plugin? Or maybe plugins should have the tests inside the
> file regardless of what we do throughout the rest of the code?

I'm sorry that I'm not acquainted with the way we use plugins. (I'm
fairly new here), and I guess I don't appreciate the situation. Are you
talking about third party contributions? Do the contributions become
part of our source tree or do the contributors do the maintenance and
upload the updates? If the former, we can add our own test code if/when
needed; if the latter, then we just (maybe) "kick bugs upstream"? One
way or another I suspect we can deal with it. If a need comes up, I
don't see why we couldn't even implement partial source trees following
different conventions. What I would like to see is some basic convention
that makes it easy for developers to write and use unit tests. There
will be difficulty enough getting buy-in (which is ok), but my hunch and
hope is that existing tests will stimulate use and extension of that usage.

Thanks and Regards,
..jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

jgsack
In reply to this post by Brian Matherly
Brian Matherly wrote:
> Jim,
>
> Thanks for stepping up for this task. You are certainly welcome to tackle it.
>
> I have a few questions/concerns:

Ahh, great questions! And a chance to preach :-)

> 1) What exactly is your end goal for unit tests?

- unit test writing probably can't be "forced", but any testing is
better than no testing
- once a developer discovers the benefits of unit tests, it is (I
believe) rare for them to abandon their use
- if a good test exists, it is *much* less traumatic to attempt bug
fixing and feature addition. "good" means it verifies correct behavior
in the vicinity of the code needing change.
- ("Fram Oil Filter rule") You can test it now or test it in
integration, or acceptance, or when it breaks; you cannot avoid testing,
so it is best to write the test along with the code (when [you think]
you know what the code is supposed to do). Be aware that test code
sometimes adds up to a significant amount of code.
- putting the tests under source control ensures they are there when a
new bug is discovered
- if you ain't tested it, you shouldn't advertise having it


> 2) What things do you hope to prove with automated unit tests?

- automated tests require a commitment to keeping tests current, so they
 prove _very little_ without ongoing maintenance of tests along with the
working code. But to the extent that it does work, one can adopt various
strategies, ranging from continuous testing to release-time checking. it
probably takes a while to mature, but my hope is that unit test
practices will eventually become considered so  important that no module
that has an associated unit test ever gets checked in unless the unit
test (is updated as required) and passes. I do know (good) programmers
who refuse to modify code that does not have any (or adequate) unit tests.
- testing of individual subtrees may still be useful even if the full
tree testing gets broken

> 3) What code should have unit tests?

- anything that can produce a OK/fail condition.
- the most common test I have written is to check that a function
returns the expected output(s) for some given input(s).
- anything that is a demonstrated bug; a bug is just some behavior that
was never tested for. Add a test to demonstrate the bug, fix the bug,
show that it's now fixed, have a permanent test that pops up if the bug
ever returns.
- especially common is to verify behavior at all input limits, and to
verify that all exceptions are raised when they should be

> 4) What code should not have unit tests?

I'm not quite sure, but...
- it is more difficult to test interactive code
- gui testing is different from anything I have done via code, but I
understand that gui testing tools are becoming more readily available
and useful
- code that operates on complicated state that is hard to separate into
simple tests will be .. hard! Interestingly, code that is hard to test
often stimulates rethinking the code design and that frequently results
in code architectural improvements </end hand waving>

> 5) I presume you don't intend to one day have a unit test for every line of code in Gramps.

- that _would_ be pretty ambitious -- I don't expect to live that long
<heh>. But some projects do aim to test every functional behavior, and
some projects do try to measure code coverage. So the goal is not that
outlandish. Realistically though, testing doesn't really guaranty
correctness.
- python has only about half as many test_XXX.py files as non test files
int their library (builtin) code. Perl6 developers claim to have 100K
test cases that they say will "define what perl6" is.

> 6) Who will maintain the unit tests as the code changes?

- that's a key question. If the tests are not visible, no one and they
might as well not exist. If that were the prospect, I would write my own
unit tests on code I worked on, and keep it to myself. If unit test code
modules are visible and start out working, there's a potential that
developers will use them to verify that their bugfixing and feature
addition hasn't broken anything. If maintenance mods render a test (or
more commonly a part of a test module) inapplicable or wrong, the
developer _should_ remove or update the test. This is the part that does
not happen without some effort. Encouragement helps. Making test writing
easy helps.  Having a test update party before release might help. Best
is if developers decide on their own that tests help them write good code.

> 7) I would hate to make unit testing a policy as it might discourage new developers. I can see the response now: "Dear Joe, your patch for the new xyz feature is great. But I can't commit it because it doesn't have a unit test. Please write a unit test and resubmit your patch."

- I agree that OSS projects can't very well implement such policies by
fiat. It probably has to be a botttom up and team-spirit effect,
although it can be encouraged at project management level. I do believe
a few good code examples and a few enthusiasts can stimulate good results.
- I would also repeat that any unit testing is better than no testing
- if a patch breaks an existing test, it may be easy for a reviewer to
update the test in the process of checking the patch. If not, maybe the
patch is too big and should be partitioned? If it comes to it, then the
test may have to be cut down or even discarded if it no longer provides
any benefit.

> All in all, I personally prefer option "a" because it is the least obtrusive and the least commitment. We don't have strict policies around here about unit test procedures. And I'm not sure that we are interested in implementing any. If the tests become stale, someone could pick them up in the future.

I've already stated my opinion that "a" (which we have now) is a major
reason that developers ignore existing tests and which discourages
writing new ones. I would like to be able to convince people that "b" or
"C" is more likely to get existing tests used and maintained and
hopefully encourage creation of new tests. I like "c" best, but know
some very respected people who like "b".

> Option "c" is tolerable for me. But I will be quite annoyed if you litter the source with subdirectories named "t" and then get tired of Gramps and move on. The contents of those directories will one day grow stale and we'll have to remove them by hand.

- sure, svn makes it pretty easy to delete things. I would say that
empty directories do no good, but that as soon as a test is deemed
necessary, the creator should go ahead and create the subdir if not present.

> Thanks again for your willingness to contribute. Please don't be discouraged by my questions...

No, questions are good! challenging questions are better!

>..I like unit tests as much as the next guy...

Me, more than the next guy, I guess. Bottom line is there is no silver
bullet in any methodology or any specific practice.

>..But in my experience, everyone is quick to say "yeah, we should do unit tests", but are slow in actually writing and maintaining them..

My answer to this is give it a try for a while. Maybe I might be the
only one who does anything with it at first. I would hope to demonstrate
with even some small efforts that there are very noticeable benefits. If
I fail to make any impact or measurable contribution, well, then I will
probably get chased away or become discouraged, and you can always
delete the tests with not much work.

>..(myself included). I hope you are prepared to run the testing project
solo if people don't put their code where their mouth is.

Well, I do _intend_ to stick around a while. And I'm am prepared to
nibble away at test creation (and maintenance) as I contribute to
mainline working code. I'm retired and don't plan on going anywhere for
a while so there's no major obstacle to putting in some time.

Really good questions. Hope my answers were the kind you were looking.
You may have to (learn to) excuse my rambling; that's not unusual for
me. ;-)

Other developers should jump in too. We could use some more opinions,
even if they are expressing reservations or even outright doubts.

In the meantime, I am working on an update to the RunAllTests.py driver,
and a demo of a small unit test development example on real code. It
would be nice to have some consensus by (say) a week from now so that I
might start some initial checkins along these lines.

Cheers, and Regards,
..jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

Benny Malengier
Hi,

my take on it:

1/tests: yes.
I just finished adding a test in 'if __main__' to Relationship.py, that is usable in all rel_xx.py modules. I fail to see if tests exist, and a patch comes in, why the test should not be rerun. If it fails it means we have a bug, so running it is good.
However, I never used the present setup in test, because frankly, I wouldn't know if a test exists for what I'm hacking on. So a) does not look like a good option to me.

So I clearly vote for c), with a name like 'unittest' for the dir. 't' is too short, it's hard to click on such small directory names.
Note that c) is important as one will need sample files to compare output, eg test xml file, test output file, .... which the test will diff to find changes. One cannot have these text files clutter the normal directories. About the name of the file, perhaps one must allow for separate test files for one module? That is, if I'm hacking on something that means only one of the 10 tests must be run, can I easily run only that one test instead of all 10? I would like that to be the case, some tests might take some time to run.

2/Maintaining tests. If you change something, you need to test it anyway. Hacking an existing tests is far easier than coming up with a test from scratch. Important is that the directory of testing is in src. I do grep searches to find code fragments only in src, as searching them higher also finds the .svn dir, the manual,  and such. Changes in functions must be changed in the test too, so the grep search should find them in src.

3/About the _ before names. Don has started to remove these. They are an artifact of the past, and new directories should not contain them anymore.

4/About testing rules. I don't think we should force tests in any way. Also not needed for everything. However, a framework you can add to the moment you think a test is needed, is very welcome.
It is a win-win situation. If you use it, Gramps get better, if you want to use it, much of the supporting code is there already, if you don't use it, it doesn't bother you.

Benny

PS: What I think is needed in the framework: ability to load a .gramps file without interface so you can do test code on it, like add a person, and check the result.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

Brian Matherly
In reply to this post by jgsack
Jim,

All your answers satisfied me.

>It would be nice to have some consensus by (say) a week from now so that I
>might start some initial checkins along these lines.

The best way to build consensus is to put together a proposal on the Wiki with tangible code examples. Explain how the framework will work, and how it will affect people's work flow. Then, come back to the list and see what people say. Once consensus is built, this page can serve as a starting point for your documentation.

Thanks again for taking charge of this.

~Brian


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

jgsack
In reply to this post by Benny Malengier
Benny Malengier wrote:

>..
> 1/tests: yes.
> I just finished adding a test in 'if __main__' to Relationship.py, that is
> usable in all rel_xx.py modules. I fail to see if tests exist, and a patch
> comes in, why the test should not be rerun. If it fails it means we have a
> bug, so running it is good.
> However, I never used the present setup in test, because frankly, I wouldn't
> know if a test exists for what I'm hacking on. So a) does not look like a
> good option to me.
>
> So I clearly vote for c), with a name like 'unittest' for the dir. 't' is
> too short, it's hard to click on such small directory names.
> Note that c) is important as one will need sample files to compare output,
> eg test xml file, test output file, .... which the test will diff to find
> changes. One cannot have these text files clutter the normal directories.
> About the name of the file, perhaps one must allow for separate test files
> for one module? That is, if I'm hacking on something that means only one of
> the 10 tests must be run, can I easily run only that one test instead of all
> 10? I would like that to be the case, some tests might take some time to
> run.

Good points. "t" is admittedly ultra short. A hangover from perl habits
I suppose. The thinking goes something like:
  The test subdirectory will be of use to programmers who will come
  to recognize it whatever it is named, and a short name is easier
  to type in editor file-load commands.
But ok, I can understand the complaint that t is both ultra cryptic, as
well as actually being hard to see!  Hmmm, my thought now is that
"unittest" and even "tests" deviate a bit from some small amount of
convention that does exist, so I offer "test" as a compromise.

I do foresee "test" being a potential source of confusion with python's
top-level test package,  but _believe_ it shouldn't really cause
problems. In this respect "unittest" might cause problems running tests
on our top-level modules. The conservative choice might be to choose a
unique name rather than a conventional one. That argument might suggest
something like "gramps_test" or "gtest" or "utest" or "gutest" ...

==> What do you (and others) think. I'm leaning toward "test".

> 2/Maintaining tests. If you change something, you need to test it anyway.
> Hacking an existing tests is far easier than coming up with a test from
> scratch. Important is that the directory of testing is in src. I do grep
> searches to find code fragments only in src, as searching them higher also
> finds the .svn dir, the manual,  and such. Changes in functions must be
> changed in the test too, so the grep search should find them in src.

Yes, good points. I would amplify that the persistent unit tests become
more valuable with time (if they are kept current), because they help
detect accidental blunders like deleting the wrong function, or any of a
million others.

By the way, it's been there all along, but I recently discovered a grep
option (--include) I like to use in combination with recursion.

 grep -rl --include='*.py' "import unittest" src

> 3/About the _ before names. Don has started to remove these. They are an
> artifact of the past, and new directories should not contain them anymore.

So if we wanted, we could use a "test_" prefix without the objections I
offered earlier.  Still, I'm thinking I actually prefer a suffix for the
display in sorted lists. While we're here, I might criticize my
suggestion of using "_Test" as violating a semi-convention of python of
using/tolerating upper case, mixed case and camel case in various places
but vigorously disliking things like "_Test"  (preferring "_test") .. so
I'm now preferring "_test". Man these trivial questions get tedious, but
now is the time to spend a few extra minutes on them, I suppose.

==> I think I'd like to use names like RelationShipView_test.py

> 4/About testing rules. I don't think we should force tests in any way. Also
> not needed for everything. However, a framework you can add to the moment
> you think a test is needed, is very welcome.
> It is a win-win situation. If you use it, Gramps get better, if you want to
> use it, much of the supporting code is there already, if you don't use it,
> it doesn't bother you.

Well put! My thinking exactly!

>..
> PS: What I think is needed in the framework: ability to load a .gramps file
> without interface so you can do test code on it, like add a person, and
> check the result.

Good suggestion. We may need various support modules like
test_utility.py that can be used by test code.

Regards and thanks for the feedback,
..jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

jgsack
In reply to this post by Brian Matherly
Brian Matherly wrote:

> Jim,
>
> All your answers satisfied me.
>
>> It would be nice to have some consensus by (say) a week from now so that I
>> might start some initial checkins along these lines.
>
> The best way to build consensus is to put together a proposal on the Wiki with tangible code examples. Explain how the framework will work, and how it will affect people's work flow. Then, come back to the list and see what people say. Once consensus is built, this page can serve as a starting point for your documentation.
>
> Thanks again for taking charge of this.
>

Now that I've made a big noise about all this, I'd like to remind
everyone that I not proposing to change the way anything or anyone
works. At least not by adding any requirements.

So as a proposal for change it's not much of a change that affects
anyone, and really may not justify bothering everybody (so much) with my
indecision on how best to do it.

..BUT..

Testing is important, and whatever is started now may have a long term
impact, so I am thinking that collecting viewpoints is worthwhile. And
preferable to just slipping something into place.

  And besides, I'm kinda new around here and do not wish to give
  an impression of trying to force anything on anyone.

If I can quote previous wisdom:
"""
I don't think we should force tests in any way. Also not needed for
everything. However, a framework you can add to the moment you think a
test is needed, is very welcome.
It is a win-win situation. If you use it, Gramps get better, if you want
to use it, much of the supporting code is there already, if you don't
use it, it doesn't bother you.
"""

I do intend start a wiki page (or several) on this.

As far as appealing for consensus or approval, I'm more thinking that
nobody really objects so I'm just going to start doing it (small). I was
mostly waiting for people to have time to criticize and suggest.

If the initial examples don't add up to a usable framework, or if major
modifications to the framework are deemed needed, well then a small
start is easy to yank out or change, and not much effort will have been
lost.

Regards,
..jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

Douglas S. Blank
James G. Sack (jim) wrote:

[snip]

> If the initial examples don't add up to a usable framework, or if major
> modifications to the framework are deemed needed, well then a small
> start is easy to yank out or change, and not much effort will have been
> lost.

Sounds like a plan! And thanks again for the energy to propose, explore,
and experiment. (I think testing may end up being the single best,
longest lasting change that we add to GRAMPS this next iteration. In 50
years, GRAMPS may be using SQL with KDE on the PlayStation, but the
testing framework will still be in place :)

-Doug

> Regards,
> ..jim


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

Brian Matherly
In reply to this post by jgsack
Jim,

>As far as appealing for consensus or approval, I'm more thinking that
>nobody really objects so I'm just going to start doing it (small). I
 was
>mostly waiting for people to have time to criticize and suggest.

I think you have hit the nail on the head. You no longer need to build consensus about whether to do unit testing. You already have that. Now, you just need to keep people involved as you implement it. Transparency is the key. I think that between using the Wiki, and taking small steps, you will be fine.

~Brian


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

Eero Tamminen-3
In reply to this post by jgsack
Hi,

On Tuesday 30 October 2007, James G. Sack (jim) wrote:

> Brian Matherly wrote:
> > Thanks for stepping up for this task. You are certainly welcome to
> > tackle it.
> >
> > I have a few questions/concerns:
>
> Ahh, great questions! And a chance to preach :-)
>
> > 1) What exactly is your end goal for unit tests?
>
> - unit test writing probably can't be "forced", but any testing is
> better than no testing
> - once a developer discovers the benefits of unit tests, it is (I
> believe) rare for them to abandon their use
> - if a good test exists, it is *much* less traumatic to attempt bug
> fixing and feature addition. "good" means it verifies correct behavior
> in the vicinity of the code needing change.
> - ("Fram Oil Filter rule") You can test it now or test it in
> integration, or acceptance, or when it breaks; you cannot avoid testing,
> so it is best to write the test along with the code (when [you think]
> you know what the code is supposed to do). Be aware that test code
> sometimes adds up to a significant amount of code.

And maintaining tests can take more time than maintaining the code it's
testing (due to needed setup, good enough parameter coverage and result
verification)...


> - putting the tests under source control ensures they are there when a
> new bug is discovered
> - if you ain't tested it, you shouldn't advertise having it
>
> > 2) What things do you hope to prove with automated unit tests?
>
> - automated tests require a commitment to keeping tests current, so they
>  prove _very little_ without ongoing maintenance of tests along with the
> working code. But to the extent that it does work, one can adopt various
> strategies, ranging from continuous testing to release-time checking. it
> probably takes a while to mature, but my hope is that unit test
> practices will eventually become considered so  important that no module
> that has an associated unit test ever gets checked in unless the unit
> test (is updated as required) and passes. I do know (good) programmers
> who refuse to modify code that does not have any (or adequate) unit
> tests. - testing of individual subtrees may still be useful even if the
> full tree testing gets broken
>
> > 3) What code should have unit tests?
>
> - anything that can produce a OK/fail condition.
> - the most common test I have written is to check that a function
> returns the expected output(s) for some given input(s).
> - anything that is a demonstrated bug; a bug is just some behavior that
> was never tested for. Add a test to demonstrate the bug, fix the bug,
> show that it's now fixed, have a permanent test that pops up if the bug
> ever returns.
> - especially common is to verify behavior at all input limits, and to
> verify that all exceptions are raised when they should be

Tests are better only in case the tests actually reduce the work (in
medium/long term). There's no point in wasting resources for writing,
maintaining and running a test unless it finds a defect.  Currently Gramps
has nearly 150 KLOCs of code, there's no point in doubling that maintenance
burden.  (Just running tests for all functions there would take a lot of
time)

So, I'd like some pre-thought on what should be tested / offer best ROI:
- Functionality that could cause data corruption or loss
- Important internal functionality which breakage is hard to catch
  and/or debug
- Functionality that covers/exercises a lot of the underlying (or above)
  code/functionality with little test-code and/or need for maintenance
- Stable internal APIs used by multiple components
- New code
?

> > 4) What code should not have unit tests?

Functionality that is very visible to user might not be a priority because
as users find faults in that easily, so test don't (necessarily) need to.


> I'm not quite sure, but...
> - it is more difficult to test interactive code

Based on what I've understood, testing is easier[1], at least if you compare
the needed effort against the test coverage.  If I remember correctly the
comments on the mailing list, at least half Gramps code deals with UI, so
you might not get very good coverage without this.

The hard thing is that it doesn't point out as well where the problem is.
So you need internal API testing also.


> - gui testing is different from anything I have done via code, but I
> understand that gui testing tools are becoming more readily available
> and useful

Gtk offers accessibility APIs for introspecting the UI contents and invoking
widget actions.


        - Eero

> - code that operates on complicated state that is hard to separate into
> simple tests will be .. hard! Interestingly, code that is hard to test
> often stimulates rethinking the code design and that frequently results
> in code architectural improvements </end hand waving>
>
> > 5) I presume you don't intend to one day have a unit test for every
> > line of code in Gramps.
>
> - that _would_ be pretty ambitious -- I don't expect to live that long
> <heh>. But some projects do aim to test every functional behavior, and
> some projects do try to measure code coverage. So the goal is not that
> outlandish. Realistically though, testing doesn't really guaranty
> correctness.
> - python has only about half as many test_XXX.py files as non test files
> int their library (builtin) code. Perl6 developers claim to have 100K
> test cases that they say will "define what perl6" is.
>
> > 6) Who will maintain the unit tests as the code changes?
>
> - that's a key question. If the tests are not visible, no one and they
> might as well not exist. If that were the prospect, I would write my own
> unit tests on code I worked on, and keep it to myself. If unit test code
> modules are visible and start out working, there's a potential that
> developers will use them to verify that their bugfixing and feature
> addition hasn't broken anything. If maintenance mods render a test (or
> more commonly a part of a test module) inapplicable or wrong, the
> developer _should_ remove or update the test. This is the part that does
> not happen without some effort. Encouragement helps. Making test writing
> easy helps.  Having a test update party before release might help. Best
> is if developers decide on their own that tests help them write good
> code.
>
> > 7) I would hate to make unit testing a policy as it might discourage
> > new developers. I can see the response now: "Dear Joe, your patch for
> > the new xyz feature is great. But I can't commit it because it doesn't
> > have a unit test. Please write a unit test and resubmit your patch."
>
> - I agree that OSS projects can't very well implement such policies by
> fiat. It probably has to be a botttom up and team-spirit effect,
> although it can be encouraged at project management level. I do believe
> a few good code examples and a few enthusiasts can stimulate good
> results. - I would also repeat that any unit testing is better than no
> testing - if a patch breaks an existing test, it may be easy for a
> reviewer to update the test in the process of checking the patch. If not,
> maybe the patch is too big and should be partitioned? If it comes to it,
> then the test may have to be cut down or even discarded if it no longer
> provides any benefit.
>
> > All in all, I personally prefer option "a" because it is the least
> > obtrusive and the least commitment. We don't have strict policies
> > around here about unit test procedures. And I'm not sure that we are
> > interested in implementing any. If the tests become stale, someone
> > could pick them up in the future.
>
> I've already stated my opinion that "a" (which we have now) is a major
> reason that developers ignore existing tests and which discourages
> writing new ones. I would like to be able to convince people that "b" or
> "C" is more likely to get existing tests used and maintained and
> hopefully encourage creation of new tests. I like "c" best, but know
> some very respected people who like "b".
>
> > Option "c" is tolerable for me. But I will be quite annoyed if you
> > litter the source with subdirectories named "t" and then get tired of
> > Gramps and move on. The contents of those directories will one day grow
> > stale and we'll have to remove them by hand.
>
> - sure, svn makes it pretty easy to delete things. I would say that
> empty directories do no good, but that as soon as a test is deemed
> necessary, the creator should go ahead and create the subdir if not
> present.
>
> > Thanks again for your willingness to contribute. Please don't be
> > discouraged by my questions...
>
> No, questions are good! challenging questions are better!
>
> >..I like unit tests as much as the next guy...
>
> Me, more than the next guy, I guess. Bottom line is there is no silver
> bullet in any methodology or any specific practice.
>
> >..But in my experience, everyone is quick to say "yeah, we should do
> > unit tests", but are slow in actually writing and maintaining them..
>
> My answer to this is give it a try for a while. Maybe I might be the
> only one who does anything with it at first. I would hope to demonstrate
> with even some small efforts that there are very noticeable benefits. If
> I fail to make any impact or measurable contribution, well, then I will
> probably get chased away or become discouraged, and you can always
> delete the tests with not much work.
>
> >..(myself included). I hope you are prepared to run the testing project
>
> solo if people don't put their code where their mouth is.
>
> Well, I do _intend_ to stick around a while. And I'm am prepared to
> nibble away at test creation (and maintenance) as I contribute to
> mainline working code. I'm retired and don't plan on going anywhere for
> a while so there's no major obstacle to putting in some time.
>
> Really good questions. Hope my answers were the kind you were looking.
> You may have to (learn to) excuse my rambling; that's not unusual for
> me. ;-)
>
> Other developers should jump in too. We could use some more opinions,
> even if they are expressing reservations or even outright doubts.
>
> In the meantime, I am working on an update to the RunAllTests.py driver,
> and a demo of a small unit test development example on real code. It
> would be nice to have some consensus by (say) a week from now so that I
> might start some initial checkins along these lines.
>
> Cheers, and Regards,
> ..jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

jgsack
Eero Tamminen wrote:
>..Be aware that test code
>> sometimes adds up to a significant amount of code.
>
> And maintaining tests can take more time than maintaining the code it's
> testing (due to needed setup, good enough parameter coverage and result
> verification)...

I would say yes and no. Writing and maintaining unnecessary tests is
certainly a waste of time. Of course the definition of "unnecessary" is
somewhat subjective as well as application-dependent. Gramps is not
quite life-critical, but on the other hand, I would say that any user
application does imply a reasonable-effort promise to safeguard the
user's data in normal operation.

The point I like to return to is that a developer will always be doing
some testing to verify the non-trivial parts of his code. In an ideal
world, that test effort could be saved under source control so that in
future maintenance, one can "painlessly" verify that new/modified code
doesn't break any existing contract.

Admittedly it takes extra effort to write a test suitable for future
use, and it isn't always painless to maintain and reuse the tests.

Presumably there will be _some_ situations where there is a clear
benefit from creating and maintaining test code. I'm happy to let that
rest with each developer. The counter argument is that if A writes a
test and B modifies code without updating the test, then the test
becomes useless. I can see that happening, and don't know any answer
other than if the test loses its value, then just throw it away (perhaps
it justified its cost via past benefits, perhaps not). The best tests
will be written so that future maintainers will find them understandable
and useful, so that they (the future maintainers) will see a value in
keeping them up-to-date.

Since we are just getting started in the unit test direction, it might
be reasonable to read your comments as advice to not invest extensively
in test code that is unlikely to be understandable and useful to someone
else. I hope you will forgive my taking liberties with your remarks. :-)

>..

> Tests are better only in case the tests actually reduce the work (in
> medium/long term). There's no point in wasting resources for writing,
> maintaining and running a test unless it finds a defect.  Currently Gramps
> has nearly 150 KLOCs of code, there's no point in doubling that maintenance
> burden.  (Just running tests for all functions there would take a lot of
> time)

I'm not sure you are objecting to something here, or just amplifying the
observation that it is undesirable to write tests just for the purpose
of having a test.

..however :-) , I do have a couple of conversational responses, anyway.

  Wheeler's sloccount shows > 100KSLOC in trunk, and measures is as
  a 3.75 million-dollar value. Nice work fellows!

There will probably never be a shortage of defects, and tests can
(sometimes) be created to demonstrate the defect, demonstrate it's
repair, and provide insurance against future regression.

Another aspect of unit tests (perhaps, mostly a personal work style
thing) is that unit tests _can_ help during the code construction phase.
You can push this to extremes (no doubt, a misuse), but whenever I have
some code to design/implement that is non-trivial, I consider the use of
TDD (/Test-Driven Development/) to guide me in the actual process. To
belabor the TDD-philosophy, a code requirement that is not yet
implemented is a "defect" -- you can write a test that fails, write the
code, demo that the code works, and not only do you have finished code,
but you also have a regression test for that feature. I don't always use
this, but it's a nice tool for anything with a little subtlety. I find
it helps me clarify my own view of the requirements, too.


> So, I'd like some pre-thought on what should be tested / offer best ROI:
> - Functionality that could cause data corruption or loss
> - Important internal functionality which breakage is hard to catch
>   and/or debug
> - Functionality that covers/exercises a lot of the underlying (or above)
>   code/functionality with little test-code and/or need for maintenance
> - Stable internal APIs used by multiple components
> - New code
> ?

All excellent suggestions. What I would like to contribute is a little
infrastructure and some coding guidelines/recipes that will make it
easier to say things like .. "hey, this is pretty important, we should
invest in a test for it".

>>> 4) What code should not have unit tests?
>
> Functionality that is very visible to user might not be a priority because
> as users find faults in that easily, so test don't (necessarily) need to.
>
>
>> I'm not quite sure, but...
>> - it is more difficult to test interactive code
>
> Based on what I've understood, testing is easier[1], at least if you compare
> the needed effort against the test coverage.  If I remember correctly the
> comments on the mailing list, at least half Gramps code deals with UI, so
> you might not get very good coverage without this.
>
> The hard thing is that it doesn't point out as well where the problem is.
> So you need internal API testing also.

Yup, agree.

I'm looking forward to future gui testing, but I think I can contribute
some preparations by concentrating first on background stuff which is
easier to test in a non-interactive way.

>> - gui testing is different from anything I have done via code, but I
>> understand that gui testing tools are becoming more readily available
>> and useful
>
> Gtk offers accessibility APIs for introspecting the UI contents and invoking
> widget actions.

Ahhh, bears looking into. I know there are other resources for gui
testing available too, but I'm personally concentrating on the non-gui
parts for now.

==> Thanks for the thoughtful comments. I hope no one thinks we're
having an argument here. I don't.

Regards,
..jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

Eero Tamminen-3
Hi,

On Friday 09 November 2007, James G. Sack (jim) wrote:
> Eero Tamminen wrote:
> >..Be aware that test code
> >
> >> sometimes adds up to a significant amount of code.
> >
> > And maintaining tests can take more time than maintaining the code it's
> > testing (due to needed setup, good enough parameter coverage and result
> > verification)...
[...]
> Since we are just getting started in the unit test direction, it might
> be reasonable to read your comments as advice to not invest extensively
> in test code that is unlikely to be understandable and useful to someone
> else. I hope you will forgive my taking liberties with your remarks. :-)

Maybe there should be also some helper code to setup the
testing "environment".  I'm often concerned about performance, so I would
suggest some code that creates 100, 1000 and 10000 person test databases
which the test code can use.  These could be then be used to test that
algorithms don't exhibit exponential behaviour etc.


[...]

> Another aspect of unit tests (perhaps, mostly a personal work style
> thing) is that unit tests _can_ help during the code construction phase.
> You can push this to extremes (no doubt, a misuse), but whenever I have
> some code to design/implement that is non-trivial, I consider the use of
> TDD (/Test-Driven Development/) to guide me in the actual process. To
> belabor the TDD-philosophy, a code requirement that is not yet
> implemented is a "defect" -- you can write a test that fails, write the
> code, demo that the code works, and not only do you have finished code,
> but you also have a regression test for that feature. I don't always use
> this, but it's a nice tool for anything with a little subtlety. I find
> it helps me clarify my own view of the requirements, too.

It will definitely help in API design phase...


[...]

> >> - gui testing is different from anything I have done via code, but I
> >> understand that gui testing tools are becoming more readily available
> >> and useful
> >
> > Gtk offers accessibility APIs for introspecting the UI contents and
> > invoking widget actions.
>
> Ahhh, bears looking into. I know there are other resources for gui
> testing available too, but I'm personally concentrating on the non-gui
> parts for now.

Both Dogtail and LDTP use the Gtk accessibility APIs for doing their work:
        http://ldtp.freedesktop.org/wiki/
        http://people.redhat.com/zcerza/dogtail/

Nice (although a bit old) article describing dogtail use:
        http://www.redhat.com/magazine/020jun06/features/dogtail/

AFAIK LDTP was originally based on verifying the test results based on
pre-generated "application map" (app widget hierarchy) dumps, whereas with
Dogtail this is done dynamically using code.  I think both ways have their
own advantages and LDTP has later added the dynamic stuff also.


        - Eero

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

Eero Tamminen-3
In reply to this post by jgsack
Hi,

On Tuesday 30 October 2007, James G. Sack (jim) wrote:
> Testing is important, and whatever is started now may have a long term
> impact, so I am thinking that collecting viewpoints is worthwhile. And
> preferable to just slipping something into place.

These papers have good points about automated testing and
testing in general:
        http://www.testing.com/writings/automate.pdf
        http://www.testing.com/writings/classic/mistakes.pdf


        - Eero

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: unit testing -- review and recommendations

jgsack
Eero Tamminen wrote:

> Hi,
>
> On Tuesday 30 October 2007, James G. Sack (jim) wrote:
>> Testing is important, and whatever is started now may have a long term
>> impact, so I am thinking that collecting viewpoints is worthwhile. And
>> preferable to just slipping something into place.
>
> These papers have good points about automated testing and
> testing in general:
> http://www.testing.com/writings/automate.pdf
> http://www.testing.com/writings/classic/mistakes.pdf
>

Yes, thanks.

This author is one of the few around who have managed to make a
profession out of testing. He is highly respected.

The site ref. for all his writings is
  http://www.testing.com/writings.html

Regards,
..jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Loading...