Testing and Coverage of Gramps

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Testing and Coverage of Gramps

DS Blank
Devs,

I've just turned on "coverage" in our continuous integration testing on Travis. Coverage tells us which lines of code are being run in our tests. Coverage of lines of code doesn't guarantee that the code is correct, but does mean that it was executed in our tests.

I've also registered gramps with codecov.io, for easy viewing of what is covered, and what needs testing:

https://codecov.io/github/gramps-project/gramps

Currently, our coverage is at 22%. Our goal should probably be to get that to at least 70% or 80%. (Testing the Gtk GUI components will require some additional work, but it looks like there are projects designed for this.)

To write tests, you can look through any of the /test/ subdirectories for examples.

I've also written a wrapper around the Gramps CLI so you can write tests that will run the command-line arguments, but get the benefit of having the results included in nose (our test framework) and thus coverage. It works like:

from gramps.test.test_util import Gramps
g = Gramps()
out, err = g.run("-l")

where out is the standard-out as a string, and err is the standard-error. run() takes the command-line args, one at a time, as strings. You can put that in a test program, and use self.assertTrue() to check particular results in the output strings. This can test a big chunk of code all at once, but more specific tests will be better to check for specific results.

Hope this helps,

-Doug

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing and Coverage of Gramps

DS Blank
Test coverage is now up to 26%:

https://codecov.io/github/gramps-project/gramps/commits

I've also added some code to make it easier to test reports. See an example here: https://github.com/gramps-project/gramps/blob/master/gramps/plugins/test/test_reports.py#L122

Works like:

reports.addtest(TestDynamic, report_name, test_function, options_as_string)

-Doug

On Sat, Feb 13, 2016 at 8:44 AM, Doug Blank <[hidden email]> wrote:
Devs,

I've just turned on "coverage" in our continuous integration testing on Travis. Coverage tells us which lines of code are being run in our tests. Coverage of lines of code doesn't guarantee that the code is correct, but does mean that it was executed in our tests.

I've also registered gramps with codecov.io, for easy viewing of what is covered, and what needs testing:

https://codecov.io/github/gramps-project/gramps

Currently, our coverage is at 22%. Our goal should probably be to get that to at least 70% or 80%. (Testing the Gtk GUI components will require some additional work, but it looks like there are projects designed for this.)

To write tests, you can look through any of the /test/ subdirectories for examples.

I've also written a wrapper around the Gramps CLI so you can write tests that will run the command-line arguments, but get the benefit of having the results included in nose (our test framework) and thus coverage. It works like:

from gramps.test.test_util import Gramps
g = Gramps()
out, err = g.run("-l")

where out is the standard-out as a string, and err is the standard-error. run() takes the command-line args, one at a time, as strings. You can put that in a test program, and use self.assertTrue() to check particular results in the output strings. This can test a big chunk of code all at once, but more specific tests will be better to check for specific results.

Hope this helps,

-Doug


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing and Coverage of Gramps

Tim Lyons
Administrator
In reply to this post by DS Blank
DS Blank wrote
Devs,

I've just turned on "coverage" in our continuous integration testing on
Travis. Coverage tells us which lines of code are being run in our tests.
Coverage of lines of code doesn't guarantee that the code is correct, but
does mean that it was executed in our tests.

I've also registered gramps with codecov.io, for easy viewing of what is
covered, and what needs testing:

https://codecov.io/github/gramps-project/gramps

Currently, our coverage is at 22%. Our goal should probably be to get that
to at least 70% or 80%. (Testing the Gtk GUI components will require some
additional work, but it looks like there are projects designed for this.)

To write tests, you can look through any of the /test/ subdirectories for
examples.
Is there an explanation anywhere of how the test code and Travis works?

(Including how Travis builds the code for execution, and how the test code is linked into the executable and how Travis finds it/links to it [I do understand that Python is interpreted so the concept of 'linking' is a bit fluid])

regards,
Tim.
Reply | Threaded
Open this post in threaded view
|

Re: Testing and Coverage of Gramps

John Ralls-2

> On Apr 3, 2016, at 7:46 AM, Tim Lyons <[hidden email]> wrote:
>
> DS Blank wrote
>> Devs,
>>
>> I've just turned on "coverage" in our continuous integration testing on
>> Travis. Coverage tells us which lines of code are being run in our tests.
>> Coverage of lines of code doesn't guarantee that the code is correct, but
>> does mean that it was executed in our tests.
>>
>> I've also registered gramps with codecov.io, for easy viewing of what is
>> covered, and what needs testing:
>>
>> https://codecov.io/github/gramps-project/gramps
>>
>> Currently, our coverage is at 22%. Our goal should probably be to get that
>> to at least 70% or 80%. (Testing the Gtk GUI components will require some
>> additional work, but it looks like there are projects designed for this.)
>>
>> To write tests, you can look through any of the /test/ subdirectories for
>> examples.
>
> Is there an explanation anywhere of how the test code and Travis works?
>
> (Including how Travis builds the code for execution, and how the test code
> is linked into the executable and how Travis finds it/links to it [I do
> understand that Python is interpreted so the concept of 'linking' is a bit
> fluid])

https://github.com/gramps-project/gramps/blob/master/.travis.yml

It just creates a clean python environment and runs setup.py build (line 38), nosetests3 (line 48) and codecov (line 51).

Regards,
John Ralls


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing and Coverage of Gramps

DS Blank
In reply to this post by DS Blank
On Fri, Mar 25, 2016 at 9:43 AM, Doug Blank <[hidden email]> wrote:
Test coverage is now up to 26%:

https://codecov.io/github/gramps-project/gramps/commits

I've also added some code to make it easier to test reports. See an example here: https://github.com/gramps-project/gramps/blob/master/gramps/plugins/test/test_reports.py#L122

Works like:

reports.addtest(TestDynamic, report_name, test_function, options_as_string)

Test coverage is now up over 30%... thanks to all of those that have written tests!


I suspect to get much higher, we'll need to mock the Gtk libs, or use some framework for doing similar.

-Doug
 

-Doug

On Sat, Feb 13, 2016 at 8:44 AM, Doug Blank <[hidden email]> wrote:
Devs,

I've just turned on "coverage" in our continuous integration testing on Travis. Coverage tells us which lines of code are being run in our tests. Coverage of lines of code doesn't guarantee that the code is correct, but does mean that it was executed in our tests.

I've also registered gramps with codecov.io, for easy viewing of what is covered, and what needs testing:

https://codecov.io/github/gramps-project/gramps

Currently, our coverage is at 22%. Our goal should probably be to get that to at least 70% or 80%. (Testing the Gtk GUI components will require some additional work, but it looks like there are projects designed for this.)

To write tests, you can look through any of the /test/ subdirectories for examples.

I've also written a wrapper around the Gramps CLI so you can write tests that will run the command-line arguments, but get the benefit of having the results included in nose (our test framework) and thus coverage. It works like:

from gramps.test.test_util import Gramps
g = Gramps()
out, err = g.run("-l")

where out is the standard-out as a string, and err is the standard-error. run() takes the command-line args, one at a time, as strings. You can put that in a test program, and use self.assertTrue() to check particular results in the output strings. This can test a big chunk of code all at once, but more specific tests will be better to check for specific results.

Hope this helps,

-Doug



------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing and Coverage of Gramps

DS Blank
On Sun, Apr 17, 2016 at 12:00 PM, Doug Blank <[hidden email]> wrote:
On Fri, Mar 25, 2016 at 9:43 AM, Doug Blank <[hidden email]> wrote:
Test coverage is now up to 26%:

https://codecov.io/github/gramps-project/gramps/commits

I've also added some code to make it easier to test reports. See an example here: https://github.com/gramps-project/gramps/blob/master/gramps/plugins/test/test_reports.py#L122

Works like:

reports.addtest(TestDynamic, report_name, test_function, options_as_string)

Test coverage is now up over 30%... thanks to all of those that have written tests!


I suspect to get much higher, we'll need to mock the Gtk libs, or use some framework for doing similar.

Thanks to new tests written by a number of developers, our percentage of coverage is now up to 36%:


This is still too little code being covered, but we are getting there (we started at about 20% 4 months ago). 

One can click on the interactive graph at the above URL to see where there are large areas of untested code. We haven't begun to test any GUI code... need to develop a framework for that.

-Doug

 

-Doug
 

-Doug

On Sat, Feb 13, 2016 at 8:44 AM, Doug Blank <[hidden email]> wrote:
Devs,

I've just turned on "coverage" in our continuous integration testing on Travis. Coverage tells us which lines of code are being run in our tests. Coverage of lines of code doesn't guarantee that the code is correct, but does mean that it was executed in our tests.

I've also registered gramps with codecov.io, for easy viewing of what is covered, and what needs testing:

https://codecov.io/github/gramps-project/gramps

Currently, our coverage is at 22%. Our goal should probably be to get that to at least 70% or 80%. (Testing the Gtk GUI components will require some additional work, but it looks like there are projects designed for this.)

To write tests, you can look through any of the /test/ subdirectories for examples.

I've also written a wrapper around the Gramps CLI so you can write tests that will run the command-line arguments, but get the benefit of having the results included in nose (our test framework) and thus coverage. It works like:

from gramps.test.test_util import Gramps
g = Gramps()
out, err = g.run("-l")

where out is the standard-out as a string, and err is the standard-error. run() takes the command-line args, one at a time, as strings. You can put that in a test program, and use self.assertTrue() to check particular results in the output strings. This can test a big chunk of code all at once, but more specific tests will be better to check for specific results.

Hope this helps,

-Doug




------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing and Coverage of Gramps

Nick Hall
On 02/06/16 21:26, Doug Blank wrote:

> Thanks to new tests written by a number of developers, our percentage
> of coverage is now up to 36%:
>
> https://codecov.io/gh/gramps-project/gramps
>
> This is still too little code being covered, but we are getting there
> (we started at about 20% 4 months ago).
>
> One can click on the interactive graph at the above URL to see where
> there are large areas of untested code. We haven't begun to test any
> GUI code... need to develop a framework for that.
>

Thanks.  This is a good start.

What plans do you have for GUI testing?


Nick.


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Testing and Coverage of Gramps

DS Blank
On Thu, Jun 2, 2016 at 5:37 PM, Nick Hall <[hidden email]> wrote:
On 02/06/16 21:26, Doug Blank wrote:
> Thanks to new tests written by a number of developers, our percentage
> of coverage is now up to 36%:
>
> https://codecov.io/gh/gramps-project/gramps
>
> This is still too little code being covered, but we are getting there
> (we started at about 20% 4 months ago).
>
> One can click on the interactive graph at the above URL to see where
> there are large areas of untested code. We haven't begun to test any
> GUI code... need to develop a framework for that.
>

Thanks.  This is a good start.

What plans do you have for GUI testing?

I don't have any plans... this is up for grabs for someone to explore, design, and build.

Some possibilities include:

* use mock to implement GUI methods
* use a Gtk testing framework (Kiwi, selenium)
* turn on the X framebuffer manager in Travis, and use methods shown here from Kiwi [1, 2]
* something else...

-Doug

 


Nick.


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel