Quantcast

Person & place selectors are horribly slow with large trees.

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Person & place selectors are horribly slow with large trees.

prculley
Gentlemen;

It seems that in the last few months that there has been some attempt to make some of the tree selectors open in the 'fully expanded' mode.

Looking over the bug tracker I note:
Someone did not like that for citations.
Others are (rightly) complaining that performance has suffered for Person and Place selectors.

While working with progress bars, I assembled a 15000 person, 13000 place, tree and discovered just how awful the performance is.

My take, opening a tree fully expanded is a -bad- idea.  If a user wants to work with a fully open tree I would think he should work in a list view instead.  The list view appears to have a lot better performance in general.  Of course we would have to give the user the option for the selectors, one way or another.

It appears that Jerome and Serge might be thinking about this already, so I don't want to preempt anything with a PR as yet.

As an interim step, has anyone considered keeping the tree view selectors collapsed on open, but using the Gtk.TreeView.set_hover_expand function?  I tried it out and it seems pretty user friendly, and is easier to use than clicking the expander icon IMHO.

P.S. I note that our tree model code has a lot of customization, compared to the standard Gtk TreeModel.  After profiling a bit, I suspect that some or all of that customization may be contributing to the performance issues, since it is python code instead of optimized 'C'.

I note that the customization is heading on to 12 years old now, seems to be originally started by Don Allingham back in 2004.

What is not clear, without a lot more study, is whether the customization is necessary today, or just there because historically Gtk did not have functions it does today.
If anyone has any insight into this, I would be interested.

Paul Culley

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Person & place selectors are horribly slow with large trees.

Nick Hall
On 16/01/17 16:12, Paul Culley wrote:

> P.S. I note that our tree model code has a lot of customization,
> compared to the standard Gtk TreeModel. After profiling a bit, I
> suspect that some or all of that customization may be contributing to
> the performance issues, since it is python code instead of optimized 'C'.
>
> I note that the customization is heading on to 12 years old now, seems
> to be originally started by Don Allingham back in 2004.
>
> What is not clear, without a lot more study, is whether the
> customization is necessary today, or just there because historically
> Gtk did not have functions it does today.
> If anyone has any insight into this, I would be interested.

The custom models only store two columns in memory, the other columns
are retrieved when needed.  A cache is used reduce database access and
improve performance.

The Gtk ListStore and TreeStore load all data into memory.  This
probably isn't a problem with modern hardware, but they also take longer
to load.  The load time is proportional to the number of cells.  In a
quick test, it took 3 secs to populate a 15,000 row and 10 column
ListStore with generated strings.  It would obviously take longer when
actually reading from a database.

In April 2015, I developed a prototype which used the standard Gtk
models.  To reduce the load time I decided to limit the number of rows
displayed.  For flat views, I decided to only display the results of a
filter instead of all rows by default.  I also rewrote the person and
citation tree views to use two flat views side-by-side.  The person view
displayed a list of surnames on the left, and all names matching the
surname selected on the right.  The citation views displayed sources on
the left and citations on the right.

Using standard Gtk models would make maintenance easier, but other
developers didn't like the idea.


Nick.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Person & place selectors are horribly slow with large trees.

jerome
In reply to this post by prculley
 > Using standard Gtk models
 > would make maintenance easier, but other
 > developers didn't like the idea.
 
Maybe because they had bad experiences in the past?

True, often related to TreeView and maybe some
gramps customizations, but some gtk/gnome choices
are sometimes strange! Like for our own interractive search
widget, it was a known issue under gtk, but never fixed.

An other annoying one has never be fixed on gtk2[1].
I suppose there is still some warnings
linked with this technology on gtk3, and maybe
current workaround could be something like:

$ export NO_AT_BRIDGE=1 | python3 Gramps.py

Maybe should also try to re-check ATK stuff on code[2]?
And remove a remaining old gnome technology[3]?

I only had a segfault on expand all nodes/rows into a tree view
with GObject < 3.4.0 and got minor gtk2 problems in the past[4].
Using gramps versions prior 4.x under Gnome2, I disabled most
ATK technologies under my desktop, but it was easy to reproduce
crash into TreeView with gtk2!


[1] https://gramps-project.org/wiki/index.php?title=Known_issues#Gramps_freezes_and_crashes_with_ATK_.2F_GNOME_.2F_GAIL_Assistive_Technologies_enabled
https://blog.inf.ed.ac.uk/chris/those-accessibility-warnings-again/
[2] https://gramps-project.org/bugs/view.php?id=6369
[3] https://gramps-project.org/bugs/view.php?id=4120
[4] https://gramps-project.org/bugs/view.php?id=5688


Jérôme
 
 
 ------------------------------------------------------------------------------
 Check out the vibrant tech community on one of
 the world's most
 engaging tech sites,
 SlashDot.org! http://sdm.link/slashdot
 _______________________________________________
 Gramps-devel mailing list
 [hidden email]
 https://lists.sourceforge.net/lists/listinfo/gramps-devel
 

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Person & place selectors are horribly slow with large trees.

Tim Lyons
Administrator
In reply to this post by Nick Hall
Nick Hall wrote
On 16/01/17 16:12, Paul Culley wrote:
> P.S. I note that our tree model code has a lot of customization,
> compared to the standard Gtk TreeModel. After profiling a bit, I
> suspect that some or all of that customization may be contributing to
> the performance issues, since it is python code instead of optimized 'C'.
>
> I note that the customization is heading on to 12 years old now, seems
> to be originally started by Don Allingham back in 2004.
>
> What is not clear, without a lot more study, is whether the
> customization is necessary today, or just there because historically
> Gtk did not have functions it does today.
> If anyone has any insight into this, I would be interested.

The custom models only store two columns in memory, the other columns
are retrieved when needed.  A cache is used reduce database access and
improve performance.

The Gtk ListStore and TreeStore load all data into memory.  This
probably isn't a problem with modern hardware, but they also take longer
to load.  The load time is proportional to the number of cells.  In a
quick test, it took 3 secs to populate a 15,000 row and 10 column
ListStore with generated strings.  It would obviously take longer when
actually reading from a database.

In April 2015, I developed a prototype which used the standard Gtk
models.  To reduce the load time I decided to limit the number of rows
displayed.  For flat views, I decided to only display the results of a
filter instead of all rows by default.  I also rewrote the person and
citation tree views to use two flat views side-by-side.  The person view
displayed a list of surnames on the left, and all names matching the
surname selected on the right.  The citation views displayed sources on
the left and citations on the right.

Using standard Gtk models would make maintenance easier, but other
developers didn't like the idea.

I didn't like the fact that with two flat side-by-side views you couldn't expand more than one surname at the same time (or source respectively).

In general, I like using a standard model, for all the normal reasons, like reduced maintenance, better able to take advantage of improvements in the standard model and it's always bad to fight what others have done.

I don't know whether conversion to side-by-side views was required with the standard model, or just something in parallel.

If the standard model loads all the data into memory, then it probably isn't much of a performance improvement.



I don't like the idea of expanding on hover - my observation of less experienced users is that they don't know why the expansion has suddenly appeared or what to do to get it to go away. Moving the cursor in what they think is the right direction to point at something else often makes the expansion just continue.

Tim.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Person & place selectors are horribly slow with large trees.

Nick Hall
On 21/01/17 23:41, Tim Lyons wrote:
In April 2015, I developed a prototype which used the standard Gtk 
models.  To reduce the load time I decided to limit the number of rows 
displayed.  For flat views, I decided to only display the results of a 
filter instead of all rows by default.  I also rewrote the person and 
citation tree views to use two flat views side-by-side.  The person view 
displayed a list of surnames on the left, and all names matching the 
surname selected on the right.  The citation views displayed sources on 
the left and citations on the right.

Using standard Gtk models would make maintenance easier, but other 
developers didn't like the idea.
I didn't like the fact that with two flat side-by-side views you couldn't
expand more than one surname at the same time (or source respectively).

In general, I like using a standard model, for all the normal reasons, like
reduced maintenance, better able to take advantage of improvements in the
standard model and it's always bad to fight what others have done.

I don't know whether conversion to side-by-side views was required with the
standard model, or just something in parallel.

The standard models are slower to load than our custom models.  This makes them unsuitable for selectors unless we reduce the number of rows displayed.  A selector with two side-by-side flat views is one way to do this.  For example, the person selector would only need to display all surnames in one view and all people matching the selected surname in the second view.

Performance of the standard models is good once the data is loaded.

Nick.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Person & place selectors are horribly slow with large trees.

jerome
In reply to this post by prculley
Someone made a pull request which could be a possible solution for 4.2.x branch.

https://gramps-project.org/bugs/view.php?id=9738


--------------------------------------------
En date de : Lun 23.1.17, Nick Hall <[hidden email]> a écrit :

 Objet: Re: [Gramps-devel] Person & place selectors are horribly slow with large trees.
 À: [hidden email]
 Date: Lundi 23 janvier 2017, 16h54
 
 
     On 21/01/17 23:41, Tim
 Lyons wrote:
 
     
     
       
         In April 2015, I developed a prototype which
 used the standard Gtk
 models.  To reduce the load time I decided to limit the
 number of rows
 displayed.  For flat views, I decided to only display the
 results of a
 filter instead of all rows by default.  I also rewrote the
 person and
 citation tree views to use two flat views side-by-side.  The
 person view
 displayed a list of surnames on the left, and all names
 matching the
 surname selected on the right.  The citation views displayed
 sources on
 the left and citations on the right.
 
 Using standard Gtk models would make maintenance easier, but
 other
 developers didn't like the idea.
 
       
       
 I didn't like the fact that with two flat side-by-side
 views you couldn't
 expand more than one surname at the same time (or source
 respectively).
 
 In general, I like using a standard model, for all the
 normal reasons, like
 reduced maintenance, better able to take advantage of
 improvements in the
 standard model and it's always bad to fight what others
 have done.
 
 I don't know whether conversion to side-by-side views
 was required with the
 standard model, or just something in parallel.
 
     
     The standard models are slower to load than our
 custom models. 
       This makes them unsuitable for selectors unless we
 reduce the
       number of rows displayed.  A selector with two
 side-by-side flat
       views is one way to do this.  For example, the person
 selector
       would only need to display all surnames in one view
 and all people
       matching the selected surname in the second view.
     Performance of the standard models is good once the
 data is
       loaded.
     Nick.
     
 
     
   
 -----La pièce jointe associée suit-----
 
 ------------------------------------------------------------------------------
 Check out the vibrant tech community on one of
 the world's most
 engaging tech sites,
 SlashDot.org! http://sdm.link/slashdot
 -----La pièce jointe associée suit-----
 
 _______________________________________________
 Gramps-devel mailing list
 [hidden email]
 https://lists.sourceforge.net/lists/listinfo/gramps-devel
 

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Person & place selectors are horribly slow with large trees.

Edmund
On , jerome wrote:
> Someone made a pull request which could be a possible solution for
> 4.2.x branch.
>
> https://gramps-project.org/bugs/view.php?id=9738
>

That would be me. :)

Can someone clarify how I can fix the two failing tests?

Thanks

Edmund

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Person & place selectors are horribly slow with large trees.

Rich Lakey
This post has NOT been accepted by the mailing list yet.
In reply to this post by prculley
The problem did not exist in 4.2.3. Perhaps that code maybe a simple solution?
2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Person & place selectors are horribly slow with large trees.

Edmund
In reply to this post by Edmund
On , Paul Culley wrote:
> There are three different tests being run on every commit and PR.  The
> most
> important is the Travis test, it runs actual tests written against a
> lot of
> our core code, to make sure that any changes don't break something.
> Unfortunately, the tests do very little against the GUI code, as we
> have
> not yet developed GUI tests yet (a hard and extensive job to volunteers
> don't usually want to do very much).

I have never done any GUI testing in python before so I do appreciate
some
guidance from anyone.

Like where to start or what framework to use.  I've looked at
https://wiki.python.org/moin/PythonTestingToolsTaxonomy#GUI_Testing_Tools
and am not sure how to go about with this.

Any help appreciated

Edmund

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Loading...