Profiling the GtkTreeView implemenation

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Profiling the GtkTreeView implemenation

Don Allingham
I've spent the evening optimizing the GtkTreeView implementation in
GRAMPS. I picked the worse case scenario, expanding all columns in the
tree.

The first run:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     5329   16.722    0.003   16.722    0.003 _PeopleModel.py:307(on_get_path)
    41241    0.340    0.000    0.340    0.000 _PeopleModel.py:359(on_iter_next)
    51899    0.177    0.000    0.177    0.000 _PeopleModel.py:378(on_iter_has_child)
    46570    0.158    0.000    0.158    0.000 _PeopleModel.py:371(on_iter_children)
     5329    0.016    0.000    0.016    0.000 _PeopleModel.py:322(on_get_iter)
        0    0.000             0.000          profile:0(profiler)

Obviously, on_get_path (a required function for the GtkTreeView was the
bottleneck. I found a few places to optimize on_get_path. The current
implementation shows:

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    41218    0.317    0.000    0.317    0.000 _PeopleModel.py:359(on_iter_next)
    51875    0.154    0.000    0.154    0.000 _PeopleModel.py:378(on_iter_has_child)
    46546    0.147    0.000    0.147    0.000 _PeopleModel.py:371(on_iter_children)
     5328    0.039    0.000    0.039    0.000 _PeopleModel.py:307(on_get_path)
     5329    0.015    0.000    0.015    0.000 _PeopleModel.py:322(on_get_iter)
        0    0.000             0.000          profile:0(profiler)

We dropped from 17.556 seconds to 0.672 seconds.

I think it was a successful evening. The display should update even
faster now.

Don

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Profiling the GtkTreeView implemenation

bm-7
Great Don,

that's quite an achievement

I nevertheless think we need a test database of 100.000+ people to test
it on. I
personally did not think people would have such databases, but from the
discussion it is clear they do.
Any ideas on how to make such a database? Just import example.gramps several
times? Or are there better ways?

The change you did will probably need some testing for stability and
such. Perhaps this is the moment to clean up some other view problems,
as there are
some bugs in category Person View, which indicate some errors in the treeview
iters and callback procedure. The pagedown/up issue Richard mentioned is also
there.

Benny

Quoting Don Allingham <[hidden email]>:

> I've spent the evening optimizing the GtkTreeView implementation in
> GRAMPS. I picked the worse case scenario, expanding all columns in the
> tree.
>
> The first run:
>
>   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>     5329   16.722    0.003   16.722    0.003 _PeopleModel.py:307(on_get_path)
>    41241    0.340    0.000    0.340    0.000
> _PeopleModel.py:359(on_iter_next)
>    51899    0.177    0.000    0.177    0.000
> _PeopleModel.py:378(on_iter_has_child)
>    46570    0.158    0.000    0.158    0.000
> _PeopleModel.py:371(on_iter_children)
>     5329    0.016    0.000    0.016    0.000 _PeopleModel.py:322(on_get_iter)
>        0    0.000             0.000          profile:0(profiler)
>
> Obviously, on_get_path (a required function for the GtkTreeView was the
> bottleneck. I found a few places to optimize on_get_path. The current
> implementation shows:
>
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>    41218    0.317    0.000    0.317    0.000
> _PeopleModel.py:359(on_iter_next)
>    51875    0.154    0.000    0.154    0.000
> _PeopleModel.py:378(on_iter_has_child)
>    46546    0.147    0.000    0.147    0.000
> _PeopleModel.py:371(on_iter_children)
>     5328    0.039    0.000    0.039    0.000 _PeopleModel.py:307(on_get_path)
>     5329    0.015    0.000    0.015    0.000 _PeopleModel.py:322(on_get_iter)
>        0    0.000             0.000          profile:0(profiler)
>
> We dropped from 17.556 seconds to 0.672 seconds.
>
> I think it was a successful evening. The display should update even
> faster now.
>
> Don
>



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Profiling the GtkTreeView implemenation

bm-7
In reply to this post by Don Allingham
I'll use this thread for some further remarks on the issue of BIG database.

Quoting Don Allingham <[hidden email]>:
[snip]
> We dropped from 17.556 seconds to 0.672 seconds.
>
> I think it was a successful evening. The display should update even
> faster now.

This is somewhat misleading as people speak of waiting 1 to 5min for startup,
and 5 sec to unfold a family in the view :-)

Kees, can you try your database again with the SVN version (Use a
BACKUP!!) and
test in practice? Don gives a reduction of 26 in time, that should mean  your
database should now open in 1 sec, no idea how it will impact unfolding. Don
expands all in his profiling if I understand correctly.

Some further ideas in case the present method still fails on large databases:
In the 90ties I programmed quite some database access. The practice at the
company I worked was that there was a custom widget available for view in a
scrolled window (no scrollbar however, only pageup/down, or buttons for
scrolling, +1,-1,...). Only 3times the amount that can be viewed was fetched
from database. The rest was only fetched if people actually start scrolling,
discarding previously fetched data
This means the window is visible immediately.
As a programmer you only had to implement 3 methods: something like
page(nrentriesonpage, keytop), pageup(nrentriesonpage, keytop),
pagedown(nrentriesonpage, keytop). This then fed the model that was used for
viewing (not GTK off course).
Of course, this means quite some changes (find must be implemented
differently,
no column sorting), but as an alternative view for large databases it should
work just fine, and have no noticable time issues.

Benny


----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Profiling the GtkTreeView implemenation

jerome
In reply to this post by Don Allingham
Hi,

With last SVN, I got two (GtkTreeView ???) issues:

1. Sometimes, on pedigree view, focus twinkle !!!

2. on my last seizure try, after sharing an existing source on event,
GRAMPS freezed every time : gtk.Entry object (GtkEntry) at 0x4364e6b,
0x4..., etc... I launch all "user-available" repair tools (tables, check
and repair, rebuild second indices) + I use a new database ... and seems
to be better :)

Without testcases (=first impression)... I thought it was:

1. my windows/dialog size (not full screen)
2. after removing an event with a primary reference (media)


Have you still have this issues ?
Is it possible to get more informations ? an existing GTK optimization
tool ?




A. windows size
Don Allingham a écrit :

> I've spent the evening optimizing the GtkTreeView implementation in
> GRAMPS. I picked the worse case scenario, expanding all columns in the
> tree.
>
> The first run:
>
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>      5329   16.722    0.003   16.722    0.003 _PeopleModel.py:307(on_get_path)
>     41241    0.340    0.000    0.340    0.000 _PeopleModel.py:359(on_iter_next)
>     51899    0.177    0.000    0.177    0.000 _PeopleModel.py:378(on_iter_has_child)
>     46570    0.158    0.000    0.158    0.000 _PeopleModel.py:371(on_iter_children)
>      5329    0.016    0.000    0.016    0.000 _PeopleModel.py:322(on_get_iter)
>         0    0.000             0.000          profile:0(profiler)
>
> Obviously, on_get_path (a required function for the GtkTreeView was the
> bottleneck. I found a few places to optimize on_get_path. The current
> implementation shows:
>
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>     41218    0.317    0.000    0.317    0.000 _PeopleModel.py:359(on_iter_next)
>     51875    0.154    0.000    0.154    0.000 _PeopleModel.py:378(on_iter_has_child)
>     46546    0.147    0.000    0.147    0.000 _PeopleModel.py:371(on_iter_children)
>      5328    0.039    0.000    0.039    0.000 _PeopleModel.py:307(on_get_path)
>      5329    0.015    0.000    0.015    0.000 _PeopleModel.py:322(on_get_iter)
>         0    0.000             0.000          profile:0(profiler)
>
> We dropped from 17.556 seconds to 0.672 seconds.
>
> I think it was a successful evening. The display should update even
> faster now.
>
> Don
>
>
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
--
Jérôme

       

       
               
___________________________________________________________________________
Yahoo! Mail r�invente le mail ! D�couvrez le nouveau Yahoo! Mail et son interface r�volutionnaire.
http://fr.mail.yahoo.com



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Profiling the GtkTreeView implemenation

Martin Hawlisch
In reply to this post by Don Allingham
Don,

the "Expand all nodes" feature is now significant slower than it was before.

I compared with older revision 7907. There klicking that takes about one second, then the whole view is redrawn in one go, and all is fine.
When updating to trunk again, all takes about 30 seconds. In that time the treeview is flickering and I can watch the nodes being expanded one by one.

"Collaps all nodes" does not have a problem.

All that was in a database of about 1600 people (A randomly googled gedcom  http://home.usit.net/~cmdaven/clyde.ged imported into a new grdb) with the view scrolled down to the bottom.

Cheers, Martin.

-------- Original-Nachricht --------
Datum: Mon, 15 Jan 2007 22:43:16 -0700
Von: Don Allingham <[hidden email]>
An: GRAMPS Development mailing list <[hidden email]>
Betreff: [Gramps-devel] Profiling the GtkTreeView implemenation

> I've spent the evening optimizing the GtkTreeView implementation in
> GRAMPS. I picked the worse case scenario, expanding all columns in the
> tree.
>
> We dropped from 17.556 seconds to 0.672 seconds.
>
> I think it was a successful evening. The display should update even
> faster now.
>
> Don

--
Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
Ideal für Modem und ISDN: http://www.gmx.net/de/go/smartsurfer

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Profiling the GtkTreeView implemenation

Richard Taylor-2
In reply to this post by bm-7
Benny

I too have done lots of database coding :-)

The primary issues with GtkTreeView is the way that it calculates how many
items there are in the data model. I think that the only reason it needs to
know this is so that it can size the scrollbar. The model has to implement a
method called get_iter_next(iter), this takes a reference to a row and
returns the reference to the next row, when there are no more rows it returns
None. When you set a model on a GtkTreeView the view repeatably calls
get_iter_next() until it gets None and it counts how many times it had to
call it. It uses this as the size of the model. (I my opinion this is
bonkers, there should be a way to tell the View how many rows there are in
the model directly, but this is not, at least I have not been able to find
it.)

Clearly this approach means that the get_iter_next will be called N times for
a model with N rows (or an index with N entries). So the best that can ever
be done with GtkTreeView is a algorithmic complexity of N. It also means that
what goes on in get_iter_next is also crucial. In Models/_FastModel.py I
developed an approach that means that the implementation of get_iter_next is:

    def on_iter_next(self, rowref):
        """
        Calculate the next iter at the same level in the tree.
        """

        # The length of the rowref (i.e. the number of elements in the path)
        # tells us the level in the tree.
        if len(rowref) == 1:

            # If we are at the top of the tree we just increment the
            # first element in the iter until we reach the total length.
            if rowref[0]+1 >= self._length:
                ret = None
            else:
                ret = [rowref[0]+1,]

        elif len(rowref) == 2:

            # If we are at the second level we first check to see if we
            # have the number of children of this row already in the cache
            if not self._num_children_cache.has_key(rowref[0]):

                # If not calculate the number of children and put it in the
cac\
he.
                self._num_children_cache[rowref[0]] =
self._cursor.get_n_childr\
en([rowref[0],])

            num_children = self._num_children_cache[rowref[0]]

            # Now increment the second element of the iter path until we
            # reach the number of children.
            if rowref[1]+1 < num_children:
                ret = [rowref[0],rowref[1]+1]
            else:
                ret = None
        else:
            # We only support two levels.
            ret = None

        return ret

len(rowref) == 1 when drawing the collapsed list and == 2 when expanding an
element.

You can see from this that, at least for the collapsed case, you can't get
much better.

The only way to improve on this performance is to implement a custom widget,
there is one already that uses a ListView and a separate ScrollBar
(http://www.earthenware-services.org/software/EarthenwareLibrary/Easy%20Grid)
but it look rather complicated to make use of last time I looked.

An alternative approach might be improve GtkTreeView and submit a patch back
to gtk, any budding C programmers fancy giving that a go?

Regards

Richard


On Tuesday 16 January 2007 09:10, [hidden email] wrote:

> Some further ideas in case the present method still fails on large
> databases: In the 90ties I programmed quite some database access. The
> practice at the company I worked was that there was a custom widget
> available for view in a scrolled window (no scrollbar however, only
> pageup/down, or buttons for scrolling, +1,-1,...). Only 3times the amount
> that can be viewed was fetched from database. The rest was only fetched if
> people actually start scrolling, discarding previously fetched data
> This means the window is visible immediately.
> As a programmer you only had to implement 3 methods: something like
> page(nrentriesonpage, keytop), pageup(nrentriesonpage, keytop),
> pagedown(nrentriesonpage, keytop). This then fed the model that was used
> for viewing (not GTK off course).
> Of course, this means quite some changes (find must be implemented
> differently,
> no column sorting), but as an alternative view for large databases it
> should work just fine, and have no noticable time issues.



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Profiling the GtkTreeView implemenation

Don Allingham
In reply to this post by Martin Hawlisch
This looks like a late night check in bug :-)

I was also playing with managing cursors to give a visual response, and
the cursor seems to be changing with every row expansion. I'll fix this
later today.

Don

On Tue, 2007-01-16 at 10:36 +0100, Martin Hawlisch wrote:

> Don,
>
> the "Expand all nodes" feature is now significant slower than it was before.
>
> I compared with older revision 7907. There klicking that takes about one second, then the whole view is redrawn in one go, and all is fine.
> When updating to trunk again, all takes about 30 seconds. In that time the treeview is flickering and I can watch the nodes being expanded one by one.
>
> "Collaps all nodes" does not have a problem.
>
> All that was in a database of about 1600 people (A randomly googled gedcom  http://home.usit.net/~cmdaven/clyde.ged imported into a new grdb) with the view scrolled down to the bottom.
>
> Cheers, Martin.
>
> -------- Original-Nachricht --------
> Datum: Mon, 15 Jan 2007 22:43:16 -0700
> Von: Don Allingham <[hidden email]>
> An: GRAMPS Development mailing list <[hidden email]>
> Betreff: [Gramps-devel] Profiling the GtkTreeView implemenation
>
> > I've spent the evening optimizing the GtkTreeView implementation in
> > GRAMPS. I picked the worse case scenario, expanding all columns in the
> > tree.
> >
> > We dropped from 17.556 seconds to 0.672 seconds.
> >
> > I think it was a successful evening. The display should update even
> > faster now.
> >
> > Don
>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (198 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Profiling the GtkTreeView implemenation

Don Allingham
In reply to this post by bm-7
No, this does not mean that everyone is going to get a 26x reduction in
time. What it means is that the Expand Nodes operation is going to get a
big reduction in time. I chose the Expand Nodes operation because it is
easy to profile (which is difficult in a multi-threaded application).
Other operations may benefit from the optimization I made, but it will
vary from operation to operation.

The object of profiling is to find items that are operations that can be
optimized. In this case, the on_get_path operation was a good candidate
for optimization. Everything that uses this operation will benefit, but
only dependent on the percentage of time that on_get_path was being
called before.

Richard is correct. The fundamental problem is the GtkTreeView. We
cannot reduce the number of calls that it is going to make. We can do
our best to reduce the time that each call will take, but fundamentally,
the number of calls is our limiting factor.

Don

On Tue, 2007-01-16 at 10:10 +0100, [hidden email] wrote:

> I'll use this thread for some further remarks on the issue of BIG database.
>
> Quoting Don Allingham <[hidden email]>:
> [snip]
> > We dropped from 17.556 seconds to 0.672 seconds.
> >
> > I think it was a successful evening. The display should update even
> > faster now.
>
> This is somewhat misleading as people speak of waiting 1 to 5min for startup,
> and 5 sec to unfold a family in the view :-)
>
> Kees, can you try your database again with the SVN version (Use a
> BACKUP!!) and
> test in practice? Don gives a reduction of 26 in time, that should mean  your
> database should now open in 1 sec, no idea how it will impact unfolding. Don
> expands all in his profiling if I understand correctly.
>
> Some further ideas in case the present method still fails on large databases:
> In the 90ties I programmed quite some database access. The practice at the
> company I worked was that there was a custom widget available for view in a
> scrolled window (no scrollbar however, only pageup/down, or buttons for
> scrolling, +1,-1,...). Only 3times the amount that can be viewed was fetched
> from database. The rest was only fetched if people actually start scrolling,
> discarding previously fetched data
> This means the window is visible immediately.
> As a programmer you only had to implement 3 methods: something like
> page(nrentriesonpage, keytop), pageup(nrentriesonpage, keytop),
> pagedown(nrentriesonpage, keytop). This then fed the model that was used for
> viewing (not GTK off course).
> Of course, this means quite some changes (find must be implemented
> differently,
> no column sorting), but as an alternative view for large databases it should
> work just fine, and have no noticable time issues.
>
> Benny
>
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (198 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Profiling the GtkTreeView implemenation

Douglas S. Blank
In reply to this post by Richard Taylor-2
I was just checking on my other BIG databases on my Linux desktop, and
wondering what they use for a toolkit. For example, my thunderbird inbox
is 852 MB, with a whopping 124,255 messages (just my mail since 2004, to
one account; I know, I know). It shows no signs of slowing, opens
instantaneously, and moves from front to back without any problem.

I don't know if use Gtk, or some custom widgets... I'll check.

I glanced at the Gtk TreeView C code. I don't think that one could jump
into that quickly...

-Doug

Richard Taylor wrote:

> Benny
>
> I too have done lots of database coding :-)
>
> The primary issues with GtkTreeView is the way that it calculates how many
> items there are in the data model. I think that the only reason it needs to
> know this is so that it can size the scrollbar. The model has to implement a
> method called get_iter_next(iter), this takes a reference to a row and
> returns the reference to the next row, when there are no more rows it returns
> None. When you set a model on a GtkTreeView the view repeatably calls
> get_iter_next() until it gets None and it counts how many times it had to
> call it. It uses this as the size of the model. (I my opinion this is
> bonkers, there should be a way to tell the View how many rows there are in
> the model directly, but this is not, at least I have not been able to find
> it.)
>
> Clearly this approach means that the get_iter_next will be called N times for
> a model with N rows (or an index with N entries). So the best that can ever
> be done with GtkTreeView is a algorithmic complexity of N. It also means that
> what goes on in get_iter_next is also crucial. In Models/_FastModel.py I
> developed an approach that means that the implementation of get_iter_next is:
>
>     def on_iter_next(self, rowref):
>         """
>         Calculate the next iter at the same level in the tree.
>         """
>
>         # The length of the rowref (i.e. the number of elements in the path)
>         # tells us the level in the tree.
>         if len(rowref) == 1:
>
>             # If we are at the top of the tree we just increment the
>             # first element in the iter until we reach the total length.
>             if rowref[0]+1 >= self._length:
>                 ret = None
>             else:
>                 ret = [rowref[0]+1,]
>
>         elif len(rowref) == 2:
>
>             # If we are at the second level we first check to see if we
>             # have the number of children of this row already in the cache
>             if not self._num_children_cache.has_key(rowref[0]):
>
>                 # If not calculate the number of children and put it in the
> cac\
> he.
>                 self._num_children_cache[rowref[0]] =
> self._cursor.get_n_childr\
> en([rowref[0],])
>
>             num_children = self._num_children_cache[rowref[0]]
>
>             # Now increment the second element of the iter path until we
>             # reach the number of children.
>             if rowref[1]+1 < num_children:
>                 ret = [rowref[0],rowref[1]+1]
>             else:
>                 ret = None
>         else:
>             # We only support two levels.
>             ret = None
>
>         return ret
>
> len(rowref) == 1 when drawing the collapsed list and == 2 when expanding an
> element.
>
> You can see from this that, at least for the collapsed case, you can't get
> much better.
>
> The only way to improve on this performance is to implement a custom widget,
> there is one already that uses a ListView and a separate ScrollBar
> (http://www.earthenware-services.org/software/EarthenwareLibrary/Easy%20Grid)
> but it look rather complicated to make use of last time I looked.
>
> An alternative approach might be improve GtkTreeView and submit a patch back
> to gtk, any budding C programmers fancy giving that a go?
>
> Regards
>
> Richard
>
>
> On Tuesday 16 January 2007 09:10, [hidden email] wrote:
>> Some further ideas in case the present method still fails on large
>> databases: In the 90ties I programmed quite some database access. The
>> practice at the company I worked was that there was a custom widget
>> available for view in a scrolled window (no scrollbar however, only
>> pageup/down, or buttons for scrolling, +1,-1,...). Only 3times the amount
>> that can be viewed was fetched from database. The rest was only fetched if
>> people actually start scrolling, discarding previously fetched data
>> This means the window is visible immediately.
>> As a programmer you only had to implement 3 methods: something like
>> page(nrentriesonpage, keytop), pageup(nrentriesonpage, keytop),
>> pagedown(nrentriesonpage, keytop). This then fed the model that was used
>> for viewing (not GTK off course).
>> Of course, this means quite some changes (find must be implemented
>> differently,
>> no column sorting), but as an alternative view for large databases it
>> should work just fine, and have no noticable time issues.
>
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Profiling the GtkTreeView implemenation

Don Allingham
Thunderbird does not use the GtkTreeView. It uses XUL.

Even Evolution does not use the GtkTreeView. It has a custom widget.

Don

On Tue, 2007-01-16 at 12:28 -0500, Douglas S. Blank wrote:

> I was just checking on my other BIG databases on my Linux desktop, and
> wondering what they use for a toolkit. For example, my thunderbird inbox
> is 852 MB, with a whopping 124,255 messages (just my mail since 2004, to
> one account; I know, I know). It shows no signs of slowing, opens
> instantaneously, and moves from front to back without any problem.
>
> I don't know if use Gtk, or some custom widgets... I'll check.
>
> I glanced at the Gtk TreeView C code. I don't think that one could jump
> into that quickly...
>
> -Doug
>
> Richard Taylor wrote:
> > Benny
> >
> > I too have done lots of database coding :-)
> >
> > The primary issues with GtkTreeView is the way that it calculates how many
> > items there are in the data model. I think that the only reason it needs to
> > know this is so that it can size the scrollbar. The model has to implement a
> > method called get_iter_next(iter), this takes a reference to a row and
> > returns the reference to the next row, when there are no more rows it returns
> > None. When you set a model on a GtkTreeView the view repeatably calls
> > get_iter_next() until it gets None and it counts how many times it had to
> > call it. It uses this as the size of the model. (I my opinion this is
> > bonkers, there should be a way to tell the View how many rows there are in
> > the model directly, but this is not, at least I have not been able to find
> > it.)
> >
> > Clearly this approach means that the get_iter_next will be called N times for
> > a model with N rows (or an index with N entries). So the best that can ever
> > be done with GtkTreeView is a algorithmic complexity of N. It also means that
> > what goes on in get_iter_next is also crucial. In Models/_FastModel.py I
> > developed an approach that means that the implementation of get_iter_next is:
> >
> >     def on_iter_next(self, rowref):
> >         """
> >         Calculate the next iter at the same level in the tree.
> >         """
> >
> >         # The length of the rowref (i.e. the number of elements in the path)
> >         # tells us the level in the tree.
> >         if len(rowref) == 1:
> >
> >             # If we are at the top of the tree we just increment the
> >             # first element in the iter until we reach the total length.
> >             if rowref[0]+1 >= self._length:
> >                 ret = None
> >             else:
> >                 ret = [rowref[0]+1,]
> >
> >         elif len(rowref) == 2:
> >
> >             # If we are at the second level we first check to see if we
> >             # have the number of children of this row already in the cache
> >             if not self._num_children_cache.has_key(rowref[0]):
> >
> >                 # If not calculate the number of children and put it in the
> > cac\
> > he.
> >                 self._num_children_cache[rowref[0]] =
> > self._cursor.get_n_childr\
> > en([rowref[0],])
> >
> >             num_children = self._num_children_cache[rowref[0]]
> >
> >             # Now increment the second element of the iter path until we
> >             # reach the number of children.
> >             if rowref[1]+1 < num_children:
> >                 ret = [rowref[0],rowref[1]+1]
> >             else:
> >                 ret = None
> >         else:
> >             # We only support two levels.
> >             ret = None
> >
> >         return ret
> >
> > len(rowref) == 1 when drawing the collapsed list and == 2 when expanding an
> > element.
> >
> > You can see from this that, at least for the collapsed case, you can't get
> > much better.
> >
> > The only way to improve on this performance is to implement a custom widget,
> > there is one already that uses a ListView and a separate ScrollBar
> > (http://www.earthenware-services.org/software/EarthenwareLibrary/Easy%20Grid)
> > but it look rather complicated to make use of last time I looked.
> >
> > An alternative approach might be improve GtkTreeView and submit a patch back
> > to gtk, any budding C programmers fancy giving that a go?
> >
> > Regards
> >
> > Richard
> >
> >
> > On Tuesday 16 January 2007 09:10, [hidden email] wrote:
> >> Some further ideas in case the present method still fails on large
> >> databases: In the 90ties I programmed quite some database access. The
> >> practice at the company I worked was that there was a custom widget
> >> available for view in a scrolled window (no scrollbar however, only
> >> pageup/down, or buttons for scrolling, +1,-1,...). Only 3times the amount
> >> that can be viewed was fetched from database. The rest was only fetched if
> >> people actually start scrolling, discarding previously fetched data
> >> This means the window is visible immediately.
> >> As a programmer you only had to implement 3 methods: something like
> >> page(nrentriesonpage, keytop), pageup(nrentriesonpage, keytop),
> >> pagedown(nrentriesonpage, keytop). This then fed the model that was used
> >> for viewing (not GTK off course).
> >> Of course, this means quite some changes (find must be implemented
> >> differently,
> >> no column sorting), but as an alternative view for large databases it
> >> should work just fine, and have no noticable time issues.
> >
> >
> >
> > -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > opinions on IT & business topics through brief surveys - and earn cash
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > Gramps-devel mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/gramps-devel
> >
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (198 bytes) Download Attachment