More on profiling

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

More on profiling

Don Allingham
Yesterday, I did an analysis on row expansion in GtkTreeView. This was
the result of several people saying that it took too long to open a sub
node (opening a Family name to see the people underneath). Profiling
showed an issue we could easily fix, and I did. As a result, expanding
names is new very quick.

Reviewing the profiling results of the full list build, we see a
different problem.

Looking at the results indicates that the majority of the time goes to
building the name for the person in the display. The actual tree
functions are much further down on the list.

This is a result of the user requirement for virtually infinite
flexibility in name display. A lot time has already been reduced by the
Lru module, which caches the last 500 built names.

Unless we can find some magical operation to reduce the time to build a
name while keeping flexibility, we are limited in what we can do here.

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    23654    1.459    0.000    1.459    0.000 NameDisplay.py:205(_format_str_base)
    23654    0.640    0.000    0.743    0.000 Lru.py:39(__setitem__)
    11829    0.583    0.000    0.583    0.000 dbshelve.py:167(get)
    11827    0.561    0.000    0.561    0.000 _GrampsBSDDB.py:93(next)
    12267    0.506    0.000    3.610    0.000 _PeopleModel.py:332(on_get_value)
        1    0.452    0.452    6.368    6.368 _PeopleModel.py:203(_build_search_sub)
    11827    0.349    0.000    1.900    0.000 _PeopleModel.py:433(column_name)
    23654    0.297    0.000    2.271    0.000 NameDisplay.py:291(raw_sorted_name)
    11827    0.242    0.000    0.242    0.000 _GrampsDbBase.py:1063(get_name_group_mapping)
    23654    0.227    0.000    1.686    0.000 NameDisplay.py:194(format_str_raw)
    23654    0.191    0.000    1.877    0.000 NameDisplay.py:101(<lambda>)
    11827    0.150    0.000    3.890    0.000 _SearchFilter.py:33(match)
    11827    0.148    0.000    0.731    0.000 _GrampsBSDDB.py:236(get_raw_person_data)
     2488    0.142    0.000    0.142    0.000 _PeopleModel.py:277(build_sub_entry)
    11827    0.132    0.000    3.740    0.000 _PeopleModel.py:171(<lambda>)
    11827    0.130    0.000    0.372    0.000 NameDisplay.py:370(name_grouping_data)
      2/1    0.112    0.056    0.118    0.118 DisplayState.py:331(modify_statusbar)
    23654    0.103    0.000    0.103    0.000 Lru.py:21(__init__)
    23654    0.102    0.000    0.102    0.000 Lru.py:36(__getitem__)
    23654    0.097    0.000    0.097    0.000 NameDisplay.py:179(_is_format_valid)
        1    0.057    0.057    6.785    6.785 _PersonView.py:521(build_tree)
     2488    0.023    0.000    0.023    0.000 _PeopleModel.py:377(on_iter_has_child)
        1    0.023    0.023    0.023    0.023 _PeopleModel.py:91(locale_sort)
     2513    0.016    0.000    0.016    0.000 _PeopleModel.py:359(on_iter_next)
        1    0.016    0.016    6.549    6.549 _PeopleModel.py:247(calculate_data)
        1    0.007    0.007    0.008    0.008 _PersonView.py:674(build_columns)
        1    0.005    0.005    0.012    0.012 _SearchBar.py:75(setup_filter)
      144    0.004    0.000    0.004    0.000 posixpath.py:168(exists)
      441    0.002    0.000    0.002    0.000 _PeopleModel.py:319(on_get_column_type)
       12    0.002    0.000    0.008    0.001 gettext.py:408(find)
        1    0.002    0.002    0.002    0.002 _PeopleModel.py:289(assign_data)
       96    0.001    0.000    0.001    0.000 posixpath.py:56(join)
       24    0.001    0.000    0.001    0.000 gettext.py:127(_expand_lang)
       44    0.000    0.000    0.000    0.000 _PeopleModel.py:625(column_header)
       12    0.000    0.000    0.008    0.001 gettext.py:472(translation)
       12    0.000    0.000    0.008    0.001 gettext.py:538(dgettext)
       12    0.000    0.000    0.000    0.000 UserDict.py:52(get)
       24    0.000    0.000    0.000    0.000 locale.py:197(normalize)
        1    0.000    0.000    0.000    0.000 _PersonView.py:846(get_selected_objects)
        1    0.000    0.000    0.000    0.000 _GrampsBSDDB.py:87(first)
        1    0.000    0.000    6.551    6.551 _PeopleModel.py:136(__init__)
       12    0.000    0.000    0.008    0.001 gettext.py:576(gettext)
        2    0.000    0.000    0.000    0.000 _GrampsDbBase.py:1959(_get_column_order)
        2    0.000    0.000    0.000    0.000 _PedigreeView.py:624(size_request_cb)
        1    0.000    0.000    0.000    0.000 _SearchBar.py:130(get_value)
        6    0.000    0.000    0.000    0.000 _PeopleModel.py:322(on_get_iter)


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (198 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More on profiling

Douglas S. Blank
Don Allingham wrote:

> Yesterday, I did an analysis on row expansion in GtkTreeView. This was
> the result of several people saying that it took too long to open a sub
> node (opening a Family name to see the people underneath). Profiling
> showed an issue we could easily fix, and I did. As a result, expanding
> names is new very quick.
>
> Reviewing the profiling results of the full list build, we see a
> different problem.
>
> Looking at the results indicates that the majority of the time goes to
> building the name for the person in the display. The actual tree
> functions are much further down on the list.
>
> This is a result of the user requirement for virtually infinite
> flexibility in name display. A lot time has already been reduced by the
> Lru module, which caches the last 500 built names.
>
> Unless we can find some magical operation to reduce the time to build a
> name while keeping flexibility, we are limited in what we can do here.

Don,

Couldn't the display_name be a field in the table so that we don't have
to build it each time we need it? If the user changes what is displayed,
then we just rebuild it once, and save it. (On a related note, maybe
person_names should be its own table, so that we could include alternate
names in the list as well.) The date display field could also be
pre-computed. Or am I misunderstanding the problem?

-Doug


>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>     23654    1.459    0.000    1.459    0.000 NameDisplay.py:205(_format_str_base)
>     23654    0.640    0.000    0.743    0.000 Lru.py:39(__setitem__)
>     11829    0.583    0.000    0.583    0.000 dbshelve.py:167(get)
>     11827    0.561    0.000    0.561    0.000 _GrampsBSDDB.py:93(next)
>     12267    0.506    0.000    3.610    0.000 _PeopleModel.py:332(on_get_value)
>         1    0.452    0.452    6.368    6.368 _PeopleModel.py:203(_build_search_sub)
>     11827    0.349    0.000    1.900    0.000 _PeopleModel.py:433(column_name)
>     23654    0.297    0.000    2.271    0.000 NameDisplay.py:291(raw_sorted_name)
>     11827    0.242    0.000    0.242    0.000 _GrampsDbBase.py:1063(get_name_group_mapping)
>     23654    0.227    0.000    1.686    0.000 NameDisplay.py:194(format_str_raw)
>     23654    0.191    0.000    1.877    0.000 NameDisplay.py:101(<lambda>)
>     11827    0.150    0.000    3.890    0.000 _SearchFilter.py:33(match)
>     11827    0.148    0.000    0.731    0.000 _GrampsBSDDB.py:236(get_raw_person_data)
>      2488    0.142    0.000    0.142    0.000 _PeopleModel.py:277(build_sub_entry)
>     11827    0.132    0.000    3.740    0.000 _PeopleModel.py:171(<lambda>)
>     11827    0.130    0.000    0.372    0.000 NameDisplay.py:370(name_grouping_data)
>       2/1    0.112    0.056    0.118    0.118 DisplayState.py:331(modify_statusbar)
>     23654    0.103    0.000    0.103    0.000 Lru.py:21(__init__)
>     23654    0.102    0.000    0.102    0.000 Lru.py:36(__getitem__)
>     23654    0.097    0.000    0.097    0.000 NameDisplay.py:179(_is_format_valid)
>         1    0.057    0.057    6.785    6.785 _PersonView.py:521(build_tree)
>      2488    0.023    0.000    0.023    0.000 _PeopleModel.py:377(on_iter_has_child)
>         1    0.023    0.023    0.023    0.023 _PeopleModel.py:91(locale_sort)
>      2513    0.016    0.000    0.016    0.000 _PeopleModel.py:359(on_iter_next)
>         1    0.016    0.016    6.549    6.549 _PeopleModel.py:247(calculate_data)
>         1    0.007    0.007    0.008    0.008 _PersonView.py:674(build_columns)
>         1    0.005    0.005    0.012    0.012 _SearchBar.py:75(setup_filter)
>       144    0.004    0.000    0.004    0.000 posixpath.py:168(exists)
>       441    0.002    0.000    0.002    0.000 _PeopleModel.py:319(on_get_column_type)
>        12    0.002    0.000    0.008    0.001 gettext.py:408(find)
>         1    0.002    0.002    0.002    0.002 _PeopleModel.py:289(assign_data)
>        96    0.001    0.000    0.001    0.000 posixpath.py:56(join)
>        24    0.001    0.000    0.001    0.000 gettext.py:127(_expand_lang)
>        44    0.000    0.000    0.000    0.000 _PeopleModel.py:625(column_header)
>        12    0.000    0.000    0.008    0.001 gettext.py:472(translation)
>        12    0.000    0.000    0.008    0.001 gettext.py:538(dgettext)
>        12    0.000    0.000    0.000    0.000 UserDict.py:52(get)
>        24    0.000    0.000    0.000    0.000 locale.py:197(normalize)
>         1    0.000    0.000    0.000    0.000 _PersonView.py:846(get_selected_objects)
>         1    0.000    0.000    0.000    0.000 _GrampsBSDDB.py:87(first)
>         1    0.000    0.000    6.551    6.551 _PeopleModel.py:136(__init__)
>        12    0.000    0.000    0.008    0.001 gettext.py:576(gettext)
>         2    0.000    0.000    0.000    0.000 _GrampsDbBase.py:1959(_get_column_order)
>         2    0.000    0.000    0.000    0.000 _PedigreeView.py:624(size_request_cb)
>         1    0.000    0.000    0.000    0.000 _SearchBar.py:130(get_value)
>         6    0.000    0.000    0.000    0.000 _PeopleModel.py:322(on_get_iter)
>
>
>
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: More on profiling

Don Allingham
Doug,

I can understand the impulse to put new entries and tables in the
database. I've thought of this myself (actually, I implemented it once).

Unfortunately, is usually boils down to being not so good in the
implementation. We have several different methods of handling names, so
a single name alone wouldn't work. For example, the tree node (what
looks like the last name, but is actually a "grouping name") is one such
name, and the display name (the expanded name as the leaves of the tree)
are another.  We have quite a few other places where different name
styles are used as well, including full customization of the names. But
assuming we ignore these, we now have two additional names to store in
the database purely for the interface display.

So now we have data and tables in the database whose sole purpose is to
make the interface's job easier. We now have additional tables that have
to be maintained. Since we don't have a full blown db to handle this for
us, we have to maintain it ourselves. Every person edit now requires us
to modify not only the person "table", but a name grouping table, and a
same sort table. If someone changes their desired format, we now have to
walk through the db updating all the grouping and display names.

So, in general, I would want to use this as a last resort.

Don

On Tue, 2007-01-16 at 12:17 -0500, Douglas S. Blank wrote:

> Don Allingham wrote:
> > Yesterday, I did an analysis on row expansion in GtkTreeView. This was
> > the result of several people saying that it took too long to open a sub
> > node (opening a Family name to see the people underneath). Profiling
> > showed an issue we could easily fix, and I did. As a result, expanding
> > names is new very quick.
> >
> > Reviewing the profiling results of the full list build, we see a
> > different problem.
> >
> > Looking at the results indicates that the majority of the time goes to
> > building the name for the person in the display. The actual tree
> > functions are much further down on the list.
> >
> > This is a result of the user requirement for virtually infinite
> > flexibility in name display. A lot time has already been reduced by the
> > Lru module, which caches the last 500 built names.
> >
> > Unless we can find some magical operation to reduce the time to build a
> > name while keeping flexibility, we are limited in what we can do here.
>
> Don,
>
> Couldn't the display_name be a field in the table so that we don't have
> to build it each time we need it? If the user changes what is displayed,
> then we just rebuild it once, and save it. (On a related note, maybe
> person_names should be its own table, so that we could include alternate
> names in the list as well.) The date display field could also be
> pre-computed. Or am I misunderstanding the problem?
>
> -Doug
>
>
> >    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
> >     23654    1.459    0.000    1.459    0.000 NameDisplay.py:205(_format_str_base)
> >     23654    0.640    0.000    0.743    0.000 Lru.py:39(__setitem__)
> >     11829    0.583    0.000    0.583    0.000 dbshelve.py:167(get)
> >     11827    0.561    0.000    0.561    0.000 _GrampsBSDDB.py:93(next)
> >     12267    0.506    0.000    3.610    0.000 _PeopleModel.py:332(on_get_value)
> >         1    0.452    0.452    6.368    6.368 _PeopleModel.py:203(_build_search_sub)
> >     11827    0.349    0.000    1.900    0.000 _PeopleModel.py:433(column_name)
> >     23654    0.297    0.000    2.271    0.000 NameDisplay.py:291(raw_sorted_name)
> >     11827    0.242    0.000    0.242    0.000 _GrampsDbBase.py:1063(get_name_group_mapping)
> >     23654    0.227    0.000    1.686    0.000 NameDisplay.py:194(format_str_raw)
> >     23654    0.191    0.000    1.877    0.000 NameDisplay.py:101(<lambda>)
> >     11827    0.150    0.000    3.890    0.000 _SearchFilter.py:33(match)
> >     11827    0.148    0.000    0.731    0.000 _GrampsBSDDB.py:236(get_raw_person_data)
> >      2488    0.142    0.000    0.142    0.000 _PeopleModel.py:277(build_sub_entry)
> >     11827    0.132    0.000    3.740    0.000 _PeopleModel.py:171(<lambda>)
> >     11827    0.130    0.000    0.372    0.000 NameDisplay.py:370(name_grouping_data)
> >       2/1    0.112    0.056    0.118    0.118 DisplayState.py:331(modify_statusbar)
> >     23654    0.103    0.000    0.103    0.000 Lru.py:21(__init__)
> >     23654    0.102    0.000    0.102    0.000 Lru.py:36(__getitem__)
> >     23654    0.097    0.000    0.097    0.000 NameDisplay.py:179(_is_format_valid)
> >         1    0.057    0.057    6.785    6.785 _PersonView.py:521(build_tree)
> >      2488    0.023    0.000    0.023    0.000 _PeopleModel.py:377(on_iter_has_child)
> >         1    0.023    0.023    0.023    0.023 _PeopleModel.py:91(locale_sort)
> >      2513    0.016    0.000    0.016    0.000 _PeopleModel.py:359(on_iter_next)
> >         1    0.016    0.016    6.549    6.549 _PeopleModel.py:247(calculate_data)
> >         1    0.007    0.007    0.008    0.008 _PersonView.py:674(build_columns)
> >         1    0.005    0.005    0.012    0.012 _SearchBar.py:75(setup_filter)
> >       144    0.004    0.000    0.004    0.000 posixpath.py:168(exists)
> >       441    0.002    0.000    0.002    0.000 _PeopleModel.py:319(on_get_column_type)
> >        12    0.002    0.000    0.008    0.001 gettext.py:408(find)
> >         1    0.002    0.002    0.002    0.002 _PeopleModel.py:289(assign_data)
> >        96    0.001    0.000    0.001    0.000 posixpath.py:56(join)
> >        24    0.001    0.000    0.001    0.000 gettext.py:127(_expand_lang)
> >        44    0.000    0.000    0.000    0.000 _PeopleModel.py:625(column_header)
> >        12    0.000    0.000    0.008    0.001 gettext.py:472(translation)
> >        12    0.000    0.000    0.008    0.001 gettext.py:538(dgettext)
> >        12    0.000    0.000    0.000    0.000 UserDict.py:52(get)
> >        24    0.000    0.000    0.000    0.000 locale.py:197(normalize)
> >         1    0.000    0.000    0.000    0.000 _PersonView.py:846(get_selected_objects)
> >         1    0.000    0.000    0.000    0.000 _GrampsBSDDB.py:87(first)
> >         1    0.000    0.000    6.551    6.551 _PeopleModel.py:136(__init__)
> >        12    0.000    0.000    0.008    0.001 gettext.py:576(gettext)
> >         2    0.000    0.000    0.000    0.000 _GrampsDbBase.py:1959(_get_column_order)
> >         2    0.000    0.000    0.000    0.000 _PedigreeView.py:624(size_request_cb)
> >         1    0.000    0.000    0.000    0.000 _SearchBar.py:130(get_value)
> >         6    0.000    0.000    0.000    0.000 _PeopleModel.py:322(on_get_iter)
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > opinions on IT & business topics through brief surveys - and earn cash
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Gramps-devel mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/gramps-devel
>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (198 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More on profiling

bm-7
Nevertheless,

It is only normal to have a key on the most important view in your
application.
The suggestion Doug had, and me too, (and probably everyone looking
into it), to
make a supper fast access method, independent of file size, is only possible
with such a construction.
You need to be able to retrieve the following.
get_next_n_occurences(namebegin, n)
returning the n names after namebegin, by only hitting the database once. So a
table 'groupname, displayname, handle', with key on groupname, and secondary
key on handle (as a reference).

It is the only way to present a view independent of database size I can think
of.

Richard gave a link to the not anymore developed easygrid treeview. I
looked at
that and it has all the ingredients we used at my previous job to show
treeviews: quick access to the the database tables, a treeview without sliders
and a small view-model with only the contents that is seen on screen, and a
slider wich is connected to the database, and whose events update the
view-model by accessing the database. In short, the database only gets hit for
the data on screen, and the slider is correct as it is initialized by the
database, not by what you see on screen.

Benny

Quoting Don Allingham <[hidden email]>:

> Doug,
>
> I can understand the impulse to put new entries and tables in the
> database. I've thought of this myself (actually, I implemented it once).
>
> Unfortunately, is usually boils down to being not so good in the
> implementation. We have several different methods of handling names, so
> a single name alone wouldn't work. For example, the tree node (what
> looks like the last name, but is actually a "grouping name") is one such
> name, and the display name (the expanded name as the leaves of the tree)
> are another.  We have quite a few other places where different name
> styles are used as well, including full customization of the names. But
> assuming we ignore these, we now have two additional names to store in
> the database purely for the interface display.
>
> So now we have data and tables in the database whose sole purpose is to
> make the interface's job easier. We now have additional tables that have
> to be maintained. Since we don't have a full blown db to handle this for
> us, we have to maintain it ourselves. Every person edit now requires us
> to modify not only the person "table", but a name grouping table, and a
> same sort table. If someone changes their desired format, we now have to
> walk through the db updating all the grouping and display names.
>
> So, in general, I would want to use this as a last resort.
>
> Don
>
> On Tue, 2007-01-16 at 12:17 -0500, Douglas S. Blank wrote:
>> Don Allingham wrote:
>> > Yesterday, I did an analysis on row expansion in GtkTreeView. This was
>> > the result of several people saying that it took too long to open a sub
>> > node (opening a Family name to see the people underneath). Profiling
>> > showed an issue we could easily fix, and I did. As a result, expanding
>> > names is new very quick.
>> >
>> > Reviewing the profiling results of the full list build, we see a
>> > different problem.
>> >
>> > Looking at the results indicates that the majority of the time goes to
>> > building the name for the person in the display. The actual tree
>> > functions are much further down on the list.
>> >
>> > This is a result of the user requirement for virtually infinite
>> > flexibility in name display. A lot time has already been reduced by the
>> > Lru module, which caches the last 500 built names.
>> >
>> > Unless we can find some magical operation to reduce the time to build a
>> > name while keeping flexibility, we are limited in what we can do here.
>>
>> Don,
>>
>> Couldn't the display_name be a field in the table so that we don't have
>> to build it each time we need it? If the user changes what is displayed,
>> then we just rebuild it once, and save it. (On a related note, maybe
>> person_names should be its own table, so that we could include alternate
>> names in the list as well.) The date display field could also be
>> pre-computed. Or am I misunderstanding the problem?
>>
>> -Doug
>>
>>
>> >    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>> >     23654    1.459    0.000    1.459    0.000
>> NameDisplay.py:205(_format_str_base)
>> >     23654    0.640    0.000    0.743    0.000 Lru.py:39(__setitem__)
>> >     11829    0.583    0.000    0.583    0.000 dbshelve.py:167(get)
>> >     11827    0.561    0.000    0.561    0.000 _GrampsBSDDB.py:93(next)
>> >     12267    0.506    0.000    3.610    0.000
>> _PeopleModel.py:332(on_get_value)
>> >         1    0.452    0.452    6.368    6.368
>> _PeopleModel.py:203(_build_search_sub)
>> >     11827    0.349    0.000    1.900    0.000
>> _PeopleModel.py:433(column_name)
>> >     23654    0.297    0.000    2.271    0.000
>> NameDisplay.py:291(raw_sorted_name)
>> >     11827    0.242    0.000    0.242    0.000
>> _GrampsDbBase.py:1063(get_name_group_mapping)
>> >     23654    0.227    0.000    1.686    0.000
>> NameDisplay.py:194(format_str_raw)
>> >     23654    0.191    0.000    1.877    0.000 NameDisplay.py:101(<lambda>)
>> >     11827    0.150    0.000    3.890    0.000 _SearchFilter.py:33(match)
>> >     11827    0.148    0.000    0.731    0.000
>> _GrampsBSDDB.py:236(get_raw_person_data)
>> >      2488    0.142    0.000    0.142    0.000
>> _PeopleModel.py:277(build_sub_entry)
>> >     11827    0.132    0.000    3.740    0.000
>> _PeopleModel.py:171(<lambda>)
>> >     11827    0.130    0.000    0.372    0.000
>> NameDisplay.py:370(name_grouping_data)
>> >       2/1    0.112    0.056    0.118    0.118
>> DisplayState.py:331(modify_statusbar)
>> >     23654    0.103    0.000    0.103    0.000 Lru.py:21(__init__)
>> >     23654    0.102    0.000    0.102    0.000 Lru.py:36(__getitem__)
>> >     23654    0.097    0.000    0.097    0.000
>> NameDisplay.py:179(_is_format_valid)
>> >         1    0.057    0.057    6.785    6.785
>> _PersonView.py:521(build_tree)
>> >      2488    0.023    0.000    0.023    0.000
>> _PeopleModel.py:377(on_iter_has_child)
>> >         1    0.023    0.023    0.023    0.023
>> _PeopleModel.py:91(locale_sort)
>> >      2513    0.016    0.000    0.016    0.000
>> _PeopleModel.py:359(on_iter_next)
>> >         1    0.016    0.016    6.549    6.549
>> _PeopleModel.py:247(calculate_data)
>> >         1    0.007    0.007    0.008    0.008
>> _PersonView.py:674(build_columns)
>> >         1    0.005    0.005    0.012    0.012
>> _SearchBar.py:75(setup_filter)
>> >       144    0.004    0.000    0.004    0.000 posixpath.py:168(exists)
>> >       441    0.002    0.000    0.002    0.000
>> _PeopleModel.py:319(on_get_column_type)
>> >        12    0.002    0.000    0.008    0.001 gettext.py:408(find)
>> >         1    0.002    0.002    0.002    0.002
>> _PeopleModel.py:289(assign_data)
>> >        96    0.001    0.000    0.001    0.000 posixpath.py:56(join)
>> >        24    0.001    0.000    0.001    0.000 gettext.py:127(_expand_lang)
>> >        44    0.000    0.000    0.000    0.000
>> _PeopleModel.py:625(column_header)
>> >        12    0.000    0.000    0.008    0.001 gettext.py:472(translation)
>> >        12    0.000    0.000    0.008    0.001 gettext.py:538(dgettext)
>> >        12    0.000    0.000    0.000    0.000 UserDict.py:52(get)
>> >        24    0.000    0.000    0.000    0.000 locale.py:197(normalize)
>> >         1    0.000    0.000    0.000    0.000
>> _PersonView.py:846(get_selected_objects)
>> >         1    0.000    0.000    0.000    0.000 _GrampsBSDDB.py:87(first)
>> >         1    0.000    0.000    6.551    6.551
>> _PeopleModel.py:136(__init__)
>> >        12    0.000    0.000    0.008    0.001 gettext.py:576(gettext)
>> >         2    0.000    0.000    0.000    0.000
>> _GrampsDbBase.py:1959(_get_column_order)
>> >         2    0.000    0.000    0.000    0.000
>> _PedigreeView.py:624(size_request_cb)
>> >         1    0.000    0.000    0.000    0.000 _SearchBar.py:130(get_value)
>> >         6    0.000    0.000    0.000    0.000
>> _PeopleModel.py:322(on_get_iter)
>> >
>> >
>> >
>> > ------------------------------------------------------------------------
>> >
>> > -------------------------------------------------------------------------
>> > Take Surveys. Earn Cash. Influence the Future of IT
>> > Join SourceForge.net's Techsay panel and you'll get the chance to
>> share your
>> > opinions on IT & business topics through brief surveys - and earn cash
>> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>> >
>> >
>> > ------------------------------------------------------------------------
>> >
>> > _______________________________________________
>> > Gramps-devel mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>
>



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: More on profiling

Alex Roitman
Benny,

On Tue, 2007-01-16 at 20:35 +0100, [hidden email] wrote:
> It is only normal to have a key on the most important view in your
> application.
> The suggestion Doug had, and me too, (and probably everyone looking
> into it), to
> make a supper fast access method, independent of file size, is only possible
> with such a construction.

No, not only with such a construction. A simple alternative
would be to have a fixed name for the purpose of using in the
people view. It would still be assembled on the fly from the
db records, but it would not be infinitely flexible like it
is now. For example, use FirstName LastName for people view,
period. Any customizations are effective everywhere else,
except the people view. This can be made optional, so the large
db users will trade off the convenience for performance,
but the small db users can stay with the luxury of flexible names.
We could even have a few common presets that are hard-coded.

> You need to be able to retrieve the following.
> get_next_n_occurences(namebegin, n)
> returning the n names after namebegin, by only hitting the database once.

All the name components are in a single person record, retrieved
by one db read. Look at Don's profiling results: db access was not
a problem. The consequential replacement of %t with title and %p
with prefix and %y with patronymic and so on, was the problem.
We retrieve records from the db amazingly fast, but then we crawl
to build the name. Because we pay for the flexibility.

With pre-set name display, all we need to do is:
   person_tuple = db.get(handle)
   name = "%s %s" % (person_tuple[3][2], person_tuple[3][3])
Of course, I made up the tuple indices above, but this is
what it would take, and it takes no time.

>  So a
> table 'groupname, displayname, handle', with key on groupname, and secondary
> key on handle (as a reference).

We could do that, but then any change to the preferred name needs
a rebuild of that index. So you click Apply in the prefs and then wait
several minutes. This can get frustrating as well.

Alex

> Quoting Don Allingham <[hidden email]>:
>
> > Doug,
> >
> > I can understand the impulse to put new entries and tables in the
> > database. I've thought of this myself (actually, I implemented it once).
> >
> > Unfortunately, is usually boils down to being not so good in the
> > implementation. We have several different methods of handling names, so
> > a single name alone wouldn't work. For example, the tree node (what
> > looks like the last name, but is actually a "grouping name") is one such
> > name, and the display name (the expanded name as the leaves of the tree)
> > are another.  We have quite a few other places where different name
> > styles are used as well, including full customization of the names. But
> > assuming we ignore these, we now have two additional names to store in
> > the database purely for the interface display.
> >
> > So now we have data and tables in the database whose sole purpose is to
> > make the interface's job easier. We now have additional tables that have
> > to be maintained. Since we don't have a full blown db to handle this for
> > us, we have to maintain it ourselves. Every person edit now requires us
> > to modify not only the person "table", but a name grouping table, and a
> > same sort table. If someone changes their desired format, we now have to
> > walk through the db updating all the grouping and display names.
> >
> > So, in general, I would want to use this as a last resort.
> >
> > Don
> >
> > On Tue, 2007-01-16 at 12:17 -0500, Douglas S. Blank wrote:
> >> Don Allingham wrote:
> >> > Yesterday, I did an analysis on row expansion in GtkTreeView. This was
> >> > the result of several people saying that it took too long to open a sub
> >> > node (opening a Family name to see the people underneath). Profiling
> >> > showed an issue we could easily fix, and I did. As a result, expanding
> >> > names is new very quick.
> >> >
> >> > Reviewing the profiling results of the full list build, we see a
> >> > different problem.
> >> >
> >> > Looking at the results indicates that the majority of the time goes to
> >> > building the name for the person in the display. The actual tree
> >> > functions are much further down on the list.
> >> >
> >> > This is a result of the user requirement for virtually infinite
> >> > flexibility in name display. A lot time has already been reduced by the
> >> > Lru module, which caches the last 500 built names.
> >> >
> >> > Unless we can find some magical operation to reduce the time to build a
> >> > name while keeping flexibility, we are limited in what we can do here.
> >>
> >> Don,
> >>
> >> Couldn't the display_name be a field in the table so that we don't have
> >> to build it each time we need it? If the user changes what is displayed,
> >> then we just rebuild it once, and save it. (On a related note, maybe
> >> person_names should be its own table, so that we could include alternate
> >> names in the list as well.) The date display field could also be
> >> pre-computed. Or am I misunderstanding the problem?
> >>
> >> -Doug
> >>
> >>
> >> >    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
> >> >     23654    1.459    0.000    1.459    0.000
> >> NameDisplay.py:205(_format_str_base)
> >> >     23654    0.640    0.000    0.743    0.000 Lru.py:39(__setitem__)
> >> >     11829    0.583    0.000    0.583    0.000 dbshelve.py:167(get)
> >> >     11827    0.561    0.000    0.561    0.000 _GrampsBSDDB.py:93(next)
> >> >     12267    0.506    0.000    3.610    0.000
> >> _PeopleModel.py:332(on_get_value)
> >> >         1    0.452    0.452    6.368    6.368
> >> _PeopleModel.py:203(_build_search_sub)
> >> >     11827    0.349    0.000    1.900    0.000
> >> _PeopleModel.py:433(column_name)
> >> >     23654    0.297    0.000    2.271    0.000
> >> NameDisplay.py:291(raw_sorted_name)
> >> >     11827    0.242    0.000    0.242    0.000
> >> _GrampsDbBase.py:1063(get_name_group_mapping)
> >> >     23654    0.227    0.000    1.686    0.000
> >> NameDisplay.py:194(format_str_raw)
> >> >     23654    0.191    0.000    1.877    0.000 NameDisplay.py:101(<lambda>)
> >> >     11827    0.150    0.000    3.890    0.000 _SearchFilter.py:33(match)
> >> >     11827    0.148    0.000    0.731    0.000
> >> _GrampsBSDDB.py:236(get_raw_person_data)
> >> >      2488    0.142    0.000    0.142    0.000
> >> _PeopleModel.py:277(build_sub_entry)
> >> >     11827    0.132    0.000    3.740    0.000
> >> _PeopleModel.py:171(<lambda>)
> >> >     11827    0.130    0.000    0.372    0.000
> >> NameDisplay.py:370(name_grouping_data)
> >> >       2/1    0.112    0.056    0.118    0.118
> >> DisplayState.py:331(modify_statusbar)
> >> >     23654    0.103    0.000    0.103    0.000 Lru.py:21(__init__)
> >> >     23654    0.102    0.000    0.102    0.000 Lru.py:36(__getitem__)
> >> >     23654    0.097    0.000    0.097    0.000
> >> NameDisplay.py:179(_is_format_valid)
> >> >         1    0.057    0.057    6.785    6.785
> >> _PersonView.py:521(build_tree)
> >> >      2488    0.023    0.000    0.023    0.000
> >> _PeopleModel.py:377(on_iter_has_child)
> >> >         1    0.023    0.023    0.023    0.023
> >> _PeopleModel.py:91(locale_sort)
> >> >      2513    0.016    0.000    0.016    0.000
> >> _PeopleModel.py:359(on_iter_next)
> >> >         1    0.016    0.016    6.549    6.549
> >> _PeopleModel.py:247(calculate_data)
> >> >         1    0.007    0.007    0.008    0.008
> >> _PersonView.py:674(build_columns)
> >> >         1    0.005    0.005    0.012    0.012
> >> _SearchBar.py:75(setup_filter)
> >> >       144    0.004    0.000    0.004    0.000 posixpath.py:168(exists)
> >> >       441    0.002    0.000    0.002    0.000
> >> _PeopleModel.py:319(on_get_column_type)
> >> >        12    0.002    0.000    0.008    0.001 gettext.py:408(find)
> >> >         1    0.002    0.002    0.002    0.002
> >> _PeopleModel.py:289(assign_data)
> >> >        96    0.001    0.000    0.001    0.000 posixpath.py:56(join)
> >> >        24    0.001    0.000    0.001    0.000 gettext.py:127(_expand_lang)
> >> >        44    0.000    0.000    0.000    0.000
> >> _PeopleModel.py:625(column_header)
> >> >        12    0.000    0.000    0.008    0.001 gettext.py:472(translation)
> >> >        12    0.000    0.000    0.008    0.001 gettext.py:538(dgettext)
> >> >        12    0.000    0.000    0.000    0.000 UserDict.py:52(get)
> >> >        24    0.000    0.000    0.000    0.000 locale.py:197(normalize)
> >> >         1    0.000    0.000    0.000    0.000
> >> _PersonView.py:846(get_selected_objects)
> >> >         1    0.000    0.000    0.000    0.000 _GrampsBSDDB.py:87(first)
> >> >         1    0.000    0.000    6.551    6.551
> >> _PeopleModel.py:136(__init__)
> >> >        12    0.000    0.000    0.008    0.001 gettext.py:576(gettext)
> >> >         2    0.000    0.000    0.000    0.000
> >> _GrampsDbBase.py:1959(_get_column_order)
> >> >         2    0.000    0.000    0.000    0.000
> >> _PedigreeView.py:624(size_request_cb)
> >> >         1    0.000    0.000    0.000    0.000 _SearchBar.py:130(get_value)
> >> >         6    0.000    0.000    0.000    0.000
> >> _PeopleModel.py:322(on_get_iter)
> >> >
> >> >
> >> >
> >> > ------------------------------------------------------------------------
> >> >
> >> > -------------------------------------------------------------------------
> >> > Take Surveys. Earn Cash. Influence the Future of IT
> >> > Join SourceForge.net's Techsay panel and you'll get the chance to
> >> share your
> >> > opinions on IT & business topics through brief surveys - and earn cash
> >> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> >> >
> >> >
> >> > ------------------------------------------------------------------------
> >> >
> >> > _______________________________________________
> >> > Gramps-devel mailing list
> >> > [hidden email]
> >> > https://lists.sourceforge.net/lists/listinfo/gramps-devel
> >>
> >
>
>
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
--
Alexander Roitman   http://www.gramps-project.org

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More on profiling

bm-7
Alex,

I agree partially with you. For 100.000 records your solution would
probably be
ok. The nice thing is, it should be easily tested if that claim is true, by
trying it. Even better, should it not be possible to offer both, the
code says:
  ngn = NameDisplay.displayer.name_grouping_data
  nsn = NameDisplay.displayer.raw_sorted_name

so with a preference, you could make this
  if fast :
    ngn = myfastroutine
  else :
    ngn = NameDisplay.displayer.name_grouping_data
etc...

Unfortunately, I do not have a large enough database to test myself.

My piont however was, even in this case, the treeview, and hence the GUI is
database dependent. Give 10 million records, and it will start to crawl again.

It is a pity that GTK does not offer a view you can dynamically pump with data
of a database without accessing all data beforehand. It would be best to code
that in C instead of trying to fix it with by inheriting treeview. It is
possible however should it be needed.
A good example of how to do it in python (viewing large data sets) is given
here: http://effbot.org/zone/wck-4.htm for Tkinter. (Look at the link, is
interesting)

And about the namedisplay routine, people will rarely change this. The wait to
change the preferred name would however not be longer than the length you wait
now to see the treeview.
Conclusion for me would be, go for easy solution if that really fixes
it, and if
people has so large databases that it is still a problem, consider the extra
table. With luck GTK3 contains such a thing. (I think to remember that in
visual basic/delphi there where specific views for access to sql databases)

Benny

Quoting Alex Roitman <[hidden email]>:

> Benny,
>
> On Tue, 2007-01-16 at 20:35 +0100, [hidden email] wrote:
>> It is only normal to have a key on the most important view in your
>> application.
>> The suggestion Doug had, and me too, (and probably everyone looking
>> into it), to
>> make a supper fast access method, independent of file size, is only possible
>> with such a construction.
>
> No, not only with such a construction. A simple alternative
> would be to have a fixed name for the purpose of using in the
> people view. It would still be assembled on the fly from the
> db records, but it would not be infinitely flexible like it
> is now. For example, use FirstName LastName for people view,
> period. Any customizations are effective everywhere else,
> except the people view. This can be made optional, so the large
> db users will trade off the convenience for performance,
> but the small db users can stay with the luxury of flexible names.
> We could even have a few common presets that are hard-coded.
>
>> You need to be able to retrieve the following.
>> get_next_n_occurences(namebegin, n)
>> returning the n names after namebegin, by only hitting the database once.
>
> All the name components are in a single person record, retrieved
> by one db read. Look at Don's profiling results: db access was not
> a problem. The consequential replacement of %t with title and %p
> with prefix and %y with patronymic and so on, was the problem.
> We retrieve records from the db amazingly fast, but then we crawl
> to build the name. Because we pay for the flexibility.
>
> With pre-set name display, all we need to do is:
>   person_tuple = db.get(handle)
>   name = "%s %s" % (person_tuple[3][2], person_tuple[3][3])
> Of course, I made up the tuple indices above, but this is
> what it would take, and it takes no time.
>
>>  So a
>> table 'groupname, displayname, handle', with key on groupname, and secondary
>> key on handle (as a reference).
>
> We could do that, but then any change to the preferred name needs
> a rebuild of that index. So you click Apply in the prefs and then wait
> several minutes. This can get frustrating as well.
>
> Alex
>
>> Quoting Don Allingham <[hidden email]>:
>>
>> > Doug,
>> >
>> > I can understand the impulse to put new entries and tables in the
>> > database. I've thought of this myself (actually, I implemented it once).
>> >
>> > Unfortunately, is usually boils down to being not so good in the
>> > implementation. We have several different methods of handling names, so
>> > a single name alone wouldn't work. For example, the tree node (what
>> > looks like the last name, but is actually a "grouping name") is one such
>> > name, and the display name (the expanded name as the leaves of the tree)
>> > are another.  We have quite a few other places where different name
>> > styles are used as well, including full customization of the names. But
>> > assuming we ignore these, we now have two additional names to store in
>> > the database purely for the interface display.
>> >
>> > So now we have data and tables in the database whose sole purpose is to
>> > make the interface's job easier. We now have additional tables that have
>> > to be maintained. Since we don't have a full blown db to handle this for
>> > us, we have to maintain it ourselves. Every person edit now requires us
>> > to modify not only the person "table", but a name grouping table, and a
>> > same sort table. If someone changes their desired format, we now have to
>> > walk through the db updating all the grouping and display names.
>> >
>> > So, in general, I would want to use this as a last resort.
>> >
>> > Don
>> >
>> > On Tue, 2007-01-16 at 12:17 -0500, Douglas S. Blank wrote:
>> >> Don Allingham wrote:
>> >> > Yesterday, I did an analysis on row expansion in GtkTreeView. This was
>> >> > the result of several people saying that it took too long to open a sub
>> >> > node (opening a Family name to see the people underneath). Profiling
>> >> > showed an issue we could easily fix, and I did. As a result, expanding
>> >> > names is new very quick.
>> >> >
>> >> > Reviewing the profiling results of the full list build, we see a
>> >> > different problem.
>> >> >
>> >> > Looking at the results indicates that the majority of the time goes to
>> >> > building the name for the person in the display. The actual tree
>> >> > functions are much further down on the list.
>> >> >
>> >> > This is a result of the user requirement for virtually infinite
>> >> > flexibility in name display. A lot time has already been reduced by the
>> >> > Lru module, which caches the last 500 built names.
>> >> >
>> >> > Unless we can find some magical operation to reduce the time to build a
>> >> > name while keeping flexibility, we are limited in what we can do here.
>> >>
>> >> Don,
>> >>
>> >> Couldn't the display_name be a field in the table so that we don't have
>> >> to build it each time we need it? If the user changes what is displayed,
>> >> then we just rebuild it once, and save it. (On a related note, maybe
>> >> person_names should be its own table, so that we could include alternate
>> >> names in the list as well.) The date display field could also be
>> >> pre-computed. Or am I misunderstanding the problem?
>> >>
>> >> -Doug
>> >>
>> >>
>> >> >    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>> >> >     23654    1.459    0.000    1.459    0.000
>> >> NameDisplay.py:205(_format_str_base)
>> >> >     23654    0.640    0.000    0.743    0.000 Lru.py:39(__setitem__)
>> >> >     11829    0.583    0.000    0.583    0.000 dbshelve.py:167(get)
>> >> >     11827    0.561    0.000    0.561    0.000 _GrampsBSDDB.py:93(next)
>> >> >     12267    0.506    0.000    3.610    0.000
>> >> _PeopleModel.py:332(on_get_value)
>> >> >         1    0.452    0.452    6.368    6.368
>> >> _PeopleModel.py:203(_build_search_sub)
>> >> >     11827    0.349    0.000    1.900    0.000
>> >> _PeopleModel.py:433(column_name)
>> >> >     23654    0.297    0.000    2.271    0.000
>> >> NameDisplay.py:291(raw_sorted_name)
>> >> >     11827    0.242    0.000    0.242    0.000
>> >> _GrampsDbBase.py:1063(get_name_group_mapping)
>> >> >     23654    0.227    0.000    1.686    0.000
>> >> NameDisplay.py:194(format_str_raw)
>> >> >     23654    0.191    0.000    1.877    0.000
>> NameDisplay.py:101(<lambda>)
>> >> >     11827    0.150    0.000    3.890    0.000
>> _SearchFilter.py:33(match)
>> >> >     11827    0.148    0.000    0.731    0.000
>> >> _GrampsBSDDB.py:236(get_raw_person_data)
>> >> >      2488    0.142    0.000    0.142    0.000
>> >> _PeopleModel.py:277(build_sub_entry)
>> >> >     11827    0.132    0.000    3.740    0.000
>> >> _PeopleModel.py:171(<lambda>)
>> >> >     11827    0.130    0.000    0.372    0.000
>> >> NameDisplay.py:370(name_grouping_data)
>> >> >       2/1    0.112    0.056    0.118    0.118
>> >> DisplayState.py:331(modify_statusbar)
>> >> >     23654    0.103    0.000    0.103    0.000 Lru.py:21(__init__)
>> >> >     23654    0.102    0.000    0.102    0.000 Lru.py:36(__getitem__)
>> >> >     23654    0.097    0.000    0.097    0.000
>> >> NameDisplay.py:179(_is_format_valid)
>> >> >         1    0.057    0.057    6.785    6.785
>> >> _PersonView.py:521(build_tree)
>> >> >      2488    0.023    0.000    0.023    0.000
>> >> _PeopleModel.py:377(on_iter_has_child)
>> >> >         1    0.023    0.023    0.023    0.023
>> >> _PeopleModel.py:91(locale_sort)
>> >> >      2513    0.016    0.000    0.016    0.000
>> >> _PeopleModel.py:359(on_iter_next)
>> >> >         1    0.016    0.016    6.549    6.549
>> >> _PeopleModel.py:247(calculate_data)
>> >> >         1    0.007    0.007    0.008    0.008
>> >> _PersonView.py:674(build_columns)
>> >> >         1    0.005    0.005    0.012    0.012
>> >> _SearchBar.py:75(setup_filter)
>> >> >       144    0.004    0.000    0.004    0.000 posixpath.py:168(exists)
>> >> >       441    0.002    0.000    0.002    0.000
>> >> _PeopleModel.py:319(on_get_column_type)
>> >> >        12    0.002    0.000    0.008    0.001 gettext.py:408(find)
>> >> >         1    0.002    0.002    0.002    0.002
>> >> _PeopleModel.py:289(assign_data)
>> >> >        96    0.001    0.000    0.001    0.000 posixpath.py:56(join)
>> >> >        24    0.001    0.000    0.001    0.000
>> gettext.py:127(_expand_lang)
>> >> >        44    0.000    0.000    0.000    0.000
>> >> _PeopleModel.py:625(column_header)
>> >> >        12    0.000    0.000    0.008    0.001
>> gettext.py:472(translation)
>> >> >        12    0.000    0.000    0.008    0.001 gettext.py:538(dgettext)
>> >> >        12    0.000    0.000    0.000    0.000 UserDict.py:52(get)
>> >> >        24    0.000    0.000    0.000    0.000 locale.py:197(normalize)
>> >> >         1    0.000    0.000    0.000    0.000
>> >> _PersonView.py:846(get_selected_objects)
>> >> >         1    0.000    0.000    0.000    0.000 _GrampsBSDDB.py:87(first)
>> >> >         1    0.000    0.000    6.551    6.551
>> >> _PeopleModel.py:136(__init__)
>> >> >        12    0.000    0.000    0.008    0.001 gettext.py:576(gettext)
>> >> >         2    0.000    0.000    0.000    0.000
>> >> _GrampsDbBase.py:1959(_get_column_order)
>> >> >         2    0.000    0.000    0.000    0.000
>> >> _PedigreeView.py:624(size_request_cb)
>> >> >         1    0.000    0.000    0.000    0.000
>> _SearchBar.py:130(get_value)
>> >> >         6    0.000    0.000    0.000    0.000
>> >> _PeopleModel.py:322(on_get_iter)
>> >> >
>> >> >
>> >> >
>> >> >
>> ------------------------------------------------------------------------
>> >> >
>> >> >
>> -------------------------------------------------------------------------
>> >> > Take Surveys. Earn Cash. Influence the Future of IT
>> >> > Join SourceForge.net's Techsay panel and you'll get the chance to
>> >> share your
>> >> > opinions on IT & business topics through brief surveys - and earn cash
>> >> >
>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>> >> >
>> >> >
>> >> >
>> ------------------------------------------------------------------------
>> >> >
>> >> > _______________________________________________
>> >> > Gramps-devel mailing list
>> >> > [hidden email]
>> >> > https://lists.sourceforge.net/lists/listinfo/gramps-devel
>> >>
>> >
>>
>>
>>
>> ----------------------------------------------------------------
>> This message was sent using IMP, the Internet Messaging Program.
>>
>>
>> -------------------------------------------------------------------------
>> Take Surveys. Earn Cash. Influence the Future of IT
>> Join SourceForge.net's Techsay panel and you'll get the chance to share your
>> opinions on IT & business topics through brief surveys - and earn cash
>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>> _______________________________________________
>> Gramps-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
> --
> Alexander Roitman   http://www.gramps-project.org
>



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: More on profiling

Alex Roitman
On Tue, 2007-01-16 at 22:28 +0100, [hidden email] wrote:
> I agree partially with you. For 100.000 records your solution would
> probably be
> ok. The nice thing is, it should be easily tested if that claim is true, by
> trying it. Even better, should it not be possible to offer both, the
> code says:
>   ngn = NameDisplay.displayer.name_grouping_data
>   nsn = NameDisplay.displayer.raw_sorted_name

Unfortunately, at present the code uses format string no matter what.
Even the "preset" formats that are not editable by the user internally
have been switched to use format string representation.

This was exactly my point: we should first get the preset
formats back to being hard-coded and see whether this is enough.

> My piont however was, even in this case, the treeview, and hence the GUI is
> database dependent. Give 10 million records, and it will start to crawl again.

Sure. But so will the secondary index. My only point was that
the secondary index does not buy us much here, compared to the
hard-coded name computation anyway.

> It is a pity that GTK does not offer a view you can dynamically pump with data
> of a database without accessing all data beforehand. It would be best to code
> that in C instead of trying to fix it with by inheriting treeview. It is
> possible however should it be needed.

If we code our extensions in C, we're back to architecture dependence.
A better thing would be to fix it in gtk :-)

> And about the namedisplay routine, people will rarely change this. The wait to
> change the preferred name would however not be longer than the length you wait
> now to see the treeview.

Actually it would, but this is probably OK. Changing the name format
would require rebuilding of the secondary index, not just accessing
the data. read+write is definitely longer than just read.

> Conclusion for me would be, go for easy solution if that really fixes
> it,

yes!

>  and if
> people has so large databases that it is still a problem, consider the extra
> table.

But how does the extra table help us? It only would if we want
to keep names fast *and* flexible. I am sure the performance-hungry
users will gladly trade name flexibility in just one view for
performance. But if this is not enough then extra table also won't be.

Alex

--
Alexander Roitman   http://www.gramps-project.org

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More on profiling

Richard Taylor-2
I have just committed a new implementation of NameDisplay that improves the
performance.

I have not made it the active method yet as it needs more testing but you can
take a look in NameDisplay.py if you are interested:

Timeing for 1000000 names gives: old - 79 secs, new - 16 secs

A reasonable improvement, if a slightly wacky implementation.

Richard

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: More on profiling

bm-7
In reply to this post by Alex Roitman
Quoting Alex Roitman <[hidden email]>:

> But how does the extra table help us? It only would if we want
> to keep names fast *and* flexible. I am sure the performance-hungry
> users will gladly trade name flexibility in just one view for
> performance. But if this is not enough then extra table also won't be.

An extra table would help in that it allows another type of view, which is in
it's easiest form a fixed list, with only page-up and page-down, and a
find-box
enabling a quick match (no loop over all names). A spinbutton could make it
nicer, ...

This is not a fancy GUI, but for quick access to a million of records,
it is ok.
You do not want to browse the data as is now possible with GRAMPS, you want to
go to the entry you look for, and edit that. Think patient record in a
hospital.

Probably not needed looking at the poll.

However, an idea for large databases just popping up in my head: allow to
disable the person view, and replace it with a find tool. User enters
beginning
of a name, and occurences with that name show up in comboentrybox. Yes
index on
name needed ;-) or it would not save much time.

Benny

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: More on profiling

Alex Roitman
In reply to this post by Richard Taylor-2
On Tue, 2007-01-16 at 22:33 +0000, Richard Taylor wrote:
> I have just committed a new implementation of NameDisplay that improves the
> performance.
>
> I have not made it the active method yet as it needs more testing but you can
> take a look in NameDisplay.py if you are interested:
>
> Timeing for 1000000 names gives: old - 79 secs, new - 16 secs
>
> A reasonable improvement, if a slightly wacky implementation.

Nice job, Richard! And that keeps the flexibility. Meaning
that with hard-coded presets one can gain yet more!

Alex

--
Alexander Roitman   http://www.gramps-project.org

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (196 bytes) Download Attachment