slow database

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

slow database

Anton Huber
Hello,

searching in the database using person view is very slow on all my personal computers and notebooks. For a Surname it takes around 20 seconds until the result is displayed. I've nearly 28000 persons in my database. Is there a way to make searching more comfortable for big database?

Best regards,

Anton

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: slow database

Benny Malengier


2012/9/30 Anton Huber <[hidden email]>
Hello,

searching in the database using person view is very slow on all my personal computers and notebooks. For a Surname it takes around 20 seconds until the result is displayed. I've nearly 28000 persons in my database. Is there a way to make searching more comfortable for big database?

It has been some time since we optimized for large databases. So some changes might have crept in that are a disaster on large databases.

Are you trying trunk or gramps34? They are quite different at the moment.

To speed things up:
1/remove calculated  columns, like spouse, ... You can do that in the preferences of the view (icon on toolbar or in menu)
2/make sure the status bar does not compute relationship, or if it does, that it does not scan too many generations back
3/make sure you don't have gramplets that change on change of active person. This can really slow things down, as the gramplet is recomputing on selection of another person in the person view. Especially a descendant gramplet and such can be problems.

We have had in the past a large .gramps to test with, but that was generated data, so behaves differently from real genealogical data. I don't have a large dataset to try with at the moment.

As you post on the devel list, can you code yourself? To understand where the bottleneck is, we need to add profiling on the searching, see
http://www.gramps-project.org/wiki/index.php?title=Debugging_GRAMPS#Use_profiling
On trunk, the import statement for that has changed however. Let us know if you are interested in profiling it.

Apart from the above, for large datasets, another view which is not a listview would not be bad. A listview after all needs to load a lot of data in memory. Typical view within companies is then a search view, where you give some limitations, and only that piece is obtained from the database, instead of the entire table. It would not be a bad addition to gramps to create such a view.

Benny

Best regards,

Anton

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel



------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: slow database

Benny Malengier
I see we have a GEP here with instructions on profiling that are perhaps easier:

http://www.gramps-project.org/wiki/index.php?title=GEPS_016:_Enhancing_GRAMPS_Processing_Speed

Benny

2012/9/30 Benny Malengier <[hidden email]>


2012/9/30 Anton Huber <[hidden email]>
Hello,

searching in the database using person view is very slow on all my personal computers and notebooks. For a Surname it takes around 20 seconds until the result is displayed. I've nearly 28000 persons in my database. Is there a way to make searching more comfortable for big database?

It has been some time since we optimized for large databases. So some changes might have crept in that are a disaster on large databases.

Are you trying trunk or gramps34? They are quite different at the moment.

To speed things up:
1/remove calculated  columns, like spouse, ... You can do that in the preferences of the view (icon on toolbar or in menu)
2/make sure the status bar does not compute relationship, or if it does, that it does not scan too many generations back
3/make sure you don't have gramplets that change on change of active person. This can really slow things down, as the gramplet is recomputing on selection of another person in the person view. Especially a descendant gramplet and such can be problems.

We have had in the past a large .gramps to test with, but that was generated data, so behaves differently from real genealogical data. I don't have a large dataset to try with at the moment.

As you post on the devel list, can you code yourself? To understand where the bottleneck is, we need to add profiling on the searching, see
http://www.gramps-project.org/wiki/index.php?title=Debugging_GRAMPS#Use_profiling
On trunk, the import statement for that has changed however. Let us know if you are interested in profiling it.

Apart from the above, for large datasets, another view which is not a listview would not be bad. A listview after all needs to load a lot of data in memory. Typical view within companies is then a search view, where you give some limitations, and only that piece is obtained from the database, instead of the entire table. It would not be a bad addition to gramps to create such a view.

Benny

Best regards,

Anton

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel




------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: slow database

jerome
In reply to this post by Anton Huber
>  For a Surname it takes around 20 seconds until the
> result is displayed. I've nearly 28000 persons in my database.

"Flat views are faster than Tree views"

http://www.gramps-project.org/wiki/index.php?title=Tips_for_large_databases

Try to rather to "display Surnames" via person list view!
Surnames will not be grouped (parent node), but I know that it will be
faster.

Note, I got this problem with Place Tree View in the past (around 38 000
rows/entries) and I did not test Citations/Sources Tree View yet, which
seems to use an other method (iteration, cursor) for displaying such
grouped data.


Jérôme

Le 30/09/2012 15:53, Anton Huber a écrit :

> Hello,
>
> searching in the database using person view is very slow on all my personal
> computers and notebooks. For a Surname it takes around 20 seconds until the
> result is displayed. I've nearly 28000 persons in my database. Is there a
> way to make searching more comfortable for big database?
>
> Best regards,
>
> Anton
>
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://ad.doubleclick.net/clk;258768047;13503038;j?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>
>
>
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>


------------------------------------------------------------------------------
Got visibility?
Most devs has no idea what their production app looks like.
Find out how fast your code is with AppDynamics Lite.
http://ad.doubleclick.net/clk;262219671;13503038;y?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: slow database

Dave Burkholder
Hello everyone. I'm Dave and I've been lurking this list for some time, considering migrating a project from another genealogy program to Gramps.  The project already has 140,000+ names.

For the most part, I hope to be using Webapp on a Postgresql database (and I would like to contribute to the Webapp in time) but I'll want to be using Gramps application also, if possible. Is that out of the question for so many names?

What's the largest project Gramps has ever handled, at any version?
 
On Mon, Oct 1, 2012 at 3:20 AM, Jérôme <[hidden email]> wrote:
>  For a Surname it takes around 20 seconds until the
> result is displayed. I've nearly 28000 persons in my database.

"Flat views are faster than Tree views"

http://www.gramps-project.org/wiki/index.php?title=Tips_for_large_databases

Try to rather to "display Surnames" via person list view!
Surnames will not be grouped (parent node), but I know that it will be
faster.

Note, I got this problem with Place Tree View in the past (around 38 000
rows/entries) and I did not test Citations/Sources Tree View yet, which
seems to use an other method (iteration, cursor) for displaying such
grouped data.


Jérôme

Le 30/09/2012 15:53, Anton Huber a écrit :
> Hello,
>
> searching in the database using person view is very slow on all my personal
> computers and notebooks. For a Surname it takes around 20 seconds until the
> result is displayed. I've nearly 28000 persons in my database. Is there a
> way to make searching more comfortable for big database?
>
> Best regards,
>
> Anton
>
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://ad.doubleclick.net/clk;258768047;13503038;j?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>
>
>
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>


------------------------------------------------------------------------------
Got visibility?
Most devs has no idea what their production app looks like.
Find out how fast your code is with AppDynamics Lite.
http://ad.doubleclick.net/clk;262219671;13503038;y?



------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: slow database

Benny Malengier


2012/10/11 Dave Burkholder <[hidden email]>
Hello everyone. I'm Dave and I've been lurking this list for some time, considering migrating a project from another genealogy program to Gramps.  The project already has 140,000+ names.

For the most part, I hope to be using Webapp on a Postgresql database (and I would like to contribute to the Webapp in time) but I'll want to be using Gramps application also, if possible. Is that out of the question for so many names?

Some things will be really slow I expect.
Biggest problem is probably the import of so many names. Import happens as one database commit, but on importing so many names, you will go over the database page limit of one commit.
That is a setting you can change however somewhere in the code. See http://gramps-project.org/wiki/index.php?title=Known_issues#GRAMPS_runs_out_of_locks
Improving the import code to have it more performant for such a case would be a great addition :-)

Once imported, I'm not sure the views will work good or not. We reworked the views to have them performant, but if that actually works on 140.000 is unknown. I'd be interested to know though.

Latest results are here: http://www.gramps-project.org/wiki/index.php?title=GRAMPS_Performance
You see that last test was with 3.3.0. on 120000+ people. Starting gramps up on the person listview was 15 sec. Switching to event view 1s (normally more events than you have people). Sorting event on date however was 30 sec, as obviously the entire table of dates must be sorted in memory.
I suppose people would expect sorting to be slow in this case.

Benny


What's the largest project Gramps has ever handled, at any version?

 
On Mon, Oct 1, 2012 at 3:20 AM, Jérôme <[hidden email]> wrote:
>  For a Surname it takes around 20 seconds until the
> result is displayed. I've nearly 28000 persons in my database.

"Flat views are faster than Tree views"

http://www.gramps-project.org/wiki/index.php?title=Tips_for_large_databases

Try to rather to "display Surnames" via person list view!
Surnames will not be grouped (parent node), but I know that it will be
faster.

Note, I got this problem with Place Tree View in the past (around 38 000
rows/entries) and I did not test Citations/Sources Tree View yet, which
seems to use an other method (iteration, cursor) for displaying such
grouped data.


Jérôme

Le 30/09/2012 15:53, Anton Huber a écrit :
> Hello,
>
> searching in the database using person view is very slow on all my personal
> computers and notebooks. For a Surname it takes around 20 seconds until the
> result is displayed. I've nearly 28000 persons in my database. Is there a
> way to make searching more comfortable for big database?
>
> Best regards,
>
> Anton
>
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://ad.doubleclick.net/clk;258768047;13503038;j?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>
>
>
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>


------------------------------------------------------------------------------
Got visibility?
Most devs has no idea what their production app looks like.
Find out how fast your code is with AppDynamics Lite.
http://ad.doubleclick.net/clk;262219671;13503038;y?



------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel



------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: slow database

jerome
You can also try the experimental 'gramps-connect'?

http://gramps-project.org/wiki/index.php?title=Gramps-Connect

--- En date de : Jeu 11.10.12, Benny Malengier <[hidden email]> a écrit :

De: Benny Malengier <[hidden email]>
Objet: Re: [Gramps-devel] slow database
À: "Dave Burkholder" <[hidden email]>
Cc: [hidden email]
Date: Jeudi 11 octobre 2012, 11h23



2012/10/11 Dave Burkholder <thinkwelldesigns@...>
Hello everyone. I'm Dave and I've been lurking this list for some time, considering migrating a project from another genealogy program to Gramps.  The project already has 140,000+ names.

For the most part, I hope to be using Webapp on a Postgresql database (and I would like to contribute to the Webapp in time) but I'll want to be using Gramps application also, if possible. Is that out of the question for so many names?

Some things will be really slow I expect.
Biggest problem is probably the import of so many names. Import happens as one database commit, but on importing so many names, you will go over the database page limit of one commit.
That is a setting you can change however somewhere in the code. See http://gramps-project.org/wiki/index.php?title=Known_issues#GRAMPS_runs_out_of_locks
Improving the import code to have it more performant for such a case would be a great addition :-)

Once imported, I'm not sure the views will work good or not. We reworked the views to have them performant, but if that actually works on 140.000 is unknown. I'd be interested to know though.

Latest results are here: http://www.gramps-project.org/wiki/index.php?title=GRAMPS_Performance
You see that last test was with 3.3.0. on 120000+ people. Starting gramps up on the person listview was 15 sec. Switching to event view 1s (normally more events than you have people). Sorting event on date however was 30 sec, as obviously the entire table of dates must be sorted in memory.
I suppose people would expect sorting to be slow in this case.

Benny


What's the largest project Gramps has ever handled, at any version?

 
On Mon, Oct 1, 2012 at 3:20 AM, Jérôme <romjerome@...> wrote:
>  For a Surname it takes around 20 seconds until the
> result is displayed. I've nearly 28000 persons in my database.

"Flat views are faster than Tree views"

http://www.gramps-project.org/wiki/index.php?title=Tips_for_large_databases

Try to rather to "display Surnames" via person list view!
Surnames will not be grouped (parent node), but I know that it will be
faster.

Note, I got this problem with Place Tree View in the past (around 38 000
rows/entries) and I did not test Citations/Sources Tree View yet, which
seems to use an other method (iteration, cursor) for displaying such
grouped data.


Jérôme

Le 30/09/2012 15:53, Anton Huber a écrit :
> Hello,
>
> searching in the database using person view is very slow on all my personal
> computers and notebooks. For a Surname it takes around 20 seconds until the
> result is displayed. I've nearly 28000 persons in my database. Is there a
> way to make searching more comfortable for big database?
>
> Best regards,
>
> Anton
>
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://ad.doubleclick.net/clk;258768047;13503038;j?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>
>
>
> _______________________________________________
> Gramps-devel mailing list
> Gramps-devel@...
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>


------------------------------------------------------------------------------
Got visibility?
Most devs has no idea what their production app looks like.
Find out how fast your code is with AppDynamics Lite.
http://ad.doubleclick.net/clk;262219671;13503038;y?



------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel



-----La pièce jointe associée suit-----

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

-----La pièce jointe associée suit-----

_______________________________________________
Gramps-devel mailing list
Gramps-devel@...
https://lists.sourceforge.net/lists/listinfo/gramps-devel

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: slow database

Robert Cawley
In reply to this post by Benny Malengier
On Thu, 2012-10-11 at 11:23 +-0200, Benny Malengier wrote:
+AD4
+AD4
+AD4 2012/10/11 Dave Burkholder +ADw-thinkwelldesigns+AEA-gmail.com+AD4
+AD4         Hello everyone. I'm Dave and I've been lurking this list for
+AD4         some time, considering migrating a project from another
+AD4         genealogy program to Gramps.  The project already has 140,000+-
+AD4         names.
+AD4        
+AD4         For the most part, I hope to be using Webapp on a Postgresql
+AD4         database (and I would like to contribute to the Webapp in
+AD4         time) but I'll want to be using Gramps application also, if
+AD4         possible. Is that out of the question for so many names?
+AD4
+AD4 Some things will be really slow I expect.
+AD4 Biggest problem is probably the import of so many names. Import
+AD4 happens as one database commit, but on importing so many names, you
+AD4 will go over the database page limit of one commit.
+AD4 That is a setting you can change however somewhere in the code. See
+AD4 http://gramps-project.org/wiki/index.php?title+AD0-Known+AF8-issues+ACM-GRAMPS+AF8-runs+AF8-out+AF8-of+AF8-locks
+AD4 Improving the import code to have it more performant for such a case
+AD4 would be a great addition :-)
+AD4
+AD4 Once imported, I'm not sure the views will work good or not. We
+AD4 reworked the views to have them performant, but if that actually works
+AD4 on 140.000 is unknown. I'd be interested to know though.

I work with a database of 141,000+-names currently without difficulty
(Gramps 3.3.1-1 on Fedora 16).
Initial start is fairly slow though.
First time to load each view is slow, but subsequent visits to views is
almost immediate.
Initial view load times -
people  11 to 12 secs
relationship abt 7 secs
family  3 to 4 secs
events  7 to 8 secs
places  3 to 4 secs
notes 11 to 12 secs
ancestry view abt 1 sec or less
Media abt 2 secs (although I only have about 1000 media in database)
Repositories almost immediate
sources about 1 sec - (time selecting a source varies according to
number references for that source - my worst case is a civil registry
which has about twice as many references as people in my database).

Avoid leaving gramplets which do a lot a database work active - deep
connections seems to be the worst case - as these slow everything down
enormously.

I do have the show relationship to home person enabled though and this
does not seem to slow things appreciably.

Inital import of the database from either gramps format or gedcom is
tiresome though and can take a few hours. (would be really good if
import could be sped up.) As stated above you will need to adjust the
number of allowable locks. I current use:
max+AF8-locks 300000 and
max+AF8-objects 300000
I used to get away with half this, but not any more.
The easiest way to do this is to create an empty database the add a file
DB+AF8-CONFIG to the database directory before importing.
Contents of DB+AF8-CONFIG

+ACM-may want to fiddle with cachesize also
+ACM-set+AF8-cachesize 0 200000000 2
set+AF8-lk+AF8-max+AF8-locks 300000
set+AF8-lk+AF8-max+AF8-objects 300000

Robert


+AD4
+AD4 Latest results are here:
+AD4 http://www.gramps-project.org/wiki/index.php?title+AD0-GRAMPS+AF8-Performance
+AD4 You see that last test was with 3.3.0. on 120000+- people. Starting
+AD4 gramps up on the person listview was 15 sec. Switching to event view
+AD4 1s (normally more events than you have people). Sorting event on date
+AD4 however was 30 sec, as obviously the entire table of dates must be
+AD4 sorted in memory.
+AD4 I suppose people would expect sorting to be slow in this case.
+AD4
+AD4 Benny
+AD4
+AD4
+AD4        
+AD4         What's the largest project Gramps has ever handled, at any
+AD4         version?
+AD4        
+AD4          
+AD4                 On Mon, Oct 1, 2012 at 3:20 AM, J+AOk-r+APQ-me
+AD4                 +ADw-romjerome+AEA-yahoo.fr+AD4 wrote:
+AD4                         +AD4  For a Surname it takes around 20 seconds
+AD4                         until the
+AD4                         +AD4 result is displayed. I've nearly 28000
+AD4                         persons in my database.
+AD4                        
+AD4                        
+AD4                         +ACI-Flat views are faster than Tree views+ACI
+AD4                        
+AD4                         http://www.gramps-project.org/wiki/index.php?title+AD0-Tips+AF8-for+AF8-large+AF8-databases
+AD4                        
+AD4                         Try to rather to +ACI-display Surnames+ACI via person
+AD4                         list view+ACE
+AD4                         Surnames will not be grouped (parent node),
+AD4                         but I know that it will be
+AD4                         faster.
+AD4                        
+AD4                         Note, I got this problem with Place Tree View
+AD4                         in the past (around 38 000
+AD4                         rows/entries) and I did not test
+AD4                         Citations/Sources Tree View yet, which
+AD4                         seems to use an other method (iteration,
+AD4                         cursor) for displaying such
+AD4                         grouped data.
+AD4                        
+AD4                        
+AD4                         J+AOk-r+APQ-me
+AD4                        
+AD4                         Le 30/09/2012 15:53, Anton Huber a +AOk-crit :
+AD4                         +AD4 Hello,
+AD4                         +AD4
+AD4                         +AD4 searching in the database using person view
+AD4                         is very slow on all my personal
+AD4                         +AD4 computers and notebooks. For a Surname it
+AD4                         takes around 20 seconds until the
+AD4                         +AD4 result is displayed. I've nearly 28000
+AD4                         persons in my database. Is there a
+AD4                         +AD4 way to make searching more comfortable for
+AD4                         big database?
+AD4                         +AD4
+AD4                         +AD4 Best regards,
+AD4                         +AD4
+AD4                         +AD4 Anton
+AD4                         +AD4
+AD4                         +AD4
+AD4                         +AD4
+AD4                        
+AD4                         +AD4
+AD4                         ------------------------------------------------------------------------------
+AD4                         +AD4 Everyone hates slow websites. So do we.
+AD4                         +AD4 Make your web apps faster with AppDynamics
+AD4                         +AD4 Download AppDynamics Lite for free today:
+AD4                         +AD4
+AD4                         http://ad.doubleclick.net/clk+ADs-258768047+ADs-13503038+ADs-j?
+AD4                         +AD4
+AD4                         http://info.appdynamics.com/FreeJavaPerformanceDownload.html
+AD4                         +AD4
+AD4                         +AD4
+AD4                         +AD4
+AD4                         +AD4
+AD4                         +AF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXw
+AD4                         +AD4 Gramps-devel mailing list
+AD4                         +AD4 Gramps-devel+AEA-lists.sourceforge.net
+AD4                         +AD4
+AD4                         https://lists.sourceforge.net/lists/listinfo/gramps-devel
+AD4                         +AD4
+AD4                        
+AD4                        
+AD4                        
+AD4                         ------------------------------------------------------------------------------
+AD4                         Got visibility?
+AD4                         Most devs has no idea what their production
+AD4                         app looks like.
+AD4                         Find out how fast your code is with
+AD4                         AppDynamics Lite.
+AD4                         http://ad.doubleclick.net/clk+ADs-262219671+ADs-13503038+ADs-y?
+AD4                         http://info.appdynamics.com/FreeJavaPerformanceDownload.html
+AD4                         +AF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXw
+AD4                         Gramps-devel mailing list
+AD4                         Gramps-devel+AEA-lists.sourceforge.net
+AD4                         https://lists.sourceforge.net/lists/listinfo/gramps-devel
+AD4                        
+AD4                
+AD4                
+AD4        
+AD4        
+AD4        
+AD4         ------------------------------------------------------------------------------
+AD4         Don't let slow site performance ruin your business. Deploy New
+AD4         Relic APM
+AD4         Deploy New Relic app performance management and know exactly
+AD4         what is happening inside your Ruby, Python, PHP, Java,
+AD4         and .NET app
+AD4         Try New Relic at no cost today and get our sweet Data Nerd
+AD4         shirt too+ACE
+AD4         http://p.sf.net/sfu/newrelic-dev2dev
+AD4         +AF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXw
+AD4         Gramps-devel mailing list
+AD4         Gramps-devel+AEA-lists.sourceforge.net
+AD4         https://lists.sourceforge.net/lists/listinfo/gramps-devel
+AD4        
+AD4
+AD4 ------------------------------------------------------------------------------
+AD4 Don't let slow site performance ruin your business. Deploy New Relic APM
+AD4 Deploy New Relic app performance management and know exactly
+AD4 what is happening inside your Ruby, Python, PHP, Java, and .NET app
+AD4 Try New Relic at no cost today and get our sweet Data Nerd shirt too+ACE
+AD4 http://p.sf.net/sfu/newrelic-dev2dev
+AD4 +AF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXwBfAF8AXw Gramps-devel mailing list Gramps-devel+AEA-lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/gramps-devel

--
Robert Cawley
Ph: +-61 2 9543 1241
rjc+AEA-cawley.id.au
rjc047+AEA-westnet.com.au
rjc047+AEA-optusnet.com.au




------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: slow database

Dave Burkholder
Thanks for your comments and suggestions, everyone. I was pretty sure Webapp wouldn't be the problem with the right database, but Gramps application performance was a question.  The http://www.gramps-project.org/wiki/index.php?title=GRAMPS_Performance wiki page was great! I hadn't encountered that page before (gramps is well documented!)

I used the db03 dataset on Gramps 3.4. My quad 925 AMD with 8 gig of ram imported the data in 50 minutes 40 seconds. Program performance after the import was satisfactory. This sets my mind at ease regarding Gramps suitability for my project.



On Fri, Oct 12, 2012 at 1:27 AM, Robert Cawley <[hidden email]> wrote:
On Thu, 2012-10-11 at 11:23 +0200, Benny Malengier wrote:
>
>
> 2012/10/11 Dave Burkholder <[hidden email]>
>         Hello everyone. I'm Dave and I've been lurking this list for
>         some time, considering migrating a project from another
>         genealogy program to Gramps.  The project already has 140,000+
>         names.
>
>         For the most part, I hope to be using Webapp on a Postgresql
>         database (and I would like to contribute to the Webapp in
>         time) but I'll want to be using Gramps application also, if
>         possible. Is that out of the question for so many names?
>
> Some things will be really slow I expect.
> Biggest problem is probably the import of so many names. Import
> happens as one database commit, but on importing so many names, you
> will go over the database page limit of one commit.
> That is a setting you can change however somewhere in the code. See
> http://gramps-project.org/wiki/index.php?title=Known_issues#GRAMPS_runs_out_of_locks
> Improving the import code to have it more performant for such a case
> would be a great addition :-)
>
> Once imported, I'm not sure the views will work good or not. We
> reworked the views to have them performant, but if that actually works
> on 140.000 is unknown. I'd be interested to know though.

I work with a database of 141,000+names currently without difficulty
(Gramps 3.3.1-1 on Fedora 16).
Initial start is fairly slow though.
First time to load each view is slow, but subsequent visits to views is
almost immediate.
Initial view load times -
people  11 to 12 secs
relationship abt 7 secs
family  3 to 4 secs
events  7 to 8 secs
places  3 to 4 secs
notes 11 to 12 secs
ancestry view abt 1 sec or less
Media abt 2 secs (although I only have about 1000 media in database)
Repositories almost immediate
sources about 1 sec - (time selecting a source varies according to
number references for that source - my worst case is a civil registry
which has about twice as many references as people in my database).

Avoid leaving gramplets which do a lot a database work active - deep
connections seems to be the worst case - as these slow everything down
enormously.

I do have the show relationship to home person enabled though and this
does not seem to slow things appreciably.

Inital import of the database from either gramps format or gedcom is
tiresome though and can take a few hours. (would be really good if
import could be sped up.) As stated above you will need to adjust the
number of allowable locks. I current use:
max_locks 300000 and
max_objects 300000
I used to get away with half this, but not any more.
The easiest way to do this is to create an empty database the add a file
DB_CONFIG to the database directory before importing.
Contents of DB_CONFIG

#may want to fiddle with cachesize also
#set_cachesize 0 200000000 2
set_lk_max_locks 300000
set_lk_max_objects 300000

Robert


>
> Latest results are here:
> http://www.gramps-project.org/wiki/index.php?title=GRAMPS_Performance
> You see that last test was with 3.3.0. on 120000+ people. Starting
> gramps up on the person listview was 15 sec. Switching to event view
> 1s (normally more events than you have people). Sorting event on date
> however was 30 sec, as obviously the entire table of dates must be
> sorted in memory.
> I suppose people would expect sorting to be slow in this case.
>
> Benny
>
>
>
>         What's the largest project Gramps has ever handled, at any
>         version?
>
>
>                 On Mon, Oct 1, 2012 at 3:20 AM, Jérôme
>                 <[hidden email]> wrote:
>                         >  For a Surname it takes around 20 seconds
>                         until the
>                         > result is displayed. I've nearly 28000
>                         persons in my database.
>
>
>                         "Flat views are faster than Tree views"
>
>                         http://www.gramps-project.org/wiki/index.php?title=Tips_for_large_databases
>
>                         Try to rather to "display Surnames" via person
>                         list view!
>                         Surnames will not be grouped (parent node),
>                         but I know that it will be
>                         faster.
>
>                         Note, I got this problem with Place Tree View
>                         in the past (around 38 000
>                         rows/entries) and I did not test
>                         Citations/Sources Tree View yet, which
>                         seems to use an other method (iteration,
>                         cursor) for displaying such
>                         grouped data.
>
>
>                         Jérôme
>
>                         Le 30/09/2012 15:53, Anton Huber a écrit :
>                         > Hello,
>                         >
>                         > searching in the database using person view
>                         is very slow on all my personal
>                         > computers and notebooks. For a Surname it
>                         takes around 20 seconds until the
>                         > result is displayed. I've nearly 28000
>                         persons in my database. Is there a
>                         > way to make searching more comfortable for
>                         big database?
>                         >
>                         > Best regards,
>                         >
>                         > Anton
>                         >
>                         >
>                         >
>
>                         >
>                         ------------------------------------------------------------------------------
>                         > Everyone hates slow websites. So do we.
>                         > Make your web apps faster with AppDynamics
>                         > Download AppDynamics Lite for free today:
>                         >
>                         http://ad.doubleclick.net/clk;258768047;13503038;j?
>                         >
>                         http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>                         >
>                         >
>                         >
>                         >
>                         _______________________________________________
>                         > Gramps-devel mailing list
>                         > [hidden email]
>                         >
>                         https://lists.sourceforge.net/lists/listinfo/gramps-devel
>                         >
>
>
>
>                         ------------------------------------------------------------------------------
>                         Got visibility?
>                         Most devs has no idea what their production
>                         app looks like.
>                         Find out how fast your code is with
>                         AppDynamics Lite.
>                         http://ad.doubleclick.net/clk;262219671;13503038;y?
>                         http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>                         _______________________________________________
>                         Gramps-devel mailing list
>                         [hidden email]
>                         https://lists.sourceforge.net/lists/listinfo/gramps-devel
>
>
>
>
>
>
>         ------------------------------------------------------------------------------
>         Don't let slow site performance ruin your business. Deploy New
>         Relic APM
>         Deploy New Relic app performance management and know exactly
>         what is happening inside your Ruby, Python, PHP, Java,
>         and .NET app
>         Try New Relic at no cost today and get our sweet Data Nerd
>         shirt too!
>         http://p.sf.net/sfu/newrelic-dev2dev
>         _______________________________________________
>         Gramps-devel mailing list
>         [hidden email]
>         https://lists.sourceforge.net/lists/listinfo/gramps-devel
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________ Gramps-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gramps-devel

--
Robert Cawley
Ph: <a href="tel:%2B61%202%209543%201241" value="+61295431241">+61 2 9543 1241
[hidden email]
[hidden email]
[hidden email]




------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel



------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel