help me understand the inner workings of gramps -- what happens when I remove...

classic Classic list List threaded Threaded
38 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Gramps : Database design principles

neut
Excuse me for butting in here, as a mere occasional (wintertime)
ancestry buff,
I know nothing of the db s/w which is being used on various platforms,
but perhaps it is time to review those which were of the lite (or light)
variety, and which may not support the facilities of larger  dbs, such
as are beginning to be created by the increasing number of users.

As a retired IT specialist, who worked alongside real db experts,  I
have dredged from the back of my memory, some principles of db design,
for where tables are growing rapidly.

My suggestion is to llok at the use of techniques such table indexes and
two-way (forward and backward) pointers  for individual larger tables
which are liable to large volumes of changes.

I am sure that there are plenty of texts on db tuning online, and if you
look at (for example) Oracle, which had, to my memory, lots of such
techniques as options (I an NOT recommending Oracle at all for Gramps
btw), it might aid you to select better db s/w to take Gramps forward

Keep up the good work



------------------------------------------------------------------------------
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: Gramps : Database design principles

Brad Rogers
On Thu, 13 Aug 2015 12:13:05 +0200
mtoby <[hidden email]> wrote:

Hello mtoby,

>I know nothing of the db s/w which is being used on various platforms,
>but perhaps it is time to review those which were of the lite (or
>light) variety, and which may not support the facilities of larger
>dbs, such as are beginning to be created by the increasing number of
>users.

Such work is already being done.  There's also some discussion regarding
the matter on the developer's list.

Although, to be fair, Tim's problems may not be solely due to poor
database scaling, but Gramps' need to update the screen constantly.
Work is also being done on that.  Improvements are being made, I believe.

--
 Regards  _
         / )           "The blindingly obvious is
        / _)rad        never immediately apparent"
You don't entertain ideas you simply bore them
I Don't Like You - Stiff Little Fingers

------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users

attachment0 (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Gramps : Database design principles

Brad Rogers
In reply to this post by neut
On Thu, 13 Aug 2015 12:13:05 +0200
mtoby <[hidden email]> wrote:

Hello mtoby,

>Keep up the good work

Hear, hear.

I should add that I'm not on the dev team, but do read messages on the
developer mailing list.

--
 Regards  _
         / )           "The blindingly obvious is
        / _)rad        never immediately apparent"
He signed up for just three years, it seemed a small amount
Tin Soldiers - Stiff Little Fingers

------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users

attachment0 (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OT: scroll speed in 4.X

enno
In reply to this post by Josip
Hi Josip,
> Now when playing with Doug's places test data i also notice terrible lag
> in scrolling. It's quite unacceptable for table with only 65K rows.
> Firefox SQLiteManager plugin for example did not have such problems with
> same database, both fill and scroll of view is fast.
>
> I agree that we do something wrong and must be that overuse one of the
> callbacks. Or in gtk-2 to gtk-3 transition one of our private function
> now overrides gtk's one (ones with "do_" prefix)
I think it's overuse, i.e. fellow developers didn't expects callbacks
being triggered for events like a row becoming visible, which happens a
lot during scrolling, or on mouse hover. You don't even have to click
for that.

In tree view, one of these callbacks tries to find a person by walking
the tree, starting at the first (or empty) surname, scanning all persons
grouped under that surname, until the person is found. The time needed
for this search has a linear relation to the place of the person in the
sorted tree, so the further you scroll to Z, the longer it takes. This
happens for all persons that appear on screen during scrolling, also
when their surname is collapsed I think, and each callback reads data
from the database.

We cannot override this callback, nor a few others, because the person
needs to found to refresh the view on delete, or when a surname is
changed, and the person has to be shown in another part of the tree. And
I bet the same is true for places shown in a hierarchy like you wrote.

A faster database like SQLite will not prevent this, but if these
callbacks use the database, scrolling will of course be faster. We
should not treat the symptom though, but change the code so that it
makes better use of gtk-3.

Nick made a prototype which scrolls very well, but the view in that
takes longer to load because all data is loaded in memory, so that we
need less callbacks to read the database, and above search can be
avoided. Loading the view takes a bit longer then, and I don't know how
good it works when you have a Million persons in your tree. A faster
database will very probably help though, in that case.
> Again for example:
> https://git.gnome.org/browse/pygobject/plain/demos/gtk-demo/demos/TreeView/treemodel_large.py
> changing number of rows (item_count) from hundred thousand to one
> million affect only loading time not scrolling one
Right, and I even tried with 10 Million, and saw no delay in scrolling.
It took a few minutes to initialize the data, and it takes more than 500
MB RAM, but it scrolls fast. Only thing is that this doesn't really look
like a tree view where you can expand rows like we can expand surnames
or places.

regards,

Enno


------------------------------------------------------------------------------
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: Gramps : Database design principles

DS Blank
In reply to this post by neut
On Thu, Aug 13, 2015 at 6:13 AM, mtoby <[hidden email]> wrote:
Excuse me for butting in here, as a mere occasional (wintertime)
ancestry buff,

Comments and questions are always welcomed!
 
I know nothing of the db s/w which is being used on various platforms,
but perhaps it is time to review those which were of the lite (or light)
variety, and which may not support the facilities of larger  dbs, such
as are beginning to be created by the increasing number of users.

For those that aren't aware, Gramps 5.0 (to be released in 2016) is planned to have database backends "plugins". That means that you can pick what database you want to use for each family tree. If you are fine using the current BSDDB backend, you can continue to use that. Others may want to use an option that is designed for huge databases, or for collaborative use.
 

As a retired IT specialist, who worked alongside real db experts,  I
have dredged from the back of my memory, some principles of db design,
for where tables are growing rapidly.

My suggestion is to llok at the use of techniques such table indexes and
two-way (forward and backward) pointers  for individual larger tables
which are liable to large volumes of changes.

Currently in Gramps "master" and "addons" in github [1,2], there are four database backends defined:

1) BSDDB - the backend that Gramps has always used. 

2) DB-API 2.0 - an API defined in Python for using a variety of databases, including sqlite3, mysql, postgresql, and others.

3) DictionaryDb - an in-memory database; saves/loads to/from Gramps XML. 

4) DjangoDb - allows regular Gramps Gtk to use the Gramps-Connect database. This database can be any valid Django database, which includes almost any database you can think of. These are designed for the web API, but will work with Gramps Gtk (not designed for Gramps Gtk, though---fast on web, slow on desktop; a trade-off).

Each of these is intended to use the best technique to make sure that all functions are fast. That means that none of them will linearly "scan" through looking for data, but will use indexes (or other means) to directly access data quickly.
 

I am sure that there are plenty of texts on db tuning online, and if you
look at (for example) Oracle, which had, to my memory, lots of such
techniques as options (I an NOT recommending Oracle at all for Gramps
btw), it might aid you to select better db s/w to take Gramps forward

Analysing database use was always a bottleneck for Gramps up to now: there aren't very many tools for optimizing BSDDB. On the other hand, there are many SQL tools for optimizing searches that "regular" programmers can use (ie, use without having to have very specific knowledge). 

Here are some timing tests for importing a medium-sized number of records from latest development:

                XML-1.5   XML-1.6   JSON
-----------------------------------------
gramps42        44m51s    4m20s     2m00s
gramps50:bsddb  45m27s    4m37s     2m10s
gramps50:dbapi     50s      48s       30s
gramps50:dict      24s      20s       26s

So, currently, DB-API is currently anywhere from 4 to 45 times faster than our BSDDB code. I am sure that with enough time and effort, BSDDB could do as well as any. But perhaps the point here is that it is really easy to make SQL fast---I've just written this code in the last few months and days, whereas we have been working on BSDDB for more than 10 years.


Keep up the good work

Thanks; much appreciated! I'm looking forward to seeing how these changes can make Gramps useful for you all... imagine many people around the world collaborating simultaneously on a single tree using Gramps Gtk.

-Doug


 



------------------------------------------------------------------------------
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users


------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: OT: scroll speed in 4.X

DS Blank
In reply to this post by enno
On Thu, Aug 13, 2015 at 7:52 AM, Enno Borgsteede <[hidden email]> wrote:
Hi Josip,
> Now when playing with Doug's places test data i also notice terrible lag
> in scrolling. It's quite unacceptable for table with only 65K rows.
> Firefox SQLiteManager plugin for example did not have such problems with
> same database, both fill and scroll of view is fast.
>
> I agree that we do something wrong and must be that overuse one of the
> callbacks. Or in gtk-2 to gtk-3 transition one of our private function
> now overrides gtk's one (ones with "do_" prefix)
I think it's overuse, i.e. fellow developers didn't expects callbacks
being triggered for events like a row becoming visible, which happens a
lot during scrolling, or on mouse hover. You don't even have to click
for that.

In tree view, one of these callbacks tries to find a person by walking
the tree, starting at the first (or empty) surname, scanning all persons
grouped under that surname, until the person is found. The time needed
for this search has a linear relation to the place of the person in the
sorted tree, so the further you scroll to Z, the longer it takes. This
happens for all persons that appear on screen during scrolling, also
when their surname is collapsed I think, and each callback reads data
from the database.

We cannot override this callback, nor a few others, because the person
needs to found to refresh the view on delete, or when a surname is
changed, and the person has to be shown in another part of the tree. And
I bet the same is true for places shown in a hierarchy like you wrote.

A faster database like SQLite will not prevent this, but if these
callbacks use the database, scrolling will of course be faster. We
should not treat the symptom though, but change the code so that it
makes better use of gtk-3.

Nick made a prototype which scrolls very well, but the view in that
takes longer to load because all data is loaded in memory, so that we
need less callbacks to read the database, and above search can be
avoided. Loading the view takes a bit longer then, and I don't know how
good it works when you have a Million persons in your tree. A faster
database will very probably help though, in that case.

We should get Nick's work in Gramps ASAP.

 
> Again for example:
> https://git.gnome.org/browse/pygobject/plain/demos/gtk-demo/demos/TreeView/treemodel_large.py
> changing number of rows (item_count) from hundred thousand to one
> million affect only loading time not scrolling one
Right, and I even tried with 10 Million, and saw no delay in scrolling.
It took a few minutes to initialize the data, and it takes more than 500
MB RAM, but it scrolls fast. Only thing is that this doesn't really look
like a tree view where you can expand rows like we can expand surnames
or places.

That makes sense. We can also add "paged views" to show only a few thousand at a time. But let's get Nick's code in first.

Thanks, Enno, for keeping on top of this.

-Doug
 

regards,

Enno


------------------------------------------------------------------------------
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users


------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: Gramps : Database design principles

Ron Johnson
In reply to this post by neut
On 08/13/2015 07:10 AM, Doug Blank wrote:
[snip]
2) DB-API 2.0 - an API defined in Python for using a variety of databases, including sqlite3, mysql, postgresql, and others.

Will the records still be in Pickle format, or individually named fields?

3) DictionaryDb - an in-memory database; saves/loads to/from Gramps XML.

Gramps still freezes on me too often to even think of using this!!  :(

[snip]
Here are some timing tests for importing a medium-sized number of records from latest development:

                XML-1.5   XML-1.6   JSON
-----------------------------------------
gramps42        44m51s    4m20s     2m00s
gramps50:bsddb  45m27s    4m37s     2m10s
gramps50:dbapi     50s      48s       30s
gramps50:dict      24s      20s       26s

Whate are XML-1.6 and JSON?

Will JSON replace XML as the backup format?

-- 
"Salads are only for murderers, cole slaw's a fascist regime!"

------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: Gramps : Database design principles

DS Blank
On Thu, Aug 13, 2015 at 10:38 AM, Ron Johnson <[hidden email]> wrote:
On 08/13/2015 07:10 AM, Doug Blank wrote:
[snip]
2) DB-API 2.0 - an API defined in Python for using a variety of databases, including sqlite3, mysql, postgresql, and others.

Will the records still be in Pickle format, or individually named fields?

There is currently little to no benefit to Gramps Gtk to have data in specific fields, and in fact would require more work to maintain those as data changes. Currently, all persistent database formats use pickle to serialize the data. 

Having said that, one could easily imagine another database backend (say using the human-oriented Sqlite export in Addons) where all of the data were in fields, and thus would not require serialization/pickle. If you know a little SQL, that backend would be easy to create from the DB-API.
 


3) DictionaryDb - an in-memory database; saves/loads to/from Gramps XML.

Gramps still freezes on me too often to even think of using this!!  :(

We need to fix whatever causes that! I have never/rarely had that happen.
 

[snip]
Here are some timing tests for importing a medium-sized number of records from latest development:

                XML-1.5   XML-1.6   JSON
-----------------------------------------
gramps42        44m51s    4m20s     2m00s
gramps50:bsddb  45m27s    4m37s     2m10s
gramps50:dbapi     50s      48s       30s
gramps50:dict      24s      20s       26s

Whate are XML-1.6 and JSON?

XML-1.5 is the version of Gramps XML with flat places; XML-1.6 is with place hierarchies [1]. 

JSON is a JSON-like format (ie, Python dictionaries and lists) with an export/import available in Addons. It was designed to be a better XML format: it is fully-specified, easy to convert to Gramps from disk, can be used for Queries (see Query Addon), and can be used for DIFFs (see Database Differences Report in Addons), and can perhaps be more useful to future software/people.
 

Will JSON replace XML as the backup format?

We haven't discussed that yet. But you can use it now, by just exporting/importing it. (To use JSON import, you must manually rebuild references, gender maps, and secondary indexes after importing). JSON import/export needs more testing overall before considering it for use in core Gramps.

-Doug




-- 
"Salads are only for murderers, cole slaw's a fascist regime!"

------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users



------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: Gramps : Database design principles

Ron Johnson
In reply to this post by Ron Johnson
On 08/13/2015 10:28 AM, Doug Blank wrote:
[snip]
Here are some timing tests for importing a medium-sized number of records from latest development:

                XML-1.5   XML-1.6   JSON
-----------------------------------------
gramps42        44m51s    4m20s     2m00s
gramps50:bsddb  45m27s    4m37s     2m10s
gramps50:dbapi     50s      48s       30s
gramps50:dict      24s      20s       26s

Whate are XML-1.6 and JSON?

XML-1.5 is the version of Gramps XML with flat places; XML-1.6 is with place hierarchies [1].

There must be something else different for the import to be 11x faster, even in v4.2!!

-- 
"Salads are only for murderers, cole slaw's a fascist regime!"

------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: Gramps : Database design principles

DS Blank
On Thu, Aug 13, 2015 at 11:42 AM, Ron Johnson <[hidden email]> wrote:
On 08/13/2015 10:28 AM, Doug Blank wrote:
[snip]
Here are some timing tests for importing a medium-sized number of records from latest development:

                XML-1.5   XML-1.6   JSON
-----------------------------------------
gramps42        44m51s    4m20s     2m00s
gramps50:bsddb  45m27s    4m37s     2m10s
gramps50:dbapi     50s      48s       30s
gramps50:dict      24s      20s       26s

Whate are XML-1.6 and JSON?

XML-1.5 is the version of Gramps XML with flat places; XML-1.6 is with place hierarchies [1].

There must be something else different for the import to be 11x faster, even in v4.2!!

This file has a large number of places... ironically 66k in v 1.6 (fast), 900 in v 1.5 (slow). When importing, the old style is expanded, but looks like BSDDB (in both gramps42 and gramps50) can't handle this very well (some indexes are reportedly turned off in BSDDB import). Problem probably stems from the issue described in this bug:


There is more discussion on the gramps-developers mailing list.

-Doug
 


-- 
"Salads are only for murderers, cole slaw's a fascist regime!"

------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users



------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: OT: scroll speed in 4.X

Josip
In reply to this post by enno
13. 08. 2015. u 13:52, Enno Borgsteede je napisao/la:
>> Again for example:
>> > https://git.gnome.org/browse/pygobject/plain/demos/gtk-demo/demos/TreeView/treemodel_large.py
>> > changing number of rows (item_count) from hundred thousand to one
>> > million affect only loading time not scrolling one
> Right, and I even tried with 10 Million, and saw no delay in scrolling.
> It took a few minutes to initialize the data, and it takes more than 500
> MB RAM, but it scrolls fast. Only thing is that this doesn't really look
> like a tree view where you can expand rows like we can expand surnames
> or places.

All i want to say is that scrolling view is fast unless slowed down with
unnecessary or badly written callback.

Speed of displaying view can be handled in two ways: only display small
portion of data or dynamically update view outer from main thread.
Best is to use both of them togheter and treeview is good for that.


Check code at:
https://gist.github.com/bpisoj/ef89112fa0b5083bed37

It is a simple file browser. It loads first directory level from homedir
quickly and is ready to use (scroll), in meantime other directory levels
is added in background not blocking gui.
To see it work in slow-motion (gui is still normally usable) uncomment
"time.sleep" lines in "_thread_scanning" method.

p.s.
It is just a quick draft, it should be recursive but i think it is good
inaf to show how new gazetter can load/shows more info





--
Josip

------------------------------------------------------------------------------
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: Gramps : Database design principles

neut
In reply to this post by DS Blank
On 13/08/2015 14:10, Doug Blank wrote:
[snip]

For those that aren't aware, Gramps 5.0 (to be released in 2016) is planned to have database backends "plugins". That means that you can pick what database you want to use for each family tree. If you are fine using the current BSDDB backend, you can continue to use that. Others may want to use an option that is designed for huge databases, or for collaborative use.
 
This raises key questions for the developers, in that you will have to issue guidance to users who

a. know nothing about databases or plugins
b. have no idea how many entries they may have to start with, and  for example, if they accept and follow up every hint that Ancestry.com supplies may end up with many distant ancestors and potential living and very distant relatives (in my case one of my ancestors seems to have had a (possibly adopted) brother who was on HMS Bounty, and who has many living descendants from Pitcairn Island still living in that area- Marlon Brando eat your heart out).
c. will need tools to help them with measuring the size of their evolving database(s), and offering easy to use tools to transit from one backend to another
d. Can they tune the size of their Gramps programs, by simple options  and in easy to understand layman's terms,  to have in-memory tables if recommended by Gramps, or setting certain preferences in Gramps ( please bear in mind that they should have no idea at what cost to their computers which can be on just about any platform and any operating system)
e. If they follow guidance and are told to select a sophisticated database because of their end-user needs, are there a good set of user friendly support tools, which may actually be Gramps front-ended to the complex underlying tools that do the real work
[snip]

Currently in Gramps "master" and "addons" in github [1,2], there are four database backends defined:

1) BSDDB - the backend that Gramps has always used. 

2) DB-API 2.0 - an API defined in Python for using a variety of databases, including sqlite3, mysql, postgresql, and others.

3) DictionaryDb - an in-memory database; saves/loads to/from Gramps XML. 

4) DjangoDb - allows regular Gramps Gtk to use the Gramps-Connect database. This database can be any valid Django database, which includes almost any database you can think of. These are designed for the web API, but will work with Gramps Gtk (not designed for Gramps Gtk, though---fast on web, slow on desktop; a trade-off).

Each of these is intended to use the best technique to make sure that all functions are fast. That means that none of them will linearly "scan" through looking for data, but will use indexes (or other means) to directly access data quickly.

Doug, I have no doubt at all that the above reply is clearly intended to be helpful, in reply to my mail, but I have used computers continuously since 1964, as a developer, pre-sales and post sales support, product introduction manager, technical manual reviewer etcetera, and I understand nothing of what has been said, in terms of how to choose which backend to use. So how will a typical Gramps user with practically no IT knowledge, be able to decide what to use !

Please everybody, my reply here is meant to be helpful to the development team , to improve the product, and make it user-friendly to all types of user, from the person who records 100 people in one locality, up to those with a million data items or more, who share them with distant relatives far far away

[snip]



Analysing database use was always a bottleneck for Gramps up to now: there aren't very many tools for optimizing BSDDB. On the other hand, there are many SQL tools for optimizing searches that "regular" programmers can use (ie, use without having to have very specific knowledge).
As I said above,  it should be possible to put a Gramps front-end on the SQL tools for all of the others who are not regular programmers to use

Here are some timing tests for importing a medium-sized number of records from latest development:

                XML-1.5   XML-1.6   JSON
-----------------------------------------
gramps42        44m51s    4m20s     2m00s
gramps50:bsddb  45m27s    4m37s     2m10s
gramps50:dbapi     50s      48s       30s
gramps50:dict      24s      20s       26s

So, currently, DB-API is currently anywhere from 4 to 45 times faster than our BSDDB code. I am sure that with enough time and effort, BSDDB could do as well as any. But perhaps the point here is that it is really easy to make SQL fast---I've just written this code in the last few months and days, whereas we have been working on BSDDB for more than 10 years.

Perhaps this example may aid in seeing where I am coming from.

20 odd years ago, the IT company that I worked for was developing a one-size fits all, DMS database, to manage corporate networks for both voice and data, and this would contain information on all nodes, of all types, and also contain information on up time, down time, connections, locations and so on.

This was in the era when relational databases were just being developed, but the developers were trying to adapt a system suitable for a particular huge corporate entity, to work with a totally different audience as well. This was all very well for the organisations who had many mainframes, dedicated DM developers, DM sizing teams, Data support controllers who tracked the the day-to-day usage of the database, and knew when to take the DB off-line to retune and reorganise it, but all of this was beyond the capability of companies with a single mainframe and no DM experience at all, and as such, I was in the product introduction team which  rejected its general release because it was not fit for the wide market it was targetted at, (and within 2 years, a sophisticated relational database, acceptable to this market, took its place)

So the fact that data can be populated faster or slower in a database means nothing much to an end user, since it is (possibly) a one-off event.

The end-user is interested in getting
fast responses to their easy to choose and use queries,
the ability to modify their data easily and quickly in a time which is acceptable (less than 10 seconds),
the revised information appear on the screen, again in a time which is acceptable (less than 10 seconds)
no need to re-organise the db (or unload and reload it) every time they modify it by adding 100 pieces of information (whether from a serial input or a screen input)
I repeat - Keep up the good work


Thanks; much appreciated! I'm looking forward to seeing how these changes can make Gramps useful for you all... imagine many people around the world collaborating simultaneously on a single tree using Gramps Gtk.

I can Doug, I can

Regards


------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: Gramps : Database design principles

Ron Johnson
In reply to this post by DS Blank
On 08/13/2015 02:58 PM, mtoby wrote:
[snip]
> Doug, I have no doubt at all that the above reply is clearly intended to
> be helpful, in reply to my mail, but I have used computers continuously
> since 1964, as a developer, pre-sales and post sales support, product
> introduction manager, technical manual reviewer etcetera, and I understand
> nothing of what has been said, in terms of how to choose which backend to
> use. So how will a typical Gramps user with practically no IT knowledge,
> be able to decide what to use !

Like with just about every other computer choice that the technically
illiterate make: accept the default.

The import program might also look at the size of the import file, and ask
some questions before making a suggestion.

[snip]
> So the fact that data can be populated faster or slower in a database
> means nothing much to an end user, since it is (possibly) a one-off event.

Well that's just not true.  As someone who's been in post-sales support, you
must know that a system is useless if it takes an impractically long time to
load do the initial load.

--
"Salads are only for murderers, cole slaw's a fascist regime!"


------------------------------------------------------------------------------
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: Gramps : Database design principles

DS Blank
In reply to this post by neut
On Thu, Aug 13, 2015 at 3:58 PM, mtoby <[hidden email]> wrote:

[snip]
 
This raises key questions for the developers, in that you will have to issue guidance to users who

[snip]

Sorry if I were unclear... my note was more of a status update, and you are right that we need to document all of that clearly. As Ron noted, we'll have sensible defaults, and for most people, that will be it. No one need to consider changing backends unless they have a specific need.
 
[snip]

So the fact that data can be populated faster or slower in a database means nothing much to an end user, since it is (possibly) a one-off event.

You are correct that we should have stats on all of the major functions. But importing data does test many of them. The GUI issues (how long to load, page through, scroll, etc) will also need to be tested. Gramps Gtk has never been used with different backends before, so there may be integration issues that emerge.
 
There is much to be done for Gramps 5.0, and we are always looking for help developing, testing, translating, and documenting. 

Stay tuned!

-Doug

------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: help me understand the inner workings of gramps -- what happens when I remove...

TJMcK
In reply to this post by Brad Rogers
Brad Rogers wrote
Yes, that's exactly what I meant.  Because, at the current rate of
excision of spurious data from your large database, it's going to take
over 1200 hours to accomplish your goal.  With two copies of Gramps
open, one with the old db and one with the new, cutting and pasting
between the two may not take any longer.
Any way you look at it, this situation will be time-consuming with the resources that I have right now.  I guess that's why I'm hoping for a solution.  For instance, if I can get the patch to work that bipasses all the unnecessary procedures for batch removals.  Or if gramps v.5 is released within a year or two, I'd consider waiting...  it's not that the database doesn't work, it's just that half the data is unnecessary.

Finally; I marvel at the size of your db.  I've only been at genealogy
for ten years or so, and have fewer than one thousand individuals
contained therein.  To get 600k people would take me over 600 years,
based on my work rate of the last decade.  And that's ignoring the laws
of diminishing returns, and the almost inevitable tree collapse.
The 600k db was mostly a cumulation of gedcoms that I imported from others (some times I didn't know for sure if anyone was related, but my pholosophy at that time was: if they could be related then import it).  At one time I had about 700k, but there were/are so many duplicate/triplicate people. (And I would spend about 80% of my time merging them -- and more times then not there was some source or piece of data that did not exist in one of the duplicates.)  About a year ago I exported all related people to me from the 600k, and arrived at 158k. (Of these 158k I have really only discovered, maybe 1/4 of them.)  But I generated a gedcom from the 600k that I currently use as a resorce, and recently I discovered a new line of relatives that was in the 600k db and so I exported it (that's where my problems all started). I still think there is a fair bit of info in the 600k db that I've missed -- and I still like the idea of having one big database so that any potential new data is always available -- it's like having my own "ancestry.com" (but currently it's clear that gramps is not made to handle 600k dbases).

BTW, I started my family history in 1994 (so about 20 years ago), and in between I did take a few long breaks from working on it.
Reply | Threaded
Open this post in threaded view
|

Re: OT: scroll speed in 4.X

TJMcK
In reply to this post by Josip
Josip wrote
13. 08. 2015. u 13:52, Enno Borgsteede je napisao/la:
>> Again for example:
>> > https://git.gnome.org/browse/pygobject/plain/demos/gtk-demo/demos/TreeView/treemodel_large.py
>> > changing number of rows (item_count) from hundred thousand to one
>> > million affect only loading time not scrolling one
> Right, and I even tried with 10 Million, and saw no delay in scrolling.
> It took a few minutes to initialize the data, and it takes more than 500
> MB RAM, but it scrolls fast. Only thing is that this doesn't really look
 
I'd be interested to know how long it would take to initialize if a SSD was used.  I found that since I switch to a SSD, gramps starts about 10x faster...  if SSDs are going to be default storage in the future, why not build future gramps with this is mind.

  tim k
Reply | Threaded
Open this post in threaded view
|

Re: Gramps : Database design principles

Peter Merchant
In reply to this post by Ron Johnson
On 13/08/15 21:19, Ron Johnson wrote:

> On 08/13/2015 02:58 PM, mtoby wrote:
> [snip]
>> Doug, I have no doubt at all that the above reply is clearly intended to
>> be helpful, in reply to my mail, but I have used computers continuously
>> since 1964, as a developer, pre-sales and post sales support, product
>> introduction manager, technical manual reviewer etcetera, and I understand
>> nothing of what has been said, in terms of how to choose which backend to
>> use. So how will a typical Gramps user with practically no IT knowledge,
>> be able to decide what to use !
> Like with just about every other computer choice that the technically
> illiterate make: accept the default.
>
--- Even the technically literate will accept the defaults in many cases
because of the hopefully correct assumptions that the defaults are the
'best' choices that the developers could make, and we assume that the
developers are technically literate.

I am very much in the same technically educated sphere as Mtoby, have
developed communications protocols, databases for Packet switch systems
management, and all sorts of other good things.

Peter M.

------------------------------------------------------------------------------
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: help me understand the inner workings of gramps -- what happens when I remove...

Brad Rogers
In reply to this post by TJMcK
On Thu, 13 Aug 2015 21:23:54 -0700 (PDT)
TJMcK <[hidden email]> wrote:

Hello TJMcK,

>all the unnecessary procedures for batch removals.  Or if gramps v.5 is
>released within a year or two, I'd consider waiting...  it's not that

There was notice given of a release date of 2016 not so long ago.  I
can't remember whether it was here on the dev list.  Either way, it
might be worth hanging on for v5.

--
 Regards  _
         / )           "The blindingly obvious is
        / _)rad        never immediately apparent"
Now would I say something that wasn't true?
Would I Lie To You - Eurythmics

------------------------------------------------------------------------------

_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users

attachment0 (484 bytes) Download Attachment
12