XML load times

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

XML load times

Joel Parker-4
My wife and I are working with a database in Gramps XML format, so we
can share the file without the grdb hassles.

The database has about 700 individuals, along with maybe 50 media links
(some of which are broken, if that matters). I'm using 2.2.8 on Ubuntu,
and she's using 2.2.7 on Windows XP.

On Windows with 2.2.7, she's getting load times of maybe 90 seconds to
open the file (that's after the app has loaded, I do File->Open from
there). On Ubuntu with 2.2.8, I get load times of a full 11 minutes at
100% CPU, sometimes ending in a "This file is either corrupt or not a
Gramps file" message (and sometimes completing successfully).

My computer is a 3GHz P4, while hers is a ~1.8GHz Pentium M, but I doubt
that would make a difference that large.

Any idea what's causing this discrepancy?



--
Joel Parker


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: XML load times

bm-5
Quoting Joel Parker <[hidden email]>:

> My wife and I are working with a database in Gramps XML format, so we
> can share the file without the grdb hassles.
>
> The database has about 700 individuals, along with maybe 50 media links
> (some of which are broken, if that matters). I'm using 2.2.8 on Ubuntu,
> and she's using 2.2.7 on Windows XP.

You should use the same version. 2.2.8 corrects not saved privacy settings on
save to xml

> On Windows with 2.2.7, she's getting load times of maybe 90 seconds to
> open the file (that's after the app has loaded, I do File->Open from
> there). On Ubuntu with 2.2.8, I get load times of a full 11 minutes at
> 100% CPU, sometimes ending in a "This file is either corrupt or not a
> Gramps file" message (and sometimes completing successfully).

I have seen this with .grdb file, but never with .gramps file.
If you open an empty grdb file, and do import of xml file, is it faster?

Is this 11 min load times always?

I do not remember, but perhaps there is a database upgrade from 2.2.7
to 2.2.8,
you should start gramps from the command line and see if some special upgrade
messages pass by. I would however expect database upgrade issues not to play a
role when working with xml files....

>
> My computer is a 3GHz P4, while hers is a ~1.8GHz Pentium M, but I doubt
> that would make a difference that large.

hmm, you said you had the slow load times, so the 3GHz would be slower
than the
1.8, no ? Anyway, there should be no difference between the two.

>
> Any idea what's causing this discrepancy?
>
>
>
> --
> Joel Parker
>

Benny


----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: XML load times

Don Allingham
Something odd is definitely going on here. There was no upgrade between
2.2.7 and 2.2.8, so that should not be a problem.

My guess is the problem is the amount of RAM on the systems. When you
use the .gramps format, GRAMPS must keep *all* data in memory at once.
So, if you have more data than you have available RAM, the system will
dramatically slow down.

Does the slow machine have less RAM than the faster machine?

Don

On Sat, 2007-08-04 at 16:38 +0200, [hidden email] wrote:

> Quoting Joel Parker <[hidden email]>:
>
> > My wife and I are working with a database in Gramps XML format, so we
> > can share the file without the grdb hassles.
> >
> > The database has about 700 individuals, along with maybe 50 media links
> > (some of which are broken, if that matters). I'm using 2.2.8 on Ubuntu,
> > and she's using 2.2.7 on Windows XP.
>
> You should use the same version. 2.2.8 corrects not saved privacy settings on
> save to xml
>
> > On Windows with 2.2.7, she's getting load times of maybe 90 seconds to
> > open the file (that's after the app has loaded, I do File->Open from
> > there). On Ubuntu with 2.2.8, I get load times of a full 11 minutes at
> > 100% CPU, sometimes ending in a "This file is either corrupt or not a
> > Gramps file" message (and sometimes completing successfully).
>
> I have seen this with .grdb file, but never with .gramps file.
> If you open an empty grdb file, and do import of xml file, is it faster?
>
> Is this 11 min load times always?
>
> I do not remember, but perhaps there is a database upgrade from 2.2.7
> to 2.2.8,
> you should start gramps from the command line and see if some special upgrade
> messages pass by. I would however expect database upgrade issues not to play a
> role when working with xml files....
>
> >
> > My computer is a 3GHz P4, while hers is a ~1.8GHz Pentium M, but I doubt
> > that would make a difference that large.
>
> hmm, you said you had the slow load times, so the 3GHz would be slower
> than the
> 1.8, no ? Anyway, there should be no difference between the two.
>
> >
> > Any idea what's causing this discrepancy?
> >
> >
> >
> > --
> > Joel Parker
> >
>
> Benny
>
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> Gramps-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-users

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: XML load times

Joel Parker-4
In reply to this post by bm-5
No, that's just it. The slow machine has 1GB RAM, while the fast one has
512MB (maybe 1GB, not sure at the moment).

I'll do some more tests today and see if I can tell what's going on.


- Joel




On Sat, 2007-08-04 at 09:16 -0600, Don Allingham wrote:

> Something odd is definitely going on here. There was no upgrade between
> 2.2.7 and 2.2.8, so that should not be a problem.
>
> My guess is the problem is the amount of RAM on the systems. When you
> use the .gramps format, GRAMPS must keep *all* data in memory at once.
> So, if you have more data than you have available RAM, the system will
> dramatically slow down.
>
> Does the slow machine have less RAM than the faster machine?
>
> Don
>
> On Sat, 2007-08-04 at 16:38 +0200, [hidden email] wrote:
> > Quoting Joel Parker <[hidden email]>:
> >
> > > My wife and I are working with a database in Gramps XML format, so we
> > > can share the file without the grdb hassles.
> > >
> > > The database has about 700 individuals, along with maybe 50 media links
> > > (some of which are broken, if that matters). I'm using 2.2.8 on Ubuntu,
> > > and she's using 2.2.7 on Windows XP.
> >
> > You should use the same version. 2.2.8 corrects not saved privacy settings on
> > save to xml
> >
> > > On Windows with 2.2.7, she's getting load times of maybe 90 seconds to
> > > open the file (that's after the app has loaded, I do File->Open from
> > > there). On Ubuntu with 2.2.8, I get load times of a full 11 minutes at
> > > 100% CPU, sometimes ending in a "This file is either corrupt or not a
> > > Gramps file" message (and sometimes completing successfully).
> >
> > I have seen this with .grdb file, but never with .gramps file.
> > If you open an empty grdb file, and do import of xml file, is it faster?
> >
> > Is this 11 min load times always?
> >
> > I do not remember, but perhaps there is a database upgrade from 2.2.7
> > to 2.2.8,
> > you should start gramps from the command line and see if some special upgrade
> > messages pass by. I would however expect database upgrade issues not to play a
> > role when working with xml files....
> >
> > >
> > > My computer is a 3GHz P4, while hers is a ~1.8GHz Pentium M, but I doubt
> > > that would make a difference that large.
> >
> > hmm, you said you had the slow load times, so the 3GHz would be slower
> > than the
> > 1.8, no ? Anyway, there should be no difference between the two.
> >
> > >
> > > Any idea what's causing this discrepancy?
> > >
> > >
> > >
> > > --
> > > Joel Parker
> > >
> >
> > Benny
> >
> >
> > ----------------------------------------------------------------
> > This message was sent using IMP, the Internet Messaging Program.
> >
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by: Splunk Inc.
> > Still grepping through log files to find problems?  Stop.
> > Now Search log events and configuration files using AJAX and a browser.
> > Download your FREE copy of Splunk now >>  http://get.splunk.com/
> > _______________________________________________
> > Gramps-users mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/gramps-users
--
Joel Parker


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: XML load times

Joel Parker-4
In reply to this post by bm-5
I did some more tests.

Again, the two systems (named from my first experience):
"fast" - 2.0 GHz Pentium M, 512 MiB RAM, Windows XP, Gramps 2.2.7
"slow" - 3.06 GHz Pentium 4, 1 GiB RAM, Ubuntu 7.04, Gramps 2.2.8

The file is a Gramps XML file with approx. 700 names. There is a linked
media folder with 67 images of various types totaling 55.1 MiB (several
high-resolution scans).

First, I made identical copies of the base .gramps XML file on each
computer and deleted the media folder, so that all the media links would
be broken.

In this case, both systems took about the same amount of time to load
the file (about 2 minutes).

Second, on the "slow" system, I copied back the media folder and
corrected all the links, so they were all working links to actual media
files, then tried opening the .gramps file again.

In this case, the file load times jumped up into the 15-minute range and
even higher (I killed it after a while). This whole time the CPU was
going at 100%, and the memory usage was creeping higher and higher,
until by the time I killed it, all of my system memory and half of my
1GB swap file was used.

I saw the same behavior on the "fast" system as well. This time it was a
little different, as Gramps loaded into Person view mode in a normal
amount of time, but went crazy with load time and memory when I pressed
the Media button.

Again, to summarize, with no media folder present, both systems loaded
the XML file in a couple minutes. With the media folder present and when
opening into Media view, the load times and memory usage went through
the roof.

What does Gramps try to do with the media during load when it's going
into Media view mode? Does it try to thumbnail all the pictures or load
them into memory or something?

- Joel

On Sat, 2007-08-04 at 09:16 -0600, Don Allingham wrote:

> Something odd is definitely going on here. There was no upgrade between
> 2.2.7 and 2.2.8, so that should not be a problem.
>
> My guess is the problem is the amount of RAM on the systems. When you
> use the .gramps format, GRAMPS must keep *all* data in memory at once.
> So, if you have more data than you have available RAM, the system will
> dramatically slow down.
>
> Does the slow machine have less RAM than the faster machine?
>
> Don
>
> On Sat, 2007-08-04 at 16:38 +0200, [hidden email] wrote:
> > Quoting Joel Parker <[hidden email]>:
> >
> > > My wife and I are working with a database in Gramps XML format, so we
> > > can share the file without the grdb hassles.
> > >
> > > The database has about 700 individuals, along with maybe 50 media links
> > > (some of which are broken, if that matters). I'm using 2.2.8 on Ubuntu,
> > > and she's using 2.2.7 on Windows XP.
> >
> > You should use the same version. 2.2.8 corrects not saved privacy settings on
> > save to xml
> >
> > > On Windows with 2.2.7, she's getting load times of maybe 90 seconds to
> > > open the file (that's after the app has loaded, I do File->Open from
> > > there). On Ubuntu with 2.2.8, I get load times of a full 11 minutes at
> > > 100% CPU, sometimes ending in a "This file is either corrupt or not a
> > > Gramps file" message (and sometimes completing successfully).
> >
> > I have seen this with .grdb file, but never with .gramps file.
> > If you open an empty grdb file, and do import of xml file, is it faster?
> >
> > Is this 11 min load times always?
> >
> > I do not remember, but perhaps there is a database upgrade from 2.2.7
> > to 2.2.8,
> > you should start gramps from the command line and see if some special upgrade
> > messages pass by. I would however expect database upgrade issues not to play a
> > role when working with xml files....
> >
> > >
> > > My computer is a 3GHz P4, while hers is a ~1.8GHz Pentium M, but I doubt
> > > that would make a difference that large.
> >
> > hmm, you said you had the slow load times, so the 3GHz would be slower
> > than the
> > 1.8, no ? Anyway, there should be no difference between the two.
> >
> > >
> > > Any idea what's causing this discrepancy?
> > >
> > >
> > >
> > > --
> > > Joel Parker
> > >
> >
> > Benny
> >
> >
> > ----------------------------------------------------------------
> > This message was sent using IMP, the Internet Messaging Program.
> >
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by: Splunk Inc.
> > Still grepping through log files to find problems?  Stop.
> > Now Search log events and configuration files using AJAX and a browser.
> > Download your FREE copy of Splunk now >>  http://get.splunk.com/
> > _______________________________________________
> > Gramps-users mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/gramps-users
--
Joel Parker


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: XML load times

Don Allingham
Can you send me the XML file so I can try some profiling on the date
here?

Don

On Sat, 2007-08-04 at 19:02 -0500, Joel Parker wrote:

> I did some more tests.
>
> Again, the two systems (named from my first experience):
> "fast" - 2.0 GHz Pentium M, 512 MiB RAM, Windows XP, Gramps 2.2.7
> "slow" - 3.06 GHz Pentium 4, 1 GiB RAM, Ubuntu 7.04, Gramps 2.2.8
>
> The file is a Gramps XML file with approx. 700 names. There is a linked
> media folder with 67 images of various types totaling 55.1 MiB (several
> high-resolution scans).
>
> First, I made identical copies of the base .gramps XML file on each
> computer and deleted the media folder, so that all the media links would
> be broken.
>
> In this case, both systems took about the same amount of time to load
> the file (about 2 minutes).
>
> Second, on the "slow" system, I copied back the media folder and
> corrected all the links, so they were all working links to actual media
> files, then tried opening the .gramps file again.
>
> In this case, the file load times jumped up into the 15-minute range and
> even higher (I killed it after a while). This whole time the CPU was
> going at 100%, and the memory usage was creeping higher and higher,
> until by the time I killed it, all of my system memory and half of my
> 1GB swap file was used.
>
> I saw the same behavior on the "fast" system as well. This time it was a
> little different, as Gramps loaded into Person view mode in a normal
> amount of time, but went crazy with load time and memory when I pressed
> the Media button.
>
> Again, to summarize, with no media folder present, both systems loaded
> the XML file in a couple minutes. With the media folder present and when
> opening into Media view, the load times and memory usage went through
> the roof.
>
> What does Gramps try to do with the media during load when it's going
> into Media view mode? Does it try to thumbnail all the pictures or load
> them into memory or something?
>
> - Joel
>
> On Sat, 2007-08-04 at 09:16 -0600, Don Allingham wrote:
> > Something odd is definitely going on here. There was no upgrade between
> > 2.2.7 and 2.2.8, so that should not be a problem.
> >
> > My guess is the problem is the amount of RAM on the systems. When you
> > use the .gramps format, GRAMPS must keep *all* data in memory at once.
> > So, if you have more data than you have available RAM, the system will
> > dramatically slow down.
> >
> > Does the slow machine have less RAM than the faster machine?
> >
> > Don
> >
> > On Sat, 2007-08-04 at 16:38 +0200, [hidden email] wrote:
> > > Quoting Joel Parker <[hidden email]>:
> > >
> > > > My wife and I are working with a database in Gramps XML format, so we
> > > > can share the file without the grdb hassles.
> > > >
> > > > The database has about 700 individuals, along with maybe 50 media links
> > > > (some of which are broken, if that matters). I'm using 2.2.8 on Ubuntu,
> > > > and she's using 2.2.7 on Windows XP.
> > >
> > > You should use the same version. 2.2.8 corrects not saved privacy settings on
> > > save to xml
> > >
> > > > On Windows with 2.2.7, she's getting load times of maybe 90 seconds to
> > > > open the file (that's after the app has loaded, I do File->Open from
> > > > there). On Ubuntu with 2.2.8, I get load times of a full 11 minutes at
> > > > 100% CPU, sometimes ending in a "This file is either corrupt or not a
> > > > Gramps file" message (and sometimes completing successfully).
> > >
> > > I have seen this with .grdb file, but never with .gramps file.
> > > If you open an empty grdb file, and do import of xml file, is it faster?
> > >
> > > Is this 11 min load times always?
> > >
> > > I do not remember, but perhaps there is a database upgrade from 2.2.7
> > > to 2.2.8,
> > > you should start gramps from the command line and see if some special upgrade
> > > messages pass by. I would however expect database upgrade issues not to play a
> > > role when working with xml files....
> > >
> > > >
> > > > My computer is a 3GHz P4, while hers is a ~1.8GHz Pentium M, but I doubt
> > > > that would make a difference that large.
> > >
> > > hmm, you said you had the slow load times, so the 3GHz would be slower
> > > than the
> > > 1.8, no ? Anyway, there should be no difference between the two.
> > >
> > > >
> > > > Any idea what's causing this discrepancy?
> > > >
> > > >
> > > >
> > > > --
> > > > Joel Parker
> > > >
> > >
> > > Benny
> > >
> > >
> > > ----------------------------------------------------------------
> > > This message was sent using IMP, the Internet Messaging Program.
> > >
> > > -------------------------------------------------------------------------
> > > This SF.net email is sponsored by: Splunk Inc.
> > > Still grepping through log files to find problems?  Stop.
> > > Now Search log events and configuration files using AJAX and a browser.
> > > Download your FREE copy of Splunk now >>  http://get.splunk.com/
> > > _______________________________________________
> > > Gramps-users mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/gramps-users


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: XML load times

Joel Parker-4
Ok, scratch all that talk about the media folder, something else is
going on here.

This time, I copied the original .gramps file from our shared folder to
each of the computers. There was no media folder, so all links were
broken. I opened each file, made a single change (changed a random media
path from absolute to relative), and exited Gramps again.

The original file was 162 KiB, consisting of 326,529 lines when
unzipped.

After the simple modification, the file had jumped to 470 KiB,
consisting of a full 1,881,729 lines, the vast majority of which are
additional <bookmark> lines.

So in the course of changing the path for a single media file, more then
1,500,000 lines had been added to my data file. This happened on both
computers, Windows and Ubuntu, 2.2.7 and 2.2.8.

Don, I'll send you both files under separate cover.

- Joel





On Sat, 2007-08-04 at 22:39 -0600, Don Allingham wrote:

> Can you send me the XML file so I can try some profiling on the date
> here?
>
> Don
>
> On Sat, 2007-08-04 at 19:02 -0500, Joel Parker wrote:
> > I did some more tests.
> >
> > Again, the two systems (named from my first experience):
> > "fast" - 2.0 GHz Pentium M, 512 MiB RAM, Windows XP, Gramps 2.2.7
> > "slow" - 3.06 GHz Pentium 4, 1 GiB RAM, Ubuntu 7.04, Gramps 2.2.8
> >
> > The file is a Gramps XML file with approx. 700 names. There is a linked
> > media folder with 67 images of various types totaling 55.1 MiB (several
> > high-resolution scans).
> >
> > First, I made identical copies of the base .gramps XML file on each
> > computer and deleted the media folder, so that all the media links would
> > be broken.
> >
> > In this case, both systems took about the same amount of time to load
> > the file (about 2 minutes).
> >
> > Second, on the "slow" system, I copied back the media folder and
> > corrected all the links, so they were all working links to actual media
> > files, then tried opening the .gramps file again.
> >
> > In this case, the file load times jumped up into the 15-minute range and
> > even higher (I killed it after a while). This whole time the CPU was
> > going at 100%, and the memory usage was creeping higher and higher,
> > until by the time I killed it, all of my system memory and half of my
> > 1GB swap file was used.
> >
> > I saw the same behavior on the "fast" system as well. This time it was a
> > little different, as Gramps loaded into Person view mode in a normal
> > amount of time, but went crazy with load time and memory when I pressed
> > the Media button.
> >
> > Again, to summarize, with no media folder present, both systems loaded
> > the XML file in a couple minutes. With the media folder present and when
> > opening into Media view, the load times and memory usage went through
> > the roof.
> >
> > What does Gramps try to do with the media during load when it's going
> > into Media view mode? Does it try to thumbnail all the pictures or load
> > them into memory or something?
> >
> > - Joel
> >
> > On Sat, 2007-08-04 at 09:16 -0600, Don Allingham wrote:
> > > Something odd is definitely going on here. There was no upgrade between
> > > 2.2.7 and 2.2.8, so that should not be a problem.
> > >
> > > My guess is the problem is the amount of RAM on the systems. When you
> > > use the .gramps format, GRAMPS must keep *all* data in memory at once.
> > > So, if you have more data than you have available RAM, the system will
> > > dramatically slow down.
> > >
> > > Does the slow machine have less RAM than the faster machine?
> > >
> > > Don
> > >
> > > On Sat, 2007-08-04 at 16:38 +0200, [hidden email] wrote:
> > > > Quoting Joel Parker <[hidden email]>:
> > > >
> > > > > My wife and I are working with a database in Gramps XML format, so we
> > > > > can share the file without the grdb hassles.
> > > > >
> > > > > The database has about 700 individuals, along with maybe 50 media links
> > > > > (some of which are broken, if that matters). I'm using 2.2.8 on Ubuntu,
> > > > > and she's using 2.2.7 on Windows XP.
> > > >
> > > > You should use the same version. 2.2.8 corrects not saved privacy settings on
> > > > save to xml
> > > >
> > > > > On Windows with 2.2.7, she's getting load times of maybe 90 seconds to
> > > > > open the file (that's after the app has loaded, I do File->Open from
> > > > > there). On Ubuntu with 2.2.8, I get load times of a full 11 minutes at
> > > > > 100% CPU, sometimes ending in a "This file is either corrupt or not a
> > > > > Gramps file" message (and sometimes completing successfully).
> > > >
> > > > I have seen this with .grdb file, but never with .gramps file.
> > > > If you open an empty grdb file, and do import of xml file, is it faster?
> > > >
> > > > Is this 11 min load times always?
> > > >
> > > > I do not remember, but perhaps there is a database upgrade from 2.2.7
> > > > to 2.2.8,
> > > > you should start gramps from the command line and see if some special upgrade
> > > > messages pass by. I would however expect database upgrade issues not to play a
> > > > role when working with xml files....
> > > >
> > > > >
> > > > > My computer is a 3GHz P4, while hers is a ~1.8GHz Pentium M, but I doubt
> > > > > that would make a difference that large.
> > > >
> > > > hmm, you said you had the slow load times, so the 3GHz would be slower
> > > > than the
> > > > 1.8, no ? Anyway, there should be no difference between the two.
> > > >
> > > > >
> > > > > Any idea what's causing this discrepancy?
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Joel Parker
> > > > >
> > > >
> > > > Benny
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > This message was sent using IMP, the Internet Messaging Program.
> > > >
> > > > -------------------------------------------------------------------------
> > > > This SF.net email is sponsored by: Splunk Inc.
> > > > Still grepping through log files to find problems?  Stop.
> > > > Now Search log events and configuration files using AJAX and a browser.
> > > > Download your FREE copy of Splunk now >>  http://get.splunk.com/
> > > > _______________________________________________
> > > > Gramps-users mailing list
> > > > [hidden email]
> > > > https://lists.sourceforge.net/lists/listinfo/gramps-users
>
--
Joel Parker


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: XML load times

Joel Parker-4
Has there been any progress on this issue?

Thanks,
- Joel


On Sun, 2007-08-05 at 00:23 -0500, Joel Parker wrote:

> Ok, scratch all that talk about the media folder, something else is
> going on here.
>
> This time, I copied the original .gramps file from our shared folder to
> each of the computers. There was no media folder, so all links were
> broken. I opened each file, made a single change (changed a random media
> path from absolute to relative), and exited Gramps again.
>
> The original file was 162 KiB, consisting of 326,529 lines when
> unzipped.
>
> After the simple modification, the file had jumped to 470 KiB,
> consisting of a full 1,881,729 lines, the vast majority of which are
> additional <bookmark> lines.
>
> So in the course of changing the path for a single media file, more then
> 1,500,000 lines had been added to my data file. This happened on both
> computers, Windows and Ubuntu, 2.2.7 and 2.2.8.
>
> Don, I'll send you both files under separate cover.
>
> - Joel
>
>
>
>
>
> On Sat, 2007-08-04 at 22:39 -0600, Don Allingham wrote:
> > Can you send me the XML file so I can try some profiling on the date
> > here?
> >
> > Don
> >
> > On Sat, 2007-08-04 at 19:02 -0500, Joel Parker wrote:
> > > I did some more tests.
> > >
> > > Again, the two systems (named from my first experience):
> > > "fast" - 2.0 GHz Pentium M, 512 MiB RAM, Windows XP, Gramps 2.2.7
> > > "slow" - 3.06 GHz Pentium 4, 1 GiB RAM, Ubuntu 7.04, Gramps 2.2.8
> > >
> > > The file is a Gramps XML file with approx. 700 names. There is a linked
> > > media folder with 67 images of various types totaling 55.1 MiB (several
> > > high-resolution scans).
> > >
> > > First, I made identical copies of the base .gramps XML file on each
> > > computer and deleted the media folder, so that all the media links would
> > > be broken.
> > >
> > > In this case, both systems took about the same amount of time to load
> > > the file (about 2 minutes).
> > >
> > > Second, on the "slow" system, I copied back the media folder and
> > > corrected all the links, so they were all working links to actual media
> > > files, then tried opening the .gramps file again.
> > >
> > > In this case, the file load times jumped up into the 15-minute range and
> > > even higher (I killed it after a while). This whole time the CPU was
> > > going at 100%, and the memory usage was creeping higher and higher,
> > > until by the time I killed it, all of my system memory and half of my
> > > 1GB swap file was used.
> > >
> > > I saw the same behavior on the "fast" system as well. This time it was a
> > > little different, as Gramps loaded into Person view mode in a normal
> > > amount of time, but went crazy with load time and memory when I pressed
> > > the Media button.
> > >
> > > Again, to summarize, with no media folder present, both systems loaded
> > > the XML file in a couple minutes. With the media folder present and when
> > > opening into Media view, the load times and memory usage went through
> > > the roof.
> > >
> > > What does Gramps try to do with the media during load when it's going
> > > into Media view mode? Does it try to thumbnail all the pictures or load
> > > them into memory or something?
> > >
> > > - Joel
> > >
> > > On Sat, 2007-08-04 at 09:16 -0600, Don Allingham wrote:
> > > > Something odd is definitely going on here. There was no upgrade between
> > > > 2.2.7 and 2.2.8, so that should not be a problem.
> > > >
> > > > My guess is the problem is the amount of RAM on the systems. When you
> > > > use the .gramps format, GRAMPS must keep *all* data in memory at once.
> > > > So, if you have more data than you have available RAM, the system will
> > > > dramatically slow down.
> > > >
> > > > Does the slow machine have less RAM than the faster machine?
> > > >
> > > > Don
> > > >
> > > > On Sat, 2007-08-04 at 16:38 +0200, [hidden email] wrote:
> > > > > Quoting Joel Parker <[hidden email]>:
> > > > >
> > > > > > My wife and I are working with a database in Gramps XML format, so we
> > > > > > can share the file without the grdb hassles.
> > > > > >
> > > > > > The database has about 700 individuals, along with maybe 50 media links
> > > > > > (some of which are broken, if that matters). I'm using 2.2.8 on Ubuntu,
> > > > > > and she's using 2.2.7 on Windows XP.
> > > > >
> > > > > You should use the same version. 2.2.8 corrects not saved privacy settings on
> > > > > save to xml
> > > > >
> > > > > > On Windows with 2.2.7, she's getting load times of maybe 90 seconds to
> > > > > > open the file (that's after the app has loaded, I do File->Open from
> > > > > > there). On Ubuntu with 2.2.8, I get load times of a full 11 minutes at
> > > > > > 100% CPU, sometimes ending in a "This file is either corrupt or not a
> > > > > > Gramps file" message (and sometimes completing successfully).
> > > > >
> > > > > I have seen this with .grdb file, but never with .gramps file.
> > > > > If you open an empty grdb file, and do import of xml file, is it faster?
> > > > >
> > > > > Is this 11 min load times always?
> > > > >
> > > > > I do not remember, but perhaps there is a database upgrade from 2.2.7
> > > > > to 2.2.8,
> > > > > you should start gramps from the command line and see if some special upgrade
> > > > > messages pass by. I would however expect database upgrade issues not to play a
> > > > > role when working with xml files....
> > > > >
> > > > > >
> > > > > > My computer is a 3GHz P4, while hers is a ~1.8GHz Pentium M, but I doubt
> > > > > > that would make a difference that large.
> > > > >
> > > > > hmm, you said you had the slow load times, so the 3GHz would be slower
> > > > > than the
> > > > > 1.8, no ? Anyway, there should be no difference between the two.
> > > > >
> > > > > >
> > > > > > Any idea what's causing this discrepancy?
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Joel Parker
> > > > > >
> > > > >
> > > > > Benny
> > > > >
> > > > >
> > > > > ----------------------------------------------------------------
> > > > > This message was sent using IMP, the Internet Messaging Program.
> > > > >
> > > > > -------------------------------------------------------------------------
> > > > > This SF.net email is sponsored by: Splunk Inc.
> > > > > Still grepping through log files to find problems?  Stop.
> > > > > Now Search log events and configuration files using AJAX and a browser.
> > > > > Download your FREE copy of Splunk now >>  http://get.splunk.com/
> > > > > _______________________________________________
> > > > > Gramps-users mailing list
> > > > > [hidden email]
> > > > > https://lists.sourceforge.net/lists/listinfo/gramps-users
> >
--
Joel Parker


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
Reply | Threaded
Open this post in threaded view
|

Re: XML load times

bm-5
Joel,

this is why you should submit an issue to the bug tracker, even after
developers
have said they will look at it.
Don has been swamped with mails, he cannot read everything, so best if
you make
a new bug issue, and then correspond with the developer that has time to take
the bug up.

Benny

Quoting Joel Parker <[hidden email]>:

> Has there been any progress on this issue?
>
> Thanks,
> - Joel
>
>
> On Sun, 2007-08-05 at 00:23 -0500, Joel Parker wrote:
>> Ok, scratch all that talk about the media folder, something else is
>> going on here.
>>
>> This time, I copied the original .gramps file from our shared folder to
>> each of the computers. There was no media folder, so all links were
>> broken. I opened each file, made a single change (changed a random media
>> path from absolute to relative), and exited Gramps again.
>>
>> The original file was 162 KiB, consisting of 326,529 lines when
>> unzipped.
>>
>> After the simple modification, the file had jumped to 470 KiB,
>> consisting of a full 1,881,729 lines, the vast majority of which are
>> additional <bookmark> lines.
>>
>> So in the course of changing the path for a single media file, more then
>> 1,500,000 lines had been added to my data file. This happened on both
>> computers, Windows and Ubuntu, 2.2.7 and 2.2.8.
>>
>> Don, I'll send you both files under separate cover.
>>
>> - Joel
>>
>>
>>
>>
>>
>> On Sat, 2007-08-04 at 22:39 -0600, Don Allingham wrote:
>> > Can you send me the XML file so I can try some profiling on the date
>> > here?
>> >
>> > Don
>> >
>> > On Sat, 2007-08-04 at 19:02 -0500, Joel Parker wrote:
>> > > I did some more tests.
>> > >
>> > > Again, the two systems (named from my first experience):
>> > > "fast" - 2.0 GHz Pentium M, 512 MiB RAM, Windows XP, Gramps 2.2.7
>> > > "slow" - 3.06 GHz Pentium 4, 1 GiB RAM, Ubuntu 7.04, Gramps 2.2.8
>> > >
>> > > The file is a Gramps XML file with approx. 700 names. There is a linked
>> > > media folder with 67 images of various types totaling 55.1 MiB (several
>> > > high-resolution scans).
>> > >
>> > > First, I made identical copies of the base .gramps XML file on each
>> > > computer and deleted the media folder, so that all the media links would
>> > > be broken.
>> > >
>> > > In this case, both systems took about the same amount of time to load
>> > > the file (about 2 minutes).
>> > >
>> > > Second, on the "slow" system, I copied back the media folder and
>> > > corrected all the links, so they were all working links to actual media
>> > > files, then tried opening the .gramps file again.
>> > >
>> > > In this case, the file load times jumped up into the 15-minute range and
>> > > even higher (I killed it after a while). This whole time the CPU was
>> > > going at 100%, and the memory usage was creeping higher and higher,
>> > > until by the time I killed it, all of my system memory and half of my
>> > > 1GB swap file was used.
>> > >
>> > > I saw the same behavior on the "fast" system as well. This time it was a
>> > > little different, as Gramps loaded into Person view mode in a normal
>> > > amount of time, but went crazy with load time and memory when I pressed
>> > > the Media button.
>> > >
>> > > Again, to summarize, with no media folder present, both systems loaded
>> > > the XML file in a couple minutes. With the media folder present and when
>> > > opening into Media view, the load times and memory usage went through
>> > > the roof.
>> > >
>> > > What does Gramps try to do with the media during load when it's going
>> > > into Media view mode? Does it try to thumbnail all the pictures or load
>> > > them into memory or something?
>> > >
>> > > - Joel
>> > >
>> > > On Sat, 2007-08-04 at 09:16 -0600, Don Allingham wrote:
>> > > > Something odd is definitely going on here. There was no
>> upgrade between
>> > > > 2.2.7 and 2.2.8, so that should not be a problem.
>> > > >
>> > > > My guess is the problem is the amount of RAM on the systems. When you
>> > > > use the .gramps format, GRAMPS must keep *all* data in memory at once.
>> > > > So, if you have more data than you have available RAM, the system will
>> > > > dramatically slow down.
>> > > >
>> > > > Does the slow machine have less RAM than the faster machine?
>> > > >
>> > > > Don
>> > > >
>> > > > On Sat, 2007-08-04 at 16:38 +0200, [hidden email] wrote:
>> > > > > Quoting Joel Parker <[hidden email]>:
>> > > > >
>> > > > > > My wife and I are working with a database in Gramps XML
>> format, so we
>> > > > > > can share the file without the grdb hassles.
>> > > > > >
>> > > > > > The database has about 700 individuals, along with maybe
>> 50 media links
>> > > > > > (some of which are broken, if that matters). I'm using
>> 2.2.8 on Ubuntu,
>> > > > > > and she's using 2.2.7 on Windows XP.
>> > > > >
>> > > > > You should use the same version. 2.2.8 corrects not saved
>> privacy settings on
>> > > > > save to xml
>> > > > >
>> > > > > > On Windows with 2.2.7, she's getting load times of maybe
>> 90 seconds to
>> > > > > > open the file (that's after the app has loaded, I do
>> File->Open from
>> > > > > > there). On Ubuntu with 2.2.8, I get load times of a full
>> 11 minutes at
>> > > > > > 100% CPU, sometimes ending in a "This file is either
>> corrupt or not a
>> > > > > > Gramps file" message (and sometimes completing successfully).
>> > > > >
>> > > > > I have seen this with .grdb file, but never with .gramps file.
>> > > > > If you open an empty grdb file, and do import of xml file,
>> is it faster?
>> > > > >
>> > > > > Is this 11 min load times always?
>> > > > >
>> > > > > I do not remember, but perhaps there is a database upgrade
>> from 2.2.7
>> > > > > to 2.2.8,
>> > > > > you should start gramps from the command line and see if
>> some special upgrade
>> > > > > messages pass by. I would however expect database upgrade
>> issues not to play a
>> > > > > role when working with xml files....
>> > > > >
>> > > > > >
>> > > > > > My computer is a 3GHz P4, while hers is a ~1.8GHz Pentium
>> M, but I doubt
>> > > > > > that would make a difference that large.
>> > > > >
>> > > > > hmm, you said you had the slow load times, so the 3GHz would
>> be slower
>> > > > > than the
>> > > > > 1.8, no ? Anyway, there should be no difference between the two.
>> > > > >
>> > > > > >
>> > > > > > Any idea what's causing this discrepancy?
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Joel Parker
>> > > > > >
>> > > > >
>> > > > > Benny
>> > > > >
>> > > > >
>> > > > > ----------------------------------------------------------------
>> > > > > This message was sent using IMP, the Internet Messaging Program.
>> > > > >
>> > > > >
>> -------------------------------------------------------------------------
>> > > > > This SF.net email is sponsored by: Splunk Inc.
>> > > > > Still grepping through log files to find problems?  Stop.
>> > > > > Now Search log events and configuration files using AJAX and
>> a browser.
>> > > > > Download your FREE copy of Splunk now >>  http://get.splunk.com/
>> > > > > _______________________________________________
>> > > > > Gramps-users mailing list
>> > > > > [hidden email]
>> > > > > https://lists.sourceforge.net/lists/listinfo/gramps-users
>> >
> --
> Joel Parker
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> Gramps-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-users
>



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users