Re: [Gramps-windows] Database Loading Issue

classic Classic list List threaded Threaded
38 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-windows] Database Loading Issue

Alex Roitman
Steve,

On Wed, 2006-09-13 at 08:48 -0700, Steve Hall wrote:

> >
> > Gramps databases are NOT portable. This means you can not move your
> > familytree.gramps file from linux to windows and expect it to be
> > safe. You MUST export to XML from Linux and import the XML on
> > Windows.
> >
> > Not only that, but you can't even move your file from one Linux box
> > to another. Any time your gramps file leaves you computer, you must
> > export it.
> >
> > Portability may change with future versions of python, but for now,
> > this is what we are stuck with.
>
> This is a horrible situation, it is unreasonable to require a user to
> manage what the application is supposed to do. Can you imagine the
> reaction if word processors required this?
We learned about this limitation a few months ago, well into developing
the 2.1.x branch. It is a horrible problem, I absolutely agree.
I've been trying to get people's attention to it for a while.
Now that few people actually tried 2.1.x we may get some meaningful
discussion on this matter. Before all my calls were just the vapor
to everybody :-)

> If GRAMPS files can not be exchanged AS IS with other GRAMPS users,
> then the program has a fundamental problem.

Let me describe the situation in detail.

The 2.0.x versions used grdb files that did not use the transactional
environment. The new 2.1/2.2 series can work both with and without the
transactional environment (TXN for short).

The benefits of the transactional environment are:
1. Data can never be corrupted on the low level.
   Before flaming me for an outright lie, let me explain
   what this means. The changes to the data are sometimes
   simple (changing a note will only affect one object,
   so exactly one piece of data will be modified). But sometimes
   they are complex (adding a new child to the family (1) adds
   a new person, (2) adds that person to the family). There
   are even more complex changes, involving the modifications
   of several records in several database tables.

   With the TXN in places, the whole transaction is either
   committed or abandoned, so the database is always in the
   consistent state. Even if the program crashes in the middle
   of the transaction, either the whole thing will be committed
   or not at all. As many people noticed when testing unstable
   2.1.x releases, no amount of crashing in whatever places
   (well, except initial upgrade from 2.0.x) can damage the data.

   This is no small matter, and is very important to anybody
   working with large data files.

2. The performance. The TXN uses logging, and we currently utilize
   asynchronous transaction logging, so that we don't have to
   physically write and clear the logs for every transaction.
   This allows us to transactionally write much faster than we did
   in 2.0.x before the transactions. For anybody who tried to
   import huge dataset in 2.0.x and in 2.1.x this performance gain
   is pretty obvious.

BUT, as we learned well into developing 2.1 branch, the TXN-capable
databases cannot be simply taken away and opened in another environment
(that is, another machine, or even another user on the same machine).
Sometimes it works, depending on how the log files are synced and
released, but we have no control over that. So we cannot count on this.

The workarounds could be either copy the whole environment directory
(the ~/.gramps/bsddbenv dir), or try to "clear" the connection between
the environment and the datafile. While the bsddb libraries (libdb4.3)
have the necessary methods, the python bindings were lacking. Because
of the bug reports that we filed the necessary method has been added
only recently.

So the next version of python (2.5) will allow us to disassociate
the database from the environment dir on cleanly closing the db.
In fact, the code is already in place, so if the method is available
in python then it will be automatically used.

Now, what are we to do in the meantime? The following steps are
the workarounds of different degree of ugliness:

1. Use XML format natively: open XML files, work with them.
Downside: slow to open, the whole data will stay in memory.
OK for small files, but impractical for large files.

2. Work in GRDB and export into XML for portability/archiving.

3. Always drag the ~/.gramps/bsddbenv directory along with the file.
Downside: very cumbersome and inconvenient.

4. We may disable the TXN functionality, or make it a user-adjustable
parameter. This way the users would decide whether they want to trade
portability of grdb files for reliability/performance. The TXN-grdb is
fast and reliable (until you move it), but not portable.


None of this is really good. But we're in a tough situation here:
we want TXN and portability. Hopefully, python2.5 will be released
soon and solve the problem. However, realistically it will be a while
until python2.5 is shipped in every distro.

I sincerely welcome any feedback on what to do here. If people want
to experiment with portable TXN-less grdbs, right now it can be done
by setting the "UseTXN = False" (instead of the current True)
in the src/GrampsDb/_GrampsBSDDB.py file in the source tree
or in /usr/share/gramps/GrampsDb/_GrampsBSDDB.py in your installed
version (or wherever it is installed -- ymmv).

If there are any bsddb experts or any database experts that can
provide useful advice, we need you now!

Alex

--
Alexander Roitman   http://www.gramps-project.org

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-windows mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-windows

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-windows] Database Loading Issue

Alex Roitman
Steve,

On Wed, 2006-09-13 at 23:10 -0400, Steve Hall wrote:
> IMO, it looks like the solution is to simply say that GRAMPS 2.1
> depends on Python 2.5. This new version of Python is already at it's
> second (and reportedly last) release candidate, I don't see that a
> little more waiting is as much of a problem as the non-portability of
> our application files.

It is likely that 2.2 will not release before python2.5 is out.
But as Don has mentioned, this does not guarantee that all distros
will ship right away with python2.5. The transition from 2.3 to 2.4
took a few years. So we would leave e.g. all the Dapper users out
in the cold, because Dapper will not switch to python2.5.

> Until then, I guess we're stuck with the export work-around as a
> hiccup of beta testing. It seems prudent to wait for Python for the
> final release though, especially since the current GRAMPS development
> pace is about the same track.

With all our hopes, the reality is that it will be probably around a
year until an average distro has python2.5. More for people who don't
like upgrading.


So maybe more is needed here. How about these alternatives:

1. We pop up a dialog on every db closing (quit or open another file,
does not matter) saying that the db is not portable to another location
and that export to XML is needed for portability? Currently, we print
this to the terminal, but few people run from the terminal anyway.

The dialog would have a checkbox "Don't mention this again" for those
who have been warned enough.

2. Have two formats: grdb and pgdb (portable gramps database).
The pgdb format will use UseTXN=False.

3. Ship gramps 2.2 with UseTXN=False and wait until python2.5 is
everywhere. This will penalize all users, even those who don't care
about portability. Both performance and data integrity will suffer.

4. Have an option to automatically export to XMl on every close.
The portability-interested users will set it on. In case the problem
is discovered with opening the file, the XML is always somewhere.

Alex

--
Alexander Roitman   http://www.gramps-project.org

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-windows] Database Loading Issue

Trevor Rhodes-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alex,

> 4. Have an option to automatically export to XMl on every close.
> The portability-interested users will set it on. In case the problem
> is discovered with opening the file, the XML is always somewhere.

This is my personal favourite.

Regards
        Trevor
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFCNt5vkusRwjhfowRAkuaAKCX1vmH8JhTFAyc3Svm9Huy0uuwywCfe0Sp
DwK4EI9WC5D0pLUDTBlGhw4=
=lblZ
-----END PGP SIGNATURE-----

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-windows] Database Loading Issue

Don Allingham
In reply to this post by Alex Roitman
I guess the fundamental question that needs to be answered is:

Is database portability of the grdb file more important to users than
performance and data integrity?

We can allow portability without going to an XML file at the expense of
performance and integrity. Either that, or we wait for everyone to
upgrade to python 2.5.

Don


On Wed, 2006-09-13 at 21:25 -0700, Alex Roitman wrote:

> Steve,
>
> On Wed, 2006-09-13 at 23:10 -0400, Steve Hall wrote:
> > IMO, it looks like the solution is to simply say that GRAMPS 2.1
> > depends on Python 2.5. This new version of Python is already at it's
> > second (and reportedly last) release candidate, I don't see that a
> > little more waiting is as much of a problem as the non-portability of
> > our application files.
>
> It is likely that 2.2 will not release before python2.5 is out.
> But as Don has mentioned, this does not guarantee that all distros
> will ship right away with python2.5. The transition from 2.3 to 2.4
> took a few years. So we would leave e.g. all the Dapper users out
> in the cold, because Dapper will not switch to python2.5.
>
> > Until then, I guess we're stuck with the export work-around as a
> > hiccup of beta testing. It seems prudent to wait for Python for the
> > final release though, especially since the current GRAMPS development
> > pace is about the same track.
>
> With all our hopes, the reality is that it will be probably around a
> year until an average distro has python2.5. More for people who don't
> like upgrading.
>
>
> So maybe more is needed here. How about these alternatives:
>
> 1. We pop up a dialog on every db closing (quit or open another file,
> does not matter) saying that the db is not portable to another location
> and that export to XML is needed for portability? Currently, we print
> this to the terminal, but few people run from the terminal anyway.
>
> The dialog would have a checkbox "Don't mention this again" for those
> who have been warned enough.
>
> 2. Have two formats: grdb and pgdb (portable gramps database).
> The pgdb format will use UseTXN=False.
>
> 3. Ship gramps 2.2 with UseTXN=False and wait until python2.5 is
> everywhere. This will penalize all users, even those who don't care
> about portability. Both performance and data integrity will suffer.
>
> 4. Have an option to automatically export to XMl on every close.
> The portability-interested users will set it on. In case the problem
> is discovered with opening the file, the XML is always somewhere.
>
> Alex
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________ Gramps-windows mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gramps-windows

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (198 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-windows] Database Loading Issue

Steve Hall-3
In reply to this post by Alex Roitman
On Wed, 2006-09-13 at 21:25 -0700, Alex Roitman wrote:
>
[snip]
>
> 4. Have an option to automatically export to XMl on every close. The
> portability-interested users will set it on. In case the problem is
> discovered with opening the file, the XML is always somewhere.

Would is the cost for this? I'm not where I can test this at the
moment, what sort of speed are we talking about? Are there any tricks
(compression?) that could be used to make read/write reasonably quick
enough to make this the default until Python 2.5?


--
Steve Hall  [ digitect dancingpaper com ]



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-users] [Gramps-windows] Database Loading Issue

bm-7
In reply to this post by Don Allingham
Just my 2 cents,

I am intrigued by what Alex said in a previous mail:

> The workarounds could be either copy the whole environment directory
> (the ~/.gramps/bsddbenv dir)

Does this mean that if instead of saying that the database is in a
file, if the
user is presented with the idea that the database is a complete directory,
things would work?

I mean, I make a database "family", gramps makes a directory family, with
contents bsddbenv dir and family.grdb. Portability means copying the dir, not
the grdb file
Open database could then work as: open directory containing database.

Is this workable as a fifth alternative?

Concerning portability, for me this is important: moving a grdb to a laptop to
do some field work/go on holiday, and back to my normal, more powerfull
PC when
I'm home.

If automatic export to xml (or portable grdb, pgdb) is fast, I would opt for
that on exit, given the alternatives that Alex lists.
I would suggest to show a dialog on opening a db for the first time
with gramps
or on creating a new database, in the line of:
"If python < 2.5: the created database is NOT PORTABLE. If you need to
transfer
data between computers, convert database to XML format, or go to menu
'preferences->save database automatically as XML on exit/close', and move the
xml file between the PCs"

Benny


>> So maybe more is needed here. How about these alternatives:
>>
>> 1. We pop up a dialog on every db closing (quit or open another file,
>> does not matter) saying that the db is not portable to another location
>> and that export to XML is needed for portability? Currently, we print
>> this to the terminal, but few people run from the terminal anyway.
>>
>> The dialog would have a checkbox "Don't mention this again" for those
>> who have been warned enough.
>>
>> 2. Have two formats: grdb and pgdb (portable gramps database).
>> The pgdb format will use UseTXN=False.
>>
>> 3. Ship gramps 2.2 with UseTXN=False and wait until python2.5 is
>> everywhere. This will penalize all users, even those who don't care
>> about portability. Both performance and data integrity will suffer.
>>
>> 4. Have an option to automatically export to XMl on every close.
>> The portability-interested users will set it on. In case the problem
>> is discovered with opening the file, the XML is always somewhere.
>>
>> Alex


----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-users] [Gramps-windows] Database Loading Issue

Michael Lightfoot
In reply to this post by Alex Roitman
On Thursday 14 September 2006 14:25, Alex Roitman wrote:

>
> So maybe more is needed here. How about these alternatives:
>
> 1. We pop up a dialog on every db closing (quit or open another file,
> does not matter) saying that the db is not portable to another location
> and that export to XML is needed for portability? Currently, we print
> this to the terminal, but few people run from the terminal anyway.
>
> The dialog would have a checkbox "Don't mention this again" for those
> who have been warned enough.
>
> 2. Have two formats: grdb and pgdb (portable gramps database).
> The pgdb format will use UseTXN=False.
>
> 3. Ship gramps 2.2 with UseTXN=False and wait until python2.5 is
> everywhere. This will penalize all users, even those who don't care
> about portability. Both performance and data integrity will suffer.
>
> 4. Have an option to automatically export to XMl on every close.
> The portability-interested users will set it on. In case the problem
> is discovered with opening the file, the XML is always somewhere.
>
None of the above.  :-)

The first option would annoy me the first time and then disappear.  
The second option would still have the problem of having to manually make a
choice.
The third option would penalise those of us who want the performance and don't
give a fig about "portability".
The fourth option would make my sloowww machine even slooowwwer.

Someone mentioned checking for the version of Python and taking action
depending on the result.  I would favour that as it would be potentially
transparent and would not affect those of us who are not interested in
portability.

--
==================================
Michael Lightfoot

[hidden email]
==================================

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Database Loading Issue

Gerald Britton-2
The portability issue matters to everyone -- even those who don't port
their dbs today. Sooner or later every one of us will have a head
crash, build a new machine, upgrade our distro or move between OSes.
As it is today, any one of these events will break our databases. Who
is comfortable with that? Not me!

I suppose that I also will be unable to restore my db from a backup
with 2.2 as well.  No amount of performance improvement is worth this.

On 9/14/06, Michael Lightfoot <[hidden email]> wrote:

> On Thursday 14 September 2006 14:25, Alex Roitman wrote:
> >
> > So maybe more is needed here. How about these alternatives:
> >
> > 1. We pop up a dialog on every db closing (quit or open another file,
> > does not matter) saying that the db is not portable to another location
> > and that export to XML is needed for portability? Currently, we print
> > this to the terminal, but few people run from the terminal anyway.
> >
> > The dialog would have a checkbox "Don't mention this again" for those
> > who have been warned enough.
> >
> > 2. Have two formats: grdb and pgdb (portable gramps database).
> > The pgdb format will use UseTXN=False.
> >
> > 3. Ship gramps 2.2 with UseTXN=False and wait until python2.5 is
> > everywhere. This will penalize all users, even those who don't care
> > about portability. Both performance and data integrity will suffer.
> >
> > 4. Have an option to automatically export to XMl on every close.
> > The portability-interested users will set it on. In case the problem
> > is discovered with opening the file, the XML is always somewhere.
> >
> None of the above.  :-)
>
> The first option would annoy me the first time and then disappear.
> The second option would still have the problem of having to manually make a
> choice.
> The third option would penalise those of us who want the performance and
> don't
> give a fig about "portability".
> The fourth option would make my sloowww machine even slooowwwer.
>
> Someone mentioned checking for the version of Python and taking action
> depending on the result.  I would favour that as it would be potentially
> transparent and would not affect those of us who are not interested in
> portability.
>
> --
> ==================================
> Michael Lightfoot
>
> [hidden email]
> ==================================
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Gramps-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-users
>

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Database Loading Issue

Michaël Van Dorpe
On 9/14/06, Gerald Britton <[hidden email]> wrote:

> I suppose that I also will be unable to restore my db from a backup
> with 2.2 as well.  No amount of performance improvement is worth this.

If you make a backup, use the xml file format. No portability problems there.

Funny how this problem gets a lot of attention now. I agree it's a big
problem, but I can live with it until python is fixed. This is a known
problem since months ago. Alex and Don filed bugs with all the
upstream problems, and I think having the right warning dialogs at the
right time is enough for now.

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Database Loading Issue

Gerald Britton-2
My backups are done using cron jobs and dar. It is not acceptable to
have to implement a special procedure for one particular app.

All the other problems I mentioned remain: switching OSes, nigrating
to a new hd, upgrading/downgrading/changing your distro etc.

On 9/14/06, Michaël Van Dorpe <[hidden email]> wrote:

> On 9/14/06, Gerald Britton <[hidden email]> wrote:
>
> > I suppose that I also will be unable to restore my db from a backup
> > with 2.2 as well.  No amount of performance improvement is worth this.
>
> If you make a backup, use the xml file format. No portability problems
> there.
>
> Funny how this problem gets a lot of attention now. I agree it's a big
> problem, but I can live with it until python is fixed. This is a known
> problem since months ago. Alex and Don filed bugs with all the
> upstream problems, and I think having the right warning dialogs at the
> right time is enough for now.
>

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-users] [Gramps-windows] Database Loading Issue

Don Allingham
In reply to this post by Michael Lightfoot
On Thu, 2006-09-14 at 18:36 +1000, Michael Lightfoot wrote:
> Someone mentioned checking for the version of Python and taking action
> depending on the result.  I would favour that as it would be potentially
> transparent and would not affect those of us who are not interested in
> portability.
>

That still would not work. This is the case:

Joe has two machines - a laptop that has python 2.4 and a desktop that
has python 2.5. He edits his database using the laptop, which has which
makes the database non-portable. He then transfers the grdb file to the
desktop, where he edits it. Since the desktop has 2.5, it assumes that
the database is portable when it really isn't. Potential crash.

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (198 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-windows] [Gramps-users] Database Loading Issue

Don Allingham
In reply to this post by bm-7
On Thu, 2006-09-14 at 09:20 +0200, [hidden email] wrote:

> Just my 2 cents,
>
> I am intrigued by what Alex said in a previous mail:
>
> > The workarounds could be either copy the whole environment directory
> > (the ~/.gramps/bsddbenv dir)
>
> Does this mean that if instead of saying that the database is in a
> file, if the
> user is presented with the idea that the database is a complete directory,
> things would work?
We used the database as a directory concept in GRAMPS 1.X, and most
people found it incredibly confusing - especially new users and casual
users. It also make it impossible to register a GRAMPS database as a
MIME type.

> Concerning portability, for me this is important: moving a grdb to a laptop to
> do some field work/go on holiday, and back to my normal, more powerfull
> PC when
> I'm home.

Is it more important than data integrity and speed? We get beat up
continuously because of the performance issues. Also, there is nothing
quite like trying to support someone who is furious because he/she lost
his database when a crash occurred.

>
> If automatic export to xml (or portable grdb, pgdb) is fast, I would opt for
> that on exit, given the alternatives that Alex lists.
> I would suggest to show a dialog on opening a db for the first time
> with gramps
> or on creating a new database, in the line of:
> "If python < 2.5: the created database is NOT PORTABLE. If you need to
> transfer
> data between computers, convert database to XML format, or go to menu
> 'preferences->save database automatically as XML on exit/close', and move the
> xml file between the PCs"
Again, checking python versions is not going to solve the problem. The
file could have been copied from a system written with 2.4, while you
are trying to open it with 2.5. 2.5 does not magically solve the problem
if it was created with 2.4

Is an automatic save on exit, which may take 30 seconds or more for
larger databases really the way to go? Or is exporting only when you
want to transfer to a different machine a better solution?

Don

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (198 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Database Loading Issue

Don Allingham
In reply to this post by Gerald Britton-2
On Thu, 2006-09-14 at 07:01 -0400, Gerald Britton wrote:
> The portability issue matters to everyone -- even those who don't port
> their dbs today. Sooner or later every one of us will have a head
> crash, build a new machine, upgrade our distro or move between OSes.
> As it is today, any one of these events will break our databases. Who
> is comfortable with that? Not me!
>
> I suppose that I also will be unable to restore my db from a backup
> with 2.2 as well.  No amount of performance improvement is worth this.
>

If your disk drive crashes, you will have lost not only the needed
environment files, but also your database. Obviously you need to make
some type of backup. Why not back up the smaller XML file? After all, if
you go to the Export menu, the selection includes "GRAMPS Package
(Portable XML)". This will package up everything including your media
files.

Upgrading your disto shouldn't hurt, unless you trash your /home
directory in the process. And in that case, you have to back up
something. Why not the smaller XML package instead of the *huge* grdb
file?

Maybe you don't care about performance, but do you care about data
integrity? This is what you will really be losing.

Don





-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (198 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-windows] Database Loading Issue

Don Allingham
In reply to this post by Gerald Britton-2
On Thu, 2006-09-14 at 07:17 -0400, Gerald Britton wrote:
> My backups are done using cron jobs and dar. It is not acceptable to
> have to implement a special procedure for one particular app.
>
> All the other problems I mentioned remain: switching OSes, nigrating
> to a new hd, upgrading/downgrading/changing your distro etc.
>

Then why not add ~/.gramps to the list of files you backup? When you
restore, restore both your db and ~/.gramps.



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (198 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Database Loading Issue

Kelvin Lawson
In reply to this post by Gerald Britton-2
> The portability issue matters to everyone -- even those who don't port
> their dbs today. Sooner or later every one of us will have a head
> crash, build a new machine, upgrade our distro or move between OSes.
> As it is today, any one of these events will break our databases. Who
> is comfortable with that? Not me!
>
> I suppose that I also will be unable to restore my db from a backup
> with 2.2 as well.  No amount of performance improvement is worth this.

I absolutely agree - previous messages have referred to "those users who
wish to achieve portability". Portability is not the issue - I've spent
a long time building my database and I'd be rather upset to find that if
my PC was damaged or stolen, the nightly backups I have carefully kept
are unusable.

I have seen the messages on exit about "Export to XML for portability"
but I had no idea that my backups might be useless until I happened
across this thread today. Thank God I did. Others might not be so lucky.

With this in mind I'd suggest that the next release must put being able
to access the database before performance.


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Database Loading Issue

Kelvin Lawson
In reply to this post by Don Allingham
> Upgrading your disto shouldn't hurt, unless you trash your /home
> directory in the process. And in that case, you have to back up
> something. Why not the smaller XML package instead of the *huge* grdb
> file?

I'll certainly export to XML from now on. I hope everyone else who just
backs up their .grdb file are aware of this for their sake.

Is the export to XML scriptable? I'd like to do this in a cron job. Also
is it necessary to back up ~/.gramps, or is the XML file enough?

> Maybe you don't care about performance, but do you care about data
> integrity? This is what you will really be losing.

The integrity doesn't matter to me if I can't open it :-) Seriously
though, I'm happy to use the new format together with a nightly export
to XML, but people need to be made aware that this is necessary for
backups (and not just "portability").


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-windows] [Gramps-users] Database Loading Issue

Steve Hall-3
In reply to this post by Don Allingham
On Thu, 2006-09-14 at 06:03 -0600, Don Allingham wrote:

> On Thu, 2006-09-14 at 09:20 +0200, [hidden email] wrote:
> >
> > Concerning portability, for me this is important: moving a grdb to
> > a laptop to do some field work/go on holiday, and back to my
> > normal, more powerfull PC when I'm home.
>
> Is it more important than data integrity and speed? We get beat up
> continuously because of the performance issues. Also, there is
> nothing quite like trying to support someone who is furious because
> he/she lost his database when a crash occurred.

They are both important, which is why attempting to trade off one with
the other is generating such a thread.

Which is also why I think the solution is to depend on Python 2.5. It
may be that distros will be slow on the uptake, but I think given the
magnitude of this problem, they should be equally as slow to upgrade
GRAMPS.

This is essentially a design/implementation problem. The design is not
flawed, but the tools to make it work are not yet available. I think
hacking a solution dilutes the good efforts and design to date,
especially since 2.5 is so close.

I understand that distros may be slow on the uptake, but maybe the big
ones could be convinced to back port a fix. (It is ironic that Windows
will not suffer from this problem.) If not, use the time to add some
more features, polish the interface some more, and up the version to
3.0. (Which it really is, the entire architecture, file format, and
interface of the program changed.)


--
Steve Hall  [ digitect dancingpaper com ]



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Database Loading Issue

Don Allingham
In reply to this post by Kelvin Lawson
Okay. Everybody forget performance. Take that completely out of the
equation.

Your choices are:

1) Protect the database integrity if a crash occurs.
2) Make the database portable without having to export to XML.

We can do one or the other. Right now, because of a bug in python - not
in GRAMPS, we cannot do both.

Right now, we provide an export mechanism and a revision control
mechanism that will solve #2. We have no other method of solving #1 than
to use Transactions, which currently causes #2.

So, Data Integrity or Portability.

The option of waiting for 2.5 does not solve #1. Data integrity is still
at risk until 2.2 is released, because 2.0 does not use transactions.

Don


On Thu, 2006-09-14 at 13:15 +0100, Kelvin Lawson wrote:

> > The portability issue matters to everyone -- even those who don't port
> > their dbs today. Sooner or later every one of us will have a head
> > crash, build a new machine, upgrade our distro or move between OSes.
> > As it is today, any one of these events will break our databases. Who
> > is comfortable with that? Not me!
> >
> > I suppose that I also will be unable to restore my db from a backup
> > with 2.2 as well.  No amount of performance improvement is worth this.
>
> I absolutely agree - previous messages have referred to "those users who
> wish to achieve portability". Portability is not the issue - I've spent
> a long time building my database and I'd be rather upset to find that if
> my PC was damaged or stolen, the nightly backups I have carefully kept
> are unusable.
>
> I have seen the messages on exit about "Export to XML for portability"
> but I had no idea that my backups might be useless until I happened
> across this thread today. Thank God I did. Others might not be so lucky.
>
> With this in mind I'd suggest that the next release must put being able
> to access the database before performance.
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

signature.asc (198 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-windows] Database Loading Issue

Gerald Britton-2
In reply to this post by Don Allingham
My gramps DB *is* backed up. However, from 2.1 on, if I understand
this thread correctly, restoring that DB from backup will not
guarantee that it can be read by gramps.  Alex even said that if I
were just to create a new user  on the same system and try to read the
database, it might fail!

As for integrity, I have yet to have a problem with data integrity
with gramps but it looks like I *will* have one as soon as I move to
2.1.  He said:

"Create a new user account, log into it and try opening the
grdb from 2.1.x. Depending on how much editing you have done
in grdb before closing, you may or may not open it. It also
may depend on the libdb version and python built, that vary
across the distros."

If I'm understanding what I'm reading, 2.1 will remove my ability to:

1. upgrade my HD and copy my gramps files from my old HD to a new one.
2. upgrade or change my distro (unless the python levels match, I suppose)
3. move to another platform (Windows, Mac, whatever)
4. rsync, scp, zip/email/unzip, my gramps db to/from another machine
5. backup my database to CD/USB drive, whatever and later restore it
-- possibly to another machine or HD

If any of these items is true, my data integrity issues will increase
significantly with 2.1

On 9/14/06, Don Allingham <[hidden email]> wrote:

> On Thu, 2006-09-14 at 07:17 -0400, Gerald Britton wrote:
> > My backups are done using cron jobs and dar. It is not acceptable to
> > have to implement a special procedure for one particular app.
> >
> > All the other problems I mentioned remain: switching OSes, nigrating
> > to a new hd, upgrading/downgrading/changing your distro etc.
> >
>
> Then why not add ~/.gramps to the list of files you backup? When you
> restore, restore both your db and ~/.gramps.
>
>
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>
>
>
>

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Gramps-users] Database Loading Issue

Brian Matherly
In reply to this post by Don Allingham
>Okay. Everybody forget performance. Take that completely out of the
>equation.
>
>Your choices are:
>
>1) Protect the database integrity if a crash occurs.
>2) Make the database portable without having to export to XML.
>
>We can do one or the other. Right now, because of a bug in python - not
>in GRAMPS, we cannot do both.
>
>Right now, we provide an export mechanism and a revision control
>mechanism that will solve #2. We have no other method of solving #1 than
>to use Transactions, which currently causes #2.
>
>So, Data Integrity or Portability.
>
>The option of waiting for 2.5 does not solve #1. Data integrity is still
>at risk until 2.2 is released, because 2.0 does not use transactions.
>
>Don
>

This is a difficult problem to solve. It has been fun watching the community come together with such passion. Although, when I see Don top-posting, I know things are getting a little tense :)

A problem like this is not going to be solved by a systematic reduction of options until only one remains.

Our cheese has moved (http://en.wikipedia.org/wiki/Who_Moved_My_Cheese).

It's time to find some new cheese. Perhaps the user paradigm needs to change. Perhaps some wizards need to be created. Maybe we need to change the user's work flow a little so things are more natural. How can we best help the user understand what is happening with their data? It's time to think outside of the box.

One common cry that we have seen over and over again is that nobody wants 2.2 to be released until they are confident that their data is safe.

I find myself agreeing with Steve on many points:
>If not, use the time to add some more features,
>polish the interface some more, and up the version to
>3.0. (Which it really is, the entire architecture, file format, and
>interface of the program changed.)

I think this situation warrants that we step back for a couple of weeks and rethink things. This hurts me to suggest because I am as excited about releasing 2.2 as anyone else. But I think with some brainstorming and a little more wrench time, we'll come up with something grand. And maybe that grand something will justify a 3.0.

Thanks again to everyone who puts their passion into this project. As a user, I know I appreciate it.

~Brian




-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
12