SQLite vs. BSDDB

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: SQLite vs. BSDDB

GRAMPS - User mailing list
It should be should start like almost like every other time Gramps has had a major upgrade: the backup and pre-flight test. Here's my first cut. Suggestions & corrections welcome! (Even the Will Smith style "Oh, HELL no!" comments are welcome.)

1) Export all your trees in Gramps XML format before installing the new version.

If it has been awhile since your last off-site backup, do that more comprehensive level! Export using the "Gramps XML Package (family-tree and media)" option. 

[I like to install the same older version of Gramps on a second computer and import there to avoid the Hotel California "programmed to receive" scenario... make sure to "find the passage back to the place I was before" before your first taste of the new stuff. This also helps remind you to copy the various Preference customizations that DON'T attach to a backup. Everything from preferred date formats & your Researcher info to your language & add-ons.

Decide whether to use the Edit->Preferences... "Add tag on import" in the General tab. I prefer to ALWAYS tag on import, use Organize Tags... to move new Tag 'up' to the top with a bold color. It makes new, unchecked data visible in the Category View lists. You can de-select the tags piecemeal to see where you have verified & Remove the tag entirely when you are done with the spot checks.

It is also good to note the record count in each Category View before export & import to verify everything moved.

Include the old Gramps download with your off-site backup. The data without the organizational system is like an unassembled haystack... and you've got to have a haystack before you can search for any needles.]

2) Use the "File->Make Backup..." on each tree, embedding the Gramps version & date in each filename. 

3) Download the new Gramps version & install the 2nd machine. It is your choice whether to follow the 'flush the old version beforehand' or 'install in parallel' method. But make sure that method works before committing the main system. 

(If you've been using a really old version, peaceful co-existence may not be viable. The migration from Python 2 to Python 3 as we evolved from Gramps 4.0 to 4.2 was a similar decision point.)

4) Choose a different default database backend, go to the Edit->Preferences... Family Tree tab. (SQLite is NOT the only choice but it will be the best explored one.)

Windows capture of the Database backend menu:

5) use Family Trees->Manage Family Trees... New button to create a Tree. 

6) Verify the new, empty tree has the correct database type.

7) Click the Rename button to rename the empty Tree. (I like to use the old name but with an underscore at the beginning to float all the converted Trees to the top of the list. Having them in the same order as the unconverted makes the dialog work as a checklist.  You can Delete the old Trees & then Rename the new ones after conversion is complete.)

8) open the newly renamed tree

9) flow the old data into the new database backend withe the Family Trees->Import...

On Wed, Sep 18, 2019 at 2:27, Peter Merchant
Do I need to take any special steps  to change from BSDDB to SQLite?
I am using Kubuntu Linux 18.04 and gramps 5.0.1. I have backed up all my databases and downloaded the new Deb for 5.1.1

Do I need to delete the old version of gramps and data before installing the new in order to do this change, or is it a case of install the new, delete all the old databases, and change the database format to SQLite before importing all the backups?

I had a quick look at the wiki, but couldn't see any advice about this.

Thanks for help.
Peter



On 03/09/2019 20:34, Enno Borgsteede wrote:




--
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org


--
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Cleaning up your Tree

GRAMPS - User mailing list
In reply to this post by Ron Johnson
I consider new revisions to be a reminder to de-clutter the data. (Just like the 2 Daylight Savings Time changes each year are my reminder to check batteries, smoke alarms, HVAC systems & filters, the water heater and less visible home system failure points.) 

An export-then-import cycle will always yeild a cleaner, more sequential database.  And that means faster too. But 'cleaner' also means that things have been thrown away. So, now that you found a good stopping point for your research & created a stable Restore Point for your Gramps data, it might be an appropriate time to do those other 'good housekeeping' chores. 

I seem to recall reading that one of the recent enhancements to the Tools--> Family Tree Processing "Reorder Gramps ID" tool was to recognize & preserve Custom IDs. If that proves true, it could be time to add another leading zero to the ID generator & re-pack the IDs. It makes the Sort-on-ID more useful for distinguishing between recently added records as opposed to recently modified records.

The wiki manual doesn't yet say anything about preserving Custom IDs.

I haven't explored the implications of the Reorder or export/import cycle on Gramps. Reordering probably remaps the ID table to the invisible underlying 'handles' in the database. But a fresh import creates a new set of handles.  It raises questions like:

What happens to Custom Filter Rules that reference IDs? If you have a central figure for your Genealogical Research and have built filter rules specifically for them, is it best to assign them a Custom ID that would survive reordering?

The Gramps Links embedded in the Notes use handles. They survive merges & reordering without issue. But what happens to the links on an import?

If you generate website or Reports with the Gramps ID, does the continuity of reference and hotlinks outweigh other benefits of a clean database?

On Wed, Sep 18, 2019 at 7:36, Ron Johnson


--
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org

Untitled Download Attachment
Untitled (206 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning up your Tree

prculley
On Wed, Sep 18, 2019 at 9:14 AM Emyoulation--- via Gramps-users <[hidden email]> wrote:
I haven't explored the implications of the Reorder or export/import cycle on Gramps. Reordering probably remaps the ID table to the invisible underlying 'handles' in the database. But a fresh import creates a new set of handles.
An import of a .gramps (XML) file does NOT change the internal handles or IDs as long as you are importing to a new clean tree.  Only the IDs can change if you do an import to an already present tree and there are ID conflicts.  Of course a GEDCOM or other source type import can change IDs and always uses new handles.
It raises questions like:

What happens to Custom Filter Rules that reference IDs? If you have a central figure for your Genealogical Research and have built filter rules specifically for them, is it best to assign them a Custom ID that would survive reordering?
A reorder that changes IDs will mess up the filters with IDs.

The Gramps Links embedded in the Notes use handles. They survive merges & reordering without issue. But what happens to the links on an import?
Since they use handles, the comments on handles above apply.

Paul C.


--
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning up your Tree

Dave Scheipers
In reply to this post by GRAMPS - User mailing list
Hi

I  filed a bug on the Reorder tool when it was revamped for 5.0. The
bug at that time was the tool would "scramble" the ID number order if
you set the tool to reuse any deleted numbers so the ID reflected to
actual number of objects.  That was on a BSDDB file. The scramble no
longer occurs on a SQLite file. Any reorder "appears" to reorder
objects in the order that they were created. I say "appears" because
there was no way for me to actually confirm this.

In the bug report I ran through all the various tool options showing
the various results. Some may find this information helpful. These
tests were done on BSDDB files.

With the switching to SQLite, the bug was closed.

https://gramps-project.org/bugs/view.php?id=10641

Filters and things like the NarWeb report that use intro/header/footer
note files will be disrupted with any change of ID numbers. .
One issue (I have not filed a bug report yet) is after a resequence of
ID numbers, the SQLite database does not start numbering at the new
last number in the sequence. It continues on from the previous higher
number leaving a gap between the resequenced numbers and the new
record numbers.

HTH Dave

On Wed, Sep 18, 2019 at 10:14 AM Emyoulation--- via Gramps-users
<[hidden email]> wrote:

>
> I consider new revisions to be a reminder to de-clutter the data. (Just like the 2 Daylight Savings Time changes each year are my reminder to check batteries, smoke alarms, HVAC systems & filters, the water heater and less visible home system failure points.)
>
> An export-then-import cycle will always yeild a cleaner, more sequential database.  And that means faster too. But 'cleaner' also means that things have been thrown away. So, now that you found a good stopping point for your research & created a stable Restore Point for your Gramps data, it might be an appropriate time to do those other 'good housekeeping' chores.
>
> I seem to recall reading that one of the recent enhancements to the Tools--> Family Tree Processing "Reorder Gramps ID" tool was to recognize & preserve Custom IDs. If that proves true, it could be time to add another leading zero to the ID generator & re-pack the IDs. It makes the Sort-on-ID more useful for distinguishing between recently added records as opposed to recently modified records.
>
> The wiki manual doesn't yet say anything about preserving Custom IDs.
> https://www.gramps-project.org/wiki/index.php/Gramps_5.1_Wiki_Manual_-_Tools#Reorder_Gramps_ID
>
> I haven't explored the implications of the Reorder or export/import cycle on Gramps. Reordering probably remaps the ID table to the invisible underlying 'handles' in the database. But a fresh import creates a new set of handles.  It raises questions like:
>
> What happens to Custom Filter Rules that reference IDs? If you have a central figure for your Genealogical Research and have built filter rules specifically for them, is it best to assign them a Custom ID that would survive reordering?
>
> The Gramps Links embedded in the Notes use handles. They survive merges & reordering without issue. But what happens to the links on an import?
>
> If you generate website or Reports with the Gramps ID, does the continuity of reference and hotlinks outweigh other benefits of a clean database?
>
> On Wed, Sep 18, 2019 at 7:36, Ron Johnson
> <[hidden email]> wrote:
> --
> Gramps-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-users
> https://gramps-project.org
>
> --
> Gramps-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-users
> https://gramps-project.org


--
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: SQLite vs. BSDDB

Peter Merchant
In reply to this post by Ron Johnson

before installing the new in order to do this change, or is it a case of install the new, delete all the old databases, and change the database format to SQLite before importing all the backups?

A "Convert" button was added to the Family Trees window in 5.0.  IOW, you can convert now to SQLite with a click of a button.

But the Convert button is grayed out. As is the Repair button.

Process: Close all databses

in Preferences, change under 'family  tree' the Database Format to 'SQLite'

Restart gramps.

Select but not open a Database, and the Convert button is live.

Do It!

Without checking everything in detail, it works.

I will now convert the remainder of my databases, then install 5.1.1


Peter



--
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning up your Tree

Brad Rogers
In reply to this post by Dave Scheipers
On Wed, 18 Sep 2019 14:56:11 -0400
Dave Scheipers <[hidden email]> wrote:

Hello Dave,

>ID numbers, the SQLite database does not start numbering at the new
>last number in the sequence. It continues on from the previous higher
>number leaving a gap between the resequenced numbers and the new
>record numbers.

Ah;  I thought it was something I'd done, but as I'm not the only one
seeing this.

I know it's only a minor inconvenience, but it would be nice to not have
to remember to 'fill in the gaps', as it were.

--
 Regards  _
         / )           "The blindingly obvious is
        / _)rad        never immediately apparent"
Life goes quick and it goes without warning
Bombsite Boy - The Adverts


--
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org

attachment0 (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SQLite vs. BSDDB

Peter Merchant
In reply to this post by Peter Merchant
On 18/09/2019 20:14, Peter Merchant wrote:

before installing the new in order to do this change, or is it a case of install the new, delete all the old databases, and change the database format to SQLite before importing all the backups?

A "Convert" button was added to the Family Trees window in 5.0.  IOW, you can convert now to SQLite with a click of a button.

But the Convert button is grayed out. As is the Repair button.

Process: Close all databses

in Preferences, change under 'family  tree' the Database Format to 'SQLite'

Restart gramps.

Select but not open a Database, and the Convert button is live.

Do It!

Without checking everything in detail, it works.

The converted database is called 'name'(Converted #1).

I then sorted the list by Database type and have deleted the BSDDB databases.

[ Yes I know I went through the process a few months ago of combining all my databses into one large master one  which is now the only 'live' one, but the others were still there. ]

I may rename them tomorrow.


I will now convert the remainder of my databases, then install 5.1.1


Peter






--
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: SQLite vs. BSDDB

Ron Johnson
On 9/18/19 2:22 PM, Peter Merchant wrote:
On 18/09/2019 20:14, Peter Merchant wrote:

before installing the new in order to do this change, or is it a case of install the new, delete all the old databases, and change the database format to SQLite before importing all the backups?

A "Convert" button was added to the Family Trees window in 5.0.  IOW, you can convert now to SQLite with a click of a button.

But the Convert button is grayed out. As is the Repair button.



When I select a BSDDB tree while in the Family Trees window in 5.0.2, the Convert button de-grays.

--
Angular momentum makes the world go 'round.


--
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning up your Tree

Dave Scheipers
In reply to this post by Brad Rogers
Hi Brad,

And you just confirmed it was not just me seeing this.

Just filed a bug report.

https://gramps-project.org/bugs/view.php?id=11346


Dave

On Wed, Sep 18, 2019 at 3:18 PM Brad Rogers <[hidden email]> wrote:

>
> On Wed, 18 Sep 2019 14:56:11 -0400
> Dave Scheipers <[hidden email]> wrote:
>
> Hello Dave,
>
> >ID numbers, the SQLite database does not start numbering at the new
> >last number in the sequence. It continues on from the previous higher
> >number leaving a gap between the resequenced numbers and the new
> >record numbers.
>
> Ah;  I thought it was something I'd done, but as I'm not the only one
> seeing this.
>
> I know it's only a minor inconvenience, but it would be nice to not have
> to remember to 'fill in the gaps', as it were.
>
> --
>  Regards  _
>          / )           "The blindingly obvious is
>         / _)rad        never immediately apparent"
> Life goes quick and it goes without warning
> Bombsite Boy - The Adverts
> --
> Gramps-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-users
> https://gramps-project.org


--
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

WebCal: invalid title and link

Patrick Gerlier
In reply to this post by prculley

index.html page for web calendar incorrectly displays "My Family Calendar" instead of the string entered in the configuration dialog for Reports>Internet>Web Calendar.

From what I could understand from source code v5.1.0 (I don't pratice Python), there may be a wrong usage in line 1601 of webcal.py as:

title = self._("My Family Tree")

where as the initilisation method __init__ retrieves the custom title and stores it in another node (line 129):

self.title_text = mgobn("title")

Perhaps replacing line 1601 with title = self.title_text would solve the issue.

Another issue is the return link to narrative web.

I chose to store the html files in a directory with a part in uppercase (like xxx_NAVWEB). This name is entered as ../../xxx_NAVWEB.index.html in the config dialog. However, the <a> href-link is converted full lowercase. Consequently, it does not work.

Even if I use an absolute URL (/NAVWEB without index.html, relying on auto selection by web server), the URL is lowercase.

I didn't find in the code where this case folding was done.

Any idea? Or suggestion for something I'm doint wrong.

Cheers,
Patrick



--
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
12