Re: New database backends for Gramps 5

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: New database backends for Gramps 5

prculley
After a very quick look, it appears that neither of these dbs is going to be a quick and easy addition.  So yes, starting with the code in gramps/plugins/db/dbapi/sqlite.py and even further back in gramps/gen/db is probably going to provide an idea of the APIs you are going to have to adapt to.

I have to wonder why you are interested though.  What advantages do you expect to get out of yet another db backend?  We are shifting to the sqlite type dbs from the bsddb backend due to a perceived ability to recover more easily from various accidents or corruptions.  I'm not sure yet if we really have enough experience to say that this is true.  We can even utilize a remote PostgreSQL db, although this has barely been tested and has its share of cautions.

I know that there has been an interest in multi-user and/or shared db in the past.  There are fundamental reasons why this is not just a db issue, but would require a lot of changes throughout Gramps code to really make it work.  I personally suspect that it has been attempts to share via Dropbox, or other methods, (even manual copying between systems) that has caused many of the reported db corruptions.

Paul C.

On Mon, Mar 5, 2018 at 2:36 AM, Anders Ardo <[hidden email]> wrote:
Hi,
  I'm interested in database backends for MongoDB and/or Neo4j.
Is it reasonable at all?
What would it take to get such beasts up and running?
Is the best place to start gramps/plugins/db/dbapi/sqlite.py?

--
Hilsen
Anders
----
Anders Ardö

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: New database backends for Gramps 5

Nick Hall
On 05/03/18 08:36, Anders Ardo wrote:
>    I'm interested in database backends for MongoDB and/or Neo4j.
> Is it reasonable at all?
> What would it take to get such beasts up and running?
> Is the best place to start gramps/plugins/db/dbapi/sqlite.py?

A MongoDB backend is probably better suited to our existing data model. 
It should be relatively easy to implement.

Have a look at he following files:

gramps/gen/db/generic.py
grampsl/plugins/db/dbapi/dbapi.py

A Neo4j backend would be interesting, but some thought would be required
for a good database design.

If you are serious about writing a new backend, then I am willing to
point you in the right direction.

Regards,


Nick.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: New database backends for Gramps 5

Nick Hall
On 06/03/18 07:55, Anders Ardo wrote:
A MongoDB backend is probably better suited to our existing data model.  It
should be relatively easy to implement.

Have a look at he following files:

gramps/gen/db/generic.py
grampsl/plugins/db/dbapi/dbapi.py
I'll start looking at those.

I have implemented a MongoDB backend.  You can find it in the third-party addons repository:

https://github.com/gramps-project/addons-source/tree/maintenance/gramps50/MongoDB

There are some performance issues that need addressing, but it appears to work.  Obviously it is not ready for production use yet.


Nick.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: New database backends for Gramps 5

Josip
8.3.2018. u 1:05, Nick Hall je napisao/la:

> I have implemented a MongoDB backend.  You can find it in the
> third-party addons repository:
>
> https://github.com/gramps-project/addons-source/tree/maintenance/gramps50/MongoDB
>
> There are some performance issues that need addressing, but it appears
> to work.  Obviously it is not ready for production use yet.
>
>
> Nick.
>

Trying to use on Windows with MongoDB v3.6.3 and PyMongo v3.6.1

I can't figure it out how to set database name?
I always get error:

2018-03-08 23:03:11.947: ERROR: dbloader.py: line 111: database names
cannot contain the character ' '
Traceback (most recent call last):
   File "C:\MSYS2\home\jole\GIT\gramps\gramps\gui\dbloader.py", line
206, in read_file
     password=password)
   File "C:\MSYS2\home\jole\GIT\gramps\gramps\gen\db\generic.py", line
593, in load
     self._initialize(directory, username, password)
   File
"C:\Users\jole\AppData\Roaming/gramps/gramps50/plugins/MongoDB\mongodb.py",
line 120, in _initialize
     self.db = self.client[dbname]
   File
"C:/MSYS2/mingw64/lib/python3.6/site-packages\pymongo\mongo_client.py",
line 996, in __getitem__
     return database.Database(self, name)
   File
"C:/MSYS2/mingw64/lib/python3.6/site-packages\pymongo\database.py", line
106, in __init__
     _check_name(name)
   File
"C:/MSYS2/mingw64/lib/python3.6/site-packages\pymongo\database.py", line
47, in _check_name
     "character %r" % invalid_char)
pymongo.errors.InvalidName: database names cannot contain the character ' '

so i resort to manually set it at line 113 in mongodb.py
Next i got error:

2018-03-08 23:18:34.407: ERROR: dbloader.py: line 111: 'Collection'
object is not callable. If you meant to call the 'list_collection_names'
method on a 'Database' object it is failing because no such method exists.
Traceback (most recent call last):
   File "C:\MSYS2\home\jole\GIT\gramps\gramps\gui\dbloader.py", line
206, in read_file
     password=password)
   File "C:\MSYS2\home\jole\GIT\gramps\gramps\gen\db\generic.py", line
595, in load
     if not self._schema_exists():
   File
"C:\Users\jole\AppData\Roaming/gramps/gramps50/plugins/MongoDB\mongodb.py",
line 129, in _schema_exists
     return "person" in self.db.list_collection_names()
   File
"C:/MSYS2/mingw64/lib/python3.6/site-packages\pymongo\collection.py",
line 2620, in __call__
     self.__name)
TypeError: 'Collection' object is not callable. If you meant to call the
'list_collection_names' method on a 'Database' object it is failing
because no such method exists.

For me "_schema_exists" should be
   return "person" in self.db.collection_names()
instead of
   return "person" in self.db.list_collection_names()

--
Josip


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: New database backends for Gramps 5

Nick Hall
On 08/03/18 22:37, Josip wrote:
8.3.2018. u 1:05, Nick Hall je napisao/la:
I have implemented a MongoDB backend.  You can find it in the third-party addons repository:

https://github.com/gramps-project/addons-source/tree/maintenance/gramps50/MongoDB

There are some performance issues that need addressing, but it appears to work.  Obviously it is not ready for production use yet.


Nick.


Trying to use on Windows with MongoDB v3.6.3 and PyMongo v3.6.1

I can't figure it out how to set database name?
I always get error:

2018-03-08 23:03:11.947: ERROR: dbloader.py: line 111: database names cannot contain the character ' '
Traceback (most recent call last):
  File "C:\MSYS2\home\jole\GIT\gramps\gramps\gui\dbloader.py", line 206, in read_file
    password=password)
  File "C:\MSYS2\home\jole\GIT\gramps\gramps\gen\db\generic.py", line 593, in load
    self._initialize(directory, username, password)
  File "C:\Users\jole\AppData\Roaming/gramps/gramps50/plugins/MongoDB\mongodb.py", line 120, in _initialize
    self.db = self.client[dbname]
  File "C:/MSYS2/mingw64/lib/python3.6/site-packages\pymongo\mongo_client.py", line 996, in __getitem__
    return database.Database(self, name)
  File "C:/MSYS2/mingw64/lib/python3.6/site-packages\pymongo\database.py", line 106, in __init__
    _check_name(name)
  File "C:/MSYS2/mingw64/lib/python3.6/site-packages\pymongo\database.py", line 47, in _check_name
    "character %r" % invalid_char)
pymongo.errors.InvalidName: database names cannot contain the character ' '

so i resort to manually set it at line 113 in mongodb.py
Next i got error:

2018-03-08 23:18:34.407: ERROR: dbloader.py: line 111: 'Collection' object is not callable. If you meant to call the 'list_collection_names' method on a 'Database' object it is failing because no such method exists.
Traceback (most recent call last):
  File "C:\MSYS2\home\jole\GIT\gramps\gramps\gui\dbloader.py", line 206, in read_file
    password=password)
  File "C:\MSYS2\home\jole\GIT\gramps\gramps\gen\db\generic.py", line 595, in load
    if not self._schema_exists():
  File "C:\Users\jole\AppData\Roaming/gramps/gramps50/plugins/MongoDB\mongodb.py", line 129, in _schema_exists
    return "person" in self.db.list_collection_names()
  File "C:/MSYS2/mingw64/lib/python3.6/site-packages\pymongo\collection.py", line 2620, in __call__
    self.__name)
TypeError: 'Collection' object is not callable. If you meant to call the 'list_collection_names' method on a 'Database' object it is failing because no such method exists.

For me "_schema_exists" should be
  return "person" in self.db.collection_names()
instead of
  return "person" in self.db.list_collection_names()

Are you sure you are using mongodb v3.6.3?  The list_collection_names() method is new in v3.6.  See:

http://api.mongodb.com/python/current/api/pymongo/database.html#pymongo.database.Database.list_collection_names

We should probably use collection_names() anyway to be compatible with older versions.

Did you create a family tree name with a space in it?  The family tree name is used as the database name.  Some more work is needed here.

Thanks for testing.  It looks like a mongodb backend may have some potential.

Regards,


Nick.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: New database backends for Gramps 5

Josip
9.3.2018. u 2:03, Nick Hall je napisao/la:
> Are you sure you are using mongodb v3.6.3?  The list_collection_names()
> method is new in v3.6.  See:
>
> http://api.mongodb.com/python/current/api/pymongo/database.html#pymongo.database.Database.list_collection_names
>
> We should probably use collection_names() anyway to be compatible with
> older versions.
>

Turns out that i was using pymongo-3.4.0
Build and install 3.6.1 over existing 3.4.0 but egg crap decided to use
older version :-O
(some time ago i was playing with MongoDB and EJDB)

> Did you create a family tree name with a space in it?  The family tree
> name is used as the database name.  Some more work is needed here.
>

Used default name "Family Tree X" (translated version also contains spaces)
Now when i create new tree and remove spaces from name it works, but
after closing Gramps it gets locked.
Info button as Gramps -L show:
   Unavailable: name '_' is not defined...

https://docs.mongodb.com/manual/reference/limits/
says that database names cannot be empty and must have fewer than 64
characters and cannot contain any of the following characters:
/\. "$*<>:|?


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: New database backends for Gramps 5

Nick Hall
On 09/03/18 20:28, Josip wrote:
9.3.2018. u 2:03, Nick Hall je napisao/la:
Are you sure you are using mongodb v3.6.3?  The list_collection_names() method is new in v3.6.  See:

http://api.mongodb.com/python/current/api/pymongo/database.html#pymongo.database.Database.list_collection_names

We should probably use collection_names() anyway to be compatible with older versions.


Turns out that i was using pymongo-3.4.0
Build and install 3.6.1 over existing 3.4.0 but egg crap decided to use older version :-O
(some time ago i was playing with MongoDB and EJDB)

Did you create a family tree name with a space in it?  The family tree name is used as the database name.  Some more work is needed here.


Used default name "Family Tree X" (translated version also contains spaces)
Now when i create new tree and remove spaces from name it works, but after closing Gramps it gets locked.
Info button as Gramps -L show:
  Unavailable: name '_' is not defined...

https://docs.mongodb.com/manual/reference/limits/
says that database names cannot be empty and must have fewer than 64 characters and cannot contain any of the following characters:
/\. "$*<>:|?

We already remove these characters from the family tree name.  See:

https://github.com/gramps-project/gramps/blob/master/gramps/gui/dbman.py#L554

I have now added code to replace spaces with underscores in the database name.

The collection_name() method bug has also been fixed.

Nick.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel