GRAMPS › GRAMPS - Dev

Current Experiments with Gramps-connect

Classic

List

Threaded

10 messages Options

DS Blank

Current Experiments with Gramps-connect

Just a short note about the current state of some gramps-connect experiments.

The current trunk has the ability to run any report using the standard
gramps API, but with the Django backend. Unfortunately, it is really
slow, even using sqlite on the same computer. Initially, it was taking
about 76 seconds to run the AncestorReport (9 pages of output, total
db size 1455 people). Turns out that about half of that time (36
seconds) was spent populating the Menu options! As one of the main
uses of Gramps-connect is to work on really large data sets, I'll have
to look at a method to only load the options if needed. The goal is to
be able to use a client-server SQL layout, so we do want to keep the
queries to a minimum.

I looked at the Django database structure and saw that we could store
birth_ref_index and death_ref_index, rather than computing those on
the fly. I was hoping to not use those at all, but to keep the Gramps
API the same, I am using them for now. That cut the total time down to
about 53 seconds---36 seconds for the menu, 17 seconds for the report.
Still too long.

I then worked on a DelayedObject interface. This uses the standard
Gramps objects, but only does database lookups when necessary. This
only got the total time down to 45 seconds (31 seconds for the menu,
and 14 for the report). Bummer. I use a lot of generator expressions,
and I think that takes some time to build.

The DelayedObject idea could no doubt be sped up, but I decided to try
a caching method originally suggested by Gerald as the main method of
representing the Django data. Basically, it just takes the serialized
version of an object and saves it. Total time: 8 seconds: 2 seconds
for the menu, 6 seconds for the report. Ok, this suggests to leave out
the complicated API, and just replicate the Gramps BSDDB data in
cache.

More later...

-Doug

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

Benny Malengier

Re: Current Experiments with Gramps-connect

2011/10/12 Doug Blank <[hidden email]>

Just a short note about the current state of some gramps-connect experiments.

The current trunk has the ability to run any report using the standard
gramps API, but with the Django backend. Unfortunately, it is really
slow, even using sqlite on the same computer. Initially, it was taking
about 76 seconds to run the AncestorReport (9 pages of output, total
db size 1455 people). Turns out that about half of that time (36
seconds) was spent populating the Menu options! As one of the main
uses of Gramps-connect is to work on really large data sets, I'll have
to look at a method to only load the options if needed. The goal is to
be able to use a client-server SQL layout, so we do want to keep the
queries to a minimum.

I looked at the Django database structure and saw that we could store
birth_ref_index and death_ref_index, rather than computing those on
the fly. I was hoping to not use those at all, but to keep the Gramps
API the same, I am using them for now. That cut the total time down to
about 53 seconds---36 seconds for the menu, 17 seconds for the report.
Still too long.

I then worked on a DelayedObject interface. This uses the standard
Gramps objects, but only does database lookups when necessary. This
only got the total time down to 45 seconds (31 seconds for the menu,
and 14 for the report). Bummer. I use a lot of generator expressions,
and I think that takes some time to build.

The DelayedObject idea could no doubt be sped up, but I decided to try
a caching method originally suggested by Gerald as the main method of
representing the Django data. Basically, it just takes the serialized
version of an object and saves it. Total time: 8 seconds: 2 seconds
for the menu, 6 seconds for the report. Ok, this suggests to leave out
the complicated API, and just replicate the Gramps BSDDB data in
cache.

Bummer.
The delayedobect idea, did it follow my suggestion of long time ago, or some other way?
Obviously, just dumping our data would be fast, but that actually means you are not using SQL at all, and only access it the same way we access data in Gramps, which is a limitation (other queries need dedicated index tables).

On the other hand, perhaps better something that works with known limitation, than something that satisfies an ideal but needs developer time to realize that is not available.

Benny

More later...

-Doug

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

DS Blank

Re: Current Experiments with Gramps-connect

On Wed, Oct 12, 2011 at 5:21 AM, Benny Malengier
<[hidden email]> wrote:

>
>
> 2011/10/12 Doug Blank <[hidden email]>
>>
>> Just a short note about the current state of some gramps-connect
>> experiments.
>>
>> The current trunk has the ability to run any report using the standard
>> gramps API, but with the Django backend. Unfortunately, it is really
>> slow, even using sqlite on the same computer. Initially, it was taking
>> about 76 seconds to run the AncestorReport (9 pages of output, total
>> db size 1455 people). Turns out that about half of that time (36
>> seconds) was spent populating the Menu options! As one of the main
>> uses of Gramps-connect is to work on really large data sets, I'll have
>> to look at a method to only load the options if needed. The goal is to
>> be able to use a client-server SQL layout, so we do want to keep the
>> queries to a minimum.
>>
>> I looked at the Django database structure and saw that we could store
>> birth_ref_index and death_ref_index, rather than computing those on
>> the fly. I was hoping to not use those at all, but to keep the Gramps
>> API the same, I am using them for now. That cut the total time down to
>> about 53 seconds---36 seconds for the menu, 17 seconds for the report.
>> Still too long.
>>
>> I then worked on a DelayedObject interface. This uses the standard
>> Gramps objects, but only does database lookups when necessary. This
>> only got the total time down to 45 seconds (31 seconds for the menu,
>> and 14 for the report). Bummer. I use a lot of generator expressions,
>> and I think that takes some time to build.
>>
>> The DelayedObject idea could no doubt be sped up, but I decided to try
>> a caching method originally suggested by Gerald as the main method of
>> representing the Django data. Basically, it just takes the serialized
>> version of an object and saves it. Total time: 8 seconds: 2 seconds
>> for the menu, 6 seconds for the report. Ok, this suggests to leave out
>> the complicated API, and just replicate the Gramps BSDDB data in
>> cache.
>
> Bummer.
> The delayedobect idea, did it follow my suggestion of long time ago, or some
> other way?

I think it was what you suggested, at least in spirit. I removed an
object's data attributes, and on getattr, evaluated a DelayedObject,
replacing the attribute. But it may be the details of this that really
slowed the process down. I tried to do this from the outside, without
changing gen.lib, but keeping the types the same. I'll put a patch on
the GEPS [1] for anyone that is interested.

> Obviously, just dumping our data would be fast, but that actually means you
> are not using SQL at all, and only access it the same way we access data in
> Gramps, which is a limitation (other queries need dedicated index tables).
>
> On the other hand, perhaps better something that works with known
> limitation, than something that satisfies an ideal but needs developer time
> to realize that is not available.

My thoughts exactly. Except it isn't too bad: we use SQL for most
everything (blazing fast, especially for huge data sets), except for
the tools and output (reports, and export) where it is fast enough
with the cache. Tools and output can be run in the background (yay,
multiuser!), and the results can be mailed or appear in a results
queue page.

It just means that if you want to use the current API as written, it
is best to have the data as it was designed. Well, I learned something
:) I think that pretty much kills GEPS 17 [1].

The good news is that this answers a long-standing question for me on
how to move forward with gramps-connect. Last night, I ran all of the
reports, and even exported to GEDCOM from the Django database.

-Doug

PS - Thanks Gioto and lcc... the GEPS overview page [2] looks great!

[1] - http://gramps-project.org/wiki/index.php?title=GEPS_017:_Flexible_gen.lib_Interface
[2] - http://gramps-project.org/wiki/index.php?title=GEPS

> Benny
>
>>
>> More later...
>>
>> -Doug
>>
>>
>> ------------------------------------------------------------------------------
>> All the data continuously generated in your IT infrastructure contains a
>> definitive record of customers, application performance, security
>> threats, fraudulent activity and more. Splunk takes this data and makes
>> sense of it. Business sense. IT sense. Common sense.
>> http://p.sf.net/sfu/splunk-d2d-oct
>> _______________________________________________
>> Gramps-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>
>

DS Blank

Re: Current Experiments with Gramps-connect

I've updated the Gramps-Connect live demo to the latest code:

http://gramps-connect.org/

Overall, I think we are about 10% done, but I think all of the major
issues have now been resolved. You can log into the site, as a
superuser (id=admin, password=gramps) or a regular user (id=admin1,
password=gramps) or just view as an anonymous user.

New: if you log in, then you can run a report. Go to Reports, and
select a report. You can enter options the way that you enter them on
the command line. For example, try the "Ahnentafel Report" and enter
for Options:

pid=I0693

The parser is very simple here. We'll eventually replace that with a
screen that looks like the options on the Gtk reports. For now, use a
space between options, and equal-sign between key and value. Note that
some of the reports won't run because they try to use Gtk widgets in
the server environment, where there is no DISPLAY.

If anyone would like to assist with Gramps-Connect, please let me know.

-Doug

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel

jerome

Re: Current Experiments with Gramps-connect

Really good, great work!

> Note that
> some of the reports won't run because they try to use Gtk
> widgets in
> the server environment, where there is no DISPLAY.

I get two errors on GEDCOM export and kinship report.
I suppose the first one is a environment issue (filename), and the second one is related to the Gtk environment (file format set)?

Environment:

Request Method: GET
Request URL: http://gramps-connect.org/report/gedcom_export/run?options=
Django Version: 1.2.2
Python Version: 2.6.4
Installed Applications:
['django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.admin',
'web.grampsdb']
Installed Middleware:
('django.middleware.csrf.CsrfViewMiddleware',
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware')

Traceback:
File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response
100. response = callback(request, *callback_args, **callback_kwargs)
File "/usr/local/wsgi/trunk/src/web/grampsdb/views.py" in process_action
273. export_file(db, filename, lambda n: n) # callback
File "/usr/local/wsgi/trunk/src/web/dbdjango.py" in export_file
109. for name, error_tuple in pmgr.get_fail_list():

Exception Type: ValueError at /report/gedcom_export/run
Exception Value: too many values to unpack

Environment:

Request Method: GET
Request URL: http://gramps-connect.org/report/kinship_report/run?options=pid%3DI0693
Django Version: 1.2.2
Python Version: 2.6.4
Installed Applications:
['django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.admin',
'web.grampsdb']
Installed Middleware:
('django.middleware.csrf.CsrfViewMiddleware',
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware')

Traceback:
File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response
100. response = callback(request, *callback_args, **callback_kwargs)
File "/usr/local/wsgi/trunk/src/web/grampsdb/views.py" in process_action
277. return send_file(request, filename, mimetype)
File "/usr/local/wsgi/trunk/src/web/grampsdb/views.py" in send_file
243. wrapper = FileWrapper(file(filename))

Exception Type: IOError at /report/kinship_report/run
Exception Value: (2, 'No such file or directory')

> If anyone would like to assist with Gramps-Connect, please
> let me know.

What is the need?

Is it possible to also run it within Gramps via HTML view ??? :-/
OK, it sounds strange, I just wanted to know if possible.

Great to see some developements for open Web applications.
Someone - who also contributed to Gramps in the past- is also making something based on Django: http://briot.github.com/geneapro/index.html
And others persons are improving GeneWeb: http://opensource.geneanet.org/projects/geneweb/wiki/En_home

Thank you!
Jérôme

--- En date de : Mer 12.10.11, Doug Blank <[hidden email]> a écrit :

> De: Doug Blank <[hidden email]>
> Objet: Re: [Gramps-devel] Current Experiments with Gramps-connect
> À: "Benny Malengier" <[hidden email]>
> Cc: "Gramps Development List" <[hidden email]>
> Date: Mercredi 12 octobre 2011, 23h09
> I've updated the Gramps-Connect live
> demo to the latest code:
>
> http://gramps-connect.org/
>
> Overall, I think we are about 10% done, but I think all of
> the major
> issues have now been resolved. You can log into the site,
> as a
> superuser (id=admin, password=gramps) or a regular user
> (id=admin1,
> password=gramps) or just view as an anonymous user.
>
> New: if you log in, then you can run a report. Go to
> Reports, and
> select a report. You can enter options the way that you
> enter them on
> the command line. For example, try the "Ahnentafel Report"
> and enter
> for Options:
>
> pid=I0693
>
> The parser is very simple here. We'll eventually replace
> that with a
> screen that looks like the options on the Gtk reports. For
> now, use a
> space between options, and equal-sign between key and
> value. Note that
> some of the reports won't run because they try to use Gtk
> widgets in
> the server environment, where there is no DISPLAY.
>
> If anyone would like to assist with Gramps-Connect, please
> let me know.
>
> -Doug
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT
> infrastructure contains a
> definitive record of customers, application performance,
> security
> threats, fraudulent activity and more. Splunk takes this
> data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>

DS Blank

Re: Current Experiments with Gramps-connect

On Thu, Oct 13, 2011 at 1:57 AM, jerome <[hidden email]> wrote:
> Really good, great work!

Thanks!

> I get two errors on GEDCOM export and kinship report.
> I suppose the first one is a environment issue (filename), and the second one is related to the Gtk environment (file format set)?

> Exception Type: ValueError at /report/gedcom_export/run
> Exception Value: too many values to unpack

Looking into this... works on one machine, but not server... and first
issue causes the second issue I think.

>> If anyone would like to assist with Gramps-Connect, please
>> let me know.
>
> What is the need?

The first goal is to get all of the data displayed. I'll update the
docs to point to where people can assist. Also, as gramps-connect uses
Gramps css, there are a number of issues that can be adjusted to make
viewing better. Gramps-connect needs to use the correct classes and
ids for items.

> Is it possible to also run it within Gramps via HTML view ??? :-/
> OK, it sounds strange, I just wanted to know if possible.

No not strange. I looked into some interoperability between Gramps Gtk
running a HTML viewer and web apps in general. But I found the HTML
viewer to be so buggy that I gave up. But I think we can do some
things directly between Gramps Gtk and a website (say, for example, of
using special GET/POST to exchange data). If you have other experience
or ideas, please let us know.

> Great to see some developements for open Web applications.
> Someone - who also contributed to Gramps in the past- is also making something based on Django: http://briot.github.com/geneapro/index.html
> And others persons are improving GeneWeb: http://opensource.geneanet.org/projects/geneweb/wiki/En_home

Although neither of those look immediately like they would be useful
for sharing, there was a pointer to:

http://www.simile-widgets.org/timeline/

which looks like a great widget for gramps-connect (and maybe
Narrative Web---but we need to refactor that before adding anything).

-Doug

>
> Thank you!
> Jérôme
>
> --- En date de : Mer 12.10.11, Doug Blank <[hidden email]> a écrit :
>
>> De: Doug Blank <[hidden email]>
>> Objet: Re: [Gramps-devel] Current Experiments with Gramps-connect
>> À: "Benny Malengier" <[hidden email]>
>> Cc: "Gramps Development List" <[hidden email]>
>> Date: Mercredi 12 octobre 2011, 23h09
>> I've updated the Gramps-Connect live
>> demo to the latest code:
>>
>> http://gramps-connect.org/
>>
>> Overall, I think we are about 10% done, but I think all of
>> the major
>> issues have now been resolved. You can log into the site,
>> as a
>> superuser (id=admin, password=gramps) or a regular user
>> (id=admin1,
>> password=gramps) or just view as an anonymous user.
>>
>> New: if you log in, then you can run a report. Go to
>> Reports, and
>> select a report. You can enter options the way that you
>> enter them on
>> the command line. For example, try the "Ahnentafel Report"
>> and enter
>> for Options:
>>
>> pid=I0693
>>
>> The parser is very simple here. We'll eventually replace
>> that with a
>> screen that looks like the options on the Gtk reports. For
>> now, use a
>> space between options, and equal-sign between key and
>> value. Note that
>> some of the reports won't run because they try to use Gtk
>> widgets in
>> the server environment, where there is no DISPLAY.
>>
>> If anyone would like to assist with Gramps-Connect, please
>> let me know.
>>
>> -Doug
>>
>> ------------------------------------------------------------------------------
>> All the data continuously generated in your IT
>> infrastructure contains a
>> definitive record of customers, application performance,
>> security
>> threats, fraudulent activity and more. Splunk takes this
>> data and makes
>> sense of it. Business sense. IT sense. Common sense.
>> http://p.sf.net/sfu/splunk-d2d-oct
>> _______________________________________________
>> Gramps-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>
>

jerome

Re: Current Experiments with Gramps-connect

> The first goal is to get all of the data displayed. I'll
> update the
> docs to point to where people can assist.

OK.

Note, for fun I generated some HTML web forms according to Gramps DTD/RelaxNG! It was only the 7/8 tables related to primary objects but also with references (all "References" Editors).
If this could help for listing missing tabs/items/secondary editors.

It was quickly generated, but this is also an alternate way for displaying at a glance, places where Gramps is storing data.

File is "forms.tar.bz2" on:
http://www.gramps-project.org/bugs/view.php?id=4234

Do not worry about all comments on #4234, it is rather a long list of current situation on the french market than a solution for web applications. ;)

I could add that 'backupmytree' is now an acquisition of MyHeritage http://blog.myheritage.com/tag/backupmytree-com/
but this is not related to 'gramps-connect'.

Thanks.
Jérôme

--- En date de : Jeu 13.10.11, Doug Blank <[hidden email]> a écrit :

> De: Doug Blank <[hidden email]>
> Objet: Re: [Gramps-devel] Current Experiments with Gramps-connect
> À: "jerome" <[hidden email]>
> Cc: "Benny Malengier" <[hidden email]>, "Gramps Development List" <[hidden email]>
> Date: Jeudi 13 octobre 2011, 13h12
> On Thu, Oct 13, 2011 at 1:57 AM,
> jerome <[hidden email]>
> wrote:
> > Really good, great work!
>
> Thanks!
>
> > I get two errors on GEDCOM export and kinship report.
> > I suppose the first one is a environment issue
> (filename), and the second one is related to the Gtk
> environment (file format set)?
>
> > Exception Type: ValueError at
> /report/gedcom_export/run
> > Exception Value: too many values to unpack
>
> Looking into this... works on one machine, but not
> server... and first
> issue causes the second issue I think.
>
> >> If anyone would like to assist with
> Gramps-Connect, please
> >> let me know.
> >
> > What is the need?
>
> The first goal is to get all of the data displayed. I'll
> update the
> docs to point to where people can assist. Also, as
> gramps-connect uses
> Gramps css, there are a number of issues that can be
> adjusted to make
> viewing better. Gramps-connect needs to use the correct
> classes and
> ids for items.
>
> > Is it possible to also run it within Gramps via HTML
> view ??? :-/
> > OK, it sounds strange, I just wanted to know if
> possible.
>
> No not strange. I looked into some interoperability between
> Gramps Gtk
> running a HTML viewer and web apps in general. But I found
> the HTML
> viewer to be so buggy that I gave up. But I think we can do
> some
> things directly between Gramps Gtk and a website (say, for
> example, of
> using special GET/POST to exchange data). If you have other
> experience
> or ideas, please let us know.
>
> > Great to see some developements for open Web
> applications.
> > Someone - who also contributed to Gramps in the past-
> is also making something based on Django: http://briot.github.com/geneapro/index.html
> > And others persons are improving GeneWeb: http://opensource.geneanet.org/projects/geneweb/wiki/En_home
>
> Although neither of those look immediately like they would
> be useful
> for sharing, there was a pointer to:
>
> http://www.simile-widgets.org/timeline/
>
> which looks like a great widget for gramps-connect (and
> maybe
> Narrative Web---but we need to refactor that before adding
> anything).
>
> -Doug
>
> >
> > Thank you!
> > Jérôme
> >
> > --- En date de : Mer 12.10.11, Doug Blank <[hidden email]>
> a écrit :
> >
> >> De: Doug Blank <[hidden email]>
> >> Objet: Re: [Gramps-devel] Current Experiments with
> Gramps-connect
> >> À: "Benny Malengier" <[hidden email]>
> >> Cc: "Gramps Development List" <[hidden email]>
> >> Date: Mercredi 12 octobre 2011, 23h09
> >> I've updated the Gramps-Connect live
> >> demo to the latest code:
> >>
> >> http://gramps-connect.org/
> >>
> >> Overall, I think we are about 10% done, but I
> think all of
> >> the major
> >> issues have now been resolved. You can log into
> the site,
> >> as a
> >> superuser (id=admin, password=gramps) or a regular
> user
> >> (id=admin1,
> >> password=gramps) or just view as an anonymous
> user.
> >>
> >> New: if you log in, then you can run a report. Go
> to
> >> Reports, and
> >> select a report. You can enter options the way
> that you
> >> enter them on
> >> the command line. For example, try the "Ahnentafel
> Report"
> >> and enter
> >> for Options:
> >>
> >> pid=I0693
> >>
> >> The parser is very simple here. We'll eventually
> replace
> >> that with a
> >> screen that looks like the options on the Gtk
> reports. For
> >> now, use a
> >> space between options, and equal-sign between key
> and
> >> value. Note that
> >> some of the reports won't run because they try to
> use Gtk
> >> widgets in
> >> the server environment, where there is no
> DISPLAY.
> >>
> >> If anyone would like to assist with
> Gramps-Connect, please
> >> let me know.
> >>
> >> -Doug
> >>
> >>
> ------------------------------------------------------------------------------
> >> All the data continuously generated in your IT
> >> infrastructure contains a
> >> definitive record of customers, application
> performance,
> >> security
> >> threats, fraudulent activity and more. Splunk
> takes this
> >> data and makes
> >> sense of it. Business sense. IT sense. Common
> sense.
> >> http://p.sf.net/sfu/splunk-d2d-oct
> >> _______________________________________________
> >> Gramps-devel mailing list
> >> [hidden email]
> >> https://lists.sourceforge.net/lists/listinfo/gramps-devel
> >>
> >
>

jerome

Re: Current Experiments with Gramps-connect

In reply to this post by DS Blank

I wonder if we should not also look at:
http://packages.python.org/django-easymode/
http://packages.python.org/django-easymode/xslt/index.html
https://github.com/specialunderwear/django-easymode/tree/master/easymode

Not for Flash/Flex but for translations and XSLT !

--- En date de : Jeu 13.10.11, Doug Blank <[hidden email]> a écrit :

------------------------------------------------------------------------------

> >> All the data continuously generated in your IT
> >> infrastructure contains a
> >> definitive record of customers, application
> performance,
> >> security
> >> threats, fraudulent activity and more. Splunk
> takes this
> >> data and makes
> >> sense of it. Business sense. IT sense. Common
> sense.
> >> http://p.sf.net/sfu/splunk-d2d-oct
> >> _______________________________________________
> >> Gramps-devel mailing list
> >> [hidden email]
> >> https://lists.sourceforge.net/lists/listinfo/gramps-devel
> >>
> >
>

DS Blank

Re: Current Experiments with Gramps-connect

On Fri, Oct 14, 2011 at 7:27 AM, jerome <[hidden email]> wrote:
> I wonder if we should not also look at:
> http://packages.python.org/django-easymode/
> http://packages.python.org/django-easymode/xslt/index.html
> https://github.com/specialunderwear/django-easymode/tree/master/easymode
>
> Not for Flash/Flex but for translations and XSLT !

Cool project, but not sure it is useful for us now. It may have been
useful as a starting point before we had the models built, but at this
point we are well beyond that stage. Even so, there is a lot of design
that goes into the tables that would be impossible to automate from
just the XML description. Or am I misunderstanding?

-Doug

>
> --- En date de : Jeu 13.10.11, Doug Blank <[hidden email]> a écrit :
>
>> De: Doug Blank <[hidden email]>
>> Objet: Re: [Gramps-devel] Current Experiments with Gramps-connect
>> À: "jerome" <[hidden email]>
>> Cc: "Benny Malengier" <[hidden email]>, "Gramps Development List" <[hidden email]>
>> Date: Jeudi 13 octobre 2011, 13h12
>> On Thu, Oct 13, 2011 at 1:57 AM,
>> jerome <[hidden email]>
>> wrote:
>> > Really good, great work!
>>
>> Thanks!
>>
>> > I get two errors on GEDCOM export and kinship report.
>> > I suppose the first one is a environment issue
>> (filename), and the second one is related to the Gtk
>> environment (file format set)?
>>
>> > Exception Type: ValueError at
>> /report/gedcom_export/run
>> > Exception Value: too many values to unpack
>>
>> Looking into this... works on one machine, but not
>> server... and first
>> issue causes the second issue I think.
>>
>> >> If anyone would like to assist with
>> Gramps-Connect, please
>> >> let me know.
>> >
>> > What is the need?
>>
>> The first goal is to get all of the data displayed. I'll
>> update the
>> docs to point to where people can assist. Also, as
>> gramps-connect uses
>> Gramps css, there are a number of issues that can be
>> adjusted to make
>> viewing better. Gramps-connect needs to use the correct
>> classes and
>> ids for items.
>>
>> > Is it possible to also run it within Gramps via HTML
>> view ??? :-/
>> > OK, it sounds strange, I just wanted to know if
>> possible.
>>
>> No not strange. I looked into some interoperability between
>> Gramps Gtk
>> running a HTML viewer and web apps in general. But I found
>> the HTML
>> viewer to be so buggy that I gave up. But I think we can do
>> some
>> things directly between Gramps Gtk and a website (say, for
>> example, of
>> using special GET/POST to exchange data). If you have other
>> experience
>> or ideas, please let us know.
>>
>> > Great to see some developements for open Web
>> applications.
>> > Someone - who also contributed to Gramps in the past-
>> is also making something based on Django: http://briot.github.com/geneapro/index.html
>> > And others persons are improving GeneWeb: http://opensource.geneanet.org/projects/geneweb/wiki/En_home
>>
>> Although neither of those look immediately like they would
>> be useful
>> for sharing, there was a pointer to:
>>
>> http://www.simile-widgets.org/timeline/
>>
>> which looks like a great widget for gramps-connect (and
>> maybe
>> Narrative Web---but we need to refactor that before adding
>> anything).
>>
>> -Doug
> ------------------------------------------------------------------------------
>> >> All the data continuously generated in your IT
>> >> infrastructure contains a
>> >> definitive record of customers, application
>> performance,
>> >> security
>> >> threats, fraudulent activity and more. Splunk
>> takes this
>> >> data and makes
>> >> sense of it. Business sense. IT sense. Common
>> sense.
>> >> http://p.sf.net/sfu/splunk-d2d-oct
>> >> _______________________________________________
>> >> Gramps-devel mailing list
>> >> [hidden email]
>> >> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>> >>
>> >
>>
>

DS Blank

Re: Current Experiments with Gramps-connect

FYI, another update on Gramps-connect.

Now that we are using an exact copy of the BSDDB data format as a
cache in the Django database, things are falling into place. This week
adds the ability for the website to run all reports (except those that
try to open up Gtk windows) in whatever format you want (text, PDF,
etc), and also export directly from the Django databases. That means
you can go to a Gramps-connect site and download an up-to-date version
of the data in Gramps XML or GEDCOM (etc).

What also works now is importing directly into the website. Prior to
this week, if you wanted to populate a Gramps-connect database, one
had to run Gramps GTK and do an Export from there into a Django
Database (sqlite, mysql, postgresql, etc) that you had configured
locally. Now, you can go to the website, upload a GEDCOM or Gramps XML
package (or provide a URL of a family tree) and Gramps-connect will
import this over the web into the database. (Well, this works in code
manually run on the server, but there isn't a web interface to the
importer yet).

Importing directly into the Django tables was an interesting problem
because the import code was not written with a relational database in
mind. In Django, you can't save a person's details until all of the
items you are going to link to have also been entered into their
tables. So, the import code for the Django tables pretends to import
all of the data. At this point you could see an overview of exactly
what is going to happen, before it actually changes any data. When the
standard Gramps import process is complete, a gramps-connect
post-process will go through the data which was attempted to be
imported, and will actually add all of the primary data to their
tables. Then the post-process goes through the data a second time, and
adds all of the linked data. The tests I have done so far shows that
this is working fine (thank you idempotent input/output!) Currently,
all of the imported data is residing in memory before the
post-processing begins, but that would be easy to add a temporary
database to remove that issue.

I think that this two-step method would also work with Gramps Gtk so
that we could have a real merge on import, or perhaps only enter some
of the data from an import, and discard the rest.

I'll hook up the importer on the website (http://gramps-connect.org)
as soon as I can, and people can try importing family trees from
around the web. This will be useful for seeing how Gramps-connect
works under large datasets. Now, I need to write some documentation...

BTW, I've moved trunk/src/web to trunk/src/webapp to make it clear
that everything in this directory is for the webapp. Trunk users will
want to ./autogen.sh and make to get everything redefined.

-Doug

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel