Help requested on database upgrade please

classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Help requested on database upgrade please

Tim Lyons
Administrator
I have made a start on implementing GEPS 023 under:
https://gramps.svn.sourceforge.net/svnroot/gramps/branches/geps/gep-023-large-sources

But I have some problems with implementing the database upgrade. In  
particular, I would appreciate some help in adding and removing the  
backlinks in the reference map. I have looked at update_reference_map  
in write.py, but this has not helped me enough in implementing the  
upgrade when the database is not yet fully connected.

As a temporary measure, I have modified write.py to connect the  
secondary tables, and have used update-reference_map in upgrade.py,  
but this is clearly too inefficient. (I guess I also need to do all  
the backlink upgrades where the new object is created).

I also have a number of questions which are interspersed in the draft  
code in upgrade.py.

(1) Does nesting BSDDBTxn inside DbTxn as I have done mean that the  
whole of the upgrade of each object instance is done as one  
transaction (and have I passed the correct transaction parameters?)
(2) Do I need to upgrade any other secondary tables (specifically the  
reference_map_primary_map)
(3) Have I implemented the creation of the new citation object in  
upgrade.py in the appropriate way?
(4) The way I find the next gramps id for the new citation is almost  
certainly not the most efficient approach, and I also don't understand  
how cmap_index should be used and maintained (it doesn't seem to be  
initialised or updated).

I am sorry that my code in upgrade.py is so untidy at present, but I  
am trying to work through how it should be implemented.

Many thanks if you are able to help.

------------------------------------------------------------------------------
Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
user administration capabilities and model configuration. Take
the hassle out of deploying and managing Subversion and the
tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Help requested on database upgrade please

Benny Malengier


2011/8/23 Tim Lyons <[hidden email]>
I have made a start on implementing GEPS 023 under:
https://gramps.svn.sourceforge.net/svnroot/gramps/branches/geps/gep-023-large-pgrade of each objesources

But I have some problems with implementing the database upgrade. In
particular, I would appreciate some help in adding and removing the
backlinks in the reference map. I have looked at update_reference_map
in write.py, but this has not helped me enough in implementing the
upgrade when the database is not yet fully connected.

As a temporary measure, I have modified write.py to connect the
secondary tables, and have used update-reference_map in upgrade.py,
but this is clearly too inefficient. (I guess I also need to do all
the backlink upgrades where the new object is created).

upgrade need not be efficient. Go for clarity of code here, not maximum speed.
Creating an object should create backlinks if needed, but that is one function like is done for the other primary objects.
 

I also have a number of questions which are interspersed in the draft
code in upgrade.py.

(1) Does nesting BSDDBTxn inside DbTxn as I have done mean that the
whole of the upgrade of each object instance is done as one
transaction (and have I passed the correct transaction parameters?)

Ask Gerald Britton  directly if that is the correct approach

(2) Do I need to upgrade any other secondary tables (specifically the
reference_map_primary_map)

you can delete the table and recreate it with the function as used in the tool with that name. That tool is a good way to test your code.
 
(3) Have I implemented the creation of the new citation object in
upgrade.py in the appropriate way?

it is as the other upgrades. Do try to be consistent with names, so call sourcereference srcref, not source, ... . Also, it is nice to show the numbers in tuples as in
        new_person = (junk_handle,        #  0
                      gramps_id,          #  1
                      gender,             #  2
 so it is well readable how the old database structure was, and what the new becomes.

(4) The way I find the next gramps id for the new citation is almost
certainly not the most efficient approach, and I also don't understand
how cmap_index should be used and maintained (it doesn't seem to be
initialised or updated).

This is the first use of citations, so why even count it and try to calculate. Just do comething like C%05d or so, and increase your own counter.
 

I am sorry that my code in upgrade.py is so untidy at present, but I
am trying to work through how it should be implemented.

About your use of global. I don't like the use of globals in python :-)
Here you can just remove it and make it local to gramps_upgrade_16, using something like

num_citations += len(new_citation_list)

A remark, every sourceref becomes a citation. Is the idea not that we can collaps sourceref in one single citation that is shared?  I'm a bit afraid users will have a huge time needed to merge citations to something manageable.

Benny

Many thanks if you are able to help.

------------------------------------------------------------------------------
Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
user administration capabilities and model configuration. Take
the hassle out of deploying and managing Subversion and the
tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel


------------------------------------------------------------------------------
Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
user administration capabilities and model configuration. Take
the hassle out of deploying and managing Subversion and the
tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2

_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Help requested on database upgrade please

Tim Lyons
Administrator

On 23 Aug 2011, at 21:29, Benny Malengier wrote:

> This is the first use of citations, so why even count it and try to  
> calculate. Just do comething like C%05d or so, and increase your own  
> counter.
>
> About your use of global. I don't like the use of globals in  
> python :-)
>

Thanks for your advice, which I have implemented.

> A remark, every sourceref becomes a citation. Is the idea not that  
> we can collaps sourceref in one single citation that is shared?  I'm  
> a bit afraid users will have a huge time needed to merge citations  
> to something manageable.
>

Yes, every sourceref becomes a citation.

A key objective in the way I have specified GEPS 023 is that it should  
be as simple as possible for users.

in fact, users should not notice any difference due to the  
introduction of citations, unless they want to take advantage of new  
features.

There is no need for users to merge citations unless they want to. I  
intend to implement a separate Gramplet to merge citations where they  
refer to the same source and have the same Volume/Page etc. I think it  
will be better to separate out merging rather that to do it  
automatically on upgrade. This will allow the user to choose whether  
they want to merge or not. It will also make it much simpler to change  
the criteria for merging without changing the main gramps code. I hope  
you think this approach is OK.

------------------------------------------------------------------------------
EMC VNX: the world's simplest storage, starting under $10K
The only unified storage solution that offers unified management
Up to 160% more powerful than alternatives and 25% more efficient.
Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Help requested on database upgrade please

Benny Malengier


2011/8/28 Tim Lyons <[hidden email]>

On 23 Aug 2011, at 21:29, Benny Malengier wrote:

This is the first use of citations, so why even count it and try to calculate. Just do comething like C%05d or so, and increase your own counter.

About your use of global. I don't like the use of globals in python :-)


Thanks for your advice, which I have implemented.


A remark, every sourceref becomes a citation. Is the idea not that we can collaps sourceref in one single citation that is shared?  I'm a bit afraid users will have a huge time needed to merge citations to something manageable.


Yes, every sourceref becomes a citation.

A key objective in the way I have specified GEPS 023 is that it should be as simple as possible for users.

in fact, users should not notice any difference due to the introduction of citations, unless they want to take advantage of new features.

There is no need for users to merge citations unless they want to. I intend to implement a separate Gramplet to merge citations where they refer to the same source and have the same Volume/Page etc. I think it will be better to separate out merging rather that to do it automatically on upgrade. This will allow the user to choose whether they want to merge or not. It will also make it much simpler to change the criteria for merging without changing the main gramps code. I hope you think this approach is OK.

Yes, this is the typical approach.

However, this is something that should be a tool in my opinion, not a gramplet, that is, something you run perhaps once a year. After the upgrade a request to the user if he wants to run the tool then right away, would be nice.
It should also be part of gramps, and ready at time of release :-)

Benny

------------------------------------------------------------------------------
EMC VNX: the world's simplest storage, starting under $10K
The only unified storage solution that offers unified management
Up to 160% more powerful than alternatives and 25% more efficient.
Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Help requested on database upgrade please

Tim Lyons
Administrator
In reply to this post by Benny Malengier
On 23 Aug 2011, at 21:29, Benny Malengier wrote:



2011/8/23 Tim Lyons <[hidden email]>

you can delete the table and recreate it with the function as used in the tool with that name. That tool is a good way to test your code.
 

Could someone please explain the difference between Tools -> Family Tree Repair -> Rebuild Reference Maps and Rebuild Secondary Indices.

I understand that the first calls write.py reindex_reference_map and the second calls rebuild_secondary. I can see that reindex_reference_map rebuilds the reference maps by going through each primary object, and calling update_reference_map, but what exactly does rebuild_secondary do?

Also, in write.py I have had to call self.__connect_secondary() and self.__open_undodb() before doing the gramps_upgrade_16. Would I have to do that if I used the reindex_reference_map function?


------------------------------------------------------------------------------
EMC VNX: the world's simplest storage, starting under $10K
The only unified storage solution that offers unified management
Up to 160% more powerful than alternatives and 25% more efficient.
Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Help requested on database upgrade please

Benny Malengier


2011/8/29 Tim Lyons <[hidden email]>
On 23 Aug 2011, at 21:29, Benny Malengier wrote:



2011/8/23 Tim Lyons <[hidden email]>

you can delete the table and recreate it with the function as used in the tool with that name. That tool is a good way to test your code.
 

Could someone please explain the difference between Tools -> Family Tree Repair -> Rebuild Reference Maps and Rebuild Secondary Indices.

I understand that the first calls write.py reindex_reference_map and the second calls rebuild_secondary. I can see that reindex_reference_map rebuilds the reference maps by going through each primary object, and calling update_reference_map, but what exactly does rebuild_secondary do?

Without looking at the code, secondary tables are index tables of bsddb which bsddb can construct automatically, eg our surname table for sorting.

Also, in write.py I have had to call self.__connect_secondary() and self.__open_undodb() before doing the gramps_upgrade_16. Would I have to do that if I used the reindex_reference_map function?

Large updates are better without the secondary tables connected, as it slows down the upgrade. Then, when finished, you connect them or have bsddb rebuild them.
I would have to dive into the code to help more.

Benny


------------------------------------------------------------------------------
EMC VNX: the world's simplest storage, starting under $10K
The only unified storage solution that offers unified management
Up to 160% more powerful than alternatives and 25% more efficient.
Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Help requested on database upgrade please

jerome
In reply to this post by Tim Lyons
Tim,


I just see that there is an "SourceTemplates" project:
http://sourcetemplates.org/details.php

I did not test GEPS 023 yet, but I guess current citation model could be
compliant with this project or their work could be improved by your
investigations, isn' it ?


Regards,
Jérôme


Tim Lyons a écrit :

> I have made a start on implementing GEPS 023 under:
> https://gramps.svn.sourceforge.net/svnroot/gramps/branches/geps/gep-023-large-sources
>
> But I have some problems with implementing the database upgrade. In  
> particular, I would appreciate some help in adding and removing the  
> backlinks in the reference map. I have looked at update_reference_map  
> in write.py, but this has not helped me enough in implementing the  
> upgrade when the database is not yet fully connected.
>
> As a temporary measure, I have modified write.py to connect the  
> secondary tables, and have used update-reference_map in upgrade.py,  
> but this is clearly too inefficient. (I guess I also need to do all  
> the backlink upgrades where the new object is created).
>
> I also have a number of questions which are interspersed in the draft  
> code in upgrade.py.
>
> (1) Does nesting BSDDBTxn inside DbTxn as I have done mean that the  
> whole of the upgrade of each object instance is done as one  
> transaction (and have I passed the correct transaction parameters?)
> (2) Do I need to upgrade any other secondary tables (specifically the  
> reference_map_primary_map)
> (3) Have I implemented the creation of the new citation object in  
> upgrade.py in the appropriate way?
> (4) The way I find the next gramps id for the new citation is almost  
> certainly not the most efficient approach, and I also don't understand  
> how cmap_index should be used and maintained (it doesn't seem to be  
> initialised or updated).
>
> I am sorry that my code in upgrade.py is so untidy at present, but I  
> am trying to work through how it should be implemented.
>
> Many thanks if you are able to help.
>
> ------------------------------------------------------------------------------
> Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
> user administration capabilities and model configuration. Take
> the hassle out of deploying and managing Subversion and the
> tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>


------------------------------------------------------------------------------
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better
price-free! And you'll get a free "Love Thy Logs" t-shirt when you
download Logger. Secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsisghtdev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Help requested on database upgrade please

Nick Hall-6
Jérôme,

Thanks for the link.  It seems to be more related to GEPS 018: Evidence
style sources.

However, they use a two-level source-citation hierarchy like Tim is
implementing with GEPS 023.  MasterSource is like our Source and
SourceDetails is like our Citation.  The also have a Repository object.

Both there Sources and Citations can have media and comments.  With GEPS
023 we will be able to attach both Notes and Media objects to Citations.

Nick.


On 31/08/11 08:51, Jérôme wrote:

> Tim,
>
>
> I just see that there is an "SourceTemplates" project:
> http://sourcetemplates.org/details.php
>
> I did not test GEPS 023 yet, but I guess current citation model could be
> compliant with this project or their work could be improved by your
> investigations, isn' it ?
>
>
> Regards,
> Jérôme
>
>
> Tim Lyons a écrit :
>> I have made a start on implementing GEPS 023 under:
>> https://gramps.svn.sourceforge.net/svnroot/gramps/branches/geps/gep-023-large-sources
>>
>> But I have some problems with implementing the database upgrade. In
>> particular, I would appreciate some help in adding and removing the
>> backlinks in the reference map. I have looked at update_reference_map
>> in write.py, but this has not helped me enough in implementing the
>> upgrade when the database is not yet fully connected.
>>
>> As a temporary measure, I have modified write.py to connect the
>> secondary tables, and have used update-reference_map in upgrade.py,
>> but this is clearly too inefficient. (I guess I also need to do all
>> the backlink upgrades where the new object is created).
>>
>> I also have a number of questions which are interspersed in the draft
>> code in upgrade.py.
>>
>> (1) Does nesting BSDDBTxn inside DbTxn as I have done mean that the
>> whole of the upgrade of each object instance is done as one
>> transaction (and have I passed the correct transaction parameters?)
>> (2) Do I need to upgrade any other secondary tables (specifically the
>> reference_map_primary_map)
>> (3) Have I implemented the creation of the new citation object in
>> upgrade.py in the appropriate way?
>> (4) The way I find the next gramps id for the new citation is almost
>> certainly not the most efficient approach, and I also don't understand
>> how cmap_index should be used and maintained (it doesn't seem to be
>> initialised or updated).
>>
>> I am sorry that my code in upgrade.py is so untidy at present, but I
>> am trying to work through how it should be implemented.
>>
>> Many thanks if you are able to help.
>>
>> ------------------------------------------------------------------------------
>> Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>> user administration capabilities and model configuration. Take
>> the hassle out of deploying and managing Subversion and the
>> tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>> _______________________________________________
>> Gramps-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>
>
> ------------------------------------------------------------------------------
> Special Offer -- Download ArcSight Logger for FREE!
> Finally, a world-class log management solution at an even better
> price-free! And you'll get a free "Love Thy Logs" t-shirt when you
> download Logger. Secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsisghtdev2dev
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel

------------------------------------------------------------------------------
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better
price-free! And you'll get a free "Love Thy Logs" t-shirt when you
download Logger. Secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsisghtdev2dev
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

GEPS 023

Tim Lyons
Administrator
In reply to this post by Benny Malengier
I have made progress on implementing GEPS 023, many thanks to Nick  
Hall for implementing treemodels with two object types and to Michiel  
Nauta for help with upgrade.

I would be very grateful if some other developers could have a look at  
the user interface and try it with their data. The GEPS is at: https://gramps.svn.sourceforge.net/svnroot/gramps/branches/geps/gep-023-large-sources

The principal user interface changes are to the Sources view, and the  
introduction of the Citations view. Key objectives are that citations  
can be shared and have Media as well as Notes.

I have implemented a 'Merge citation' tool for use after database  
upgrade, as suggested by Benny on the developers list on 29 Aug.

Database upgrade, xml import and export and the various editors are  
all done. The citation filter is mostly done, but some of the other  
filters still need some work.

GEDCOM import and export and the various reports are the main  
outstanding things to be implemented.

Would it be possible for this GEPS to be added to the roadmap for  
Gramps 3.4?

Tim.



------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: GEPS 023

robhealey1
Greetings:

I know that I am not exactly what you were asking about, as in a developer to test and give a complete analysis and help...

I am willing to do and try whatever parts that you would need feedback from a user standpoint...  I have a good working backup before I even start to use your new LARGE SOURCES!

I have been looking forward to this so I have a +1 for implementing this into Gramps-3.4.0 ...

Sincerely yours,
Rob G. Healey


On Sat, Oct 1, 2011 at 7:45 AM, Tim Lyons <[hidden email]> wrote:
I have made progress on implementing GEPS 023, many thanks to Nick
Hall for implementing treemodels with two object types and to Michiel
Nauta for help with upgrade.

I would be very grateful if some other developers could have a look at
the user interface and try it with their data. The GEPS is at: https://gramps.svn.sourceforge.net/svnroot/gramps/branches/geps/gep-023-large-sources

The principal user interface changes are to the Sources view, and the
introduction of the Citations view. Key objectives are that citations
can be shared and have Media as well as Notes.

I have implemented a 'Merge citation' tool for use after database
upgrade, as suggested by Benny on the developers list on 29 Aug.

Database upgrade, xml import and export and the various editors are
all done. The citation filter is mostly done, but some of the other
filters still need some work.

GEDCOM import and export and the various reports are the main
outstanding things to be implemented.

Would it be possible for this GEPS to be added to the roadmap for
Gramps 3.4?

Tim.



------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel



--
Sincerely yours,
Rob G. Healey

"Always surround yourself with people that inspire you to
greatness!"


------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: GEPS 023

Helge@Gramps
In reply to this post by Tim Lyons
This version looks really nice having this new features. I'll try next few days how to work using my available data.
Here a first error issue:
If I try to add the gramplet "Person Sources" I get this message:
2585399: ERROR: gramps.py: line 142: Unhandled exception
Traceback (most recent call last):
  File "C:\ProgrammeNeutral\GrampsGEP023\src\gui\grampsbar.py", line 459, in __a
dd_clicked
    self.add_gramplet(gname)
  File "C:\ProgrammeNeutral\GrampsGEP023\src\gui\grampsbar.py", line 260, in add
_gramplet
    self.dbstate, self.uistate)
  File "C:\ProgrammeNeutral\GrampsGEP023\src\gui\widgets\grampletpane.py", line
167, in make_requested_gramplet
    getattr(module, opts["content"])(gui)
  File "C:\ProgrammeNeutral\GrampsGEP023\src\gen\plug\_gramplet.py", line 72, in
 __init__
    self._db_changed(self.dbstate.db)
  File "C:\ProgrammeNeutral\GrampsGEP023\src\gen\plug\_gramplet.py", line 377, i
n _db_changed
    self.db_changed()
  File "C:\ProgrammeNeutral\GrampsGEP023\src\plugins\gramplet\Sources.py", line
226, in db_changed
    self.update()
  File "C:\ProgrammeNeutral\GrampsGEP023\src\gen\plug\_gramplet.py", line 286, i
n update
    self.update_has_data()
  File "C:\ProgrammeNeutral\GrampsGEP023\src\plugins\gramplet\Sources.py", line
234, in update_has_data
    self.set_has_data(self.get_has_data(active))
  File "C:\ProgrammeNeutral\GrampsGEP023\src\plugins\gramplet\Sources.py", line
270, in get_has_data
    if self.check_sources(person):
  File "C:\ProgrammeNeutral\GrampsGEP023\src\plugins\gramplet\Sources.py", line
123, in check_sources
    return True if obj.get_source_references() else False
AttributeError: 'Person' object has no attribute 'get_source_references'

Shall we use the bug tracker for further issues?
___

Finally a question:
Citations are more and better in focus than before (having their own user table, having filters etc.). That's great.
For my own working process I'm not sure where to place the media reference of documents now. In the past I put them all into the gallery of the source (using one source for one original document part). Now it would be more interesting to have one source having several citations including the media reference in the the gallery of the citation.
But it would be hard to change the system for older data. And if I would use the new way for new data, than I would have my media references some time in the gallery of the sources and some time in the gallery of citation.
And during genaolgic work it's often necessary to check and validate data to have access to the original document (via e. g. source -> media --> external document). It's not easy and short today and could be more difficult if using several places for the media reference.
I hope I could explain my thinking good enough. And may be I'm lost in my own thinking.

-Helge
Reply | Threaded
Open this post in threaded view
|

Re: GEPS 023

Tim Lyons
Administrator
In reply to this post by Tim Lyons
Thanks for giving it a try, it is really helpful for someone else to  
have a look.

On 1 Oct 2011, at 19:34, "Helge@Gramps" wrote:

> This version looks really nice having this new features. I'll try  
> next few
> days how to work using my available data.
> Here a first error issue:
<snip>
> AttributeError: 'Person' object has no attribute  
> 'get_source_references'
>

Thanks for the bug report - I will fix that. (Note that the change is  
not yet complete, so there will doubtless be other things that don't  
work).

> Shall we use the bug tracker for further issues?

I don't know. However, since this is not even in the roadmap for 3.4 I  
imagine that it should NOT go in the bug tracker yet. In fact. I would  
imagine that it would only go in the bug tracker once the change was  
committed to trunk, but perhaps someone else could comment.

> ___
>
> Finally a question:
> Citations are more and better in focus than before (having their own  
> user
> table, having filters etc.). That's great.
> For my own working process I'm not sure where to place the media  
> reference
> of documents now. In the past I put them all into the gallery of the  
> source
> (using one source for one original document part). Now it would be  
> more
> interesting to have one source having several citations including  
> the media
> reference in the the gallery of the citation.
> But it would be hard to change the system for older data. And if I  
> would use
> the new way for new data, than I would have my media references some  
> time in
> the gallery of the sources and some time in the gallery of citation.
> And during genaolgic work it's often necessary to check and validate  
> data to
> have access to the original document (via e. g. source -> media -->  
> external
> document). It's not easy and short today and could be more difficult  
> if
> using several places for the media reference.
> I hope I could explain my thinking good enough. And may be I'm lost  
> in my
> own thinking.
>

Im my view, the answer is simple. If there is only ONE citation for a  
given source, then it doesn't really matter whether the media is in  
the Source or in the Citation. At the moment, reports etc will be  
designed for displaying the media with the source, so that is the way  
to go.

However, the real power of the change comes when there are several  
different Citations for a single source. In that case, when the media  
is different for the different Citations, then it should be stored  
with the Citation.

> -Helge


Tim.

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: GEPS 023

John Ralls-2
In reply to this post by Tim Lyons

On Oct 1, 2011, at 7:45 AM, Tim Lyons wrote:

> I have made progress on implementing GEPS 023, many thanks to Nick  
> Hall for implementing treemodels with two object types and to Michiel  
> Nauta for help with upgrade.
>
> I would be very grateful if some other developers could have a look at  
> the user interface and try it with their data. The GEPS is at: https://gramps.svn.sourceforge.net/svnroot/gramps/branches/geps/gep-023-large-sources
>
> The principal user interface changes are to the Sources view, and the  
> introduction of the Citations view. Key objectives are that citations  
> can be shared and have Media as well as Notes.
>
> I have implemented a 'Merge citation' tool for use after database  
> upgrade, as suggested by Benny on the developers list on 29 Aug.
>
> Database upgrade, xml import and export and the various editors are  
> all done. The citation filter is mostly done, but some of the other  
> filters still need some work.
>
> GEDCOM import and export and the various reports are the main  
> outstanding things to be implemented.
>
> Would it be possible for this GEPS to be added to the roadmap for  
> Gramps 3.4?

Tim,

Sorry I didn't get to this right away... something about getting a release
out. ;-)

There was a typo in displaytabs/Makefile.am, which I fixed for you, but it won't launch:

456: ERROR: grampsgui.py: line 316: Gramps failed to start.
Traceback (most recent call last):
 File "/Volumes/RAID1/local/gtk-stable/inst/share/gramps/gui/grampsgui.py", line 292, in __startgramps
   Gramps(argparser)
 File "/Volumes/RAID1/local/gtk-stable/inst/share/gramps/gui/grampsgui.py", line 209, in __init__
   from viewmanager import ViewManager
 File "/Volumes/RAID1/local/gtk-stable/inst/share/gramps/gui/viewmanager.py", line 69, in <module>
   from gui.plug import tool
 File "/Volumes/RAID1/local/gtk-stable/inst/share/gramps/gui/plug/__init__.py", line 32, in <module>
   from _guioptions import GuiMenuOptions, make_gui_option
 File "/Volumes/RAID1/local/gtk-stable/inst/share/gramps/gui/plug/_guioptions.py", line 60, in <module>
   from Filters import GenericFilterFactory, GenericFilter, Rules
 File "/Volumes/RAID1/local/gtk-stable/inst/share/gramps/Filters/__init__.py", line 31, in <module>
   from _FilterList import FilterList
 File "/Volumes/RAID1/local/gtk-stable/inst/share/gramps/Filters/_FilterList.py", line 37, in <module>
   from Filters._FilterParser import FilterParser
 File "/Volumes/RAID1/local/gtk-stable/inst/share/gramps/Filters/_FilterParser.py", line 37, in <module>
   from Filters import Rules
 File "/Volumes/RAID1/local/gtk-stable/inst/share/gramps/Filters/Rules/__init__.py", line 42, in <module>
   from Filters.Rules import (Person, Family, Event, Source, Citation, Place,
ImportError: cannot import name Citation

Looks like something's missing?

Regards,
John Ralls



------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: GEPS 023

Tim Lyons
Administrator
In reply to this post by Tim Lyons
John,

On 2 Oct 2011, at 22:30, [hidden email]  
wrote:

>
> Tim,
>
> Sorry I didn't get to this right away... something about getting a  
> release
> out. ;-)
>
> There was a typo in displaytabs/Makefile.am, which I fixed for you,  
> but it won't launch:
>
> <snip>
> Looks like something's missing?
>

Sorry, I haven't done a build from scratch, I have just run python on  
gramps.py.

I missed out adding the new subdirectory to src/Filters/Rules/
Makefile.am. Should be:
SUBDIRS = Person Family Event Place Source MediaObject Repository Note  
Citation
(Commit 18241)


I had added the new individual files to the various Makefiles.am, but  
didn't know about needing to add the new subdirectories.

Thanks for doing that kind of build.

Tim.



------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: GEPS 023

John Ralls-2

On Oct 2, 2011, at 3:22 PM, Tim Lyons wrote:

> John,
>
> On 2 Oct 2011, at 22:30, [hidden email] wrote:
>
>>
>> Tim,
>>
>> Sorry I didn't get to this right away... something about getting a release
>> out. ;-)
>>
>> There was a typo in displaytabs/Makefile.am, which I fixed for you, but it won't launch:
>>
>> <snip>
>> Looks like something's missing?
>>
>
> Sorry, I haven't done a build from scratch, I have just run python on gramps.py.
>
> I missed out adding the new subdirectory to src/Filters/Rules/Makefile.am. Should be:
> SUBDIRS = Person Family Event Place Source MediaObject Repository Note Citation
> (Commit 18241)
>
>
> I had added the new individual files to the various Makefiles.am, but didn't know about needing to add the new subdirectories.
>
> Thanks for doing that kind of build.

Tim,

Ah. Not that used to autotools, I guess, because there were a couple more fixes (r18242).

Then it whined about
"ERROR: Python file /Volumes/RAID1/local/gtk-stable/inst/share/gramps/plugins/tool/MergeCitations.py in register file /Volumes/RAID1/local/gtk-stable/inst/share/gramps/plugins/tool/tools.gpr.py does not exist"
A bit of digging found another piece of Makefile you left out (r18243).

And then I got tripped by mergecitation.glade. I looked for it before committing 18243, but I must have gotten the capitalization wrong. Oh well. (r18244)

Now it builds and works. Yay.

The merge citation tool window should either close on its own after running a merge or the buttons should say "close" and "run" instead of "cancel" and "OK".

An observation about the citation list: By default, the first column is volume/page. Not very useful unless you know volume/page of what. Easily fixed, of course, with the configure view dialog, but I suggest that either the source title or abbreviation should be the first column by default. (I chose abbreviation.)

I like this a lot already. +1 on adding it to the roadmap for 3.4.

Hmm, do we really need a separate view of citations given that they're listed under sources?

General observation not specific to this, we need a "General" filter rule for finding targets matching key/value pair fields. ("Data" in sources and citations, "Attributes" in people, families, and events. Maybe we should use a consistent name, too?)

I'll bang on this some more as I have time.

Regards,
John Ralls


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2dcopy1
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: GEPS 023

Cedric Scott
In reply to this post by Tim Lyons
Tim,

On Saturday 01 October 2011 15:45:10 Tim Lyons wrote:

> I have made progress on implementing GEPS 023, many thanks to Nick
> Hall for implementing treemodels with two object types and to Michiel
> Nauta for help with upgrade.
>
> I would be very grateful if some other developers could have a look at
> the user interface and try it with their data. The GEPS is at:
> https://gramps.svn.sourceforge.net/svnroot/gramps/branches/geps/gep-023-la
> rge-sources
>
> The principal user interface changes are to the Sources view, and the
> introduction of the Citations view. Key objectives are that citations
> can be shared and have Media as well as Notes.
>
> I have implemented a 'Merge citation' tool for use after database
> upgrade, as suggested by Benny on the developers list on 29 Aug.
>
> Database upgrade, xml import and export and the various editors are
> all done. The citation filter is mostly done, but some of the other
> filters still need some work.
>
> GEDCOM import and export and the various reports are the main
> outstanding things to be implemented.
>
> Would it be possible for this GEPS to be added to the roadmap for
> Gramps 3.4?
>
> Tim.
>  

I tried gep-023-large-sources today and I found one problem which you may not
know about. The link from an event to people is missing.

1. Click on Event View
2. Click on an event
3. Click on References and the reference list is empty; in other versions of
gramps it has at least one entry

You can also see the problem if you add 'Main Participants' to an the Event
View; there aren't any main participants.

Do you want me to enter a bug report?

Finally, I'd like to thank you for taking on the job of implementing this - I
think that it's a great addition to gramps and I hope that it gets accepted
into trunk in the near future.

Cedric

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2dcopy1
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: GEPS 023

Tim Lyons
Administrator
In reply to this post by John Ralls-2

On 3 Oct 2011, at 01:19, John Ralls wrote:

> Ah. Not that used to autotools, I guess, because there were a couple  
> more fixes (r18242).
> <snip>
> Now it builds and works. Yay.
>

Thanks for all the work on the autotools. Glad you were able to get it  
all working in the end.

> The merge citation tool window should either close on its own after  
> running a merge or the buttons should say "close" and "run" instead  
> of "cancel" and "OK".

I will have another look at this, because I thought I had used the  
same workflow as some of the other tools. But I agree with you.

> An observation about the citation list: By default, the first column  
> is volume/page. Not very useful unless you know volume/page of what.  
> Easily fixed, of course, with the configure view dialog, but I  
> suggest that either the source title or abbreviation should be the  
> first column by default. (I chose abbreviation.)

Yes, I prefer source title, because abbreviation doesn't work well in  
the example database.

> I like this a lot already. +1 on adding it to the roadmap for 3.4.
>
> Hmm, do we really need a separate view of citations given that  
> they're listed under sources?

Good question. I had some other combinations of views before, so this  
is the result of a bit of experimentation. The only problem is that  
you can't filter citations in the source view, so the citations view  
does give you that extra power (you will be able to filter on source  
as well using a custom filter in the citations view, when I get that  
part going).

> General observation not specific to this, we need a "General" filter  
> rule for finding targets matching key/value pair fields. ("Data" in  
> sources and citations, "Attributes" in people, families, and events.  
> Maybe we should use a consistent name, too?)

I don't really understand why "Data" and "Attributes" are two  
separate, and different, things. I still have some more to do with  
filters, so I may make a single filter for the two, if it is simple to  
do.

> I'll bang on this some more as I have time.

That would be very helpful.

However, do remember that it is not complete yet - the sort of  
comments that you are giving are really helpful, but there will be  
things that aren't working yet (it may help to send me a private email  
if you spot such things, just in case I don't find it later when I  
search methodically for omitted changes).

Tim.



------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2dcopy1
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: GEPS 023

Tim Lyons
Administrator
In reply to this post by Cedric Scott

On 3 Oct 2011, at 13:09, Cedric Scott wrote:

> I tried gep-023-large-sources today and I found one problem which  
> you may not
> know about. The link from an event to people is missing.

Well spotted, there is a problem with the construction of the  
secondary index - I will fix it. Thanks very much for this.

> Do you want me to enter a bug report?

I would imagine not, as it is still in the branch directory at the  
moment, but I would be very grateful for similar omissions. (although  
I know that the change is not complete, so there are things that just  
don't work at the moment).

> Finally, I'd like to thank you for taking on the job of implementing  
> this - I
> think that it's a great addition to gramps and I hope that it gets  
> accepted
> into trunk in the near future.

Thanks for your comment.
Tim.



------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2dcopy1
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Data and Attribute fields

John Ralls-2
In reply to this post by Tim Lyons

On Oct 3, 2011, at 6:58 AM, Tim Lyons wrote:

>
> On 3 Oct 2011, at 01:19, John Ralls wrote:
>
>> General observation not specific to this, we need a "General" filter rule for finding targets matching key/value pair fields. ("Data" in sources and citations, "Attributes" in people, families, and events. Maybe we should use a consistent name, too?)
>
> I don't really understand why "Data" and "Attributes" are two separate, and different, things. I still have some more to do with filters, so I may make a single filter for the two, if it is simple to do.

Are they separate things? I thought they were the same thing with different names -- but I admit that I didn't go look at the code to see.

If they are separate, I agree that that's dumb, since the functionality seems to be the same. It should be just one class (or interface, depending on where the resulting data live).

Regards,
John Ralls


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2dcopy1
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|

Re: Data and Attribute fields

jerome
Hi,


If I remember well, Benny made a clear answer on devel list (archives ?
bug tracker ?).

There is some differences, the main one is that we cannot re-order/sort
data into a "data list". I do not know the technical word or the name
for this type of key/value issue, but it is a different behavior.


I hope this could help.

Jérôme

John Ralls a écrit :

> On Oct 3, 2011, at 6:58 AM, Tim Lyons wrote:
>
>> On 3 Oct 2011, at 01:19, John Ralls wrote:
>>
>>> General observation not specific to this, we need a "General" filter rule for finding targets matching key/value pair fields. ("Data" in sources and citations, "Attributes" in people, families, and events. Maybe we should use a consistent name, too?)
>> I don't really understand why "Data" and "Attributes" are two separate, and different, things. I still have some more to do with filters, so I may make a single filter for the two, if it is simple to do.
>
> Are they separate things? I thought they were the same thing with different names -- but I admit that I didn't go look at the code to see.
>
> If they are separate, I agree that that's dumb, since the functionality seems to be the same. It should be just one class (or interface, depending on where the resulting data live).
>
> Regards,
> John Ralls
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2dcopy1
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2dcopy1
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
12