Quantcast

GEPS023 merged into trunk

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

GEPS023 merged into trunk

Tim Lyons
Administrator
I am pleased to announce that commit 18548 merges GESP023 into trunk.


This means that databases will be upgraded to the new citation format  
when using the latest trunk. You may need to remove the ini files in  
your .gramps directory for Gramps to work properly. There are  
something like 160 files changed, so no doubt there may be some  
anomalies noticed. As always, you should backup your database before  
running the new trunk and getting your database upgraded, and should  
not use it for production work.

There are a few known issues which are documented in the wiki:
http://www.gramps-project.org/wiki/index.php?title=GEPS_023:_Storing_data_from_large_sources#Known_Issues
so before reporting any problems, please be sure to check the list of  
known issues. The issues include a few imports/exports (ProGen, FTree,  
Grdb), simple report support, some webapp modules, and tool/Check.  
However, I do not believe that these issues should prevent normal use  
of trunk.

I would like to thank Nick Hall for the enormous help and  
encouragement I have received from him, especially in implementing the  
tree model for two different primary objects which is largely his  
code. Without his help, this enhancement would not have been possible.

Also thanks for help, testing and support from Michiel Nauta (advice  
on upgrade transactions), John Ralls (fixes to some make files),  
Jerome Rapinat (fixes to translation files etc.), Benny Malengier  
(advice on upgrade), Brian Matherly (svn admin), Rob Healey, Gary  
Burton, Doug Blank and Cedric Scott and everyone else whose names I  
may have missed.




On 28 Nov 2011, at 13:43, Doug Blank wrote:

> If we are going to merge GEPS023 into trunk, it would also be
> convenient to do this sooner rather than later so that those of us
> that have time over the winter break/holiday seasons will be able to
> update related code, and test.
>
> Tim, as part of this transition, it would be quite useful to have a
> brief summary of the changes to the database, and how data is mapped
> to the new structures.
>
> -Doug

The user interface has been designed on the principle that Aunt Martha  
should not need to know or even be aware that citations have changed.  
Hence she will see a Source navigation with similar information to the  
current view. There will be a disclosure triangle, but she can ignore  
that. When she wants to add a source to a person or event, she clicks  
on the add or add existing button, and gets a similar choice of source  
as at present. (Again she can ignore the disclosure triangles). The  
editor that appears is then very similar to the existing editor. Only  
if a user particularly wants to share citations between objects do  
they need to be aware of the change. The change is also intended to be  
entirely compatible with GEDCOM.

Simple access to, and updating of, citations or sources does not  
involve very extensive changes to existing code. The database changes,  
the various editors, filters and proxies etc. have involved more  
extensive changes.

The Citation primary object corresponds pretty closely to the  
SourceRef secondary object. Data in the SourceRef is mapped to the  
Citation. A Citation has the additional possibility of having media  
objects attached (which is not possible with SourceRefs). A Citation  
points to exactly one Source (the code may assume that here is always  
a source object present).

The 'shape' of existing objects is unchanged. The change is that  
SourceBase is replaced by CitationBase.

For creating data, where you previously created a SourceRef, you now  
create a Citation object, and set_reference_handle() to point to the  
Source object. There is a difference that you then have to do a  
db.add_citation(citation, transaction). You then do an  
obj.add_citation(citation_handle) instead of an  
obj.add_source_reference(sourcerefobj) to add the SourceRef/Citation  
to the object.

An example of the changes is:
http://gramps.svn.sourceforge.net/viewvc/gramps/branches/geps/gep-023-large-sources/src/plugins/lib/libgedcom.py?r1=18405&r2=18415


As far as data access is concerned, you do obj.get_citation_list()  
instead of obj.get_source_references() to get a list of Citation  
handles/SourceRefs respectively. With the citation, you then need to  
do a db.get_citation_from_handle(citation_handle). Having got the  
object, you do get_reference_handle() to get the source object handle.

An example of the changes is:
http://gramps.svn.sourceforge.net/viewvc/gramps/branches/geps/gep-023-large-sources/src/plugins/export/ExportGedcom.py?r1=18405&r2=18408


Note 1. The names of the functions and the return values are mostly  
chosen to match the way notes are attached to objects, so  
obj.add_citation(citation_handle) corresponds with add_note - both  
taking handles as parameters, and obj.get_citation_list() corresponds  
with get_note_list() - both returning handles.

Note 2. I am aware of PEP 8 and pylint and have followed them for new  
code. However, where I have just been modifying existing code, or have  
largely copied some existing code, I have retained the style of the  
existing code, for example in terms of code layout, naming conventions  
and the number of parameters or methods in a class. This may result in  
a lowered pylint score.

[N.B. This is only a guide, no guarantee!]

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GEPS023 merged into trunk

DS Blank
Congrats, Tim! That was a huge amount of work. I also see that you
cleaned up a lot of code along the way. Bonus!

Many of the Addons will need to be updated, of course. I'm currently
working on Sqlite import/export, Django, and Data Entry Gramplet.

-Doug

On Sun, Dec 4, 2011 at 12:46 PM, Tim Lyons <[hidden email]> wrote:

> I am pleased to announce that commit 18548 merges GESP023 into trunk.
>
>
> This means that databases will be upgraded to the new citation format
> when using the latest trunk. You may need to remove the ini files in
> your .gramps directory for Gramps to work properly. There are
> something like 160 files changed, so no doubt there may be some
> anomalies noticed. As always, you should backup your database before
> running the new trunk and getting your database upgraded, and should
> not use it for production work.
>
> There are a few known issues which are documented in the wiki:
> http://www.gramps-project.org/wiki/index.php?title=GEPS_023:_Storing_data_from_large_sources#Known_Issues
> so before reporting any problems, please be sure to check the list of
> known issues. The issues include a few imports/exports (ProGen, FTree,
> Grdb), simple report support, some webapp modules, and tool/Check.
> However, I do not believe that these issues should prevent normal use
> of trunk.
>
> I would like to thank Nick Hall for the enormous help and
> encouragement I have received from him, especially in implementing the
> tree model for two different primary objects which is largely his
> code. Without his help, this enhancement would not have been possible.
>
> Also thanks for help, testing and support from Michiel Nauta (advice
> on upgrade transactions), John Ralls (fixes to some make files),
> Jerome Rapinat (fixes to translation files etc.), Benny Malengier
> (advice on upgrade), Brian Matherly (svn admin), Rob Healey, Gary
> Burton, Doug Blank and Cedric Scott and everyone else whose names I
> may have missed.
>
>
>
>
> On 28 Nov 2011, at 13:43, Doug Blank wrote:
>
>> If we are going to merge GEPS023 into trunk, it would also be
>> convenient to do this sooner rather than later so that those of us
>> that have time over the winter break/holiday seasons will be able to
>> update related code, and test.
>>
>> Tim, as part of this transition, it would be quite useful to have a
>> brief summary of the changes to the database, and how data is mapped
>> to the new structures.
>>
>> -Doug
>
> The user interface has been designed on the principle that Aunt Martha
> should not need to know or even be aware that citations have changed.
> Hence she will see a Source navigation with similar information to the
> current view. There will be a disclosure triangle, but she can ignore
> that. When she wants to add a source to a person or event, she clicks
> on the add or add existing button, and gets a similar choice of source
> as at present. (Again she can ignore the disclosure triangles). The
> editor that appears is then very similar to the existing editor. Only
> if a user particularly wants to share citations between objects do
> they need to be aware of the change. The change is also intended to be
> entirely compatible with GEDCOM.
>
> Simple access to, and updating of, citations or sources does not
> involve very extensive changes to existing code. The database changes,
> the various editors, filters and proxies etc. have involved more
> extensive changes.
>
> The Citation primary object corresponds pretty closely to the
> SourceRef secondary object. Data in the SourceRef is mapped to the
> Citation. A Citation has the additional possibility of having media
> objects attached (which is not possible with SourceRefs). A Citation
> points to exactly one Source (the code may assume that here is always
> a source object present).
>
> The 'shape' of existing objects is unchanged. The change is that
> SourceBase is replaced by CitationBase.
>
> For creating data, where you previously created a SourceRef, you now
> create a Citation object, and set_reference_handle() to point to the
> Source object. There is a difference that you then have to do a
> db.add_citation(citation, transaction). You then do an
> obj.add_citation(citation_handle) instead of an
> obj.add_source_reference(sourcerefobj) to add the SourceRef/Citation
> to the object.
>
> An example of the changes is:
> http://gramps.svn.sourceforge.net/viewvc/gramps/branches/geps/gep-023-large-sources/src/plugins/lib/libgedcom.py?r1=18405&r2=18415
>
>
> As far as data access is concerned, you do obj.get_citation_list()
> instead of obj.get_source_references() to get a list of Citation
> handles/SourceRefs respectively. With the citation, you then need to
> do a db.get_citation_from_handle(citation_handle). Having got the
> object, you do get_reference_handle() to get the source object handle.
>
> An example of the changes is:
> http://gramps.svn.sourceforge.net/viewvc/gramps/branches/geps/gep-023-large-sources/src/plugins/export/ExportGedcom.py?r1=18405&r2=18408
>
>
> Note 1. The names of the functions and the return values are mostly
> chosen to match the way notes are attached to objects, so
> obj.add_citation(citation_handle) corresponds with add_note - both
> taking handles as parameters, and obj.get_citation_list() corresponds
> with get_note_list() - both returning handles.
>
> Note 2. I am aware of PEP 8 and pylint and have followed them for new
> code. However, where I have just been modifying existing code, or have
> largely copied some existing code, I have retained the style of the
> existing code, for example in terms of code layout, naming conventions
> and the number of parameters or methods in a class. This may result in
> a lowered pylint score.
>
> [N.B. This is only a guide, no guarantee!]
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure
> contains a definitive record of customers, application performance,
> security threats, fraudulent activity, and more. Splunk takes this
> data and makes sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-novd2d
> _______________________________________________
> Gramps-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gramps-devel

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GEPS023 merged into trunk

Paul Franklin-5
In reply to this post by Tim Lyons
On 12/4/11, Tim Lyons <[hidden email]> wrote:

> I am pleased to announce that commit 18548 merges GESP023 into trunk.

> ... There are
> something like 160 files changed, so no doubt there may be some
> anomalies noticed. ...

One anomaly I noticed is that some files seem to have lost
their SVN "history" and their SVN properties.

I haven't made a rigorous check -- and don't plan to --
but I happened to notice the three "user" files (src/gen/user.py,
src/cli/user.py, and src/gui/user.py) all had lost their SVN
record of what changes had previously been made.

Likewise, none of the three now have either their svn:mime-type
(of "text/plain") or their svn:keywords (of "Id"), which I added
a month or so ago.

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GEPS023 merged into trunk

Brian Matherly
Paul,


>One anomaly I noticed is that some files seem to have lost
>their SVN "history" and their SVN properties.
>
>I haven't made a rigorous check -- and don't plan to --
>but I happened to notice the three "user" files (src/gen/user.py,
>src/cli/user.py, and src/gui/user.py) all had lost their SVN
>record of what changes had previously been made.
>
>Likewise, none of the three now have either their svn:mime-type
>(of "text/plain") or their svn:keywords (of "Id"), which I added
>a month or so ago.


That's strange. It must be a result of the database upgrade. Feel free to fix the broken properties. You might want to try making a fresh checkout first.

Regards,

~Brian


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GEPS023 merged into trunk

Tim Lyons
Administrator

On 5 Dec 2011, at 16:05, Brian Matherly wrote:

> Paul,
>
>
>> One anomaly I noticed is that some files seem to have lost
>> their SVN "history" and their SVN properties.
>>
>> I haven't made a rigorous check -- and don't plan to --
>> but I happened to notice the three "user" files (src/gen/user.py,
>> src/cli/user.py, and src/gui/user.py) all had lost their SVN
>> record of what changes had previously been made.
>>
>> Likewise, none of the three now have either their svn:mime-type
>> (of "text/plain") or their svn:keywords (of "Id"), which I added
>> a month or so ago.
>
>
> That's strange. It must be a result of the database upgrade. Feel  
> free to fix the broken properties. You might want to try making a  
> fresh checkout first.
>
> Regards,
>
> ~Brian
>


When I did the commit, most of the files were reported as 'Sending' or  
'Adding', but a few were reported as 'Replacing':

[Mon Dec 05 16:07:38]$ grep Replacing /Users/xxx/Desktop/temp.txt
     Replacing      /Users/tim/gramps/trunk/src/DateHandler/_Date_zh.py
     Replacing      /Users/tim/gramps/trunk/src/Filters/Rules/Person/
_IsRelatedWith.py
     Replacing      /Users/tim/gramps/trunk/src/Filters/Rules/
_HasSourceBase.py
     Replacing      /Users/tim/gramps/trunk/src/cli/user.py
     Replacing      /Users/tim/gramps/trunk/src/data/templates/
browse_page.html
     Replacing      /Users/tim/gramps/trunk/src/data/templates/
process_action.html
     Replacing      /Users/tim/gramps/trunk/src/data/templates/
view_report.html
     Replacing      /Users/tim/gramps/trunk/src/data/templates/
view_report_detail.html
     Replacing      /Users/tim/gramps/trunk/src/data/templates/
view_tag.html
     Replacing      /Users/tim/gramps/trunk/src/data/templates/
view_tag_detail.html
     Replacing      /Users/tim/gramps/trunk/src/data/templates/
view_tags.html
     Replacing      /Users/tim/gramps/trunk/src/gen/user.py
     Replacing      /Users/tim/gramps/trunk/src/gui/user.py
     Replacing      /Users/tim/gramps/trunk/src/gui/widgets/
basicentry.py
     Replacing      /Users/tim/gramps/trunk/src/plugins/lib/
libmetadata.py
     Replacing      /Users/tim/gramps/trunk/src/plugins/webstuff/css/
Web_Horizontal-Menus.css
     Replacing      /Users/tim/gramps/trunk/src/plugins/webstuff/css/
Web_Vertical-Menus.css
[Mon Dec 05 16:07:56]$


I have no idea why this should have been. I have checked a few of  
these files, and none of them appeared in the diff I did before the  
commit. I am not sure what I have done to these files, except for the  
Filters files, I don't think I have changed any of them. They seem to  
have come from the merge I did in 5 Nov. Not sure what else to look at.

Replacing is difficult to find in the svn documentation, but it seems  
to mean something like it is scheduled for deletion, and then inserted  
(or something like that).

Regard
Tim.

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GEPS023 merged into trunk

Paul Franklin-5
After I wrote those words ("I don't plan to") I realized
that I could still run my script which crawls over my copy
of trunk even if I didn't want to change anything.

So I append a list of the files it noticed, none of which
have either the SVN "svn:mime-type" "text/plain" property or
(more importantly) the SVN "svn:keywords" "Id" property.

I could easily add those two things to these files but what
I am hoping is that somebody who understands SVN (not me)
can (somehow) restore the "change history" of those files
(maybe manually?), which might also restore their missing
properties.  But let me know if I should just add them.

It should also be noted that my script only crawls the "src"
sub-tree and not any of the "top" files or the other
sub-trees, so nothing has been checked for any of them.

I suspect my list is a subset of yours, but haven't checked.

===========================

cli/user.py
gen/user.py
gui/user.py
gui/widgets/basicentry.py
plugins/lib/libmetadata.py
plugins/gramplet/PopulateGramplet.gpr.py
plugins/gramplet/PopulateGramplet.py
DateHandler/_Date_zh.py
Filters/Rules/Event/_HasCitation.py
Filters/Rules/Person/_IsRelatedWith.py

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GEPS023 merged into trunk

Brian Matherly
>I could easily add those two things to these files but what
>I am hoping is that somebody who understands SVN (not me)
>can (somehow) restore the "change history" of those files
>(maybe manually?), which might also restore their missing
>properties.  But let me know if I should just add them.

Well, what's done is done. I would guess that the files that were "replaced" were created in trunk after the GEPS023 branch was created. So there probably isn't much history to be lost.

Paul, If you would run your script and add the missing properties, it would be appreciated.

If anyone is inspired to dig into the nitty-gritty and figure out what happened, feel free - and please report back. But I don't think I'll lose any sleep over it.

~BM

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GEPS023 merged into trunk

Tim Lyons
Administrator

On 5 Dec 2011, at 18:07, Brian Matherly wrote:

>> I could easily add those two things to these files but what
>> I am hoping is that somebody who understands SVN (not me)
>> can (somehow) restore the "change history" of those files
>> (maybe manually?), which might also restore their missing
>> properties.  But let me know if I should just add them.
>
> Well, what's done is done. I would guess that the files that were  
> "replaced" were created in trunk after the GEPS023 branch was  
> created. So there probably isn't much history to be lost.

I think this is probably true. Certainly the webapp directory which  
accounts for several files was created after I made the branch.

>
> Paul, If you would run your script and add the missing properties,  
> it would be appreciated.
>
> If anyone is inspired to dig into the nitty-gritty and figure out  
> what happened, feel free - and please report back. But I don't think  
> I'll lose any sleep over it.
>
> ~BM


By the way, Paul, in your list, the two PopulateGramplet files, and  
the _HasCitation file are files that I introduced, and it is probably  
just my mistake that I didn't assign the svn properties (probably  
nothing to do with the loss of history).


------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: GEPS023 merged into trunk

Paul Franklin-5
In reply to this post by Brian Matherly
On 12/5/11, Brian Matherly <[hidden email]> wrote:

> Paul, If you would run your script and add the missing properties, it would
> be appreciated.

This has been done.

Also, I slightly modified my script so it noticed Makefile.am
files (but I only ever run it on the "src" sub-tree, nowhere else).
So they too got an "Id" line if they had none, and properties.

------------------------------------------------------------------------------
Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/
_______________________________________________
Gramps-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-devel
Loading...