Json Export

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Json Export

StoltHD
When I export a Json from Gramps 5.0.2 database that contain only a place hierarchy, I get an error in the json when trying to open it in other programs i.e. vs code, excel, notepad++

The error is that its not a valid json because of multiple json roots...

I also run it on the curiousconcept web page and the validator there come with the same error...
 Invalid JSON (RFC 4627) 
 
  • Error:Multiple JSON root elements[Code 22, Structure 110]
  • Error:Multiple JSON root elements[Code 22, Structure 276]
  • Error:Multiple JSON root elements[Code 22, Structure 442]
  • Error:Multiple JSON root elements[Code 22, Structure 608]

    the json from Gramps, first 4 elements:
    {
    "_class": "Place",
    "handle": "dd5f00ca7166cf6cee27202ce5b",
    "change": 1527336664,
    "private": false,
    "tag_list": [],
    "gramps_id": "NO",
    "citation_list": [],
    "note_list": [],
    "media_list": [],
    "urls": [],
    "long": "",
    "lat": "",
    "title": "",
    "name": {
    "_class": "PlaceName",
    "date": null,
    "value": "Norge",
    "lang": ""
    },
    "alt_names": [],
    "placeref_list": [],
    "place_type": {
    "_class": "PlaceType",
    "string": "Land"
    },
    "code": "",
    "alt_loc": []
    } {
    "_class": "Place",
    "handle": "dd5f00ca7701458200d36ce94a4",
    "change": 1521250655,
    "private": false,
    "tag_list": [],
    "gramps_id": "NO-KRK-01",
    "citation_list": [],
    "note_list": [],
    "media_list": [],
    "urls": [],
    "long": "",
    "lat": "",
    "title": "",
    "name": {
    "_class": "PlaceName",
    "date": null,
    "value": "Kristiania stift",
    "lang": ""
    },
    "alt_names": [],
    "placeref_list": [{
    "_class": "PlaceRef",
    "ref": "dd5f00ca7166cf6cee27202ce5b",
    "date": {
    "_class": "Date",
    "format": null,
    "calendar": 0,
    "modifier": 1,
    "quality": 0,
    "dateval": [
    0,
    0,
    1918,
    false
    ],
    "text": "Before 1918",
    "sortval": 2421595,
    "newyear": 0
    }
    }],
    "place_type": {
    "_class": "PlaceType",
    "string": "Stift"
    },
    "code": "",
    "alt_loc": []
    } {
    "_class": "Place",
    "handle": "dd5f00ca7843cbe4d53f01f69b0",
    "change": 1521250655,
    "private": false,
    "tag_list": [],
    "gramps_id": "NO-KRK-01.01",
    "citation_list": [],
    "note_list": [],
    "media_list": [],
    "urls": [],
    "long": "",
    "lat": "",
    "title": "",
    "name": {
    "_class": "PlaceName",
    "date": null,
    "value": "Mellem Borgesyssel provsti",
    "lang": ""
    },
    "alt_names": [],
    "placeref_list": [{
    "_class": "PlaceRef",
    "ref": "dd5f00ca7701458200d36ce94a4",
    "date": {
    "_class": "Date",
    "format": null,
    "calendar": 0,
    "modifier": 1,
    "quality": 0,
    "dateval": [
    0,
    0,
    1918,
    false
    ],
    "text": "Before 1918",
    "sortval": 2421595,
    "newyear": 0
    }
    }],
    "place_type": {
    "_class": "PlaceType",
    "string": "Provsti"
    },
    "code": "",
    "alt_loc": []
    } {
    "_class": "Place",
    "handle": "dd5f00ca78e1e9d0f11090baab4",
    "change": 1521250655,
    "private": false,
    "tag_list": [],
    "gramps_id": "NO-KRK-01.01.01",
    "citation_list": [],
    "note_list": [],
    "media_list": [],
    "urls": [],
    "long": "",
    "lat": "",
    "title": "",
    "name": {
    "_class": "PlaceName",
    "date": null,
    "value": "Trøgstad Prestegjeld",
    "lang": ""
    },
    "alt_names": [],
    "placeref_list": [{
    "_class": "PlaceRef",
    "ref": "dd5f00ca7843cbe4d53f01f69b0",
    "date": {
    "_class": "Date",
    "format": null,
    "calendar": 0,
    "modifier": 1,
    "quality": 0,
    "dateval": [
    0,
    0,
    1918,
    false
    ],
    "text": "Before 1918",
    "sortval": 2421595,
    "newyear": 0
    }
    }],
    "place_type": {
    "_class": "PlaceType",
    "string": "Prestegjeld"
    },
    "code": "",
    "alt_loc": []
    }
  • Im not a developer, so if anyone can give me an easy fix, just so that I can open this file in other software than Gramps, it would be nice... and maybe someone could fix this problem in the export gramplet?

    J


  • _______________________________________________
    Gramps-devel mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/gramps-devel
    Reply | Threaded
    Open this post in threaded view
    |

    Re: Json Export

    Nick Hall
    On 14/08/2019 18:53, StoltHD wrote:
    > Im not a developer, so if anyone can give me an easy fix, just so that
    > I can open this file in other software than Gramps, it would be
    > nice... and maybe someone could fix this problem in the export gramplet?

    The JSON export contains one object per line.  Each line should be valid
    JSON.  The JSON import reads one line at a time.

    Nick.




    _______________________________________________
    Gramps-devel mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/gramps-devel
    Reply | Threaded
    Open this post in threaded view
    |

    Re: Json Export

    prculley
    Expanding on Nick's answer, the Gramps JSON export addon does not have a top level JSON object for the tree.  It uses JSON for each of Gramps primary and secondary objects.  And that is what is stored in the file, just concatenated together.

    It might be more 'valid' to store each object in its own file, but that would be a very large number of files.

    I think the JSON addons were done for development and testing purposes, and were not meant for general usage.  If you could explain what you hoped to get from that export, perhaps we could assist further.

    Paul C.

    On Wed, Aug 14, 2019 at 1:40 PM Nick Hall <[hidden email]> wrote:
    On 14/08/2019 18:53, StoltHD wrote:
    > Im not a developer, so if anyone can give me an easy fix, just so that
    > I can open this file in other software than Gramps, it would be
    > nice... and maybe someone could fix this problem in the export gramplet?

    The JSON export contains one object per line.  Each line should be valid
    JSON.  The JSON import reads one line at a time.

    Nick.




    _______________________________________________
    Gramps-devel mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/gramps-devel


    _______________________________________________
    Gramps-devel mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/gramps-devel
    Reply | Threaded
    Open this post in threaded view
    |

    Re: Json Export

    Jonathan Biegert
    Am 14.08.19 um 22:12 schrieb Paul Culley:
    > Expanding on Nick's answer, the Gramps JSON export addon does not have a
    > top level JSON object for the tree.  It uses JSON for each of Gramps
    > primary and secondary objects.  And that is what is stored in the file,
    > just concatenated together.

    ... and this is not valid JSON (see json.org or
    <https://tools.ietf.org/html/rfc8259#section-2>, which apparently breaks
    syntax highlighting and prevents usage of standard validation tools.

    Since an array is a valid json root object, simply joining the
    primary/secondary gramps objects with a comma and enclosing the file in
    [] should be enough to fix this.

    Patch for JSONExport below.
    JSONImport is not so straightforward: I'd replace its reading code by a
    single `json.load(stream)`, but therefore I'd need access to
    `gramps.gen.lib.serialize.__object_hook`.

      azrdev

    --
    diff --git a/JSON/JSONExport.py b/JSON/JSONExport.py
    index 3ec0c56e..32ea9a52 100644
    --- a/JSON/JSONExport.py
    +++ b/JSON/JSONExport.py
    @@ -33,6 +33,7 @@ def exportData(db, filename,
             callback = lambda percent: None # dummy

         with OpenFileOrStdout(filename, encoding="utf-8") as fp:
    +        fp.write('[')

             total = (db.get_number_of_notes() +
                      db.get_number_of_people() +
    @@ -126,10 +127,12 @@ def exportData(db, filename,
                 count += 1
                 callback(100 * count/total)

    +        fp.write(']')
    +
         return True

     def write_line(fp, obj):
         """
         Write a single object to the file.
         """
    -    fp.write(to_json(obj) + "\n")
    +    fp.write(to_json(obj) + ",\n")


    _______________________________________________
    Gramps-devel mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/gramps-devel
    Reply | Threaded
    Open this post in threaded view
    |

    Re: Json Export

    Nick Hall
    On 14/08/2019 22:35, [hidden email] wrote:
    > ... and this is not valid JSON

    The file as a whole was never intended to be valid JSON. However, each
    individual line is valid JSON and represents a single primary object.

    Then the importer only needs to read one line/object at a time.

    Nick.




    _______________________________________________
    Gramps-devel mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/gramps-devel
    Reply | Threaded
    Open this post in threaded view
    |

    Re: Json Export

    StoltHD
    Nick, 
    If the json wasn't intended to use for external usage, that's good enough for me...

    azrdev 
    Thanks for the solution of making it a array, and import that way do work with Excel and vs code is reading it correctly...

    The  DTD XML schema is no longer supported by any Microsoft product that I use, and converting it to some supported format I gave up...

    I'm creating a place database of different types of Norwegian districts, farm names and places, street names and buildings for the period between approx 1756 and 1923, in addition to adding all Public Repositories (Archives) and the most common sources for that time, ...but the csv export/import makes double entries for every import I do, no handler to control the entities, and it doesnt seem that Gramps looks at the codes when importing for the second time... that resulting in that I need to delete the Gramps database or run a manual merge every time I run a csv import...
    There are some 100k + place names with time based alternative names and hierarchy relations, so manual merge is "problematic"... I was thinking that I could use the json import/export for this since the Gramps Handlers was in that file and I hoped Gramps then only would import the additional new entities from that file... I also hoped that I could generate handlers In a "readable" format with some prefix and a serial number as identifiers... but this was my "next project"

    I have got my  answer, I can't use json nor xml (lack of support from Microsoft) from Gramps for this project as long as I use Access, Excel or other Microsoft tools,  It might be I can do this via Excel by creating an array in the json and just edit the json file before importing to Gramps, I will test out that... if that doesn't work, I just need to figure out some other way to do it...  no stress.

    Just one question... wouldn't it be okay to create a json export from Gramps that actually could be used by external software and research tools?
    I'm looking at multiple solutions like Arches, Vistorian and Histograph, and it would be great to be able to reuse the data from Gramps in other research tools... And yes, I do know this is not the scope of Gramps, but it would be great if I could export one file instead of exporting to csv, then edit that file to multiple csv or when supported make multiple selections every time I shall exchange data...

    And yes, I will set up a mongodb enviroment for Gramps, and that will give me some easier ways to connect different systems, but at this point, that is way over my limited knowledge... I am looking into Python, R and some other languages, but I'm actually more interested in the research than in programming...

    I have two other projects also that I hoped to use the json format in, but I will not use it as long as it is not a valid json format...

    j


    _______________________________________________
    Gramps-devel mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/gramps-devel
    Reply | Threaded
    Open this post in threaded view
    |

    Re: Json Export

    prculley
    The csv import should NOT create double places if the 'Place' column includes valid Gramps IDs.
    There is a small bug in the csv importer that creates additional place enclosures if the same data is imported more than once.  That can be corrected with the following patch:
    --- a/gramps/plugins/importer/importcsv.py
    +++ b/gramps/plugins/importer/importcsv.py
    @@ -800,13 +800,17 @@ class CSVParser:
             if place_enclosed_by_id is not None:
                 place_enclosed_by = self.lookup("place", place_enclosed_by_id)
                 if place_enclosed_by is None:
    -                raise Exception("cannot enclose %s in %s as it doesn't exist" % (place.gramps_id, place_enclosed_by_id))
    -            if not place_enclosed_by.handle in place.placeref_list:
    +                raise Exception("cannot enclose %s in %s as it doesn't exist" %
    +                                (place.gramps_id, place_enclosed_by_id))
    +            for placeref in place.placeref_list:
    +                if place_enclosed_by.handle == placeref.ref:
    +                    break
    +            else:
                     placeref = PlaceRef()
                     placeref.ref = place_enclosed_by.handle
    -                if place_date:
    -                    placeref.date = _dp.parse(place_date)
                     place.placeref_list.append(placeref)
    +            if place_date:
    +                placeref.date = _dp.parse(place_date)
             #########################################################
             self.db.commit_place(place, self.trans)


    Try exporting to csv and note that the 'Place' column has the Gramps ID in brackets.  With this patch, you can import the same file multiple times with no additional data being created.  If you make changes in other columns the new data will overwrite the old.

    If you want to add places, the 'Place' column containing the Gramps ID must have a previously unused ID.
    Below is a tiny example of a csv exported from Gramps with four places (I used the Place Cleanup tool to load these based on the GeoNames data).  It can be imported multiple times without creating copies.

    Place Title Name Type Latitude Longitude Code Enclosed_by
    [GEO6252001] United States United States of America Country 39.76 -98.5

    [GEO4736286] Texas, United States Texas State 31.25044 -99.25061
    [GEO6252001]
    [GEO4696376] Harris, Texas, United States Harris County County 29.85728 -95.39234
    [GEO4736286]
    [GEO4737094]
    Tomball City 30.09716 -95.61605 77375 77377 [GEO4696376]

    Also note that the GetGOV addon can be used to import places from their database with all the enclosures and whatever historical names and enclosures it has.

    Paul C.

    On Wed, Aug 14, 2019 at 6:59 PM StoltHD <[hidden email]> wrote:
    Nick, 
    If the json wasn't intended to use for external usage, that's good enough for me...

    azrdev 
    Thanks for the solution of making it a array, and import that way do work with Excel and vs code is reading it correctly...

    The  DTD XML schema is no longer supported by any Microsoft product that I use, and converting it to some supported format I gave up...

    I'm creating a place database of different types of Norwegian districts, farm names and places, street names and buildings for the period between approx 1756 and 1923, in addition to adding all Public Repositories (Archives) and the most common sources for that time, ...but the csv export/import makes double entries for every import I do, no handler to control the entities, and it doesnt seem that Gramps looks at the codes when importing for the second time... that resulting in that I need to delete the Gramps database or run a manual merge every time I run a csv import...
    There are some 100k + place names with time based alternative names and hierarchy relations, so manual merge is "problematic"... I was thinking that I could use the json import/export for this since the Gramps Handlers was in that file and I hoped Gramps then only would import the additional new entities from that file... I also hoped that I could generate handlers In a "readable" format with some prefix and a serial number as identifiers... but this was my "next project"

    I have got my  answer, I can't use json nor xml (lack of support from Microsoft) from Gramps for this project as long as I use Access, Excel or other Microsoft tools,  It might be I can do this via Excel by creating an array in the json and just edit the json file before importing to Gramps, I will test out that... if that doesn't work, I just need to figure out some other way to do it...  no stress.

    Just one question... wouldn't it be okay to create a json export from Gramps that actually could be used by external software and research tools?
    I'm looking at multiple solutions like Arches, Vistorian and Histograph, and it would be great to be able to reuse the data from Gramps in other research tools... And yes, I do know this is not the scope of Gramps, but it would be great if I could export one file instead of exporting to csv, then edit that file to multiple csv or when supported make multiple selections every time I shall exchange data...

    And yes, I will set up a mongodb enviroment for Gramps, and that will give me some easier ways to connect different systems, but at this point, that is way over my limited knowledge... I am looking into Python, R and some other languages, but I'm actually more interested in the research than in programming...

    I have two other projects also that I hoped to use the json format in, but I will not use it as long as it is not a valid json format...

    j
    _______________________________________________
    Gramps-devel mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/gramps-devel


    _______________________________________________
    Gramps-devel mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/gramps-devel
    Reply | Threaded
    Open this post in threaded view
    |

    Re: Json Export

    StoltHD
    Thanks Paul,,
    I will test this out as soon as I can,...

    Will you be able to add this fix to a later version of Gramps? maybe 5.1 or something?  

    My Place (not as i wrote "code", just an type error) column and Enclosed_by have the unik ID in square brackets, the csv works flawless for the first run, but not for the following imports, so I will absolute make this change if it can not be implemented in a near future release

    jaran

    tor. 15. aug. 2019 kl. 18:27 skrev Paul Culley <[hidden email]>:
    The csv import should NOT create double places if the 'Place' column includes valid Gramps IDs.
    There is a small bug in the csv importer that creates additional place enclosures if the same data is imported more than once.  That can be corrected with the following patch:
    --- a/gramps/plugins/importer/importcsv.py
    +++ b/gramps/plugins/importer/importcsv.py
    @@ -800,13 +800,17 @@ class CSVParser:
             if place_enclosed_by_id is not None:
                 place_enclosed_by = self.lookup("place", place_enclosed_by_id)
                 if place_enclosed_by is None:
    -                raise Exception("cannot enclose %s in %s as it doesn't exist" % (place.gramps_id, place_enclosed_by_id))
    -            if not place_enclosed_by.handle in place.placeref_list:
    +                raise Exception("cannot enclose %s in %s as it doesn't exist" %
    +                                (place.gramps_id, place_enclosed_by_id))
    +            for placeref in place.placeref_list:
    +                if place_enclosed_by.handle == placeref.ref:
    +                    break
    +            else:
                     placeref = PlaceRef()
                     placeref.ref = place_enclosed_by.handle
    -                if place_date:
    -                    placeref.date = _dp.parse(place_date)
                     place.placeref_list.append(placeref)
    +            if place_date:
    +                placeref.date = _dp.parse(place_date)
             #########################################################
             self.db.commit_place(place, self.trans)


    Try exporting to csv and note that the 'Place' column has the Gramps ID in brackets.  With this patch, you can import the same file multiple times with no additional data being created.  If you make changes in other columns the new data will overwrite the old.

    If you want to add places, the 'Place' column containing the Gramps ID must have a previously unused ID.
    Below is a tiny example of a csv exported from Gramps with four places (I used the Place Cleanup tool to load these based on the GeoNames data).  It can be imported multiple times without creating copies.

    Place Title Name Type Latitude Longitude Code Enclosed_by
    [GEO6252001] United States United States of America Country 39.76 -98.5

    [GEO4736286] Texas, United States Texas State 31.25044 -99.25061
    [GEO6252001]
    [GEO4696376] Harris, Texas, United States Harris County County 29.85728 -95.39234
    [GEO4736286]
    [GEO4737094]
    Tomball City 30.09716 -95.61605 77375 77377 [GEO4696376]

    Also note that the GetGOV addon can be used to import places from their database with all the enclosures and whatever historical names and enclosures it has.

    Paul C.

    On Wed, Aug 14, 2019 at 6:59 PM StoltHD <[hidden email]> wrote:
    Nick, 
    If the json wasn't intended to use for external usage, that's good enough for me...

    azrdev 
    Thanks for the solution of making it a array, and import that way do work with Excel and vs code is reading it correctly...

    The  DTD XML schema is no longer supported by any Microsoft product that I use, and converting it to some supported format I gave up...

    I'm creating a place database of different types of Norwegian districts, farm names and places, street names and buildings for the period between approx 1756 and 1923, in addition to adding all Public Repositories (Archives) and the most common sources for that time, ...but the csv export/import makes double entries for every import I do, no handler to control the entities, and it doesnt seem that Gramps looks at the codes when importing for the second time... that resulting in that I need to delete the Gramps database or run a manual merge every time I run a csv import...
    There are some 100k + place names with time based alternative names and hierarchy relations, so manual merge is "problematic"... I was thinking that I could use the json import/export for this since the Gramps Handlers was in that file and I hoped Gramps then only would import the additional new entities from that file... I also hoped that I could generate handlers In a "readable" format with some prefix and a serial number as identifiers... but this was my "next project"

    I have got my  answer, I can't use json nor xml (lack of support from Microsoft) from Gramps for this project as long as I use Access, Excel or other Microsoft tools,  It might be I can do this via Excel by creating an array in the json and just edit the json file before importing to Gramps, I will test out that... if that doesn't work, I just need to figure out some other way to do it...  no stress.

    Just one question... wouldn't it be okay to create a json export from Gramps that actually could be used by external software and research tools?
    I'm looking at multiple solutions like Arches, Vistorian and Histograph, and it would be great to be able to reuse the data from Gramps in other research tools... And yes, I do know this is not the scope of Gramps, but it would be great if I could export one file instead of exporting to csv, then edit that file to multiple csv or when supported make multiple selections every time I shall exchange data...

    And yes, I will set up a mongodb enviroment for Gramps, and that will give me some easier ways to connect different systems, but at this point, that is way over my limited knowledge... I am looking into Python, R and some other languages, but I'm actually more interested in the research than in programming...

    I have two other projects also that I hoped to use the json format in, but I will not use it as long as it is not a valid json format...

    j
    _______________________________________________
    Gramps-devel mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/gramps-devel


    _______________________________________________
    Gramps-devel mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/gramps-devel