ancestry import media list

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

ancestry import media list

Joel Ford

I recently accomplished a successful gedcom import from ancestry.com to gramps. One issue was problematic, however, that being the media. All media entries and references imported accuirately, however, every one of them links to an ancesty.com URL which is, apparently, not accessible by gramps. Correct this requires that I edit each media item by:

  1. copying the URL in the gramps media reference entry and pasting it into a browser.
  2. clicking on the "View original" link in the ancestry.com media browser window.
  3. saving the full-size image to the hard drive.
  4. editing the gramps media entry to link to the saved image.

This works like a charm, but... as you can imagine, it is a tedious process, to say the least. I have found that the ancestry URL in each media entry is only slightly different from the "View original" media link on the ancestry page. Theoretically, one could edit a long list of imported media entries, replacing the existing URL with the "original image" URL and then download all the images at once using, for example, wget .

The problem is, I can see no way to access the Gramps media reference list. By this, I mean the list that is shown when one clicks on "Media" in the main left hand column of gramps. Is there any way to do so or is there some other means of solving this issue? Overall, the importation was excellent, with this one exception. 


Thanks in advance!



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

Ron Johnson
One possible solution is to search/replace the slightly different URLs in the GEDCOM before importation.  This would only work, though, if the slight difference between then URLs is the same.

On 03/16/2018 09:47 AM, Joel Ford wrote:

I recently accomplished a successful gedcom import from ancestry.com to gramps. One issue was problematic, however, that being the media. All media entries and references imported accuirately, however, every one of them links to an ancesty.com URL which is, apparently, not accessible by gramps. Correct this requires that I edit each media item by:

  1. copying the URL in the gramps media reference entry and pasting it into a browser.
  2. clicking on the "View original" link in the ancestry.com media browser window.
  3. saving the full-size image to the hard drive.
  4. editing the gramps media entry to link to the saved image.

This works like a charm, but... as you can imagine, it is a tedious process, to say the least. I have found that the ancestry URL in each media entry is only slightly different from the "View original" media link on the ancestry page. Theoretically, one could edit a long list of imported media entries, replacing the existing URL with the "original image" URL and then download all the images at once using, for example, wget .

The problem is, I can see no way to access the Gramps media reference list. By this, I mean the list that is shown when one clicks on "Media" in the main left hand column of gramps. Is there any way to do so or is there some other means of solving this issue? Overall, the importation was excellent, with this one exception.


--
Angular momentum makes the world go 'round.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

Joel Ford

Yes, that did occur to me, as the gedcom is just a text file, With a little regex magic, I could find/replace the urls with the new url. Then, however, I would need to extract all the media URLs in order to create a bash script to download the media. I guess there really is no other way. In theory, this should be an easy script fix for gramps, although I suspect ancestry.com is a moving target...


On 03/16/2018 11:36 AM, Ron Johnson wrote:
One possible solution is to search/replace the slightly different URLs in the GEDCOM before importation.  This would only work, though, if the slight difference between then URLs is the same.
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

paul womack
Joel Ford wrote:
> Yes, that did occur to me, as the gedcom is just a text file, With a little regex magic, I could find/replace the urls with the new url. Then, however, I would need to extract all the media URLs in order to create a bash script to download the media. I guess there really is no other way. In theory, this should be an easy script fix for gramps, although I suspect ancestry.com is a moving target...

I also suspect ancestry has assorted protection measures against being scraped like this.

  BugBear


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

Joel Ford
In reply to this post by Ron Johnson

Hmmmm, might be worth looking at.  I use linux, so editing and regex is an easy task. The addon solution is interesting, although I'm hesitant to try that first. Also, hadn't thought about the ancestry authorization issue, although I think that can be dealt with using wget in linux. Okay, nothing left to do but give a good hack and see what happens...

I'll try to post back if I finda solution...


On 03/16/2018 11:53 AM, Paul Culley wrote:

I see this as a two part process.  The first part is getting the URLs to be the ‘right’ ones for direct access to the files.  You might be able to use the Gramps Tools/Media Manager/replace substrings function to modify the URLs, it depends on exactly how you need to change things.  If that doesn’t work, you may have to either edit the original Gedcom, or a Gramps XML (export/edit/import into a new tree) to fix up the URLs using your favorite editing tools.  Notepad++ on Windows offers regular expression functions to make this possibly easier as well as a good replace function.

 

The second part would be to download the files.  Again there is a Gramps Addon called the “Download Media files from internet” you can install to actually download the files.  If you have trouble with these steps and cannot get things working, you can send me your Gramps XML or at least a subset of it containing a few of the media items with the original ancestry URLs and what you think the correct URLs should look like.

 

I have not tried this myself, one area of potential concern might be that ancestry would need your login credentials to download the files, and as the browser is not used to do the work, the Gramps addon might not have a way to supply the credentials.

 

Paul Culley.

 


On 03/16/2018 09:47 AM, Joel Ford wrote:

I recently accomplished a successful gedcom import from ancestry.com to gramps. One issue was problematic, however, that being the media. All media entries and references imported accuirately, however, every one of them links to an ancesty.com URL which is, apparently, not accessible by gramps. Correct this requires that I edit each media item by:

  1. copying the URL in the gramps media reference entry and pasting it into a browser.
  2. clicking on the "View original" link in the ancestry.com media browser window.
  3. saving the full-size image to the hard drive.
  4. editing the gramps media entry to link to the saved image.

This works like a charm, but... as you can imagine, it is a tedious process, to say the least. I have found that the ancestry URL in each media entry is only slightly different from the "View original" media link on the ancestry page. Theoretically, one could edit a long list of imported media entries, replacing the existing URL with the "original image" URL and then download all the images at once using, for example, wget .

The problem is, I can see no way to access the Gramps media reference list. By this, I mean the list that is shown when one clicks on "Media" in the main left hand column of gramps. Is there any way to do so or is there some other means of solving this issue? Overall, the importation was excellent, with this one exception.

 

--
Angular momentum makes the world go 'round.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

Joel Ford
In reply to this post by Ron Johnson

It's appears that the media manager plugin string replacement function cannot manage regex of any kind. The URL replacement requires a change both before and after  a  media ID# as follows:

ORIGINAL URL: http://trees.ancestry.com/rd?f=image&guid=d5ec9399-2497-431e-a38e-09b6d3894a74&tid=109252978&pid=111

ACTUAL IMAGE URL: https://mediasvc.ancestry.com/v2/image/namespaces/1093/media/d5ec9399-2497-431e-a38e-09b6d3894a74?client=TreesUI

So, the replacement of the first part is straight forward and works like a charm using the string function in media manager plugin. However, I see no way to accomplish the replacement of the second part using the media manager string function as the &pid entries are unique to each media, so regex is definitely needed. 

So, it backs to the gedcom text regex replacement...


On 03/16/2018 11:53 AM, Paul Culley wrote:

I see this as a two part process.  The first part is getting the URLs to be the ‘right’ ones for direct access to the files.  You might be able to use the Gramps Tools/Media Manager/replace substrings function to modify the URLs, it depends on exactly how you need to change things.  If that doesn’t work, you may have to either edit the original Gedcom, or a Gramps XML (export/edit/import into a new tree) to fix up the URLs using your favorite editing tools.  Notepad++ on Windows offers regular expression functions to make this possibly easier as well as a good replace function.

 

The second part would be to download the files.  Again there is a Gramps Addon called the “Download Media files from internet” you can install to actually download the files.  If you have trouble with these steps and cannot get things working, you can send me your Gramps XML or at least a subset of it containing a few of the media items with the original ancestry URLs and what you think the correct URLs should look like.

 

I have not tried this myself, one area of potential concern might be that ancestry would need your login credentials to download the files, and as the browser is not used to do the work, the Gramps addon might not have a way to supply the credentials.

 

Paul Culley.

 




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

Joel Ford
In reply to this post by Joel Ford

Script is a no go. too many errors, feels iffy, but thanks!


On 03/16/2018 12:31 PM, Paul Culley wrote:

Your question got me going on a bit of research.
... 

Paul C.

 




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

GRAMPS - User mailing list
In reply to this post by Joel Ford

You could look at using rootsmagic to download your ancestry tree. You don't say what operating system you use, but rootsmagic works on windows and (I believe) macs. There is a free trial version of rootsmagic that will download all media connected to your ancestry tree. Then you could export a gedcom from rootsmagic and import it to gramps.

Mary
PS apologies if I have sent two copies of this.

On 16/03/2018 14:47, Joel Ford wrote:

I recently accomplished a successful gedcom import from ancestry.com to gramps. One issue was problematic, however, that being the media. All media entries and references imported accuirately, however, every one of them links to an ancesty.com URL which is, apparently, not accessible by gramps. Correct this requires that I edit each media item by:

  1. copying the URL in the gramps media reference entry and pasting it into a browser.
  2. clicking on the "View original" link in the ancestry.com media browser window.
  3. saving the full-size image to the hard drive.
  4. editing the gramps media entry to link to the saved image.

This works like a charm, but... as you can imagine, it is a tedious process, to say the least. I have found that the ancestry URL in each media entry is only slightly different from the "View original" media link on the ancestry page. Theoretically, one could edit a long list of imported media entries, replacing the existing URL with the "original image" URL and then download all the images at once using, for example, wget .

The problem is, I can see no way to access the Gramps media reference list. By this, I mean the list that is shown when one clicks on "Media" in the main left hand column of gramps. Is there any way to do so or is there some other means of solving this issue? Overall, the importation was excellent, with this one exception. 


Thanks in advance!




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
roy
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

roy
In reply to this post by Joel Ford
I have an import from Ancestry with the same problem. I figured I was just out of luck at ever transferring the images. I don't even understand half of what you're talking about. I will be watching this thread for a possible solution that a non-guru can understand

Roy Jackson
Naples, FL



Sent from my Verizon, Samsung Galaxy smartphone

-------- Original message --------
From: Joel Ford <[hidden email]>
Date: 3/16/18 12:05 PM (GMT-05:00)
To: GRAMPS user list <[hidden email]>
Subject: Re: [Gramps-users] ancestry import media list

Hmmmm, might be worth looking at.  I use linux, so editing and regex is an easy task. The addon solution is interesting, although I'm hesitant to try that first. Also, hadn't thought about the ancestry authorization issue, although I think that can be dealt with using wget in linux. Okay, nothing left to do but give a good hack and see what happens...

I'll try to post back if I finda solution...


On 03/16/2018 11:53 AM, Paul Culley wrote:

I see this as a two part process.  The first part is getting the URLs to be the ‘right’ ones for direct access to the files.  You might be able to use the Gramps Tools/Media Manager/replace substrings function to modify the URLs, it depends on exactly how you need to change things.  If that doesn’t work, you may have to either edit the original Gedcom, or a Gramps XML (export/edit/import into a new tree) to fix up the URLs using your favorite editing tools.  Notepad++ on Windows offers regular expression functions to make this possibly easier as well as a good replace function.

 

The second part would be to download the files.  Again there is a Gramps Addon called the “Download Media files from internet” you can install to actually download the files.  If you have trouble with these steps and cannot get things working, you can send me your Gramps XML or at least a subset of it containing a few of the media items with the original ancestry URLs and what you think the correct URLs should look like.

 

I have not tried this myself, one area of potential concern might be that ancestry would need your login credentials to download the files, and as the browser is not used to do the work, the Gramps addon might not have a way to supply the credentials.

 

Paul Culley.

 


On 03/16/2018 09:47 AM, Joel Ford wrote:

I recently accomplished a successful gedcom import from ancestry.com to gramps. One issue was problematic, however, that being the media. All media entries and references imported accuirately, however, every one of them links to an ancesty.com URL which is, apparently, not accessible by gramps. Correct this requires that I edit each media item by:

  1. copying the URL in the gramps media reference entry and pasting it into a browser.
  2. clicking on the "View original" link in the ancestry.com media browser window.
  3. saving the full-size image to the hard drive.
  4. editing the gramps media entry to link to the saved image.

This works like a charm, but... as you can imagine, it is a tedious process, to say the least. I have found that the ancestry URL in each media entry is only slightly different from the "View original" media link on the ancestry page. Theoretically, one could edit a long list of imported media entries, replacing the existing URL with the "original image" URL and then download all the images at once using, for example, wget .

The problem is, I can see no way to access the Gramps media reference list. By this, I mean the list that is shown when one clicks on "Media" in the main left hand column of gramps. Is there any way to do so or is there some other means of solving this issue? Overall, the importation was excellent, with this one exception.

 

--
Angular momentum makes the world go 'round.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

Joel Ford
In reply to this post by GRAMPS - User mailing list
Thanks Mary. I do appreciate your suggestion, but downloading and
installing a second genealogy applicaiton in order to import an ancestry
gedcom into my already existing genealogy applications just seems
unreasonable to me. ESPECIALLY if I one plans to continue to use
ancestry. If rootsmagic can do it, why can't gramps?

Joel


On 03/16/2018 01:25 PM, e-mail mary_lynch via Gramps-users wrote:

>
> You could look at using rootsmagic to download your ancestry tree. You
> don't say what operating system you use, but rootsmagic works on
> windows and (I believe) macs. There is a free trial version of
> rootsmagic that will download all media connected to your ancestry
> tree. Then you could export a gedcom from rootsmagic and import it to
> gramps.
>
> Mary
> PS apologies if I have sent two copies of this.
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

Joel Ford
In reply to this post by Joel Ford

Okay, so here's an update on the adventure of importing media files from ancestry to gramps after importing an acestry gedcom... Suffice to say that, at the moment, it's an imperfect solution. That said, it gets me a lot farther down the road than I was before. I have a lot of media attachments in ancestry and that makes this both important and arduous.

As stated before, I use Linux, so you windoze user will need to adapt to your environement. that said, everything I've done here can be replicated in a windows environment. Using a variety of regex "Find and replace" functions, some linux wizardry via commands with sed and wget, I am able to:

  1. edit the downloaded gedcom to correct the media URLS.
  2. Extract the lines of media URLS from the gedcom file into a text file.
  3. Use wget to download from ancestry the URLS in the said text file.
  4. Use a couple of linux file commands to clean up the file names and then move them to the gramps media folder.
  5. Run the gramps media manager string plugin to replace the media URL strings with a directory path.

        !!VOILA!! the media appears. However, it not yet time for the happy dance...

I had no issues with ancestry authentication EXCEPT when trying to download media files that are attached to my tree FROM another ancestry user. As you know, ancestry allows you to connect another users' shared media files to your tree. I don't have a lot of these, but I do have some and wget will not download them without some authentication. The strange thing is, the download URL is EXACTLY the same as all the other media... wget can handle authentication, so I'll try that next...

I'll continue to look at this. The good news is, there is a way, the bad news is it currently requires a lot of steps. Most of these can be scripted into a single command file, but first, I need to figure out the solution to downloading the shared media files. My hope it to create a script that will manage this in one command.

Joel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

enno
In reply to this post by Joel Ford
Op 16-03-18 om 21:39 schreef Joel Ford:
> Thanks Mary. I do appreciate your suggestion, but downloading and
> installing a second genealogy applicaiton in order to import an
> ancestry gedcom into my already existing genealogy applications just
> seems unreasonable to me. ESPECIALLY if I one plans to continue to use
> ancestry. If rootsmagic can do it, why can't gramps?
To connect to Ancestry, or FamilySearch, we need to write software that
can be certified by either Ancestry or FamilySearch, and that requires a
legal agreement between them and us, and quite an investment too. And at
the moment, we're not a legal entity that can sign such an agreement,
nor do we have the funds to enter into a formal software certification
process like they require.

In other words, if you like to work on-line on Ancestry or FamilySearch,
or Find-My-Past, or My Heritage, you will have to use RootsMagic, or
another commercial program that can work the site(s) of your choice. You
can then import GEDCOMs into Gramps, but if you plan to continue using
Ancestry, I suggest that you keep things simple and just stick with
RootsMagic. You can use the free version, if you want.

I often use RootsMagic to import branches from FamilySearch into Gramps,
but that is a one-way street for me, so I can keep Gramps as my main
program, which is a choice driven by my mother's language. If English
were my main language, I would probably use RootsMagic instead, and if
it's yours, I strongly suggest that you do the same.

Cheers,

Enno


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

enno
In reply to this post by roy
Op 16-03-18 om 19:09 schreef roy:
I have an import from Ancestry with the same problem. I figured I was just out of luck at ever transferring the images. I don't even understand half of what you're talking about. I will be watching this thread for a possible solution that a non-guru can understand
If you have an account on Ancestry, RootsMagic is the simplest way. You can use the free version to download tree and images, and then import the whole lot into Gramps.

Cheers,

Enno


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

prculley
In reply to this post by Joel Ford

I think you could have skipped steps 2-5 if you had used the Gramps Addon called the “Download Media files from internet”.

 

If you determine the fixes for URLs (I saw the one you mentioned already) we can add some string replace code to that addon to skip the first step as well.   As well if you can find appropriate authentication methods, please let us know that.

 

Paul C.

 

From: Joel Ford [mailto:[hidden email]]
Sent: Friday, March 16, 2018 4:02 PM
To: GRAMPS user list
Subject: Re: [Gramps-users] ancestry import media list

 

Okay, so here's an update on the adventure of importing media files from ancestry to gramps after importing an acestry gedcom... Suffice to say that, at the moment, it's an imperfect solution. That said, it gets me a lot farther down the road than I was before. I have a lot of media attachments in ancestry and that makes this both important and arduous.

As stated before, I use Linux, so you windoze user will need to adapt to your environement. that said, everything I've done here can be replicated in a windows environment. Using a variety of regex "Find and replace" functions, some linux wizardry via commands with sed and wget, I am able to:

  1. edit the downloaded gedcom to correct the media URLS.
  2. Extract the lines of media URLS from the gedcom file into a text file.
  3. Use wget to download from ancestry the URLS in the said text file.
  4. Use a couple of linux file commands to clean up the file names and then move them to the gramps media folder.
  5. Run the gramps media manager string plugin to replace the media URL strings with a directory path.

        !!VOILA!! the media appears. However, it not yet time for the happy dance...

I had no issues with ancestry authentication EXCEPT when trying to download media files that are attached to my tree FROM another ancestry user. As you know, ancestry allows you to connect another users' shared media files to your tree. I don't have a lot of these, but I do have some and wget will not download them without some authentication. The strange thing is, the download URL is EXACTLY the same as all the other media... wget can handle authentication, so I'll try that next...

I'll continue to look at this. The good news is, there is a way, the bad news is it currently requires a lot of steps. Most of these can be scripted into a single command file, but first, I need to figure out the solution to downloading the shared media files. My hope it to create a script that will manage this in one command.

Joel


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

Joel Ford
In reply to this post by enno

Thanks for the info. I do understand the licensing issues. As stated previously, I use Linux and linux only. To my knowledge, rootsmagic does not offer a native linux application and I do not run wine under my linux. I will, however, look into running a windows VM with rootsmagic and see how that works.

Joel


On 03/16/2018 05:18 PM, Enno Borgsteede wrote:
Op 16-03-18 om 21:39 schreef Joel Ford:
Thanks Mary. I do appreciate your suggestion, but downloading and installing a second genealogy applicaiton in order to import an ancestry gedcom into my already existing genealogy applications just seems unreasonable to me. ESPECIALLY if I one plans to continue to use ancestry. If rootsmagic can do it, why can't gramps?
To connect to Ancestry, or FamilySearch, we need to write software that can be certified by either Ancestry or FamilySearch, and that requires a legal agreement between them and us, and quite an investment too. And at the moment, we're not a legal entity that can sign such an agreement, nor do we have the funds to enter into a formal software certification process like they require.

In other words, if you like to work on-line on Ancestry or FamilySearch, or Find-My-Past, or My Heritage, you will have to use RootsMagic, or another commercial program that can work the site(s) of your choice. You can then import GEDCOMs into Gramps, but if you plan to continue using Ancestry, I suggest that you keep things simple and just stick with RootsMagic. You can use the free version, if you want.

I often use RootsMagic to import branches from FamilySearch into Gramps, but that is a one-way street for me, so I can keep Gramps as my main program, which is a choice driven by my mother's language. If English were my main language, I would probably use RootsMagic instead, and if it's yours, I strongly suggest that you do the same.

Cheers,

Enno


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

Joel Ford
In reply to this post by prculley

Paul,

I cannot see how that addon helps. I did run it while the corrected urls were in the media listing, but it did nothing. I can't see switches or config for it anywhere.


On 03/16/2018 05:34 PM, Paul Culley wrote:

I think you could have skipped steps 2-5 if you had used the Gramps Addon called the “Download Media files from internet”.

 

If you determine the fixes for URLs (I saw the one you mentioned already) we can add some string replace code to that addon to skip the first step as well.   As well if you can find appropriate authentication methods, please let us know that.

 

Paul C.




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

Ron Johnson
In reply to this post by Joel Ford
Could you show us what regex patterns you had to use?

On 03/16/2018 04:01 PM, Joel Ford wrote:

Okay, so here's an update on the adventure of importing media files from ancestry to gramps after importing an acestry gedcom... Suffice to say that, at the moment, it's an imperfect solution. That said, it gets me a lot farther down the road than I was before. I have a lot of media attachments in ancestry and that makes this both important and arduous.

As stated before, I use Linux, so you windoze user will need to adapt to your environement. that said, everything I've done here can be replicated in a windows environment. Using a variety of regex "Find and replace" functions, some linux wizardry via commands with sed and wget, I am able to:

  1. edit the downloaded gedcom to correct the media URLS.
  2. Extract the lines of media URLS from the gedcom file into a text file.
  3. Use wget to download from ancestry the URLS in the said text file.
  4. Use a couple of linux file commands to clean up the file names and then move them to the gramps media folder.
  5. Run the gramps media manager string plugin to replace the media URL strings with a directory path.

        !!VOILA!! the media appears. However, it not yet time for the happy dance...

I had no issues with ancestry authentication EXCEPT when trying to download media files that are attached to my tree FROM another ancestry user. As you know, ancestry allows you to connect another users' shared media files to your tree. I don't have a lot of these, but I do have some and wget will not download them without some authentication. The strange thing is, the download URL is EXACTLY the same as all the other media... wget can handle authentication, so I'll try that next...

I'll continue to look at this. The good news is, there is a way, the bad news is it currently requires a lot of steps. Most of these can be scripted into a single command file, but first, I need to figure out the solution to downloading the shared media files. My hope it to create a script that will manage this in one command.


--
Angular momentum makes the world go 'round.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

enno
In reply to this post by Joel Ford
Op 16-03-18 om 23:49 schreef Joel Ford:

Thanks for the info. I do understand the licensing issues. As stated previously, I use Linux and linux only. To my knowledge, rootsmagic does not offer a native linux application and I do not run wine under my linux. I will, however, look into running a windows VM with rootsmagic and see how that works.

Good choice. I have Wine installed, but RootsMagic's FamilySearch module gave a lot of problems here, so now I only use it in a Windows 10 VM.

I write Windows software for a living, so I have dual boot on all PCs, but I prefer Linux Mint for Gramps, and safety too.

Cheers,

Enno




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org
Reply | Threaded
Open this post in threaded view
|

Re: ancestry import media list

Joel Ford
In reply to this post by Ron Johnson

Ron,

For correcting the URL to a down-loadable link, there are two steps: 1) the part of the URL before the guid, and 2) the part after the. the regex used for the other parts are noted after these

The original URL had two variations similar to this:

<a class="moz-txt-link-rfc2396E" href="http://trees.ancestry.com/rd\?f=image&amp;guid=bb7c762g-33c6-5d88-7654-8aa1fg4ae5bb&amp;tid=12&amp;pid=34">"http://trees.ancestry.com/rd\?f=image&guid=bb7c762g-33c6-5d88-7654-8aa1fg4ae5bb&tid=12&pid=34"
AND
<a class="moz-txt-link-rfc2396E" href="http://trees.ancestry.com/rd\?f=document&amp;guid=bb7c762g-33c6-5d88-7654-8aa1fg4ae5bb&amp;tid=12&amp;pid=34">"http://trees.ancestry.com/rd\?f=document&guid=bb7c762g-33c6-5d88-7654-8aa1fg4ae5bb&tid=12&pid=34"

The desired full image URL (in both cases) was this:

"https://mediasvc.ancestry.com/v2/image/namespaces/1921/media/bb7c762g-33c6-5d88-7654-8aa1fg4ae5bb?client=TreesUI"

However, the above desired URL points to full-sized images that were uploaded or added to my tree BY ME. There is a different URL for full sized images that were uploaded by another member and have been shared to my tree. That URL is:

"https://mediasvc.ancestry.com/v2/image/namespaces/1093/media/bb7c762g-33c6-5d88-7654-8aa1fg4ae5bb?client=TreesUI"

Note that the only difference is the part before the guid, (note that the guid is the same). Here's the interesting part; the original URL does not provide anything to indicate that an image is an owner or a shared image. The original URL will take you to the main page for that item, but the "View Original..." link is different. Consequently, the only way I could identify shared items was when they failed to download using the first URL. I had to manually download these this time. I suppose I can hack the wget command to do an IF, THEN and try the second URL if the first one fails.

BREAKING NEWS:

Just discovered this before hitting send... There is also a third URL, which is the URL from the "Download to Desktop" link on the ancestry media viewer. I suspect this URL is an easier way to approach this task. That link includes the filename, which is derived from the media title and is included in the gedcom under the "TITL" tage for each media.  That URL is:

"https://mediasvc.ancestry.com/v2/image/namespaces/1093/media/30dd9ab5-ee57-4165-8c35-66ca1122dae3.jpg?client=Trees&filename=Will%20of%20John%20Doe"

Ideally, if one could extract the FILE, TITL, and FORM data from the gedcom and concatenate a universally downloadable URL to match this one. That file would download with file name and extension intact.

So, what I initially did was to use following replacements: (note escaped "?" in both)

    REPLACE: <a class="moz-txt-link-rfc2396E" href="http://trees.ancestry.com/rd\?f=document&amp;guid=">"http://trees.ancestry.com/rd\?f=document&guid="
    WITH: "https://mediasvc.ancestry.com/v2/image/namespaces/1921/media/"

    REPLACE: <a class="moz-txt-link-rfc2396E" href="http://trees.ancestry.com/rd\?f=image&amp;guid=">"http://trees.ancestry.com/rd\?f=image&guid="
    WITH: "https://mediasvc.ancestry.com/v2/image/namespaces/1921/media/"

This can be combined into one regex easily. I just did it this way for speed and awareness. Maybe like:

    REPLACE: "http://trees.?&guid="
    WITH: "https://mediasvc.ancestry.com/v2/image/namespaces/1921/media/"

The second part after the guid is straightforward:

    REPLACE: "&tid=[0-9]+&pid=[0-9]+"
    WITH: "?client=TreesUI"

That resolved the gedcom file URL listings. Now, when the gedcom is imported, the media reflect the URL that leads directly to the full size image.


On 03/16/2018 07:50 PM, Ron Johnson wrote:
Could you show us what regex patterns you had to use?



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Gramps-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gramps-users
https://gramps-project.org