Links from PDF documents to external images/documents
I have some experience with PDFs and in my experience, best results are obtained by postprocessing: assembling different documents converted to PDF, then assembling them in a master document with an indexed table of contents and other optimization (linearization), rather than generating a document directly from a software package. Usually, on OS X this is done with Adobe Pro DC ($$) but cheaper tools are available such as PDF Expert for Mac (free basic, $ pro). I myself use commandline tools such as sips (part of OS X) and imagemagick for image processing, Python img2pdf which is a great tool for generating a PDF from an image, and xpdf / pdftk / qpdf, versatile PDF toolboxes for rotating and combining PDFs and creating a table of contents linked to pages and optimizing final output. Other tools are useful: ghostscript, exiftool. I give my visuals long descriptive filenames, and if there is a standardized naming scheme, it is straightforward to generate a PDF table of contents textfile directly from the filenames using bash. Of course, commandline tools require some experience...
I am not sure linking to external images from within a PDF is the best approach for your use case. If your goal is to distribute physical media (optical or USB thumb drive, since many modern computers don't have optical drives) to multiple recipients using different systems, why not just build a static website instead? You will have far greater compatibility with various operating systems and browsers, while external PDF linking is reputedly unreliable depending on the viewer used (e.g. Preview on OS X). You can tweak the code, and store images in an easy-to-browse subfolder. I understand that if you already have images linked to from within gramps, it would be a bit of work to rebuild a structure. But I think gramps has options to export to HTML/XHTML.
Another alternative is EPUB format, which I have not used for authoring, but which seems far more hyperlink-friendly.
The PDF spec  says that PDF hyperlinks are drawn around a fixed region of a page, not an object (such as text). So I think you may need one of the authoring tools above to reliably embed links, and even then you would need to test if those links work on different systems.