py-novas_py fetches files during its build phase

[ryandesign (Ryan Carsten Schmidt)]

py-novas_py fetches files during its build phase, which is troublesome and should be avoided if possible. (Fetch during the fetch phase instead.)

For some systems with more restrictive networks, like the buildbot, this even causes a build failure:

​http://build.macports.org/builders/buildports-snowleopard-x86_64/builds/5749/steps/compile/logs/stdio

[lpsinger (Leo Singer)]

NOVAS_Py's setup.py script downloads the entire contents of the directory ​ftp://ssd.jpl.nasa.gov/pub/eph/planets/ascii/de405. Is there a way to retrieve all of the contents of a directory during the fetch phase, or do I need to override or augment the fetch phase?

As an alternative, I could hard-code the paths of each of the files; the last modification times in FTP are all in 2007.

[ryandesign (Ryan Carsten Schmidt)]

Each file that needs to be downloaded would be listed in the port's distfiles line, and checksums for each file would be listed as well. Then if necessary in post-extract you would move these files into places where the build script will recognize them, perhaps even needing to patch the build script to not download the files.

If the distfile names do not contain the novas_py version number (and these don't), then you have a problem with unversioned distfiles and need to set dist_subdir, but that would then cause unnecessary re-downloading and re-mirroring of all those files when the novas_py version is increased but those other files don't change.

How are these files used by novas_py? Are they merely installed into a particular location and used at runtime? If so, perhaps a separate port could be created that only installs these data files, and novas_py could then depend on it. That new port (novas_py-data?) could set dist_subdir, but wouldn't need to be updated until those files actually change.

[lpsinger (Leo Singer)]

Replying to ryandesign@…:

How are these files used by novas_py?

These are ASCII data files generated by JPL (they are a solar system ephemeris, or a table of the calculated positions of solar system objects, spanning many centuries). The NOVAS_Py setup.py script downloads them and packages them into a single binary file that is used by the underlying C NOVAS library. There are certain NOVAS calls that don't require an ephemeris, but many do. Most users would want to have an ephemeris.

To complicate matters, the code for packing the ASCII files into the one binary file lives in the NOVAS_Py package.

Another option would be for a third party (e.g., myself) to generate a binary ephemeris file, with a filename that reflects the date on which the JPL data files were downloaded. Then a Port could be created for this. Yet another complication: the file's endianness matters; it has to match the native endianness of the platform.

[ryandesign (Ryan Carsten Schmidt)]

Simplest then is to just list all the files as distfiles, list their checksums, and inform the build script to use those files (possibly patching it). If the data files ever change, then we can add "dist_subdir ${name}/${version}" but since the files haven't changed in years we can skip that for now.

[lpsinger (Leo Singer)]

Replying to ryandesign@…:

Simplest then is to just list all the files as distfiles, list their checksums, and inform the build script to use those files (possibly patching it). If the data files ever change, then we can add "dist_subdir ${name}/${version}" but since the files haven't changed in years we can skip that for now.

If I don't specify dist_subdir, will those files just be silently carried over from one version of the Port to another? That is exactly the behavior that I want.

[ryandesign (Ryan Carsten Schmidt)]

Yes, and that won't be a problem, unless upsteam at some point updates one of those files.

[lpsinger (Leo Singer)]

new Portfile

[lpsinger (Leo Singer)]

additional patch file

[lpsinger (Leo Singer)]

I just added a new Portfile and one extra patch file implementing what was suggested.

[ryandesign (Ryan Carsten Schmidt)]

Looks good to me. I'm attaching a diff of your Portfile changes, with the following changes:

• simplify post-extract using eval ... glob
• use "-W" to change directory in a "system" command instead of manually calling "cd"

[lpsinger (Leo Singer)]

Fixed in r91231. Thanks for your help!

[lpsinger (Leo Singer)]

The ​buildbot fails when attempting to retrieve the JPL ephemeris files via FTP. Is this a problem with this Port?

[mf2k (Frank Schima)]

Unfortunately the buildbot cannot download from FTP sources. This is a general issue and has to do with security. Ideally a http source can be added for this port.

[lpsinger (Leo Singer)]

Replying to macsforever2000@…:

Unfortunately the buildbot cannot download from FTP sources. This is a general issue and has to do with security. Ideally a http source can be added for this port.

I just checked; I do not think that there is an alternate http URL.

[ryandesign (Ryan Carsten Schmidt)]

The buildbot was fixed to be able to download from FTP sources some time ago; I believe Daniel Luke provided a proxy server. So I don't know why this is now failing.