fallback to build from source when binary package not available failing when primary archive source is FTP

[JohnPritchard (Johnny P)]

First and foremost, apologies if this is repeating another ticket, but I have searchedanc can not find anything relevant.

I an not certain exactly when this problem started but I believe it was with Macports 2.4.

My system: Mac OS Sierra, 10.12.3, Xcode 8.2.1, Macports 2.4.0, in more detail:

$ uname -a 
Darwin ma018012.ads.eso.org 16.4.0 Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64 x86_64
$ port version
Version: 2.4.0
$ xcodebuild -version
Xcode 8.2.1
Build version 8C1002

I have installed the "external" ESO repository following the instructions at ​http://www.eso.org/sci/software/pipelines/installation/macports.html, which I repeat here for convenience:

curl ftp://ftp.eso.org/pub/dfs/pipelines/repositories/stable/macports/setup/Portfile -o Portfile
sudo port install
sudo port sync

This installation provides both binary and source repositories, however the binary repository is maintained on a ~6monthly basis, meaning it can be some time before binary versions of the packages are available for the latest OS release, in this case Sierra.

In my /opt/local/etc/macports/conf, buildfromsource is set to ifneeded,

$ grep buildfromsource /opt/local/etc/macports/macports.conf
buildfromsource         ifneeded

While the behaviour is normal for "standard" packages, i.e. those in the MacPorts repository, installation of ESO packages is failing to fallback to source installation when the binary package is not available, example:

$ sudo port install esopipe-isaac-datastatic
--->  Fetching archive for esopipe-isaac-datastatic
--->  Attempting to fetch esopipe-isaac-datastatic-6.1.5_5.darwin_16.noarch.tbz2 from ftp://ftp.eso.org/pub/dfs/pipelines/repositories/stable/macports/packages/esopipe-isaac-datastatic
Error: Failed to archivefetch esopipe-isaac-datastatic: version @6.1.5_5: RETR response: 550
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_ftp.eso.org_pub_dfs_pipelines_repositories_stable_macports_ports_science_esopipe-isaac-datastatic/esopipe-isaac-datastatic/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets to report a bug.
Error: Processing of port esopipe-isaac-datastatic failed

Installation from source, works normally:

$ sudo port install -s esopipe-isaac-datastatic
--->  Fetching distfiles for esopipe-isaac-datastatic
--->  Attempting to fetch isaac-kit-6.1.5-4.tar.gz from ftp://ftp.eso.org/pub/dfs/pipelines/instruments/isaac
--->  Verifying checksums for esopipe-isaac-datastatic                               
--->  Extracting esopipe-isaac-datastatic
--->  Configuring esopipe-isaac-datastatic
--->  Building esopipe-isaac-datastatic
--->  Staging esopipe-isaac-datastatic into destroot
--->  Installing esopipe-isaac-datastatic @6.1.5_5
--->  Activating esopipe-isaac-datastatic @6.1.5_5
--->  Cleaning esopipe-isaac-datastatic
--->  Updating database of binaries
--->  Scanning binaries for linking errors
--->  No broken files found.                             

Please note this issue is only applicable/reproducible on Mac OS Sierra at this point in time as binary packages are available for previous Mac OS X releases.

So, is this an issue with MacPorts itself? Or with the repository?

Many thanks in advance for assistance John Pritchard

[raimue (Rainer Müller)]

Could you please attach a main.log showing the error? You can produce and find the file the following way:

$ sudo port archivefetch esopipe-isaac-datastatic
$ port logfile esopipe-isaac-datastatic

This particular error message is raised at this point, which leads to the abort: ​https://github.com/macports/macports-base/blob/v2.4.0/src/package1.0/portarchivefetch.tcl#L298-L300

This should only happen when the fetch failed, but _archive_available (effectively a curl getsize as provided by Pextlib) said that the file exists on the archive site. This might be caused by the corresponding archive site being accessed over FTP.

[JohnPritchard (Johnny P)]

main.log

[JohnPritchard (Johnny P)]

main.log generated by the commands you instructed me to use attached.

Cheers John

[jmroot (Joshua Root)]

Well this is interesting.

% curl getsize ftp://ftp.eso.org/pub/dfs/pipelines/repositories/stable/macports/packages/esopipe-isaac-datastatic/esopipe-isaac-datastatic-6.1.5_5.darwin_16.noarch.tbz2
-1
% curl getsize https://packages.macports.org/esopipe-isaac-datastatic/esopipe-isaac-datastatic-6.1.5_5.darwin_16.noarch.tbz2
HTTP response code said error

We assume curl will throw an error when the file doesn't exist, but apparently it doesn't with FTP.

[jmroot (Joshua Root)]

In 6718cc6/macports-base:

Check for file size sanity in _archive_available

It seems that if 'curl getsize' is called with an FTP URL that points
to a nonexistent file, it will return a size of -1 rather than throw
an error like it does with HTTP.

Fixes: #53530