gpsd: python34 variant depends on nonexistent py34-serial port

[ryandesign (Ryan Carsten Schmidt)]

​https://build.macports.org/builders/jobs-mirror/builds/150253/steps/mirror/logs/stdio

--->  Fetching distfiles for gpsd
Error: No such port: py34-serial
./mpbb/mpbb: error: `mirror-distfiles' failed to run successfully

[fhgwright (Fred Wright)]

The obvious fix is to remove the python34 variant, though I'm not particularly a fan of the zealous removal of perfectly functional subports/variants based on older Python versions (it was the gratuitous removal of py34-serial that caused this in the first place). If the same philosophy were applied to OS versions, MacPorts wouldn't run on anything older than 10.13.

This is actually a "soft" dependency in the sense that the absence of py-serial only causes the loss of a subset of the functionality of two of the vendor-specific GPSD programs (which most people probably don't even use), and everything else works without it. It's included as a dependency mainly as a convenience, since it's a fairly lightweight dependency. Replacing the dependency with an info message in the +python34 case would be another possibility.

I always test the upstream GPSD code against Python 2.6, 2.7, and 3.2+ (though there are currently issues with 3.8), which requires some local patches to put back removed subports of some dependencies.

BTW, does the fact that this arose at all on the buildbots mean that consideration is being given to offering precompiled binaries for non-default variants (which is otherwise extremely rare)?

[jmroot (Joshua Root)]

Replying to fhgwright:

BTW, does the fact that this arose at all on the buildbots mean that consideration is being given to offering precompiled binaries for non-default variants (which is otherwise extremely rare)?

At the moment it just means that ports' distfiles are mirrored for all variants. Actually building non-default variants is something we'd like to do, but only for curated selections since building all combinations would be mathematically problematic. That's not yet possible.

[fhgwright (Fred Wright)]

Replying to jmroot:

Replying to fhgwright:

BTW, does the fact that this arose at all on the buildbots mean that consideration is being given to offering precompiled binaries for non-default variants (which is otherwise extremely rare)?

At the moment it just means that ports' distfiles are mirrored for all variants.

If it's just about distfile mirroring, why is it paying attention to dependencies at all? With the possible exception of fetch dependencies (if they even exist), dependencies should be completely irrelevant for what's essentially a version of "port fetch".

Actually building non-default variants is something we'd like to do, but only for curated selections since building all combinations would be mathematically problematic. That's not yet possible.

Indeed providing *all* variants would be problematic, though I think I've seen isolated cases of more than one. Criteria for inclusion might be based on popularity of the variant, time required to build from source, and size of the binary archive. It would make sense to boost the priority of anything that can't be built from source for an upgrade without manual intervention, such as anything involving the dreaded libunwind-headers, or poppler, or the ld64 stuff that Ken's been talking about. And "gratuitous universality" makes precompiled +universal binaries a lot more important than they really ought to be.

[ryandesign (Ryan Carsten Schmidt)]

Replying to fhgwright:

If it's just about distfile mirroring, why is it paying attention to dependencies at all? With the possible exception of fetch dependencies (if they even exist), dependencies should be completely irrelevant for what's essentially a version of "port fetch".

The buildbot uses MacPorts base to do the work, and I guess MacPorts base ensures that a port's dependencies exist before doing any of the phases. And yes, there is such a thing as a fetch dependency (depends_fetch).

Indeed providing *all* variants would be problematic, though I think I've seen isolated cases of more than one. Criteria for inclusion might be based on popularity of the variant, time required to build from source, and size of the binary archive. It would make sense to boost the priority of anything that can't be built from source for an upgrade without manual intervention, such as anything involving the dreaded libunwind-headers, or poppler, or the ld64 stuff that Ken's been talking about.

I don't think we have any of the data that would be required to do any of the above, except for variant installation statistics which are available on ports.macports.org but only for those very few users who have opted in to send us their stats.

And "gratuitous universality" makes precompiled +universal binaries a lot more important than they really ought to be.

If a port requires its dependencies to be universal (either by forcing its own universal variant on or by requiring a 32-bit build on a 64-bit system) then, since the buildbot is using MacPorts to do the building, MacPorts would first install the dependencies universal, and those universal archives would get uploaded to the packages server so that you could have them. See also #35897.

[ryandesign (Ryan Carsten Schmidt)]

Replying to fhgwright:

"gratuitous universality"

Perhaps you're referring to #48051. If so, that's certainly an area someone could look into improving.

[fhgwright (Fred Wright)]

Replying to ryandesign:

Replying to fhgwright:

If it's just about distfile mirroring, why is it paying attention to dependencies at all? With the possible exception of fetch dependencies (if they even exist), dependencies should be completely irrelevant for what's essentially a version of "port fetch".

The buildbot uses MacPorts base to do the work, and I guess MacPorts base ensures that a port's dependencies exist before doing any of the phases. And yes, there is such a thing as a fetch dependency (depends_fetch).

I just did a test where I verified that "port fetch gpsd +python34" (after "port clean --all gpsd" of course) has no problem fetching the distfile for gpsd with py34-serial as a nonexistent port. So the mirroring stuff is clearly trying to enforce dependencies beyond what a normal port command would do. I don't see the value in this.

Indeed providing *all* variants would be problematic, though I think I've seen isolated cases of more than one. Criteria for inclusion might be based on popularity of the variant, time required to build from source, and size of the binary archive. It would make sense to boost the priority of anything that can't be built from source for an upgrade without manual intervention, such as anything involving the dreaded libunwind-headers, or poppler, or the ld64 stuff that Ken's been talking about.

I don't think we have any of the data that would be required to do any of the above, except for variant installation statistics which are available on ports.macports.org but only for those very few users who have opted in to send us their stats.

Certainly the (relative) build time and the archive size are available from the buildbots. Ports which can't be built or upgraded without manual intervention would probably be best handled by flagging them explicitly in the Portfile.

There is a form of information available about variant usage that's both better in some ways and worse in some ways than the opt-in reports from users. Every install or upgrade without -s (or equivalent) attempts to fetch a binary archive before resorting to building from source. The URL of the archive being requested contains the port name, the version, the OS version, the CPU architecture, and the variants. So all this information probably already exists in the server logs without any action on the part of the users, though it's only sampled at install/upgrade time, and absent with -s.

[michaelld (Michael Dickens)]

In 1184767e309aada665373d2fdf8e235e00c1ee44/macports-ports (master):

gpsd: remove support for Py34

Some Py34 dependencies no longer exist.

Fixes: #60044

[fhgwright (Fred Wright)]

Well, OK, but I'd originally left this alone since I figured it would be a good test case for fixing the *real* bug, which is the inappropriate use of non-fetch dependencies in mirroring.