Add versions to platforms

[raimue (Rainer Müller)]

Currently the platforms statement can only specify a platform like darwin, macosx, linux, freebsd and others, but cannot require a version of the platform. So this is a proposal to extend it to also include platform version numbers.

This would look like this:

platforms    darwin                  # as it is now
platforms    {darwin 7} {darwin 8}   # Darwin 7 or 8
platforms    {darwin >=8}            # Darwin 8 or higher
platforms    {macosx >=9.3}          # Mac OS X 10.5.3 or higher

If a platform does not match the requirements, the port should error out automatically when trying to install it. To enforce a specifc version, some ports are using pre-fetch {} blocks at the moment, these could be replaced with this new syntax.

[afb@…]

Replying to raimue@…:

platforms    {macosx >=9.3}          # Mac OS X 10.5.3 or higher

Shouldn't that be {macosx >= 10.5.3} (or {darwin >= 9.3} perhaps) ?

[ryandesign (Ryan Carsten Schmidt)]

I like this idea, especially the ability to specify an operator (e.g. ">=", "<", etc.). It doesn't let you specify a range... like "darwin 8 through 10". But that may be uncommon enough that we don't need to worry about it.

Has duplicate #29466.

[rudloff@…]

Cc Me!

[raimue (Rainer Müller)]

Has duplicate #38508.

[cooljeanius (Eric Gallager)]

Cc Me!

[larryv (Lawrence Velázquez)]

Cc Me!

[mojca (Mojca Miklavec)]

Cc Me!

[ci42]

Cc Me!

[neverpanic (Clemens Lang)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Replying to afb@…:

Replying to raimue@macports.org:

platforms    {macosx >=9.3}          # Mac OS X 10.5.3 or higher

Shouldn't that be {macosx >= 10.5.3}

No. We're proposing that the platforms variable should be able to check os.major, which will be 9 on Darwin 9, regardless of whether it's a mythical open-source-only Darwin 9 or the full Mac OS X Leopard.

If you want to query the Mac OS X version, and not the Darwin version, you can do so using the macosx_version variable and the vercmp procedure, but I don't think we're considering doing that as part of the platforms variable changes proposed in this ticket.

(or {darwin >= 9.3} perhaps) ?

Just as it has always been, you would use darwin in the platforms statement if your software does not need the proprietary Apple frameworks that are part of macOS, or macosx if the software does need those frameworks.

Replying to ryandesign:

It doesn't let you specify a range... like "darwin 8 through 10".

It does if we do it the way we already do it in the compiler_blacklist_versions portgroup. In that case, it would be {darwin >= 8 <= 10}.

[neverpanic (Clemens Lang)]

We've discussed having this a couple of times, so let's plan it for a milestone. Since there's no code written yet, assign 2.6, rather than 2.5 which we want to get out sooner.

[pmetzger (Perry E. Metzger)]

Given this, will it be possible to change the buildbot infrastructure not to even try to build on unsupported platforms? That will make life much easier by suppressing spurious errors.

[mojca (Mojca Miklavec)]

Yes, that's the whole idea.

[ctreleaven (Craig Treleaven)]

To implement this, is the idea to have a new phase (maybe "check") that is evaluated before the fetch phase?

[raimue (Rainer Müller)]

For platforms, I think the best is to make this a declarative option in the Portfile. Specifiying support by matching on platform name and version should be sufficient. That would also make it possible to write this information into the PortIndex, so you could query for ports with this information (or present it accordingly on a web page).

A new phase would mean that the Portfile needs to be evaluated, therefore you could not easily list all (non-)supported ports for a platform. Also, if the result of such a check phase is just "yes" or "no", it does not represent what platform would be required to be compatible.

[mojca (Mojca Miklavec)]

One thought. A good point has been raised: the platforms variable currently represents the platforms where the port has been tested rather than platforms where the port works. If we try to be strict, most likely it will no longer be possible to install any port on linux since linux is almost nowhere listed as a platform (then again, if we did that, we would suddenly get enough pull requests to list the missing platform variable and thus get feedback when something actually works).

We could potentially introduce something like

platforms-blacklist {darwin < 10} darwin12

to signal that Mac OS X < 10.6 and 10.8 don't work with this port. Or

platforms-whitelist darwin15 darwin16 linux

to signal that the port only works on linux and macOS 10.11 and 10.12. I don't know whether

platform-blacklist {darwin > 11}

should match darwin11.1 in that case or not.

I'm not saying that this is necessarily the best idea, just thinking loud.

[mojca (Mojca Miklavec)]

One potential advantage would be that, say, Qt5 requires nearly the latest OS. If one port includes the PortGroup to request Qt5 (unless special measures are taken and an older version is provided, but forget about that for a moment), the PortGroup could blacklist some darwin versions and the dependent port would be blacklisted automatically. Similarly, if a port asks for wxWidgets 3.2, the PortGroup could add

platforms-blacklist-append {darwin < 11}

and the port would automatically cease to exist on 10.6 and lower.

[raimue (Rainer Müller)]

Not sure about having both -whitelist and -blacklist. What would be the result if the expressions overlap? And yes, the meaning of the "negated" comparison of the version number could indeed become a problem.

I propose the following: every supported platform should be added explicitly to platforms. Each entry specifies requirements. A plain platform name means any version of this platform. Specifying {darwin >= 11} implicitly assumes that {darwin < 11} is not supported, but does not imply anything for any other platform. To express that a port will never work on a particular platform, we could use platforms !linux. If there is no entry for the current platform, the user will see a warning, but the build continues.

• a) The default is empty. The option would still be mandatory in a Portfile as it is now.
• b) Make the default platforms darwin on all platforms. The option is not mandatory and would not be required in almost all Portfiles.

Realistically, we are only writing Portfiles for darwin/macOS, so b) would be a sane assumption in my opinion.

[mojca (Mojca Miklavec)]

In my opinion, if blacklisting and whitelisting are both present, then start with whitelist and additionally remove all blacklisted ones from that list. After all, if one dependency can only be installed on 10.13 (say that's the whitelist) and the other dependency cannot be installed on anything newer than 10.9 (blacklist) ... you cannot really install that port, can you? Same with

if (darwin_version == 13) {
  if (!(darwin_version > 9)) {
    printf("supported"); // this will not be printed
  }
}

What we could also do is support just blacklisting. If one particular port is supported on darwin10, darwin12 and darwin13, you do something in this sense (please ignore the exact syntax, I just want to convey the meaning for now):

platforms-blacklist[-append] {darwin < 10} darwin11 {darwin >= 14}

I would really like to see the ability to proliferate this information from PortGroups or potentially even dependencies. If one port depends on qt5 which cannot be compiled on older systems, then this port (or any other dependent port of qt5) should have those darwin versions blacklisted as well.

As far as the current meaning of platforms is concerned: given that we are developing and testing the packages almost exclusively on macOS anyway, I indeed find the current value of platforms of limited value. It's officially forbidden to exclude it, but we currently don't accept any package that would only compile on linux for example, so darwin would always be implied, while the lack of linux does not mean that the package cannot be compiled on linux and nobody would ever notice that this value is "missing".

It could make more sense to use (pseudocode):

also-tested-on linux freebsd

or something along those lines. We currently have absolutely no way to actually tell that one particular package doesn't compile on linux or solaris (and most packages don't - not because the software would not compile, but because the Portfile is totally mac-centric).

Totally off-topic. I was just reading Bootstrap tutorial, according to their logic we would specify (in pseudocode):

platforms blacklist-darwin-9-down blacklist-darwin-11 blacklist-darwin-14-up

They support whitelisting as well, but I didn't test what happens when you do both in the same string. If you do them separately (nested), blacklisting would always win.

[ryandesign (Ryan Carsten Schmidt)]

Replying to mojca:

the platforms variable currently represents the platforms where the port has been tested rather than platforms where the port works.

Not particularly. platforms macosx means the port works on macOS, and does not work on pure Darwin because of the use of proprietary framework or other non-open-source technologies only available in macOS. platforms darwin means the port is not believed to use proprietary frameworks or other closed-source features so it should work on pure Darwin, though in practice nobody tests this of course, and I expect many ports that declare platforms darwin do actually use proprietary macOS features and don't work on pure Darwin.

My guess is that most ports that list platforms other than darwin or macosx don't actually work on those other platforms because the portfile and the underlying software have undergone changes since those other platforms were added to the platforms directive. For example, the zlib port claims to work on freebsd, but that claim was added 11 years ago, and I certainly haven't ever tested it on freebsd. It might have worked on freebsd back then, but who knows if it does now.

I'm not particularly concerned about addressing that problem. Everybody knows MacPorts is for macOS and it's not terrible that our support for non-Mac platforms isn't as good.

I agree with Mojca that it would be good if it were possible for portgroups or subports to alter the set of supported macOS versions set by the port. As long as we design the syntax with that in mind, I don't think it particularly matters whether we design the syntax to use blacklisting or whitelisting. Using a blacklisting syntax seems to have worked well for choosing compilers, but I can't offer an opinion at this point as to whether it would make sense for platform versions.

[raimue (Rainer Müller)]

Replying to mojca:

I would really like to see the ability to proliferate this information from PortGroups or potentially even dependencies. If one port depends on qt5 which cannot be compiled on older systems, then this port (or any other dependent port of qt5) should have those darwin versions blacklisted as well.

I agree the platform information should be used early in a port install and error out before even installing dependencies. But in my opinion, this is out of scope of defining a syntax for platforms.

I am still not convinced of the -blacklist/-whitelist approach. What if I only specify a whitelist? Would it imply all other versions are not supported?

Also, consider this example:

platforms-blacklist {darwin >= 10}
platforms-whitelist {darwin 11}

Of course this is an artificial example, but the syntax supports it. You said start with whitelist, then apply blacklist. So, the blacklist overrides {darwin 11} in the whitelist and it is not supported? Or is {darwin 11} supported although it is matched by the blacklist? Implementing and documenting this behavior seems complicated and overly complex.

With a platforms option taking entries for supported platforms as as proposed above, there is no room for ambiguity:

platforms {darwin < 10} {darwin 11}

[pmetzger (Perry E. Metzger)]

Side note: The OCaml OPAM people have a pretty rich language for specifying versions for platforms and dependencies. I'm not sure their syntax is worth stealing, but the ability to say things like "this requires a dependent package or platform not less than X in version and not greater than Y" is kind of cool. (When I work with opam, I long for many things that are built in to macports, and when I work with macports, I long for things built in to opam. opam's ability to specify versioning nicely is such a feature.)

[ryandesign (Ryan Carsten Schmidt)]

Perry, can you show us specifically what that syntax looks like, or provide a link to its documentation?

What we implement for MacPorts will be constrained by what makes sense in Tcl.

[pmetzger (Perry E. Metzger)]

Link to the documentation below. I stole these examples from the web site. The syntax they use probably isn't feasible to steal for Tcl, but some of the ideas might be interesting to steal.

Typically, you specify a package dependency constraint like:

"pkg1" {>= "3.2" & != "3.7"}

They can constrain multiple packages, eg:

("pkg1" & "pkg2") | "pkg3" {>= "3.2" & != "3.7"}

You constrain where your package is available with separate constraints described in stanzas like this:

available: [ os != "darwin" | ocaml-version >= "4.00" ]

They can also do things like constraining individual arguments to build commands based on version constraints, like:

["./configure" "--with-foo" {ocaml-version > "3.12"} "--prefix=%{prefix}%"]

which will only include --with-foo in the command for 3.12 and above.

​Full Manual.

[pmetzger (Perry E. Metzger)]

Oh, one other example from the manual:

"foo" { >= "3.12" } & ("bar" | "baz" { !(> "2" & < "3.5") & != "5.1" })

[mojca (Mojca Miklavec)]

Rainer, yes, in your example above I would consider all platforms blacklisted. If you don't like "whitelisting" (you could call it platforms-blacklisteverythingbut :), we can drop it. My idea only needs blacklisting to work.

Now, to explain why your suggestion might be tricky to implement and understand, imagine that your port declares

platforms {darwin < 10} {darwin 11}

and one the portgroup it includes declares

platforms {darwin > 10}

and one dependency would declare

platforms {darwin > 12}

etc. How do you then combine these together? Users might just as well start thinking of using something like platforms-append {darwin 13}

How will you determine whether the port can be installed on a particular platform? You would need to use cross-sections of options which is not quite easy for users to understand either.

[raimue (Rainer Müller)]

You would have to use platforms-append like we do for any other option that comes from PortGroups. Why would we want to handle platforms differently than other options?

# PortGroup ...
platforms {darwin > 10}

# Portfile
platforms-append {darwin < 10} {darwin 11}

Overall, we mostly care whether the current platform is supported or not. There is no need for complicated blacklists or whitelists. There is usually only a need to express a maximum or minimum version requirement for each platform.

To find out if the current platform is compatible, we walk through this full list of requirements as explained above. We check each item of this list ({darwin > 10} {darwin < 10} {darwin 11}) against the current platform.

Let me write some pseudo-code for the algorithm:

state = MAYBE
foreach item in $platforms
    supported = check_if_supported(current_platform, current_version, item)
    if supported == YES
        state = YES
    else if supported == NO
        if state == MAYBE
            state = NO
        endif
    else
        // MAYBE is ignored
    endif
endforeach

define check_if_supported(current_platform, current_version, item)
    platform, operator, version = split(item)
    if current_platform != platform
        return MAYBE
    endif
    if current_version operator version    // think of: 1.5 < 2.0
        return YES
    else
        return NO
    endif
enddefine

If we end with state == NO, the current platform is definitely not supported. If we end with state == MAYBE, this build might work and we warn the user, but continue. If we end with state == yes, we obviously just continue.

The example list from above would always evaluate to YES on macOS except for darwin 10 (or darwin 10.0.0, depending on how we compare the version).

I do not see a use case when you would need to broaden the list of supported versions later. If a PortGroup says that it requires darwin < 10, under what circumstances would the port need to say that an older version is actually supported? Then the requirement in the PortGroup was likely bogus and needs to be fixed. And even then, you can still override it in the Portfile by just using platforms (without -append) or delete the specific item with a -delete or -replace. It is just like any other option.

I do not understand why you also bring dependencies into this... They can declare whatever they want and yes, it might stop us from installing dependents. But this has no influence on the way we specify platforms in this port. We will check whether the current platform is supported each time we want to install ports, so we automatically do it for the dependency. If it does not support it, we stop at this point and all the dependents cannot be installed. To give an appropriate error message we have to evaluate this for each port separately anway, so I do not see when we would have to merge anything.

[mojca (Mojca Miklavec)]

Replying to raimue:

The example list from above would always evaluate to YES on macOS except for darwin 10

See, that's where we completely disagree. The point of my example above is that the PortGroup specifying

platforms {darwin > 10}

means that a dependency (like Qt or wxWidgets or so) cannot be compiled on Snow Leopard or earlier, while (sorry for a not too realistic example) the following code in the Portfile

platforms {darwin < 10} {darwin 11}

means that the code in that port either requires some ancient SDK or no longer compiles with a newer compiler. So basically it means that it would only compile on darwin 11, except that another dependency requires a newer OS, so this port could not be compiled at all.

The message I want to convey is that a PortGroup specifying {darwin > 10} should not mean that a port using that PG can additionally (unconditionally) be compiled on any OS 10.7 or newer. It should mean that the port cannot be compiled on 10.6 and lower under any circumstances. Like the latest version of Qt.

You are doing union. I claim we should be doing cross-section on compiler selection, else specifying any platforms in the PortGroup will be completely useless. This is why I consider blacklisting a better option.

[mojca (Mojca Miklavec)]

OK, we might not need to merge the lists with dependencies, but we do need to make sure that multiple PortGroups included in the same port could specify that the port cannot be built on some platforms.

Building gnuplot +qt will certainly have different coverage of supported platforms than gnuplot -qt or gnuplot +wxt.

Then again ... if we evaluate dependencies separately anyway, if a port includes qt5 PG and that one pulls in dependency on qt5 anyway, the PG might not actually need to blacklist any platforms since the platform will already be blacklisted by qt5 itself.

[raimue (Rainer Müller)]

What did you expect as result for the example? Should it be supported on darwin 11 or not? And why would it not be supported on darwin 10? I am confused by your description.

Okay, I understand your point now that it should be allowed to add more restrictions. This is indeed not possible with my proposal.

Replying to mojca:

Then again ... if we evaluate dependencies separately anyway, if a port includes qt5 PG and that one pulls in dependency on qt5 anyway, the PG might not actually need to blacklist any platforms since the platform will already be blacklisted by qt5 itself.

Exactly this is what I was trying to say. We do not need to care about dependencies, we only need to specify what *this* port supports.

[ryandesign (Ryan Carsten Schmidt)]

It was said at one point that implementing this type of restriction on whether a port can build or not (whether it's based on OS version or other criteria) could not be done until we had merged the GSoC '15 libsolv branch. Do we still think that's a requirement, or no?

[raimue (Rainer Müller)]

We can certainly check whether the current platform is supported when trying to install a port. This would be the same what we already do for supported_archs.

However, checking the whole dependency tree before attempting to install any port, even any dependency, should only be attempted with the libsolv branch as that inherently collects the whole dependency tree first.

[mojca (Mojca Miklavec)]

Just to "defend" myself a bit. My "frustration" with blacklisting comes from the compiler background. Assume port foo which cannot be built with the latest compiler & macOS 10.13, providing two non-conflicting variantes, +qt5 (or +qt4) and +wxwidgets. This particular port would generally only blacklist that particular compiler and specify that it cannot be built on 10.13.

Both qt5-base and wxWidgets have their own set of defunct compilers and unsupported macOS versions.

Now, wxWidgets gets upgraded to a newer version after which it no longer compiles with clang 425. This means that foo and all other ports depending on wxWidgets need to have an additional compiler blacklisted. This information does not proliferate from one port to another (from dependencies), so all the compiler blacklisting needs to be done inside the PortGroup. At the moment we would have three compiler blacklistings: for foo, for wxWidgets and for qt5. Only compilers (and OSes) that have not been blacklisted anywhere should "survive" after both PortGroups have been included.

I initially failed to realize that blacklisting OS versions can be done in an easier way since the system should be designed in a way that will not allow installing any port that depends on something that's not supported on this platform. That is: it's not necessary to blacklist OSes inside the wxWidgets PortGroup.

But I do imagine that there will be still be some PortGroups that will require blacklisting OSes, cxx11 comes to mind as one (not 100% real) potential example. If we did not have this sophisticated machinery to allow building ports with C++11 on < 10.9, we would specify in the cxx11 PortGroup that those ports cannot be built on {darwin < 13}. Now imagine a port that needs C++11 and cannot be built on 10.13. My approach would be to only specify

platforms {darwin < 17}

or

platforms-blacklist {darwin >= 17}

in the port itself, and leave blacklisting of {darwin < 13} to the PortGroup.

I still believe that blacklisting is easier to implement and easier to understand than if you specify {darwin < 17} in the port and {darwin >= 13} in the PortGroup if that should mean only darwin 13 up to 16.

[janstary (Jan Starý)]

I agree it is useful to be able to express some global restriction, such as "you need at least _this_ version of darwin to build this".

But the proposed whitelisting and blacklisting sounds like the opposite of keeping it simple.

The default should be _empty_ . If the port does not specify "platforms" (i.e. the Portfile does not even use the platforms keyword) it should mean the port has no requirements about the platform it is being built on. Most ports should be like that.

The ports which do need a specfic platform would then say darwin 8 or darwin >= 10 or darwin < 7. Does anyone know about a particular port that needs to express something more complex than the simple inequalities?

The distinction between darwin and macosx is not realistic. Most ports say darwin, and imho only do so because the platform field is required (and ignored, LOL). Currently, 155 ports include macosx among their platforms; some of those have darwin macosx. Does that mean all the other ports do specifically _not_ require the Apple Frameworks? No it doesn't.

This is what the Guide says about platforms now, confusingly:

A list of the platforms on which the port has been tested. Required, but not interpreted in any way by the software at this time; it is purely informational for users. Possible values: darwin (= macosx or puredarwin), macosx, puredarwin, freebsd, linux, sunos, netbsd. In general, it can just be set to darwin. (puredarwin is an OS based on Apple's open-source Darwin releases without any of Apple's proprietary bits.) See also os.platform.

So people have been putting in darwin, or macosx which is _the_same_, or puredarwin which is also the same but does not exist.

[janstary (Jan Starý)]

In fact, the specification should express what _doesn't_ work, the default being that the ports works, everywhere. For example, a port saying

broken darwin < 7 # Does not have strsep (or whatever)

would then stop building immediately on darwin < 7, because we know it does not build on this platform.

I believe it is simpler and better than enumerating the working cases, which should be the vast majority, and the default.

For comparison, OpenBSD has a BROKEN field for that. For example in audio/xmms2/Makefile:

BROKEN-sparc64 = waf build goes into an infinite loop

"We don't build here, and this is why." Isn't that the purpose of this proposal as well?

[raimue (Rainer Müller)]

Regarding darwin vs. macosx: the keyword is "require". Most of the other ports will also build without Apple Frameworks, but will use them if available. Applications with a native GUI will choose macosx, as they cannot work without Apple Frameworks.

As I see it, the guide is correct. Maybe it is not properly phrased, but darwin means any of macosx or puredarwin and these are not the same. They are also not handled the same in MacPorts base, which has the notion of macosx and puredarwin each being a subplatform of darwin. See ${os.subplatform}.

I agree that the specification is to exclude platforms. However, we always list the working cases. We also have supported_archs and not broken_archs.

I do not think we need a way to express the reason why a port is broken. A comment in the Portfile should suffice. If someone encounters an unsupported platform and want to look into it, they will find the comment anyway.

[mojca (Mojca Miklavec)]

I'm not aware of any case of one PortGroup specifying supported_archs i386 x86_64 and another one specifying supported_archs ppc i386 which would mean that the port would only support i386. This variable needs much less intervention in general, just occasionally specifying that the port could not be built 64-bit.

@janstary: there's no need to do any whitelisting. For me it's enough to keep just blacklisting or specifying "broken" state in which case my proposal is basically equivalent to yours, just using a different keyword (blacklist instead of broken). I don't care which keyword exactly is used, but it's nice if it's clear that

whatever-keywoard {darwin >= 13}

doesn't mean that the port will unconditionally build on any given 10.9 or above since another expression in a PortGroup could additionally declare a broken support for another OS version.

[janstary (Jan Starý)]

I like "broken" more than "blacklist" because "blacklist" seems to imply we intentionally don't want it, as opposed to "broken".

At any rate, we should enumerate the bad, not the good; it should be understood that the port just works if it does not say anything.

[jmroot (Joshua Root)]

Ticket retargeted after milestone closed

[jmroot (Joshua Root)]

In 28666188b01ef181a93b36b7468807ae558c5d3b/macports-base (master):

Support version specifiers in platforms option

To use this functionality, add one or more elements to platforms of the
form: platform operator version [operator version] ...

Elements that do not have all three of these components are ignored, so
existing usage like 'platforms darwin' continues to be displayed but
doesn't do anything else.

Each element where the platform matches the current os.platform is
processed by applying each operator to compare os.version with the
corresponding version. Operators supported are ==, !=, >, <, >= and <=.
For == and !=, a string comparison that allows globs is used. For the
rest, vercmp is used.

If any of the comparisons evaluates to false, the default value of
known_fail is set to yes.

Usage example: A port that works on Darwin versions between 16 and 22
inclusive, except for 18.2.x, and doesn't work on Linux:
platforms {darwin >= 16 != 18.2.* < 23} {linux != *}

The special value 'any' can also be used to indicate broad
compatibility of binary archives (not of the port itself). This only
has an effect on the archive name, allowing the same archive to be used
on either all platforms or all versions of one or more platforms.
Example, the same archive will work anywhere:
platforms any

An archive built on darwin will work on any darwin version, but not
on other platforms:
platforms {darwin any}

This can be combined with version specifiers if needed:
platforms {freebsd any} {darwin > 15}

Fixes: #15712