Change filenames of binary packages built against libc++ on < 10.9

[mojca (Mojca Miklavec)]

This ticket is meant for discussion about the implementation of unique filenames for packages built against libc++ on OS X versions prior to 10.9, so that we could eventually set up a buildslave and distribute binaries for that configuration as described in LibcxxOnOlderSystems.

There are three (four) categories of packages:

• noarch
• C++ programs and libraries (that link against libc++ or libstdc++)
  • packages that require C++11 and thus link against libc++ even where default is libstdc++ (like clang-3.X, root6, ... )
• packages that don't require stdlib

My suggestion:

• no changes are needed for noarch (but when setting up a new buildslave we should make sure to only upload packages from one of them)
• packages that link against stdlib
  • I would suggest to make no changes on >= 10.9
  • we should add something like libc++ to the name (or potentially something shorter)
  • while we could add stdlibc++ to the name of default configuration, I don't see any reason to do so as it would only require extra work (unless we start differentiating between package with/without the need for stdlib)
• package that don't link against stdlib
  • we could do a similar analysis as for noarch and differentiate between packages that link against stdlib and those that don't with a special flag; but we need to remember that there might be further differences between the packages built on the old buildslaves and those built on the new one, for example due to delete_la_files setting; my suggestion would be not to worry about differentiating packages that link against stdlib and those that don't, but that's certainly debatable
  • If it's easy enough to implement, we could do it for reasons of "purism". The question is then: should packages contain a similar keyword as noarch? Who would add those flags? If we do something about that, OS X versions >= 10.9 should be affected as well and we should change the names (and either rebuild or rename the packages) there as well.
• packages that link against libc++
  • if we want a quck & dirty solution: don't bother about those special cases (the package would only have libc++ in the name if it was built on the new buildslave and we would still ship two version)
  • if we want to optimize space and only upload the package once, we would have to do some major cleanup on the existing buildbots (due to delete_la_files etc) and most likely ship the files from the new buildbot

Basically we have two ways:

• do it right (quite a bit of effort)
• do it in a quick and dirty way (but make it work faster)

While discussing this it might be worth thinking about potential (simultaneous) switch to *.tar.xz for archives which could save quite some space both for end users as well as on the buildbots / servers with packages.

[michaelld (Michael Dickens)]

Cc Me!

[RJVB (René Bertin)]

I have the impression that more and more software assumes C++11 will work, and pretty soon that will be true also for later C++ flavours. I think that increases the pressure to tackle this issue because motivation to do so will only decrease...

One should probably use a less confusion term than stdlib; either talk about libstdc++ vs libc++ or about the C++ runtime library.

Do I understand correctly that you do not suggest changing the default, at least for OS versions that do not ship libc++ at all? Is there any scenario in which it would be possible (and feasible) to make using libc++ the default when a system libc++ *is* available?

While discussing this it might be worth thinking about potential (simultaneous) switch to *.tar.xz for archives which could save quite some space both for end users as well as on the buildbots / servers with packages.

This has already been discussed on the ML and/or Trac, but: In that case it might be even more beneficial to consider unpacking the software images with --hfsCompression, possibly under optional control of a setting in macports.conf . Xz'ped archives can be considerably smaller than bzip2'ed archives, but HFS compression can reduce the space taken by installed ports by quite a bit more.

[RJVB (René Bertin)]

Speaking of quick and "dirty" ways to implement this kind of change: I think that can be done using existing mechanisms:

• the cxx11 PortGroup could detect the presence of a "system" libc++ on OS versions where this is not the default.
• If that library is present, a variant could be set through default_variants

Supposing the PortGroup already does all the required additional things this should give rise to distinguishable libc++ and libstdc++ packages provided there are ditto slaves.

On user systems this mechanism should have the same effect: after doing the conversion to libc++ any port requiring C++11 will set the libc++ variant by default, and thus request the ditto packages.

What bothers me here is the case of older C++ software. I'm guessing it will continue to work if it only links to system libraries or other ports using older C++, but won't it get into trouble if you try to link it with a library from a port using the C++11 portgroup?

In other words, shouldn't the discussion focus on ports using C++, or at least on some provision (through a portgroup at first) to toggle the default for configure.cxx_stdlib persistently when the first port requiring libc++ gets installed? (Sic, *installed*, ideally ...)

[mojca (Mojca Miklavec)]

I have absolutely no experience about hfs, so I cannot comment on that.

Please also note that I opened this ticket before we had the MacPorts meeting. At the meeting we proposed the following solution:

• Don't bother about the situation with noarch, about whether the port requires C++ runtime at all and whether the port can be built against libstdc++ at all (C++11 ports cannot and some already switch to libc++ even under default settings): getting it right is way too complex to handle in any easy way. (My personal addition: Debian splits all of their packages into arch part and the generic part and then ships files of the generic part just once. We could probably even do that for most of the ports and ship generic parts for all OSes, all architectures and both configurations (libc++ vs. libstdc++), but then we always have a zillion exceptions where one port version doesn't build on the older OS and another version doesn't build on a newer OS, so this would inevitably cause all kinds of problems anyway.)
• My personal suggestion was to switch to .tar.xz and ship .tar.xz for libc++ (on < 10.9) and keep distributing .tar.bz2 for existing default setups (and later gradually switch everything else to .tar.xz as well). Then we would have no name clashes at all. But most other devs didn't like this idea at all, saying that we should not mix the extension/compression format with the rest.
• We proposed to put new binaries under a new prefix. That is, put binaries to:

  http://packages.macports.org/10.5/ppc
  http://packages.macports.org/10.5/i386
  http://packages.macports.org/10.6/i386/
  http://packages.macports.org/10.6/x86_64/
  http://packages.macports.org/10.7/x86_64/
  http://packages.macports.org/10.8/x86_64/
  

or perhaps darwin9 to darwin12 (rather than OS version numbers), architecture is also not required.

The (most likely) reason why this hasn't been implemented yet is because we currently have absolutely no overview/summary of successful and failed jobs on the buildbot. At the moment you can look at http://packages.macports.org/<packagename> and check quickly whether a particular port is missing on a particular OS version without having to implement the new (but sorely missing) functionality.

We argued that the lack of functionality on our website shouldn't be the reason for keeping the "poor man's method" around and denying support to users of old OSes.

Surely there are other problems. This still doesn't provide any means to support installations with non-default prefixes, it doesn't tell anything about other user settings etc.

PS: and of course 10.5 PPC comes with its own set of problems, starting with the fact that clang doesn't work properly there, so there are now "movements" to allow building C++11 software by setting gcc6 (or earlier) as the default compiler, complicating things even further.

[RJVB (René Bertin)]

Well, yes, PPC is now sufficiently old almost to warrant forking MacPorts completely and trusting it to maintenance by the (dwindling number of?) users of such systems ...

And to be honest, the same could almost be said for sufficiently old Intel OS versions like those that don't ship libc++ .

I don't see the problem with non-default prefixes. Those have never had the benefit of using prebuilt packages, but as long as ports use ${prefix} the way it's intended to be used there shouldn't be a problem building them regardless of the what C++ runtime is used, no?

But most other devs didn't like this idea at all, saying that we should not mix the extension/compression format with the rest.

For once I'm with "most other devs" ... Sounds way too delicate to get right. How for instance would you handle user set-ups that are configured to use xz for compressing software images built locally?

[kencu (Ken)]

Another instantly doable option (that I would probably prefer) would be to just insist people set up older systems (10.6+) with LibcxxOnOlderSystems as the only supported macports configuration going forward, and only host libc++ linked binaries on the buildbots.

Then nothing needs to change at all, except the macports.conf stdlib line on the buildbots.

Just have to think for a second about bootstrapping these systems...but I think that would just happen automatically from the buildbots ...

[mojca (Mojca Miklavec)]

We cannot do that because people would then end up with broken installations without any warning (and many of them probably switch to another package manager after bad experience).

We could stop providing packages for libstdc++ installations, but not replace them with incompatible ones and break everyone's computer.

[RJVB (René Bertin)]

Exactly.

[kencu (Ken)]

Of course we could do it. Just update the buildbots to libc++ - all at once - and instruct the few remaining people using these systems to change over to libc++ (or make it the default in base). No worse than the usual significant upgrade. Once the buildbots are updated to libc++-linked binaries, the upgrade would be fast and essentially effortless. It's just a chore now because of the three-day builds on these older systems (which is why we're even talking about libstdc++ in libgcc in the first place).

Done. Never look back. They won't go anywhere, I don't think. Homebrew, far as I can tell, has poor support for these systems. And as everyone points out, more and more software requires cxx11+ anyway, so they are going to move to a cxx11-supporting one way or another. Might as well be the right one.

[kencu (Ken)]

Better that than start force-linking to libgcc's libstdc++ on the current old build systems, which is the other option on the table that has a chance of moving forward.

That might be interesting, but in the end, will wind up causing troubles as ports try to start building against that standard library instead of libc++ -- and run into the inevitable errors doing so -- and maintainers lose enthusiasm for trying to support two standard library systems.

I think that road leads to dropping support for <10.9 completely, (we already hear that quite often), which is way worse than moving to libc++.

[neverpanic (Clemens Lang)]

We should have a resolution for this for MacPorts 2.5.0. That solution may be a decision to not rename at all and just change the default C++ stdlib of the existing archives at some cutoff date.

[ryandesign (Ryan Carsten Schmidt)]

Well, 2.5.0 will have an enhanced rev-upgrade that can detect cxx_stdlib mismatches. Some time after everybody has upgraded to 2.5.0, we can release a new version that switches the default to libc++, and throw out the old libstdc++ archives on the packages server and build new libc++ ones.

[RJVB (René Bertin)]

maintainers lose enthusiasm for trying to support two standard library systems.

I'd use the term "motivation" for myself ;) and more for maintaining support for legacy OS versions for which my own OS version isn't representative. Supporting 2 different C++ runtimes on those older systems only makes things worse.

There's another solution which requires less hacking: branch off port trees for legacy systems, and incite port maintainers to move the latest version of their port that support a particular OS X version to the corresponding tree.

So if 10.y were to be the latest version supported officially, users of an earlier versions 10.x would have a 10.x tree that they'd install as a default tree ahead of the main tree. For example, port:audacity could provide 2.1.3 in the 10.6 through 10.8 trees (last version to build without C++11) or to use Qt5 as an example in a more general context: port:qt5 would be providing Qt 5.8 in the 10.9 tree and 5.10 in the 10.10 tree (instead of having a bunch of qt5x ports in addition to port:qt5).

Just mentioning it because I realise this will come with more administrative work, but that could in large part be supported by the users of those systems. And it's also something that should be required only once, in principle.

[ryandesign (Ryan Carsten Schmidt)]

Replying to RJVB:

branch off port trees for legacy systems

I don't want to discuss that suggestion in this ticket.

[RJVB (René Bertin)]

How is that more constructive than simply ignoring my comment?

[ryandesign (Ryan Carsten Schmidt)]

Not responding to your comment might lead others to respond to it here. It is not related to this ticket so please discuss it on the mailing list instead.

[jmroot (Joshua Root)]

Ticket retargeted after milestone closed

[jmroot (Joshua Root)]

This was never done and the libc++ transition has happened.