Use separate portindexes for libc++ on older systems

[ryandesign (Ryan Carsten Schmidt)]

All of our portindexes are generated by mprsyncup on a server running a version of macOS that uses libc++ by default. As such, all of the portindexes contain information that applies to systems running libc++. This means the portindexes are inaccurate on default installations of 10.8 and earlier, which use libstdc++ (but are accurate for 10.8 and earlier systems using the configuration in LibcxxOnOlderSystems).

Some ports may wish to offer a newer version of software (which requires C++11) to libc++ users while continuing to offer an older version (which does not require C++11) to libstdc++ users—gnome-online-accounts and libomp are examples. Or a port may need a different compiler, or other different dependencies, depending on the C++ library—rethinkdb is an example.

The result of this is that, on 10.8 and earlier systems with libstdc++ today, port outdated will show that e.g. libomp 5.0.0 is available, but when the user tries to upgrade, nothing happens, because the user already has the latest version of libomp that is compatible with libstdc++.

We need to change portindex, to be able to simulate an alternate C++ standard library; mprsyncup, to generate separate libstdc++ and libc++ portindexes for 10.8 and earlier systems; and port sync, to download the right portindex for the chosen C++ library.

We would need to decide what the parent directory name should be so that we can make equivalent changes in all the relevant places. Currently, we have PortIndex files in the following directories:

$ rsync --no-motd rsync://rsync.macports.org/macports/release/ports/PortIndex*
drwxr-xr-x          4,096 2017/11/24 13:04:06 PortIndex_darwin_10_i386
drwxr-xr-x          4,096 2017/11/24 13:04:06 PortIndex_darwin_11_i386
drwxr-xr-x          4,096 2017/11/24 13:04:03 PortIndex_darwin_12_i386
drwxr-xr-x          4,096 2017/11/24 13:04:04 PortIndex_darwin_13_i386
drwxr-xr-x          4,096 2017/11/24 13:04:06 PortIndex_darwin_14_i386
drwxr-xr-x          4,096 2017/11/24 13:04:06 PortIndex_darwin_15_i386
drwxr-xr-x          4,096 2017/11/24 13:04:06 PortIndex_darwin_16_i386
drwxr-xr-x          4,096 2017/11/24 13:04:03 PortIndex_darwin_17_i386
drwxr-xr-x          4,096 2017/11/24 13:04:03 PortIndex_darwin_8_i386
drwxr-xr-x          4,096 2017/11/24 13:04:06 PortIndex_darwin_8_powerpc
drwxr-xr-x          4,096 2017/11/24 13:04:06 PortIndex_darwin_9_i386
drwxr-xr-x          4,096 2017/11/24 13:04:06 PortIndex_darwin_9_powerpc

It makes the most sense to me to consider the C++ library part of the platform, i.e. "darwin 10" (libstdc++) and "darwin 10 libc++" are two different platforms.

If we agree on that, then we could put older systems' libc++ portindexes into e.g.:

PortIndex_darwin_10_libcxx_i386
PortIndex_darwin_11_libcxx_i386
PortIndex_darwin_12_libcxx_i386
PortIndex_darwin_8_libcxx_i386
PortIndex_darwin_8_libcxx_powerpc
PortIndex_darwin_9_libcxx_i386
PortIndex_darwin_9_libcxx_powerpc

[ryandesign (Ryan Carsten Schmidt)]

We need to be careful that, in simulating another C++ library, we only change the default C++ library, and do not disturb ports that override that default. For example:

• mongodb always uses libc++, even on systems that default to libstdc++
• giza-pp always uses libstdc++, even on systems that default to libc++

[mojca (Mojca Miklavec)]

I would say that your proposal makes perfect sense. Name for portindex is a much less tricky issue than the binary packages.

I would keep the default names at their current values and would change the one for libcxx (even though: doesn't libc++ work as folder name?), according to your proposal, except that I don't know if it makes any sense whatsoever to do libc++ on PPC.

It would potentially make sense to support gcc's libstdc++v3 on PPC and that one could be called

PortIndex_darwin_9_libstdcxx_powerpc

where libstdcxx would stand for libstdc++v3. I don't know if we have any explicit support for that in base though.

[mojca (Mojca Miklavec)]

Modifying the following chunk of code of mprsyncup seems trivial:

PLATFORMS="8_powerpc 8_i386 9_powerpc 9_i386 10_i386 11_i386 12_i386 13_i386 14_i386 15_i386 16_i386 17_i386"
for PLATFORM in $PLATFORMS; do
    INDEX="PortIndex_darwin_${PLATFORM}"
    ${PORTINDEX} -p "macosx_${PLATFORM}" -o "${INDEX}" \
        | ${AWK} '{ print "Updating " idx ":\t" $0 }' idx="$INDEX" \
        | expand -t 40,48,56,64,72,80 &
done

Looking at the sources of portindex:

} elseif {$arg eq "-p"} { # Set platform
    incr i
    set platlist [split [lindex $argv $i] _]
    set os_platform [lindex $platlist 0]
    set os_major [lindex $platlist 1]
    set os_arch [lindex $platlist 2]
    if {$os_platform eq "macosx"} {
        lappend port_options os.subplatform $os_platform os.universal_supported yes
        set os_platform darwin
    }

it kind of looks like a bad idea to try to squeeze _libc++ into 10_i386 as in

portindex -p macosx_10_libcxx_i386

and break indexing. Maybe something like

portindex -p macosx_10_i386 -stdlib libc++

or (less ideal)

portindex -p macosx-libc++_10_i386

would be a better fit?

I also wouldn't really mind having a directory structure like this one:

    PortIndex_darwin_8_i386
    PortIndex_darwin_8_i386/libc++
    PortIndex_darwin_8_powerpc
    PortIndex_darwin_8_powerpc/libstdc++
    PortIndex_darwin_9_i386
    PortIndex_darwin_9_i386/libc++
    PortIndex_darwin_9_powerpc
    PortIndex_darwin_9_powerpc/libstdc++

if that doesn't bring too many additional complications for generating the tarballs or calling selfupdate (one can exclude folders when using rsync).

While at it: why do we have a separate portindex for powerpc, but not for x86_64?

One further question. While writing some Portfiles I remember experiencing problems by not being able to tell the difference between the globally set stdlib and the one that could have come from a PortGroup. I hope that won't be a problem for Portindex, I need to check.

[ryandesign (Ryan Carsten Schmidt)]

Replying to mojca:

I would keep the default names at their current values and would change the one for libcxx (even though: doesn't libc++ work as folder name?)

It would. But elsewhere in MacPorts, we have adopted the convention that "+" becomes "x" so I adopted it here. We did that in port names so that the "+" would not be confused with the syntax for specifying a variant; that reason doesn't apply here. I also wasn't sure if PortIndex files would ever be transferred over http; if so, the "+" would need to be percent-encoded to "%20", and I wasn't sure if that might introduce problems.

according to your proposal, except that I don't know if it makes any sense whatsoever to do libc++ on PPC.

As far as I know, at this time, libc++ doesn't work on PowerPC generally. I think Ken had found a way to make it work with some additional hacking.

It would potentially make sense to support gcc's libstdc++v3 on PPC and that one could be called

PortIndex_darwin_9_libstdcxx_powerpc

where libstdcxx would stand for libstdc++v3. I don't know if we have any explicit support for that in base though.

I don't know; I hadn't heard of that before.

[ryandesign (Ryan Carsten Schmidt)]

Replying to mojca:

Modifying the following chunk of code of mprsyncup seems trivial:

You did see the attached patches, where I already made the modifications?

it kind of looks like a bad idea to try to squeeze _libc++ into 10_i386 as in

portindex -p macosx_10_libcxx_i386

and break indexing.

What do you mean, "break indexing"?

Maybe something like

portindex -p macosx_10_i386 -stdlib libc++

or (less ideal)

portindex -p macosx-libc++_10_i386

would be a better fit?

I did consider and did initially implement a -stdlib flag, before deciding that I liked it better next to the OS version number. The stdlib is no less a part of the PortIndex identification than the OS name, version, or arch; since those three don't have individual flags, I didn't think we should add a separate flag for stdlib.

I also wouldn't really mind having a directory structure like this one:

    PortIndex_darwin_8_i386
    PortIndex_darwin_8_i386/libc++
    PortIndex_darwin_8_powerpc
    PortIndex_darwin_8_powerpc/libstdc++
    PortIndex_darwin_9_i386
    PortIndex_darwin_9_i386/libc++
    PortIndex_darwin_9_powerpc
    PortIndex_darwin_9_powerpc/libstdc++

if that doesn't bring too many additional complications for generating the tarballs or calling selfupdate (one can exclude folders when using rsync).

I had not considered this. But again, I see the stdlib as no less important to the description of the PortIndex than the other three components, so I would not demote one to a subdirectory.

While at it: why do we have a separate portindex for powerpc, but not for x86_64?

"x86_64" is not a valid value for os.arch. The only valid values are powerpc (meaning any 32-bit or 64-bit PowerPC processor) and i386 (meaning any 32-bit or 64-bit Intel processor).

One further question. While writing some Portfiles I remember experiencing problems by not being able to tell the difference between the globally set stdlib and the one that could have come from a PortGroup. I hope that won't be a problem for Portindex, I need to check.

If I remember correctly, there is cxx_stdlib (the value set in macports.conf; ports and portgroups should not use this) and configure.cxx_stdlib (which defaults to cxx_stdlib but which ports and portgroups can override as needed). My patches have portindex change the value of cxx_stdlib when so requested, and ports like those mentioned in comment:1 that want to override it still can.

[mojca (Mojca Miklavec)]

I'm sorry, I totally missed the patches. You could potentially open a Pull request against base to simplify some further testing, but it would be awesome if these patches were actually deployed, soon.

Regarding the usage of variable in the port: in the past (before we had cxx11 1.1) I would switch to libc++ for some ports just because that would allow me to compile them and if they had no dependency on another C++11 api, this was never a problem. I found it weird that I had to duplicate the code in the PortGroup just for the sake of compiler blacklisting, but if I only used the cxx 1.0 PortGroup, the port would not compile at all, so I had to duplicate the code anyway. At some point I found a workaround to first change configure.cxx_stdlib and then include the PortGroup as that would trick the PortGroup into believing everything was fine. But I could no longer distinguish whether it was the port that set that variable or if it was set globally. I forgot all the details and I'm not willing to go back to history (this was the case with root6 for sure), but just to give a rough idea.

[ryandesign (Ryan Carsten Schmidt)]

I knew I had talked about this topic before but couldn't find a ticket. Turns out it wasn't a ticket but a mailing list thread:

​https://lists.macports.org/pipermail/macports-dev/2016-August/033324.html

Replying to mojca:

I'm sorry, I totally missed the patches. You could potentially open a Pull request against base to simplify some further testing, but it would be awesome if these patches were actually deployed, soon.

It would be two PRs, one for base and one for infrastructure. For now I would probably rather keep everything together in this ticket. Applying a few patchfiles for testing should be too difficult.

Regarding the usage of variable in the port: in the past (before we had cxx11 1.1) I would switch to libc++ for some ports just because that would allow me to compile them and if they had no dependency on another C++11 api, this was never a problem. I found it weird that I had to duplicate the code in the PortGroup just for the sake of compiler blacklisting, but if I only used the cxx 1.0 PortGroup, the port would not compile at all, so I had to duplicate the code anyway. At some point I found a workaround to first change configure.cxx_stdlib and then include the PortGroup as that would trick the PortGroup into believing everything was fine.

The ability to tell the cxx11-1.* portgroup what cxx_stdlib you want to use shouldn't be considered a hack or a workaround; it's the way it's meant to be used. The fact that you have to set that variable before you include the portgroup is an undesirable API quirk, the same quirk present in the obsolete-1.0 portgroup where you have to set replaced_by before including it. It would be great if those quirks could be fixed.

But I could no longer distinguish whether it was the port that set that variable or if it was set globally.

I'm not sure why you would need to distinguish that. But as I said ${cxx_stdlib} should be the value from macports.conf and ${configure.cxx_stdlib} should be the value that the port wants to use.

I forgot all the details and I'm not willing to go back to history (this was the case with root6 for sure), but just to give a rough idea.

[mojca (Mojca Miklavec)]

Regarding

mprsyncup

@@ -59 +59 @@
 
 # Platforms we generate indexes for. This is intentionally split on
 # whitespace later.
-PLATFORMS="8_powerpc 8_i386 9_powerpc 9_i386 10_i386 11_i386 12_i386 13_i386 14_i386 15_i386 16_i386 17_i386"
+PLATFORMS="8_powerpc 8_libcxx_powerpc 8_i386 8_libcxx_i386 9_powerpc 9_libcxx_powerpc 9_i386 9_libcxx_i386 10_i386 10_libcxx_i386 11_i386 11_libcxx_i386 12_i386 12_libcxx_i386 13_i386 14_i386 15_i386 16_i386 17_i386"
 
 # Sources information:
 BASEURL=https://github.com/macports/macports-base.git

I would leave the libcxx_powerpc platforms out. I find it nearly impossible to use them and the chances to ever distribute the binaries with libc++ are super minimal. If anything, it would make sense to support binaries compiled with gcc7.

[mojca (Mojca Miklavec)]

Ryan, I'm not sure if I know how to properly test this (I'm not 100% sure that I'm in fact patching the correct files).

Here's the Portfile I'm testing

PortSystem          1.0
name                foo
version             1.0
categories          devel
platforms           darwin

if {${os.platform} eq "darwin" && ${os.major} > 10} {
    subport foo-bar {}
}

if {${cxx_stdlib} eq "libc++"} {
    subport foo-libcxx {}
}

The result is

> portindex -p macosx_12_i386 -o PortIndex.12_i386
Creating port index in /path/to/PortIndex.12_i386
Adding port devel/foo
Adding subport foo-bar
Adding subport foo-libcxx

The second subport shouldn't have been there. But again, I might have not tested this correctly.

[ryandesign (Ryan Carsten Schmidt)]

Replying to mojca:

if {${cxx_stdlib} eq "libc++"} {

Test ${configure.cxx_stdlib} instead. ${cxx_stdlib} is the value in macports.conf and does not represent the possibly-modified value the port will actually build with.

[mojca (Mojca Miklavec)]

Still, when I run

portindex -p macosx_12_i386 -o PortIndex.12_i386

on a file containing

subport foo-${os.platform}-${os.major}-${configure.cxx_stdlib}-${cxx_stdlib} {}

then I get

Adding subport foo-darwin-12-libc++-libc++

so either I did not patch my MacPorts correctly (which is quite possible) or the code does not set/use the correct stdlib.

One thing that still slightly bothers me (but I would not worry about it yet) is that we would sometimes need a conditional like

if {c++11 is supported} {
    version 1.2
} else {
    version 0.9
}

Making the decision based on stdlib alone will not work correctly on, say, 10.5/ppc/gcc7 configuration, nor with cxx11 1.1 PortGroup on 10.8/libstdc++.

Lower priority though, nothing to worry about at this very moment.

But I would like to know whether you can reproduce the above behaviour or not.

[mojca (Mojca Miklavec)]

OK, it seems to work ok with

portindex -p macosx_12_libstdcxx_i386

but I guess the idea was to use just macosx_12. The problem is that in the else part of

if {[llength $platlist] > 3} {...}

the cxx_stdlib never gets set to its default value based on OS version, so it stays at libc++ when the script is executed on >= 10.9.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #56155.

[ryandesign (Ryan Carsten Schmidt)]

Replying to mojca:

One thing that still slightly bothers me (but I would not worry about it yet) is that we would sometimes need a conditional like

if {c++11 is supported} {
    version 1.2
} else {
    version 0.9
}

Why does this bother you?

Making the decision based on stdlib alone will not work correctly on, say, 10.5/ppc/gcc7 configuration, nor with cxx11 1.1 PortGroup on 10.8/libstdc++.

Why would it not work? Doesn't it currently work, in the ports that do it?

[ryandesign (Ryan Carsten Schmidt)]

Replying to mojca:

OK, it seems to work ok with

portindex -p macosx_12_libstdcxx_i386

but I guess the idea was to use just macosx_12_i386. The problem is that in the else part of

if {[llength $platlist] > 3} {...}

the cxx_stdlib never gets set to its default value based on OS version, so it stays at libc++ when the script is executed on >= 10.9.

I think you're right. I'm not sure why didn't notice that problem before.

[mojca (Mojca Miklavec)]

It's also not so difficult to fix (I was planning to do it, but kept procrastinating). Once that gets fixed, I'm in favour of committing it soon.

[mojca (Mojca Miklavec)]

I committed some changes to ​https://github.com/macports/macports-base/pull/63

[mojca (Mojca Miklavec)]

Ryan, do you have time to review my changes and test? I believe the changes should be good to go.

[neverpanic (Clemens Lang)]

We want to get ​https://github.com/macports/macports-base/pull/63 into 2.5.0. Once this is done, the ticket can be moved to the 2.6.0 milestone.

[raimue (Rainer Müller)]

I created a pull request with only the changes to portindex: ​https://github.com/macports/macports-base/pull/83

[raimue (Rainer Müller)]

As Mojca ​asked why I did not want to commit the changes to macports.tcl, let me explain that here.

In the proposed changes to macports.tcl, ${platindex_cxxlib} will be added to the remote PortIndex filename. It will only do this if the value of cxx_stdlib does not match the default.

Side note: I do not understand why you would want to hardcode the default cxx_stdlib at this place. I would just always add ${platindex_cxxlib} to the remote filename, isn't this much simpler? Wasn't your plan to eventually change the default on older systems at some point? Hardcoding default values seems to work against that.

I also have reservations because of users that already have a non-default setting for cxx_stdlib. With this change, mportsync would expect a remote filename that does not yet exist. What are the consequences of this? Is the plan to deploy the changes to mprsyncup immediately after the 2.5.0 release without ever testing the changes against rsync? Will mportsync fail gracefully or hard without a remote PortIndex file? Will base fall back to generate a PortIndex locally in this case? I see no reports of tests or consideration of it in this ticket.

The changes to macports.tcl require the changes to mprsyncup, but that requires the changes to portindex.

I can only review the change to portindex as ready-to-go. The new portindex would already fix the problem that PortIndex was generated with the wrong value of cxx_stdlib in mprsyncup for legacy platforms. This seemed to be the most important fix of this pull request to go into a release. I also thought this is what we discussed to do for MacPorts 2.5.0 in our online meeting.

[mojca (Mojca Miklavec)]

Thanks a lot for the explanation. I didn't catch all the details of the discussion during the meeting (and didn't want to ask too much as it was already taking way too much time about this particular ticket).

I'll respond to other questions later, but feel free to proceed with your PR.

The issue won't be solved for users of libc++ on legacy systems, but it's the first step in the right direction and we might indeed want to coordinate a few other steps. (If you always append cxx_stdlib, you'll need a change everywhere, even for 10.13.)

[ryandesign (Ryan Carsten Schmidt)]

Replying to raimue:

Side note: I do not understand why you would want to hardcode the default cxx_stdlib at this place. I would just always add ${platindex_cxxlib} to the remote filename, isn't this much simpler? Wasn't your plan to eventually change the default on older systems at some point? Hardcoding default values seems to work against that.

My intention was to not change the remote Portindex filename for current users, so that existing MacPorts 2.4.x installations would not break.

I think when I made this ticket, there had not yet been an initiative to change the default value of cxx_stdlib for 10.6-10.8, so I did not consider that that might be how we ultimately implement the migration and that it might affect these Portindex filenames.

We could go all out: put the cxx_stdlib into every Portindex filename as you suggest, to be used by future versions of MacPorts, and also hardlink the old Portindex name without cxx_stdlib to what MacPorts 2.4.x expects.

I also have reservations because of users that already have a non-default setting for cxx_stdlib. With this change, mportsync would expect a remote filename that does not yet exist. What are the consequences of this? Is the plan to deploy the changes to mprsyncup immediately after the 2.5.0 release without ever testing the changes against rsync? Will mportsync fail gracefully or hard without a remote PortIndex file? Will base fall back to generate a PortIndex locally in this case? I see no reports of tests or consideration of it in this ticket.

I did not test it but when I made these patches I assumed MacPorts would explode if the remote Portindex does not exist, which is why I created all three patches at once in this ticket and expected the changes to mprsyncup and portindex to be deployed immediately to the server, so that the Portindexes with the new names would be generated and ready to go before users would update to a new version of MacPorts that would use them. I'm happy to manually patch the copy of portindex that mprsyncup uses on the server.

The changes to macports.tcl require the changes to mprsyncup, but that requires the changes to portindex.

I can only review the change to portindex as ready-to-go. The new portindex would already fix the problem that PortIndex was generated with the wrong value of cxx_stdlib in mprsyncup for legacy platforms. This seemed to be the most important fix of this pull request to go into a release.

I just want to make clear again that deploying only this part will fix the portindex for ≤ 10.8 with libstdc++ but break the portindex for any users of ≤ 10.8 already using libc++. I'm ok with that, because it will help more users than it harms, and the breakage is so minor (a handful of ports not being properly displayed in port outdated) that I'm only aware of one ≤ 10.8 libstdc++ user reporting the problem so far. But I want to make sure we're all clear about what will happen if we make this change.

I also thought this is what we discussed to do for MacPorts 2.5.0 in our online meeting.

We did discuss it, but I, like Mojca, wasn't entirely clear on why the three parts of the fix were to be separated from one another. One reason could be if we have not yet agreed that my proposed Portindex filenames are the ones we want to use, since after we release a version of MacPorts that uses them, we don't want to change them again.

[mojca (Mojca Miklavec)]

Rainer, can you please suggest how exactly you would solve this in what you think is ideal case? I would like to avoid spending another year or two with zero activity just because we cannot even decide what to do, while the patch would in fact be super straightforward.

We can of course always add the cxx_stdlib to the name, but then you also need to change selfupdate (or whatever all those places are) to fetch a different PortIndex file from the server.

In any case, can you please merge one of the two PRs and then we proceed with discussion about the changes in remaining two files.

[mojca (Mojca Miklavec)]

Can we please proceed on this? By merging at least one PR, to start with?

[raimue (Rainer Müller)]

I do not know how to proceed here, this needs to be worked out by those interested in legacy systems. I offered the PR against portindex only to make progress towards 2.5.0 as discussed in our online meeting, but I do not want to lead planning of this change.

[neverpanic (Clemens Lang)]

Merging ​https://github.com/macports/macports-base/pull/63 without the changes in mprsyncup will cause systems that do not get a portindex from the server to generate one locally. Systems won't break, but generating a portindex locally takes ~15 minutes on SSDs already and will be even slower on the typical machine still running an old setup. Since the portindex is first generated in the postflight of the installer, this leads to a bad user experience.

Merging ​https://github.com/macports/macports-base/pull/83 will improve correctness of the server-generated portindexes when merged as-is, so we should probably do that. The portindex files without an explicit mention of C++ stdlib in its filename should always contain data for the default on this system, which is what PR 83 achieves. Even if we move to always include the C++ stdlib in the filename for all platforms later on, this change is correct for now.

As such, to move the 2.5.0 release forward, I'm merging PR 83.

[ryandesign (Ryan Carsten Schmidt)]

In d49c4fb8a1b05684bb8a6d0c542b20218b886bc0/macports-base (master):

portindex: support creating PortIndex for libc++

See: #55471

[neverpanic (Clemens Lang)]

Replying to neverpanic:

We want to get ​https://github.com/macports/macports-base/pull/63 into 2.5.0. Once this is done, the ticket can be moved to the 2.6.0 milestone.

Now that PR 83 (which replaces PR 63) has been merged, let's move the ticket to 2.6.0 as discussed.

[jmroot (Joshua Root)]

Ticket retargeted after milestone closed

[jmroot (Joshua Root)]

This was never done and the libc++ transition has happened.