Need to delete 10.6-10.8 archives for ports that used the cxx11 1.1 portgroup and rebuild them

[ryandesign (Ryan Carsten Schmidt)]

At least on 10.8 (e.g. boost-1.71.0_0+no_single+no_static+python27.darwin_12.x86_64), boost is linked with MacPorts libstdc++, so of course programs that use libc++ ​fail to link with it.

I don't understand why gcc got anywhere near this build.

I have seen a similar problem on 10.6-10.8 with graphite2 (e.g. graphite2-1.3.13_0.darwin_12.x86_64.tbz2), and I don't understand why that happened either.

It looks like these archives were not identified for deletion and rebuild during our switchover to libc++. Josh, do you know why?

[jmroot (Joshua Root)]

In the metadata of the graphite2 archive it has cxx_stdlib_overridden 1. So apparently something changed configure.cxx_stdlib to something different to the global cxx_stdlib at the portfile level when it was built. Beyond that no idea sorry.

[ryandesign (Ryan Carsten Schmidt)]

Looks like the cxx11 1.1 portgroup sets cxx_stdlib to macports-libstdc++ if it was libstdc++. The portgroup has done this ever since it was added; I don't know why. Marcus, do you remember?

I think I should identify and remove all 10.6-10.8 binaries for ports that override cxx_stdlib, and rebuild them. Or, restrict it to only those that override it to macports-libstdc++, if that information is available in the metadata.

[jmroot (Joshua Root)]

Overriding cxx_stdlib is a perfectly valid thing to do in some cases and rebuilding ports that still do it won't change anything. Just removing the 10.6-10.8 archives for all ports that used the cxx11 portgroup at the time of the changeover and haven't been updated since should be fine.

[ryandesign (Ryan Carsten Schmidt)]

Replying to jmroot:

Overriding cxx_stdlib is a perfectly valid thing to do in some cases and rebuilding ports that still do it won't change anything.

I know, I just didn't think of another way to identify the ones that need rebuilds.

Just removing the 10.6-10.8 archives for all ports that use the cxx11 portgroup and haven't been updated since the changeover should be fine.

That's a better way, except I'll have to use an old checkout of ports to identify those that used the portgroup at the time of the changeover, since several ports have had that portgroup removed since then without a revbump that would have rebuilt them.

[jmroot (Joshua Root)]

Right. And then delete the corresponding archives if their mtime is before the changeover.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

Looks like the cxx11 1.1 portgroup sets cxx_stdlib to macports-libstdc++ if it was libstdc++. The portgroup has done this ever since it was added; I don't know why. Marcus, do you remember?

notEnoughCoffeeErr: Obviously it's because macOS libstdc++ doesn't support C++11 so MacPorts libstdc++ has to be used.

[mojca (Mojca Miklavec)]

I tried to compile boost with the following patch:

devel/boost/Portfile

@@ -1 +1 @@
 # -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:filetype=tcl:et:sw=4:ts=4:sts=4
 
 PortSystem      1.0
-PortGroup       compiler_blacklist_versions 1.0
 PortGroup       mpi 1.0
-PortGroup       cxx11 1.1
 PortGroup       active_variants 1.1
+compiler.cxx_standard   2011
+configure.cxxflags-append -std=c++11
 
 name            boost
 
@@ -104 +104 @@
 # Further: Building Boost using C++11 compliance does not seem to then
 # require ports depending on Boost to also require C++11 compliance,
 # and requiring it does make such building easier for those ports.
-configure.cxxflags-append -std=gnu++11
+#configure.cxxflags-append -std=gnu++11
 
 # It turns out that ccache and distcc can produce boost libraries that, although they
 # compile without warning, have all sorts of runtime errors especially with pointer corruption.

but it fails to build due to some errors with (potentially misidentified?) thread-local support:

    "/opt/local/bin/clang++-mp-9.0" -dynamiclib -Wl,-single_module -install_name "/opt/local/lib/libboost_context-mt.dylib"  -o "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/libboost_context-mt.dylib" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/asm/make_x86_64_sysv_macho_gas.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/asm/jump_x86_64_sysv_macho_gas.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/asm/ontop_x86_64_sysv_macho_gas.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/execution_context.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/posix/stack_traits.o"      -headerpad_max_install_names -fPIC -m64 -fvisibility=hidden -fvisibility-inlines-hidden -Wl,-dead_strip -no_dead_strip_inits_and_terms -L/opt/local/lib -Wl,-headerpad_max_install_names -stdlib=libc++ -arch x86_64 

Undefined symbols for architecture x86_64:
  "___cxa_thread_atexit", referenced from:
      boost::context::detail::ecv1_activation_record::current() in execution_context.o
      boost::context::detail::ecv1_activation_record_initializer::ecv1_activation_record_initializer() in execution_context.o
      boost::context::detail::ecv1_activation_record_initializer::ecv1_activation_record_initializer() in execution_context.o
      boost::context::detail::ecv1_activation_record_initializer::~ecv1_activation_record_initializer() in execution_context.o
      boost::context::detail::ecv1_activation_record_initializer::~ecv1_activation_record_initializer() in execution_context.o
      boost::context::v1::execution_context::current() in execution_context.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
...failed darwin.link.dll bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/libboost_context-mt.dylib...
...skipped

[jmroot (Joshua Root)]

Even though the compiler may support TLS, libcxx probably doesn't.

[kencu (Ken)]

Mojca is the first to run into this 58898

[cjones051073 (Chris Jones)]

why is everyone removing me from the cc list ??

[kencu (Ken)]

simultaneous posting to the ticket

[cjones051073 (Chris Jones)]

well when that happens you get a warning.... Its best I find to not ignore that warning and refresh before posting (e.g. copy the message and then retype it again).

[kencu (Ken)]

Apologies. Everything harder on this ipad (an also I often miss spelling errors as the writing is so small). I'll try to do better!

[cjones051073 (Chris Jones)]

no apologies needed, I was just saying what works for me.

(I mostly blame trac here. I'm not a fan of its UI.... Its pretty backward I would say that it cannot properly handle simultaneous posts without losing information ).

[mojca (Mojca Miklavec)]

I just wanted to thank you for the quick hint. I was able to compile boost with the above mentioned patches by just running sudo port install libcxx again.

This also means that now liblas works with the above patch for boost (so far reported broken by our buildbots).

[ryandesign (Ryan Carsten Schmidt)]

This ticket is about the need to identify the ports that used the cxx11 1.1 portgroup at the time of the MacPorts 2.6.0 release; uninstall them from the 10.6-10.8 builders; delete the archives from the server; delete failcache entries for any ports that depend on them; and schedule builds to rebuild them.

Any discussion of proposed modifications to boost or another port need to go elsewhere.

[mojca (Mojca Miklavec)]

It might also generally be about all the ports that still use the cxx11 1.1 PortGroup which needs to be removed, else the removal itself won't really help. As these ports need to be modified, I suspect that we need to revbump them (and dependencies) anyway, since users do need to upgrade them on their machine (unless revupgrade will do that?). I'm not sure about dependencies. The one dependency I tried failed to build due to stdlib mismatch anyway.

I'm waiting for #58898 to be resolved before being able to change boost / before opening a PR.

[jmroot (Joshua Root)]

The portgroup doesn't need to be removed, though there's also no reason to use it now. The ports don't need to be modified, the archives just need to be deleted and the builds can be forced. Rev-upgrade will take care of any mismatches on users' machines.

[ryandesign (Ryan Carsten Schmidt)]

I downloaded the ​2.6.0 archive of ports, extracted it, ran portindex in it, set up sources.conf to use it as my primary ports tree, and ran these commands in it to find all the portfiles that used the cxx11 1.1 portgroup at the time that MacPorts 2.6.0 was released:

find . -name Portfile \
| xargs grep -E '^[[:space:]]*PortGroup[[:space:]]+cxx11[[:space:]]+1\.1[[:space:]]*$' \
| cut -d / -f 3 \
| tee ~/cxx11-1.1-ports.txt

Output: cxx11-1.1-ports.txt​

Then I used these commands to identify all the subports:

xargs -I % port echo % subportof:% < ~/cxx11-1.1-ports.txt \
| sed -E 's/ +//' \
| tee ~/cxx11-1.1-subports.txt

Output: cxx11-1.1-subports.txt​

I didn't attempt to distinguish which subports used the portgroup. If the portgroup is mentioned at all in a portfile, I'm counting all of its subports as using it. This means some ports will get uninstalled / deleted that didn't need to be. That's fine. They'll get rebuilt.

I copied cxx11-1.1-subports.txt​ to the 10.6-10.8 builders and uninstalled those subports using these commands (run once on each builder):

xargs /opt/local/bin/port -q installed < ~/cxx11-1.1-subports.txt \
| sed -E 's/ \(active\)$//' \
| xargs sudo /opt/local/bin/port -f uninstall \
| tee ~/$(hostname -s)-uninstall.log

Output: snowleopard-i386-uninstall.log​ snowleopard-uninstall.log​ lion-uninstall.log​ mountainlion-uninstall.log​

Even ports that had already been updated/fixed after the 2.6.0 release were uninstalled. That's fine. They'll get reinstalled automatically as needed by future builds.

Other ports had failed to build because of this problem with these cxx11 ports, so I deleted the entire failcache on the 10.6-10.8 builders. This seemed like the simplest way to ensure no builds are skipped because of this problem.

I copied cxx11-1.1-subports.txt​ to the packages server and deleted the public and private archives for 10.6-10.8 older than the 2.6.0 release date using these commands (run once in each directory):

# 2.6.0 release date as mmddHHMMccyy (UTC)
release_date=$(date -ju 092102492019 +%s)
while read p; do
  for f in $p/*.darwin_1[012].*.tbz2; do
    if [[ $f != *.noarch.tbz2 && -f $f ]]; then
      mod_date=$(stat -f %m $f)
      if [ $mod_date -lt $release_date ]; then
        echo $f
        echo $f.rmd160
      fi
    fi
  done
done < ~/cxx11-1.1-subports.txt \
| xargs rm -fv \
| tee ~/delete-$(basename $PWD).log

Output: delete-packages.log​ delete-packages-private.log​

I'm now waiting for the deletion of the packages to sync to the public server. After that, I'll purge the MaxCDN cache for the packages zone, reactivate the 10.6-10.8 builders, and schedule new builds of cxx11-1.1-ports.txt​ on the 10.6-10.8 builders. And that should take care of this ticket.

[mojca (Mojca Miklavec)]

Thank you very much!

Out of curiosity: what's the status of the 10.5/ppc and 10.6/i386 builders?

[ryandesign (Ryan Carsten Schmidt)]

Replying to mojca:

what's the status of the 10.5/ppc and 10.6/i386 builders?

10.6 i386 is now online along with 10.6-10.8 x86_64. I was bringing them online one at a time to make sure things were working.

10.5 ppc remains offline due to a failed power supply and the power supply is the most ridiculously hard component of a G5 to remove/replace. I have a replacement G5 that I'd like to bring online, but it needs to be vigorously cleaned first.

I'll wait to reschedule the cxx11-1.1-ports.txt​​ ports until the queues are clear. I think that will probably help minimize any problems caused by #59286 / #53587 / #59287.

[mojca (Mojca Miklavec)]

I'll try to speed up my BB 2 setup which should hopefully address most of the issues in the tickets you linked (I really hope that the last two won't even happen).

[ryandesign (Ryan Carsten Schmidt)]

All of the builds have been rescheduled.

[jmroot (Joshua Root)]

Great; thanks Ryan.