#59217 closed defect (fixed)
Need to delete 10.6-10.8 archives for ports that used the cxx11 1.1 portgroup and rebuild them
Reported by: | ryandesign (Ryan Carsten Schmidt) | Owned by: | ryandesign (Ryan Carsten Schmidt) |
---|---|---|---|
Priority: | Normal | Milestone: | |
Component: | server/hosting | Version: | 2.6.1 |
Keywords: | snowleopard lion mountainlion | Cc: | michaelld (Michael Dickens), jmroot (Joshua Root), MarcusCalhoun-Lopez (Marcus Calhoun-Lopez), mojca (Mojca Miklavec), cjones051073 (Chris Jones) |
Port: |
Description
At least on 10.8 (e.g. boost-1.71.0_0+no_single+no_static+python27.darwin_12.x86_64), boost is linked with MacPorts libstdc++, so of course programs that use libc++ fail to link with it.
I don't understand why gcc got anywhere near this build.
I have seen a similar problem on 10.6-10.8 with graphite2 (e.g. graphite2-1.3.13_0.darwin_12.x86_64.tbz2), and I don't understand why that happened either.
It looks like these archives were not identified for deletion and rebuild during our switchover to libc++. Josh, do you know why?
Attachments (8)
Change History (36)
comment:1 Changed 5 years ago by jmroot (Joshua Root)
comment:2 follow-up: 6 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Cc: | MarcusCalhoun-Lopez added |
---|
Looks like the cxx11 1.1 portgroup sets cxx_stdlib to macports-libstdc++ if it was libstdc++. The portgroup has done this ever since it was added; I don't know why. Marcus, do you remember?
I think I should identify and remove all 10.6-10.8 binaries for ports that override cxx_stdlib, and rebuild them. Or, restrict it to only those that override it to macports-libstdc++, if that information is available in the metadata.
comment:3 follow-up: 4 Changed 5 years ago by jmroot (Joshua Root)
Overriding cxx_stdlib is a perfectly valid thing to do in some cases and rebuilding ports that still do it won't change anything. Just removing the 10.6-10.8 archives for all ports that used the cxx11 portgroup at the time of the changeover and haven't been updated since should be fine.
comment:4 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Replying to jmroot:
Overriding cxx_stdlib is a perfectly valid thing to do in some cases and rebuilding ports that still do it won't change anything.
I know, I just didn't think of another way to identify the ones that need rebuilds.
Just removing the 10.6-10.8 archives for all ports that use the cxx11 portgroup and haven't been updated since the changeover should be fine.
That's a better way, except I'll have to use an old checkout of ports to identify those that used the portgroup at the time of the changeover, since several ports have had that portgroup removed since then without a revbump that would have rebuilt them.
comment:5 Changed 5 years ago by jmroot (Joshua Root)
Right. And then delete the corresponding archives if their mtime is before the changeover.
comment:6 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Replying to ryandesign:
Looks like the cxx11 1.1 portgroup sets cxx_stdlib to macports-libstdc++ if it was libstdc++. The portgroup has done this ever since it was added; I don't know why. Marcus, do you remember?
notEnoughCoffeeErr: Obviously it's because macOS libstdc++ doesn't support C++11 so MacPorts libstdc++ has to be used.
comment:7 Changed 5 years ago by mojca (Mojca Miklavec)
Cc: | mojca added |
---|
comment:8 Changed 5 years ago by mojca (Mojca Miklavec)
I tried to compile boost with the following patch:
-
devel/boost/Portfile
1 1 # -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:filetype=tcl:et:sw=4:ts=4:sts=4 2 2 3 3 PortSystem 1.0 4 PortGroup compiler_blacklist_versions 1.05 4 PortGroup mpi 1.0 6 PortGroup cxx11 1.17 5 PortGroup active_variants 1.1 6 compiler.cxx_standard 2011 7 configure.cxxflags-append -std=c++11 8 8 9 9 name boost 10 10 … … mpi.setup -gcc 104 104 # Further: Building Boost using C++11 compliance does not seem to then 105 105 # require ports depending on Boost to also require C++11 compliance, 106 106 # and requiring it does make such building easier for those ports. 107 configure.cxxflags-append -std=gnu++11107 #configure.cxxflags-append -std=gnu++11 108 108 109 109 # It turns out that ccache and distcc can produce boost libraries that, although they 110 110 # compile without warning, have all sorts of runtime errors especially with pointer corruption.
but it fails to build due to some errors with (potentially misidentified?) thread-local support:
"/opt/local/bin/clang++-mp-9.0" -dynamiclib -Wl,-single_module -install_name "/opt/local/lib/libboost_context-mt.dylib" -o "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/libboost_context-mt.dylib" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/asm/make_x86_64_sysv_macho_gas.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/asm/jump_x86_64_sysv_macho_gas.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/asm/ontop_x86_64_sysv_macho_gas.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/execution_context.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/posix/stack_traits.o" -headerpad_max_install_names -fPIC -m64 -fvisibility=hidden -fvisibility-inlines-hidden -Wl,-dead_strip -no_dead_strip_inits_and_terms -L/opt/local/lib -Wl,-headerpad_max_install_names -stdlib=libc++ -arch x86_64 Undefined symbols for architecture x86_64: "___cxa_thread_atexit", referenced from: boost::context::detail::ecv1_activation_record::current() in execution_context.o boost::context::detail::ecv1_activation_record_initializer::ecv1_activation_record_initializer() in execution_context.o boost::context::detail::ecv1_activation_record_initializer::ecv1_activation_record_initializer() in execution_context.o boost::context::detail::ecv1_activation_record_initializer::~ecv1_activation_record_initializer() in execution_context.o boost::context::detail::ecv1_activation_record_initializer::~ecv1_activation_record_initializer() in execution_context.o boost::context::v1::execution_context::current() in execution_context.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) ...failed darwin.link.dll bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/libboost_context-mt.dylib... ...skipped
comment:9 Changed 5 years ago by mojca (Mojca Miklavec)
Cc: | kencu cjones051073 added |
---|
comment:10 Changed 5 years ago by jmroot (Joshua Root)
Cc: | kencu cjones051073 removed |
---|
Even though the compiler may support TLS, libcxx probably doesn't.
comment:11 Changed 5 years ago by cjones051073 (Chris Jones)
Cc: | cjones051073 added |
---|
comment:12 Changed 5 years ago by kencu (Ken)
Cc: | cjones051073 removed |
---|
Mojca is the first to run into this 58898
comment:13 Changed 5 years ago by cjones051073 (Chris Jones)
why is everyone removing me from the cc list ??
comment:14 Changed 5 years ago by cjones051073 (Chris Jones)
Cc: | cjones051073 added |
---|
comment:16 Changed 5 years ago by cjones051073 (Chris Jones)
well when that happens you get a warning.... Its best I find to not ignore that warning and refresh before posting (e.g. copy the message and then retype it again).
comment:17 Changed 5 years ago by kencu (Ken)
Apologies. Everything harder on this ipad (and also I often miss spelling errors as the writing is so small). I'll try to do better!
comment:18 Changed 5 years ago by cjones051073 (Chris Jones)
no apologies needed, I was just saying what works for me.
(I mostly blame trac here. I'm not a fan of its UI.... Its pretty backward I would say that it cannot properly handle simultaneous posts without losing information ).
comment:19 Changed 5 years ago by mojca (Mojca Miklavec)
I just wanted to thank you for the quick hint. I was able to compile boost
with the above mentioned patches by just running sudo port install libcxx
again.
This also means that now liblas
works with the above patch for boost (so far reported broken by our buildbots).
comment:20 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Owner: | set to ryandesign |
---|---|
Port: | boost graphite2 removed |
Status: | new → accepted |
Summary: | boost, graphite2, possibly other ports use macports-libstdc++ instead of libc++ → Need to delete 10.6-10.8 archives for ports that used the cxx11 1.1 portgroup and rebuild them |
This ticket is about the need to identify the ports that used the cxx11 1.1 portgroup at the time of the MacPorts 2.6.0 release; uninstall them from the 10.6-10.8 builders; delete the archives from the server; delete failcache entries for any ports that depend on them; and schedule builds to rebuild them.
Any discussion of proposed modifications to boost or another port need to go elsewhere.
comment:21 Changed 5 years ago by mojca (Mojca Miklavec)
It might also generally be about all the ports that still use the cxx11 1.1
PortGroup which needs to be removed, else the removal itself won't really help. As these ports need to be modified, I suspect that we need to revbump them (and dependencies) anyway, since users do need to upgrade them on their machine (unless revupgrade will do that?). I'm not sure about dependencies. The one dependency I tried failed to build due to stdlib mismatch anyway.
I'm waiting for #58898 to be resolved before being able to change boost / before opening a PR.
comment:22 Changed 5 years ago by jmroot (Joshua Root)
The portgroup doesn't need to be removed, though there's also no reason to use it now. The ports don't need to be modified, the archives just need to be deleted and the builds can be forced. Rev-upgrade will take care of any mismatches on users' machines.
Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Attachment: | cxx11-1.1-ports.txt added |
---|
Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Attachment: | cxx11-1.1-subports.txt added |
---|
Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Attachment: | snowleopard-i386-uninstall.log added |
---|
Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Attachment: | snowleopard-uninstall.log added |
---|
Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Attachment: | lion-uninstall.log added |
---|
Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Attachment: | mountainlion-uninstall.log added |
---|
Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Attachment: | delete-packages.log added |
---|
Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Attachment: | delete-packages-private.log added |
---|
comment:23 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Component: | ports → server/hosting |
---|
I downloaded the 2.6.0 archive of ports, extracted it, ran portindex
in it, set up sources.conf to use it as my primary ports tree, and ran these commands in it to find all the portfiles that used the cxx11 1.1 portgroup at the time that MacPorts 2.6.0 was released:
find . -name Portfile \ | xargs grep -E '^[[:space:]]*PortGroup[[:space:]]+cxx11[[:space:]]+1\.1[[:space:]]*$' \ | cut -d / -f 3 \ | tee ~/cxx11-1.1-ports.txt
Output: cxx11-1.1-ports.txt
Then I used these commands to identify all the subports:
xargs -I % port echo % subportof:% < ~/cxx11-1.1-ports.txt \ | sed -E 's/ +//' \ | tee ~/cxx11-1.1-subports.txt
Output: cxx11-1.1-subports.txt
I didn't attempt to distinguish which subports used the portgroup. If the portgroup is mentioned at all in a portfile, I'm counting all of its subports as using it. This means some ports will get uninstalled / deleted that didn't need to be. That's fine. They'll get rebuilt.
I copied cxx11-1.1-subports.txt to the 10.6-10.8 builders and uninstalled those subports using these commands (run once on each builder):
xargs /opt/local/bin/port -q installed < ~/cxx11-1.1-subports.txt \ | sed -E 's/ \(active\)$//' \ | xargs sudo /opt/local/bin/port -f uninstall \ | tee ~/$(hostname -s)-uninstall.log
Output: snowleopard-i386-uninstall.log snowleopard-uninstall.log lion-uninstall.log mountainlion-uninstall.log
Even ports that had already been updated/fixed after the 2.6.0 release were uninstalled. That's fine. They'll get reinstalled automatically as needed by future builds.
Other ports had failed to build because of this problem with these cxx11 ports, so I deleted the entire failcache on the 10.6-10.8 builders. This seemed like the simplest way to ensure no builds are skipped because of this problem.
I copied cxx11-1.1-subports.txt to the packages server and deleted the public and private archives for 10.6-10.8 older than the 2.6.0 release date using these commands (run once in each directory):
# 2.6.0 release date as mmddHHMMccyy (UTC) release_date=$(date -ju 092102492019 +%s) while read p; do for f in $p/*.darwin_1[012].*.tbz2; do if [[ $f != *.noarch.tbz2 && -f $f ]]; then mod_date=$(stat -f %m $f) if [ $mod_date -lt $release_date ]; then echo $f echo $f.rmd160 fi fi done done < ~/cxx11-1.1-subports.txt \ | xargs rm -fv \ | tee ~/delete-$(basename $PWD).log
Output: delete-packages.log delete-packages-private.log
I'm now waiting for the deletion of the packages to sync to the public server. After that, I'll purge the MaxCDN cache for the packages zone, reactivate the 10.6-10.8 builders, and schedule new builds of cxx11-1.1-ports.txt on the 10.6-10.8 builders. And that should take care of this ticket.
comment:24 follow-up: 25 Changed 5 years ago by mojca (Mojca Miklavec)
Thank you very much!
Out of curiosity: what's the status of the 10.5/ppc and 10.6/i386 builders?
comment:25 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Replying to mojca:
what's the status of the 10.5/ppc and 10.6/i386 builders?
10.6 i386 is now online along with 10.6-10.8 x86_64. I was bringing them online one at a time to make sure things were working.
10.5 ppc remains offline due to a failed power supply and the power supply is the most ridiculously hard component of a G5 to remove/replace. I have a replacement G5 that I'd like to bring online, but it needs to be vigorously cleaned first.
I'll wait to reschedule the cxx11-1.1-ports.txt ports until the queues are clear. I think that will probably help minimize any problems caused by #59286 / #53587 / #59287.
comment:26 Changed 5 years ago by mojca (Mojca Miklavec)
I'll try to speed up my BB 2 setup which should hopefully address most of the issues in the tickets you linked (I really hope that the last two won't even happen).
comment:27 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Resolution: | → fixed |
---|---|
Status: | accepted → closed |
All of the builds have been rescheduled.
In the metadata of the graphite2 archive it has
cxx_stdlib_overridden 1
. So apparently something changed configure.cxx_stdlib to something different to the global cxx_stdlib at the portfile level when it was built. Beyond that no idea sorry.