clang-11-bootstrap links opportunistically against libxml2

[cave-canem]

I also tried to build libxml2 with 'clang-11-bootstrap @11.1.0_3' add the following lines to the Portfile:

  depends_build-append    port:clang-11-bootstrap
  configure.cc            ${prefix}/libexec/clang-11-bootstrap/bin/clang
  configure.cxx           ${prefix}/libexec/clang-11-bootstrap/bin/clang++

but it didn't work.

sudo port activate icu @72.1_0

port installed active and icu
The following ports are currently installed:
  icu @72.1_0 (active)

sudo port -vd upgrade -n --force libxml2
...

checking for gcc... /opt/MacPorts/libexec/clang-11-bootstrap/bin/clang
checking whether the C compiler works... no
configure: error: in `/opt/MacPorts/var/macports/build/_opt_macports-ports_textproc_libxml2/libxml2/work/libxml2-2.10.3':
configure: error: C compiler cannot create executables
See `config.log' for more details
Command failed:  cd "/opt/MacPorts/var/macports/build/_opt_macports-ports_textproc_libxml2/libxml2/work/libxml2-2.10.3" && ./configure --prefix=/opt/MacPorts --disable-silent-rules --enable-static --with-ftp --with-icu --without-python
Exit code: 77
Error: Failed to configure libxml2: consult /opt/MacPorts/var/macports/build/_opt_macports-ports_textproc_libxml2/libxml2/work/libxml2-2.10.3/config.log
Error: Failed to configure libxml2: configure failure: command execution failed
DEBUG: Error code: NONE
DEBUG: Backtrace: configure failure: command execution failed
    while executing
"$procedure $targetname"
Error: See /opt/MacPorts/var/macports/logs/_opt_macports-ports_textproc_libxml2/libxml2/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there is a bug.

But, although clang-11-bootstrap supposedly does not depend on libxml2, I see with my own eyes:

/opt/MacPorts/libexec/clang-11-bootstrap/bin/clang ./hello.c -o ./hello
dyld: Library not loaded: /opt/MacPorts/lib/libicui18n.67.dylib
  Referenced from: /opt/MacPorts/lib/libxml2.2.dylib
  Reason: image not found
clang-11: error: unable to execute command: Trace/BPT trap: 5
clang-11: error: linker command failed due to signal (use -v to see invocation)

[jmroot (Joshua Root)]

Why wouldn't it just build with the system clang?

[kencu (Ken)]

clang will find and link against macports libxml2 if it is installed.

Therefore it was listed as a dep for most clang ports.

But macports libxml2 is not really needed, as the system libxml2 works 100% fine.

I forced clang to ignore MacPorts libxml2 in some of the (bootstrappy) clangs:

​https://github.com/macports/macports-ports/blob/c2e71cacbad12f2c5664f3330d7e9ec9078eba29/lang/llvm-5.0/Portfile#L344

But it appears this didn't get carried into all new clangs, etc; probably should be.

[cave-canem]

Joshua Root, see:

sudo port -vd upgrade -n --force libxml2
...
DEBUG: Environment:
CC='/usr/bin/clang'
CC_PRINT_OPTIONS='YES'
CC_PRINT_OPTIONS_FILE='/opt/MacPorts/var/macports/build/_opt_macports-ports_textproc_libxml2/libxml2/work/.CC_PRINT_OPTIONS'
CFLAGS='-pipe -Os -arch x86_64'
CPATH='/opt/MacPorts/include'
CPPFLAGS='-I/opt/MacPorts/include'
CXX='/usr/bin/clang++'
CXXFLAGS='-pipe -Os -stdlib=libc++ -arch x86_64'
DEVELOPER_DIR='/Library/Developer/CommandLineTools'
F90FLAGS='-pipe -Os -m64'
FCFLAGS='-pipe -Os -m64'
FFLAGS='-pipe -Os -m64'
INSTALL='/usr/bin/install -c'
LDFLAGS='-L/opt/MacPorts/lib -Wl,-headerpad_max_install_names -arch x86_64'
LIBRARY_PATH='/opt/MacPorts/lib'
MACOSX_DEPLOYMENT_TARGET='10.9'
OBJC='/usr/bin/clang'
OBJCFLAGS='-pipe -Os -arch x86_64'
OBJCXX='/usr/bin/clang++'
OBJCXXFLAGS='-pipe -Os -stdlib=libc++ -arch x86_64'
Executing:  cd "/opt/MacPorts/var/macports/build/_opt_macports-ports_textproc_libxml2/libxml2/work/libxml2-2.10.3" && ./configure --prefix=/opt/MacPorts --disable-silent-rules --enable-static --with-ftp --with-icu --without-python
DEBUG: system:  cd "/opt/MacPorts/var/macports/build/_opt_macports-ports_textproc_libxml2/libxml2/work/libxml2-2.10.3" && ./configure --prefix=/opt/MacPorts --disable-silent-rules --enable-static --with-ftp --with-icu --without-python
checking build system type... x86_64-apple-darwin13.4.0
checking host system type... x86_64-apple-darwin13.4.0
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /opt/MacPorts/bin/gmkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether to enable maintainer-specific portions of Makefiles... yes
checking whether make supports nested variables... (cached) yes

checking for gcc... /usr/bin/clang

checking whether the C compiler works... no

configure: error: in `/opt/MacPorts/var/macports/build/_opt_macports-ports_textproc_libxml2/libxml2/work/libxml2-2.10.3':
configure: error: C compiler cannot create executables
See `config.log' for more details
Command failed:  cd "/opt/MacPorts/var/macports/build/_opt_macports-ports_textproc_libxml2/libxml2/work/libxml2-2.10.3" && ./configure --prefix=/opt/MacPorts --disable-silent-rules --enable-static --with-ftp --with-icu --without-python
Exit code: 77
Error: Failed to configure libxml2: consult /opt/MacPorts/var/macports/build/_opt_macports-ports_textproc_libxml2/libxml2/work/libxml2-2.10.3/config.log
Error: Failed to configure libxml2: configure failure: command execution failed
DEBUG: Error code: NONE
DEBUG: Backtrace: configure failure: command execution failed
    while executing
"$procedure $targetname"
Error: See /opt/MacPorts/var/macports/logs/_opt_macports-ports_textproc_libxml2/libxml2/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there is a bug.

[jmroot (Joshua Root)]

Well it worked on the buildbot. The config.log will have details about what went wrong with yours.

[cave-canem]

Hello Ken,

In fact, clang-5.0 depends on llvm-5.0, llvm-5.0 depends on xar, while xar depends on libxml2, and thus everything does not work again.

It can still be done with clang-11-bootstrap, but libxml2 must be deactivated first (I did not do this at first, and this was my error).

The ticket can be closed as the problem has been solved (in my case).

To avoid this situation, in perhaps makes sense to adjust the port in such a way that libxml2 would be deactivated forcibly and clang-11-bootstrap would be used forcibly.

[kencu (Ken)]

clang-11-bootstrap could be changed as per what I did in llvm/clang-5.0 to ignore macports libxml2, rather than making folks deactivate libxml2 to build it cleanly, if it is truly being picked up during the build.

[kencu (Ken)]

I believe what happened here is clang-11-bootstrap picked up libxml2 during the build, even though the environment is sterilized.

This would have to be verified with a test build of clang-11-bootstrap while libxml2 is installed to see if that happens, as we have not confirmed that yet.

[catap (Kirill A. Korinsky)]

Ken, I don't think that clang-11-bootstrap is linked against libxml:

catap@Mac-mini ~ % port installed
The following ports are currently installed:
  cmake-bootstrap @3.9.6_0 (active)
  gettext-runtime @0.21_0 (active)
  icu @72.1_0 (active)
  libiconv @1.17_0 (active)
  libxml2 @2.10.3_1 (active)
  python27-bootstrap @2.7.18_10 (active)
  xz @5.2.7_0 (active)
  xz-bootstrap @5.2.7_0 (active)
  zlib @1.2.13_0 (active)
catap@Mac-mini ~ % sudo port -s install clang-11-bootstrap
--->  Computing dependencies for clang-11-bootstrap
--->  Extracting clang-11-bootstrap
--->  Applying patches to clang-11-bootstrap
--->  Configuring clang-11-bootstrap
--->  Building clang-11-bootstrap                        
--->  Staging clang-11-bootstrap into destroot           
--->  Installing clang-11-bootstrap @11.1.0_3+universal  
--->  Activating clang-11-bootstrap @11.1.0_3+universal
--->  Cleaning clang-11-bootstrap
--->  Scanning binaries for linking errors
--->  No broken files found.
--->  No broken ports found.
--->  Some of the ports you installed have notes:
  clang-11-bootstrap has the following notes:
    To use this bootstrap version of clang instead of the default compiler, add the  following lines to the Portfile:
    
    depends_build-append    port:clang-11-bootstrap
    configure.cc            ${prefix}/libexec/clang-11-bootstrap/bin/clang
    configure.cxx           ${prefix}/libexec/clang-11-bootstrap/bin/clang++
catap@Mac-mini ~ % sudo port deactivate libxml2
--->  Deactivating libxml2 @2.10.3_1
--->  Cleaning libxml2
catap@Mac-mini ~ % sudo port rev-upgrade
--->  Scanning binaries for linking errors
--->  No broken files found.
--->  No broken ports found.
catap@Mac-mini ~ % 

[kencu (Ken)]

Ok.

we’re trying to guess how this happened to this fellow:

opt/MacPorts/libexec/clang-11-bootstrap/bin/clang ./hello.c -o ./hello
dyld: Library not loaded: /opt/MacPorts/lib/libicui18n.67.dylib
  Referenced from: /opt/MacPorts/lib/libxml2.2.dylib
  Reason: image not found

but perhaps he just force- installed the wrong icu or something.

We’ll close this then, and see if it turns up on anyone else ‘s system, as the issue seems gone on this system.

[catap (Kirill A. Korinsky)]

Anyway, it I still would like to see otool -L /opt/MacPorts/libexec/clang-11-bootstrap/bin/clang and /opt/MacPorts/libexec/clang-11-bootstrap/bin/clang -v ./hello.c -o ./hello

What might explain a bit that's going on here.

[kencu (Ken)]

me too, but can’t anymore, as problem is gone

[cave-canem]

Hello Ken, Hello Kirill:

cat ./hello.c
#include <unistd.h>
#include <string.h>
#include <errno.h>

int main()
{
    const char * const msg = "Hello World!\n";
    const char * begin = msg;
    const char * const end = begin + strlen(msg);
   
    while (begin < end)
    {
        size_t remaining = end - begin;
        ssize_t res = write(STDOUT_FILENO, begin, remaining);
        if (res >= 0)
        {
            begin += res;
            continue; // Let's send the remaining part of this message
        }
        if (EINTR == errno)
        {
            continue; // It's just a signal, try again
        }
        return 1; // It's a real error
    }
    return 0; // OK! Let's celebrate and drink some beer!
}

otool -L /opt/MacPorts/libexec/clang-11-bootstrap/bin/clang
/opt/MacPorts/libexec/clang-11-bootstrap/bin/clang:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

/opt/MacPorts/libexec/clang-11-bootstrap/bin/clang -v ./hello.c -o ./hello
clang version 11.1.0
Target: x86_64-apple-darwin13.4.0
Thread model: posix
InstalledDir: /opt/MacPorts/libexec/clang-11-bootstrap/bin
 "/opt/MacPorts/libexec/clang-11-bootstrap/bin/clang-11" -cc1 -triple x86_64-apple-macosx10.9.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name hello.c -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-rounding-math -munwind-tables -faligned-alloc-unavailable -fcompatibility-qualified-id-block-type-checking -target-cpu core2 -debugger-tuning=lldb -target-linker-version 450.3 -v -resource-dir /opt/MacPorts/libexec/clang-11-bootstrap/lib/clang/11.1.0 -internal-isystem /usr/local/include -internal-isystem /opt/MacPorts/libexec/clang-11-bootstrap/lib/clang/11.1.0/include -internal-externc-isystem /usr/include -fdebug-compilation-dir /Volumes/WORK/WORK -ferror-limit 19 -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fmax-type-align=16 -fcolor-diagnostics -o /var/folders/j2/whh7d9lj5mvfvtn9yc3dsxt00000gn/T/hello-d9d3ea.o -x c ./hello.c
clang -cc1 version 11.1.0 based upon LLVM 11.1.0 default target x86_64-apple-darwin13.4.0
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /opt/MacPorts/libexec/clang-11-bootstrap/lib/clang/11.1.0/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.
 "/opt/MacPorts/bin/ld" -demangle -lto_library /opt/MacPorts/libexec/clang-11-bootstrap/lib/libLTO.dylib -no_deduplicate -dynamic -arch x86_64 -macosx_version_min 10.9.0 -o ./hello /var/folders/j2/whh7d9lj5mvfvtn9yc3dsxt00000gn/T/hello-d9d3ea.o -lSystem /opt/MacPorts/libexec/clang-11-bootstrap/lib/clang/11.1.0/lib/darwin/libclang_rt.osx.a

[catap (Kirill A. Korinsky)]

Am I wrong but it works as expected?

[cave-canem]

Yes, clang-11-bootstrap works great.

The problem was libxml2, which had to be forced deactivated for a rebuild libxml2 when icu changed ( icu @67.1_4 < icu @72.1 )

All compilers (except clang-11-bootstrap) depend on libxml2 (including clang-5.0, which Ken wrote about) and we get ouroboros (οὐροβόρος).

[kencu (Ken)]

of course, libxml2 is in the clang dependency group:

PortGroup clang_dependency 1.0

so exactly how you got yourself into this situation remains a complete mystery to us all. Must have been some forcing of things.

But nevertheless, doesn't matter, you are fixed up and nobody else sees this issue!

[cave-canem]

No Ken, others also see this issue.

For example (quote):

Artemio González López via macports-users macports-users@… ( 10.11.22, 20:52 )

artemiog@…

...

checking for gcc... /opt/local/bin/clang-mp-14

checking whether the C compiler works... no

configure: error: in `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_textproc_libxml2/libxml2/work/libxml2-2.10.3':

configure: error: C compiler cannot create executables

...

Library not loaded: /opt/local/lib/libicui18n.71.dylib

Referenced from: <45F9A0DF-B06D-34C8-946F-88FFA574E722> /opt/local/lib/libxml2.2.dylib

Reason: tried: '/opt/local/lib/libicui18n.71.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/local/lib/libicui18n.71.dylib' (no such file), '/opt/local/lib/libicui18n.71.dylib' (no such file), '/usr/local/lib/libicui18n.71.dylib' (no such file), '/usr/lib/libicui18n.71.dylib' (no such file, not in dyld cache)

[kencu (Ken)]

libxml2 is in the clang-dependency PortGroup.

​https://github.com/macports/macports-ports/blob/b01bdb8562ea4b611b5f406d70ba0cf3276e73c2/textproc/libxml2/Portfile#L4

​https://github.com/macports/macports-ports/blob/master/_resources/port1.0/group/clang_dependency-1.0.tcl

MacPorts will never attempt to build libxml2 with any compiler listed in that PortGroup unless you try to force it to do so.

after ICU is updated, libxml2 is revbumped, and will be built by a compiler that does not rely on libxml2. All is well, unless you take steps to break it, which you can probably do if you try.

Now regarding clang-11-bootstrap, the idea of clang-11-bootstrap in the first place was that I envisioned it as an end-run around all this, and it would be a modern clang that work no matter what the status of any ports in Macports happened to be. clang-11-bootstrap tries to go to great lengths to avoid being entangled in any MacPorts ports.

​https://github.com/macports/macports-ports/blob/b01bdb8562ea4b611b5f406d70ba0cf3276e73c2/lang/clang-11-bootstrap/Portfile#L103

That fact that a broken libxml2/icu situation made your clang-11-bootstrap not function is an interesting observation that we might have benefitted from sorting out, to see if there is in fact some way that clang-11-bootstrap did depend on something in macports.

But we will never know what happened to you now. Nothing in the ticket above explains it. So we are lost and have to wait to see if someone else finds this issue again and perhaps we can work with them to see what is up.

Maybe something to do with all the forcing of build compilers and forcing of versions of icu and possibly other similar things you were doing broke it?

At any rate, this works for (almost) everyone throughout MacPorts. If it happens to you again, we would appreciate you trying to see what is going on for us, as if there is something that needs fixing, at present from this ticket, we can't know what it is.

[kencu (Ken)]

Well, I wonder if it could have been something in the cctools/ld64/libtapi arena that did this? Did we somehow let a bootstrap loop creep in that nobody has picked up on yet?

There are a lot of moving parts here that fit together in subtle ways. People can build cctools and ld64 variants against different llvms, and there are all kinds of combinations and permutations that could bite (which is why I am leaning towards getting rid of all of them).

for libtapi, I specifically made sure it would not depend on libxml2 here:

​https://github.com/macports/macports-ports/blob/b01bdb8562ea4b611b5f406d70ba0cf3276e73c2/devel/libtapi/Portfile#L42

ld64 doesn't seem to have any interest in icu or libxml2...cctools conceivably might through it's dependency on an llvm version though...

One thing is for sure -- whenever you upgrade ICU, an uproar is inevitable somewhere :>