libgit2 @1.7.1: Undefined symbols _libiconv _libiconv_close _libiconv_open due to errant symlink in cmake installation

[jrabinow]

Attempting to build libgit2 fails:

info:build ld: Undefined symbols:
:info:build   _libiconv, referenced from:
:info:build       _git_fs_path_iconv in fs_path.c.o
:info:build       _unicode_iconv_encoding_convert in unicode_iconv.c.o
:info:build   _libiconv_close, referenced from:
:info:build       _git_fs_path_direach in fs_path.c.o
:info:build       _git_fs_path_iconv_clear in fs_path.c.o
:info:build       _git_fs_path_diriter_free in fs_path.c.o
:info:build       _git_fs_path_dirload in fs_path.c.o
:info:build       _ntlm_unicode_shutdown in unicode_iconv.c.o
:info:build       _ntlm_unicode_shutdown in unicode_iconv.c.o
:info:build   _libiconv_open, referenced from:
:info:build       _git_fs_path_direach in fs_path.c.o
:info:build       _git_fs_path_iconv_init_precompose in fs_path.c.o
:info:build       _git_fs_path_diriter_init in fs_path.c.o
:info:build       _ntlm_unicode_init in unicode_iconv.c.o
:info:build       _ntlm_unicode_init in unicode_iconv.c.o
:info:build clang: error: linker command failed with exit code 1 (use -v to see invocation)

Digging deeper, it fails at exactly this part of the build process:

cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgit2/libgit2/work/build/tests/libgit2 && /opt/local/bin/cmake -E cmake_link_script CMakeFiles/libgit2_tests.dir/link.txt --verbose=ON

I tried manually adding variations of -l /opt/local/lib/libiconv.dylib (-l libiconv.dylib, -l libiconv) to CMakeFiles/libgit2_tests.dir/link.txt without success:

/opt/local/bin/cmake -E cmake_link_script CMakeFiles/libgit2_tests.dir/link.txt --verbose=ON -v
ld: library '/opt/local/lib/libiconv.dylib' not found

This one is quite surprising, because libiconv is installed on my system:

$ port installed libiconv
The following ports are currently installed:
  libiconv @1.17_0+universal (active)
$ file /opt/local/lib/libiconv.*
/opt/local/lib/libiconv.2.dylib: Mach-O universal binary with 2 architectures: [x86_64:\012- Mach-O 64-bit x86_64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>] [\012- arm64:\012- Mach-O 64-bit arm64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>]
/opt/local/lib/libiconv.a:       Mach-O universal binary with 2 architectures: [x86_64:\012- current ar archive random library] [\012- arm64:\012- current ar archive random library]
/opt/local/lib/libiconv.dylib:   symbolic link to libiconv.2.dylib

I thought it might be related to the recent change [abd25f7f4a164c7ab2a8a8146fcbd746f45035a0/macports-ports], reverting that commit didn't seem to affect anything.

macOS 14.0 23A344 arm64
Xcode 15.0 15A240d

[jrabinow]

build log

[jrabinow]

manual invocation of cmake: trace

[jrabinow]

I just went back and looked at old tickets I had created. By complete chance, I'd created #66931 a few months back, and bnoctis was kind enough to provide a solution which I just tested and works.

I've created a PR at ​https://github.com/macports/macports-ports/pull/20805

[jrabinow]

Would it make sense to leave this ticket open until we can explain why libgit2 builds on the build bots but not on some people's personal laptops?

[ryandesign (Ryan Carsten Schmidt)]

As was discussed in #66931, the problem appears to be unique to a few users' systems. The build system is supposed to be adding -liconv on its own. Manually adding that flag is a poor workaround; we would prefer to figure out why the build system is not adding that flag when it should and fix it to do so and submit that fix to the developers so that everyone, not just MacPorts users, can benefit.

[ryandesign (Ryan Carsten Schmidt)]

Comparing the buildbot log of libgit2 1.7.1 on macOS 13 with your log from macOS 14, the buildbot log says:

-- Checking for module 'heimdal-gssapi'
--   No package 'heimdal-gssapi' found
-- Could NOT find GSSAPI (missing: GSSAPI_LIBRARIES) 

while yours says:

-- Found GSSAPI: /opt/local/lib/libgssapi_krb5.dylib;/opt/local/lib/libkrb5.dylib;/opt/local/lib/libk5crypto.dylib;/opt/local/lib/libcom_err.dylib  

So this port is using kerberos5 opportunistically. It must be modified to either depend on kerberos5 or not to use kerberos5 even if it is installed. But this does not appear to be the cause of the libiconv problem because I have kerberos5 installed and was able to build libgit2 from source successfully.

Continuing on, the buildbot log says:

-- Looking for iconv_open
-- Looking for iconv_open - not found
-- Found Iconv: -L/opt/local/lib -liconv

while yours says:

-- Performing Test Iconv_IS_BUILT_IN
-- Performing Test Iconv_IS_BUILT_IN - Failed
-- Found Iconv: /opt/local/lib/libiconv.dylib (found version "1.17") 

Strange that libiconv was found in both cases but apparently using different methods, one of which apparently works and the other doesn't. More investigation needed.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

Continuing on, the buildbot log says:

-- Looking for iconv_open
-- Looking for iconv_open - not found
-- Found Iconv: -L/opt/local/lib -liconv

I believe this is coming from the FindIconv.cmake file included in the libgit2 source distribution.

while yours says:

-- Performing Test Iconv_IS_BUILT_IN
-- Performing Test Iconv_IS_BUILT_IN - Failed
-- Found Iconv: /opt/local/lib/libiconv.dylib (found version "1.17") 

I believe this is coming from the FindIconv.cmake file installed as part of the cmake port.

Why are there two versions of this file? How is it determined or specified which one will be used?

[ryandesign (Ryan Carsten Schmidt)]

This appears to have come up many times before:

• ​https://github.com/libgit2/libgit2/issues/2918
• ​https://github.com/libgit2/libgit2/issues/6472
• ​https://github.com/libgit2/libgit2/issues/6466

It's not clear whether anyone there understands the cause.

[jrabinow]

build/libgit2.pc from raw repo

[jrabinow]

build/libgit2.pc

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

Replying to ryandesign:

while yours says:

-- Performing Test Iconv_IS_BUILT_IN
-- Performing Test Iconv_IS_BUILT_IN - Failed
-- Found Iconv: /opt/local/lib/libiconv.dylib (found version "1.17") 

I believe this is coming from the FindIconv.cmake file installed as part of the cmake port.

I am able to reproduce the error if I delete the FindIconv.cmake file included with libgit2, thus forcing it to use the one that comes with cmake. I'll file an issue with the developers of libgit2.

[jrabinow]

I'm afraid I'm quite out of my depth here.

When I dug into the cmake recipe the other day, I landed on ​this call to ​find_library

I'm really not sure why it's giving me a full path to libiconv on my system. I just tried to reproduce it and got the same thing. I tried removing ​this line to check if libiconv might be found by ./configure rather than by cmake, but the full path showed up in my main.log again. That's as far as I got so far.

[ryandesign (Ryan Carsten Schmidt)]

Replying to jrabinow:

When I dug into the cmake recipe the other day, I landed on ​this call to ​find_library

Right, but the problem is that that FindIconv.cmake is not being used at all in your case; your build is, for some reason, using a different FindIconv.cmake provided by the cmake port, with which libgit2 is not compatible. I'll explain this to the developer in my bug report and suggest two possible solutions: either forcing, somehow, that the bundled copy of FindIconv.cmake is always used, or else deleting the bundled copy and rewriting the rest of the libgit2 cmake files so that they are compatible with cmake's FindIconv.cmake.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

I'll file an issue with the developers of libgit2.

​https://github.com/libgit2/libgit2/issues/6644

[jrabinow]

Going through the issues you linked to: ​https://stackoverflow.com/a/57734435 this suggests that it's not (just) finding the wrong FindIconv.cmake but finding the wrong libiconv.dylib. Could the opportunistic linking to gssapi you brought up earlier be a clue there's opportunistic linking to the first libiconv.dylib that comes along?

(side note, I just got bitten by GNU find ​https://superuser.com/q/1679798):

$ fd '.*iconv\..*' /
/Library/Developer/CoreSimulator/Volumes/iOS_21A328/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 17.0.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libiconv.2.dylib
/Library/Developer/CoreSimulator/Volumes/iOS_21A328/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 17.0.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libiconv.dylib
/Users/user/Library/Developer/Xcode/iOS DeviceSupport/16.5.1 (20F75) arm64e/Symbols/usr/lib/libiconv.2.dylib
/Users/user/Library/Developer/Xcode/iOS DeviceSupport/16.2 (20C65) arm64e/Symbols/usr/lib/libiconv.2.dylib
/System/Volumes/Data/Users/user/Library/Developer/Xcode/iOS DeviceSupport/16.5.1 (20F75) arm64e/Symbols/usr/lib/libiconv.2.dylib
/System/Volumes/Data/Users/user/Library/Developer/Xcode/iOS DeviceSupport/16.2 (20C65) arm64e/Symbols/usr/lib/libiconv.2.dylib
/System/Volumes/Data/Library/Developer/CoreSimulator/Volumes/iOS_21A328/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 17.0.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libiconv.dylib
/System/Volumes/Data/Library/Developer/CoreSimulator/Volumes/iOS_21A328/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 17.0.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libiconv.2.dylib
/System/Volumes/Data/opt/local/lib/libiconv.dylib
/System/Volumes/Data/opt/local/lib/libiconv.2.dylib
/opt/local/lib/libiconv.dylib
/opt/local/lib/libiconv.2.dylib

None of these look like good candidates. If there was a libiconv.dylib under /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk or /usr/{local/,}lib/ that would have been an interesting find. Then again, my system is the one with the failing build so it makes sense there's no other good candidate.

I can't do the following experiment on my system since it will only use the libiconv under my /opt/local, but I'd be curious what otool -L /opt/local/lib/libgit2.1.7.1.dylib says after a clean build on your machine. Or if you renamed /opt/local/lib/libiconv* to something else, does macports libgit2 still build? ​This issue you linked to above might be ​relevant in this rebuild scenario.

Word of caution if you choose to run the rename experiment: anything linked to libiconv will be unable to run, and it's a dependency for a lot of things potentially including the shell, GNU coreutils binaries and terminal multiplexers.

​https://github.com/libgit2/libgit2/issues/6644

Many thanks for digging and for putting it all together.

[ryandesign (Ryan Carsten Schmidt)]

Replying to jrabinow:

Going through the issues you linked to: ​https://stackoverflow.com/a/57734435 this suggests that it's not (just) finding the wrong FindIconv.cmake but finding the wrong libiconv.dylib.

I do not believe we are dealing with a two-different-versions-of-libiconv problem here.

Could the opportunistic linking to gssapi you brought up earlier be a clue there's opportunistic linking to the first libiconv.dylib that comes along?

No, I checked for that before. The opportunistic linking to gssapi happened on my computer as well, yet the build succeeded.

The problem is, as I said, that on your system it is using the wrong FindIconv.cmake, for reasons not yet understood.

[kencu (Ken)]

As per the PR discussion, for some reason he has rogue files installed into:

/opt/local/share/cmake

one of which is FindIconv.cmake and these are being picked up and making the build fail.

[jrabinow]

$ file /opt/local/share/cmake
/opt/local/share/cmake: symbolic link to cmake-3.24/

I didn't create that link. It was created with the port install time

$ port provides /opt/local/share/cmake-3.24/Modules/FindIconv.cmake
/opt/local/share/cmake-3.24/Modules/FindIconv.cmake is provided by: cmake

No rogue files here. This is all stock cmake and stock libgit2.

I might ask, why do you not have the files? They come from upstream ​https://github.com/Kitware/CMake/blob/master/Modules/FindIconv.cmake why are they not interfering with your build?

[mascguy (Christopher Nielsen)]

Replying to jrabinow:

$ file /opt/local/share/cmake
/opt/local/share/cmake: symbolic link to cmake-3.24/

I didn't create that link. It was created with the port install time

$ port provides /opt/local/share/cmake-3.24/Modules/FindIconv.cmake
/opt/local/share/cmake-3.24/Modules/FindIconv.cmake is provided by: cmake

No rogue files here. This is all stock cmake and stock libgit2.

Ah, but ${prefix}/share/cmake isn't created by cmake. You can verify this yourself, by checking the list of installed items via: port contents cmake | less. (If cmake created that directory, you'd see it in the list, along with a dummy "turd" file.)

There are a small number of ports that may create that directory, and install items into it. But those are few and far between.

In short: Most of the time, ${prefix}/share/cmake shouldn't exist at all. Or if it does, it should be a separate directory, with a very small number of items.

So that symlink is your problem, and it shouldn't exist. Can you try removing it, and re-test?

[kencu (Ken)]

Funny, eh?

Late one night, you create a symlink because you think it is a good idea, probably years ago -- and it messes you up in ways you can never imagine for years later... all your issues with libgit2 stem from that. There is nothing wrong with macports, libgit2 upstream, or cmake -- it's just your setup that is hosed up.

The problem now is that all the software you have built with cmake ever since that day is possibly built wrongly.

So I continue to echo my endless refrain -- blow out your /opt/local and start over, because your installation is hopelessly compromised.

You will never be able to tell exactly what software is built incorrectly... maybe lots, maybe none.

[jrabinow]

I can deal with port breakage, but I don't go around creating random symlinks. Stuff I build/download goes under /usr/local (fbinfer and docker) if no other choice, ~/bin if I want it in my PATH, or stays in its build directory.

I might wind up rm -rf ing /opt/local because at this point yes, my system is tainted, but this cruft you're so wary of is macports own. That's fine, it's not a blame game, let's keep it that way.

Moving on.

$ port provides /opt/local/share/cmake-3.24/Modules/FindIconv.cmake
/opt/local/share/cmake-3.24/Modules/FindIconv.cmake is provided by: cmake
$ sudo port uninstall cmake
Password:
--->  Deactivating cmake @3.24.4_0
--->  Cleaning cmake
--->  Uninstalling cmake @3.24.4_0
--->  Cleaning cmake
$ ls -l /opt/local/share/cmake
lrwxr-xr-x 1 root wheel 11 Dec 14  2022 /opt/local/share/cmake -> cmake-3.24/    # I would have expected this to be gone if it was provided by the port
$ sudo rm /opt/local/share/cmake
$ sudo port install cmake
[...]
$ ls -l /opt/local/share/cmake
ls: cannot access '/opt/local/share/cmake': No such file or directory
[Exit 2]

So indeed, this is an errant symlink issue. Where did it come from? Here, the creation date gives us a hint: Dec 14 2022

$ git log --oneline --pretty=reference devel/cmake
85f77b08dcc (cmake: take co-ownership, to assist @michaelld, 2023-10-07)
5b8d8ed41e2 (cmake: Update to 3.24.4, 2023-03-14)
9c0f9ab73a6 (cmake: tweak patches to apply cleanly, 2023-01-04)
9cac879c1a7 (cmake: update extract rename comment again to be more precise, 2022-12-30)
5e5b4cb4f3f (cmake: fix extract.rename comment to be more accurate, 2022-12-28)
4580bcd8cdb (cmake: prepare for 'extract.rename' feature per the comment, 2022-12-13)
44aef1f0e46 (cmake: add comment for why the move to C++14, 2022-12-02)
38f70c2aef5 (cmake: limit livecheck to the current major.minor version, 2022-12-02)
b12c4a40155 (cmake: when bootstrapping disable new Python variants, 2022-11-26)
d9e3218c0d5 (cmake{,-devel}: fix build before 10.10, 2022-11-26)

I update my ports once a week max, and usually once every 2 days or so. The symlink was created a day after ​4580bcd8cdb so maybe it's related?

What's ​https://trac.macports.org/ticket/66415, could this be relevant?

[mascguy (Christopher Nielsen)]

Relative to the source of the symlink, it's possible that a port un-helpfully created it.

Your port installations also have a date, visible by running port -v installed. And if any ports were installed on that date, those might be a place for us to look.

[jrabinow]

port -v installed|grep '2022-12-14' came up empty.

I looked at registry/registry.db from the oldest time machine backup I had available, I was able to find this:

select * from ports where date > 1670918400 and date < 1671091200;
2344|arp-scan|3744df2bd6f8207873a676fc8239ecdc7c276f4d425faaa2c4619d69c45d7dfb-865|/opt/local/var/macports/software/arp-scan/arp-scan-1.10.0_0.darwin_22.arm64.tbz2|0|1.10.0|0|||installed|1671059636|image|arm64|1|darwin|22|none|0

But I somewhat doubt it's related.