net-snmp @5.8 fails to address the correct SDK

[TP75]

This happened on macOS Majove 10.14.6 with MacPorts 2.6.2 after a fresh macOS install.

The CommandLine tools are installed with 'xcode-select --install' but not XCode intentionally.

     net-snmp @5.8 +ssl
--->  Computing dependencies for net-snmp
--->  Fetching distfiles for net-snmp
--->  Verifying checksums for net-snmp
--->  Extracting net-snmp
--->  Applying patches to net-snmp
--->  Configuring net-snmp
--->  Building net-snmp
Error: Failed to build net-snmp: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_net_net-snmp/net-snmp/main.log for details.
Error: rev-upgrade failed: Error rebuilding net-snmp
Error: Follow https://guide.macports.org/#project.tickets to report a bug.

The excerpt from the a.m. log shows:

:info:build Running Mkbootstrap for default_store ()
:info:build cp MakefileSubs.pm blib/lib/Bundle/MakefileSubs.pm
:info:build chmod 644 "default_store.bs"
:info:build "/opt/local/bin/perl5.28" -MExtUtils::Command::MM -e 'cp_nonempty' -- default_store.bs ../blib/arch/auto/NetSNMP/default_store/default_store.bs 644
:info:build cp default_store.pm ../blib/lib/NetSNMP/default_store.pm
:info:build AutoSplitting ../blib/lib/NetSNMP/default_store.pm (../blib/lib/auto/NetSNMP/default_store)
:info:build mv default_store.xsc default_store.c
:info:build /usr/bin/clang -c   -I../../include -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -arch x86_64 -DNETSNMP_ENABLE_IPV6 -fno-strict-aliasing -DNETSNMP_REMOVE_U64 -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -arch x86_64 -Udarwin18 -Ddarwin18=darwin18 -I/opt/local/include -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -I. -I/opt/local/include -fno-common -DPERL_DARWIN -mmacosx-version-min=10.14 -pipe -Os -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk  -fno-strict-aliasing -fstack-protector-strong -I/opt/local/include -DPERL_USE_SAFE_PUTENV -Wformat -O3   -DVERSION=\"5.08\" -DXS_VERSION=\"5.08\"  "-I/opt/local/lib/perl5/5.28/darwin-thread-multi-2level/CORE"   default_store.c
:info:build clang: warning: no such sysroot directory: '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk' [-Wmissing-sysroot]
:info:build In file included from default_store.xs:6:
:info:build /opt/local/lib/perl5/5.28/darwin-thread-multi-2level/CORE/perl.h:684:10: fatal error: 'sys/types.h' file not found
:info:build #include <sys/types.h>
:info:build          ^~~~~~~~~~~~~
:info:build 1 error generated.
:info:build make[2]: *** [default_store.o] Error 1
:info:build make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_net_net-snmp/net-snmp/work/net-snmp-5.8/perl/default_store'
:info:build make[1]: *** [subdirs] Error 2
:info:build make[1]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_net_net-snmp/net-snmp/work/net-snmp-5.8/perl'
:info:build make: *** [perlmodules] Error 1
:info:build make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_net_net-snmp/net-snmp/work/net-snmp-5.8'
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_net_net-snmp/net-snmp/work/net-snmp-5.8" && /usr/bin/make -j4 -w all 
:info:build Exit code: 2
:error:build Failed to build net-snmp: command execution failed
:debug:build Error code: CHILDSTATUS 80031 2
:debug:build Backtrace: command execution failed
:debug:build     while executing
:debug:build "system {*}$notty {*}$nice $fullcmdstring"
:debug:build     invoked from within
:debug:build "command_exec build"
:debug:build     (procedure "portbuild::build_main" line 8)
:debug:build     invoked from within
:debug:build "$procedure $targetname"
:error:build See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_net_net-snmp/net-snmp/main.log for details.

Apparently the clang is addressing the non-existent XCode SDK and not the CommandLine tools SDK, I presume.

-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk

Any advice?

[ryandesign (Ryan Carsten Schmidt)]

I see it's attempting to use multiple different SDKs:

-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
-isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk

Certainly it shouldn't be doing that, but do either of those paths exist on your system? Which version of Xcode do you have?

[ryandesign (Ryan Carsten Schmidt)]

Also, I'm curious why you're needing to build this from source, since we have binaries available.

[ryandesign (Ryan Carsten Schmidt)]

Since the net-snmp portfile doesn't do anything special related to SDKs, but is using the perl5 portgroup, I'm going to assume this is a bug in either the perl5.* ports for baking the SDK path into the config files that get installed, and/or a bug in the perl5 portgroup for not successfully ignoring that baked-in SDK path when building other ports.

[TP75]

Replying to ryandesign:

I see it's attempting to use multiple different SDKs:

-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
-isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk

Certainly it shouldn't be doing that, but do either of those paths exist on your system? Which version of Xcode do you have?

Thank you for your kind advice and request. Please find the results below.

$ ls -la /Library/Developer/CommandLineTools/SDKs/

drwxr-xr-x  7 root  wheel  224  9 Jul 02:07 MacOSX.sdk
lrwxr-xr-x  1 root  wheel   10 23 Nov 17:53 MacOSX10.14.sdk -> MacOSX.sdk


$ xcode-select --version
xcode-select version 2354.

Please recall there is no XCode installed intentionally.

Replying to TP75:

This happened on macOS Majove 10.14.6 with MacPorts 2.6.2 after a fresh macOS install.

The CommandLine tools are installed with 'xcode-select --install' but not XCode intentionally.

Hope this helps.

[TP75]

Replying to ryandesign:

Also, I'm curious why you're needing to build this from source, since we have binaries available.

Thank you again.

$ port installed

  libressl @2.8.3_0 (active)

  perl5 @5.26.1_0+perl5_28 (active)
  perl5.28 @5.28.2_0 (active)

This is a ​LibreSSL install instead of OpenSSL intentionally.

[TP75]

Off-topic: You had Black Friday, I presume. -- Please note Europe is more relaxed. Today we celebrate the ​First Advent. Happy holiday.

Not installing XCode was intentionally but not according the MacPorts installation procedure, obviously. This was done deliberately as an experiment.

However, after install of XCode from the XCode_11-2.1.xip package the net-snmp build still fails.

[kencu (Ken)]

despite our best efforts for the past 6 or 8 weeks, there are still many places where the wrong "isysroot" is baked into various MacPorts-installed software, in particular things like the python setup scrips, and (obviously) still in perl. It's a frustrating battle. Some software, like perl, does it's own dance to find an SDK, and it's not usually the one we want the software to find.

We decided to use /Library/Developer/CommandLineTools/SDKs/ as our settled path to SDK, and to build all of our software using that path to the SDK needed (which is supposed to be the SDK that matches the system it is running on). That way, people could install the command line tools, and not install xcode, and most ports (that don't need a full Xcode) would build.

However, as you are finding, it's not yet perfect. Please feel free to contribute, and find out where on your system's perl files that darned SDK path into Xcode has been baked, and if you're really helpful, put up a PR that fixes it. Then we can rebuild that software and hopefully, finally, get the SDK path we want baked into it.

Or, if your time is valuable, and whose time is not, just accept the inevitable, and either install Xcode or make a symlink from the SDK in the commandline tools sdk folder to that path

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk

and the hardcoded build files will find it.

Now -- if you have already done either of those, and you are still getting a build error, presumably it must not be due to the SDK path (that you just fixed), so in that case we'd need to see another log with whatever on earth might be going wrong now to make it fail, as it must be something different.

[kencu (Ken)]

I suspect that you have installed Xcode 11, and you don't have a MacOSX10.14.sdk now in the Xcode bundle. Instead, you have a MacOSX10.15.sdk in the Xcode bundle, which is symlinked to MacOSX.sdk.

This path will not work for software looking for the MacOSX10.14.sdk, naturally.

So -- symlink as above from the proper MacOSX10.14.sdk in the command line tools folder into the Xcode bundle, and your software will be happy. Or help us find out how to fix the piece of software with the wrong baked in path.

Don't be tempted to symlink the MacOSX10.15.sdk in the Xcode bundle to MacOSX10.14.sdk as a quick hack -- they are too different, and it will break.

[TP75]

Replying to kencu:

Or, if your time is valuable, and whose time is not, just accept the inevitable, and either install Xcode or make a symlink from the SDK in the commandline tools sdk folder to that path

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk

and the hardcoded build files will find it.

THX -- However, I already made a test of my own before your kind advice. That trial failed utterly and I will have to present you the long long log in due time when I find the time.

Happy hacking.

[TP75]

Replying to kencu:

I suspect that you have installed Xcode 11, and you don't have a MacOSX10.14.sdk now in the Xcode bundle. Instead, you have a MacOSX10.15.sdk in the Xcode bundle, which is symlinked to MacOSX.sdk.

Please note and recall: A fresh macOS install - no full XCode and only 'xcode-select --install' ...

---

Don't be tempted to symlink the MacOSX10.15.sdk in the Xcode bundle to MacOSX10.14.sdk as a quick hack -- they are too different, and it will break.

You got me and it worked against all odds. However, not in this experiment and not on this machine. And was mentioned by me somewhere else in this huge mount of tickets where after some time I always get lost and never find again may old comments to other tickets unfortunately.

[kencu (Ken)]

Replying to TP75:

Replying to kencu:

I suspect that you have installed Xcode 11, and you don't have a MacOSX10.14.sdk now in the Xcode bundle. Instead, you have a MacOSX10.15.sdk in the Xcode bundle, which is symlinked to MacOSX.sdk.

Please note and recall: A fresh macOS install - no full XCode and only 'xcode-select --install' ...

Must have been misled by the fact you said you had installed XCode11....

[ryandesign (Ryan Carsten Schmidt)]

Replying to TP75:

Replying to ryandesign:

I see it's attempting to use multiple different SDKs:

-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
-isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk

Certainly it shouldn't be doing that, but do either of those paths exist on your system? Which version of Xcode do you have?

Thank you for your kind advice and request. Please find the results below.

$ ls -la /Library/Developer/CommandLineTools/SDKs/

drwxr-xr-x  7 root  wheel  224  9 Jul 02:07 MacOSX.sdk
lrwxr-xr-x  1 root  wheel   10 23 Nov 17:53 MacOSX10.14.sdk -> MacOSX.sdk


$ xcode-select --version
xcode-select version 2354.

Please recall there is no XCode installed intentionally.

Ok. I wasn't sure how it would behave with multiple -isysroot flags—whether it would try to use them all or just the first or last one. Since you definitely do have the path specified in the first, second and third -isysroot flags, but not the last one, I suspect only the last flag takes effect.

The bad flag is likely coming from perl's config files. Perl loves to build its modules with the flags that had been used to build perl (which might have been built on our build server, where we do have Xcode installed). I would love it if perl would cease and desist doing that, but I don't know how to make it do that.

Replying to TP75:

Not installing XCode was intentionally but not according the MacPorts installation procedure, obviously. This was done deliberately as an experiment.

That's fine, assuming you'd like to encounter problems and help us work through resolving them. :) We would like to be able to support users who don't have Xcode installed, but have only just begun to add support for that in MacPorts 2.6.0. Prior to that, it was assumed that you have Xcode installed, and Xcode-relative SDK paths were used and in many cases got baked into ports inadvertently. As of MacPorts 2.6.0, we will use command-line-tools-relative SDK paths instead (unless a port indicates the need for Xcode by using use_xcode yes), but we still have an unknown probably large number of binary packages that we distribute that were built prior to MacPorts 2.6.0 and so still use Xcode-based SDK paths.

However, after install of XCode from the XCode_11-2.1.xip package the net-snmp build still fails.

That's not surprising, since Xcode 11 doesn't contain the 10.14 SDK. Try installing ​Xcode 10.3, which does contain the 10.14 SDK.

[TP75]

Replying to ryandesign:

Replying to TP75:

Not installing XCode was intentionally but not according the MacPorts installation procedure, obviously. This was done deliberately as an experiment.

That's fine, assuming you'd like to encounter problems and help us work through resolving them. :) We would like to be able to support users who don't have Xcode installed, but have only just begun to add support for that in MacPorts 2.6.0. Prior to that, it was assumed that you have Xcode installed, and Xcode-relative SDK paths were used and in many cases got baked into ports inadvertently. As of MacPorts 2.6.0, we will use command-line-tools-relative SDK paths instead (unless a port indicates the need for Xcode by using use_xcode yes), but we still have an unknown probably large number of binary packages that we distribute that were built prior to MacPorts 2.6.0 and so still use Xcode-based SDK paths.

However, after install of XCode from the XCode_11-2.1.xip package the net-snmp build still fails.

That's not surprising, since Xcode 11 doesn't contain the 10.14 SDK. Try installing ​Xcode 10.3, which does contain the 10.14 SDK.

Cool. THX.

IMHO this XCode issue is quite redundant. One could wonder if blaming Apple for pushing poor users and agile developers down the road to 10.15 and apparently into the dark bucket of an ugly copy of Micro-Somethings business model, I presume.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

The bad flag is likely coming from perl's config files.

So fixing #62440 would probably fix this issue.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

So fixing #62440 would probably fix this issue.

#62440 was fixed. Did that fix this issue?

[ryandesign (Ryan Carsten Schmidt)]

Presumed fixed.