openssl3 @ 3.0.0_2+universal.darwin_15.i386-x86_64: dyld: Library not loaded: /opt/local/lib/libcrypto.1.1.dylib

[thetrial (alabay)]

I’m stuck with openssl3. I had that yesterday and replayed /opt from backup. Today I tried it step by step, first installing openssl11 due to I have that unsolved problem: #62220, and due to the error yesterday. But it happened again.

:info:build /usr/bin/perl apps/progs.pl "-H" "apps/openssl" > apps/progs.h
:info:build ranlib -c test/libtestutil.a || echo Never mind.
:info:build /opt/local/bin/clang-mp-9.0  -Iinclude  -arch x86_64 -pipe -Os -arch x86_64 -D_REENTRANT -DOPENSSL_BUILDING_OPENSSL -DZLIB -DNDEBUG -I/opt/local/include -MMD -MF test/buildtest_c_aes-bin-buildtest_aes.d.tmp -MT test/buildtest_c_aes-bin-buildtest_aes.o -c -o test/buildtest_c_aes-bin-buildtest_aes.o test/buildtest_aes.c
:info:build dyld: Library not loaded: /opt/local/lib/libcrypto.1.1.dylib
:info:build   Referenced from: /opt/local/lib/libxar.1.dylib
:info:build   Reason: image not found
:info:build clang: error: unable to execute command: Trace/BPT trap: 5
:info:build clang: error: linker command failed due to signal (use -v to see invocation)
:info:build make[1]: *** [test/p_test.dylib] Error 254
:info:build make[1]: *** Waiting for unfinished jobs....
:info:build make[1]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3/work/openssl-3.0.0-x86_64'
:info:build make: *** [build_sw] Error 2
:info:build make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3/work/openssl-3.0.0-x86_64'
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3/work/openssl-3.0.0-x86_64" && /usr/bin/make -j4 -w all 
:info:build Exit code: 2
:error:build Failed to build openssl3: command execution failed
:debug:build Error code: NONE
:debug:build Backtrace: command execution failed

I’ll attach the logfile, maybe I oversee something. And after this ticket, I’ll replay /opt again. Really stuck. And no, I cannot start from scratch … because of #62220 I’d lose.

[jmroot (Joshua Root)]

Already fixed by [9f126afd71e24200a66d5965ae9dd034501dc904/macports-ports]. Disallowing use of Xcode clang previous to that created a circular dependency with clang-9.0.

[thetrial (alabay)]

What does that mean? Which clang is used now? Apple LLVM version 8.0.0 (clang-800.0.42.1) or clang version 9.0.1 (MP)? By the way … should one adjust or switch something regarding that? Or does this happen automatically?

So we’ll have to wait until an update plays in the changes? How long could this take?

[mouse07410 (Mouse)]

It means that openssl11 port forgot to symlink /opt/local/libexec/openssl11/lib/libcrypto.1.1.dylib to /opt/local/lib/libcrypto.1.1.dylib.

The workaround is to add those symlinks manually. The solution is for the port to ensure it symlinks libcrypto.1.1.dylib and libssl.1.1.dylib to /opt/local/lib.

[cjones051073 (Chris Jones)]

Replying to mouse07410:

It means that openssl11 port forgot to symlink /opt/local/libexec/openssl11/lib/libcrypto.1.1.dylib to /opt/local/lib/libcrypto.1.1.dylib.

The workaround is to add those symlinks manually. The solution is for the port to ensure it symlinks libcrypto.1.1.dylib and libssl.1.1.dylib to /opt/local/lib.

No, this is not correct. The openssl11 port intentionally does not install anything in the primary prefix. It *only* installs to its isolated install area under libexec. Any build using openssl11 neds to be configured to use the libraries from that area.

[cjones051073 (Chris Jones)]

... and you absolutely should *never* *ever* create sym links under the macports prefix (/opt/local) by hand as this will cause you issues later on when you forget you did it...

[cjones051073 (Chris Jones)]

from the above

:info:build dyld: Library not loaded: /opt/local/lib/libcrypto.1.1.dylib
:info:build   Referenced from: /opt/local/lib/libxar.1.dylib
:info:build   Reason: image not found

the fix for the above is to rebuild libxar, against the new openssl version.

[cjones051073 (Chris Jones)]

the xar port has infact already been rebuilt. i.e. from an up to date install

Oberon ~/Projects/MacPorts/ports > otool -L  /opt/local/lib/libxar.1.dylib
/opt/local/lib/libxar.1.dylib:
	/opt/local/lib/libxar.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/opt/local/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.8)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/opt/local/libexec/openssl3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/libxml2.2.dylib (compatibility version 12.0.0, current version 12.12.0)
	/opt/local/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.5.0)
	/opt/local/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
	/opt/local/lib/libicui18n.67.dylib (compatibility version 67.0.0, current version 67.1.0)
	/opt/local/lib/libicuuc.67.dylib (compatibility version 67.0.0, current version 67.1.0)
	/opt/local/lib/libicudata.67.dylib (compatibility version 67.0.0, current version 67.1.0)

so the *fix* is for the OP just to update their ports, as the solution is already out...

[mouse07410 (Mouse)]

The openssl11 port intentionally does not install anything in the primary prefix. It *only* installs to its isolated install area under libexec. Any build using openssl11 neds to be configured to use the libraries from that area

Respectfully disagree.

openssl11 and openssl3 must co-exist, at least for some time. It makes perfect sense to make both available in the "main" Macports library dir /opt/local/lib. Especially since there are plenty of (non-Macports) apps that cannot (yet?) switch to OpenSSL-3.0. E.g., I've been linking my stuff that uses OpenSSL against Macports-installed OpenSSL for ages.

The whole reason for "dotted" versions for .dylib is to support exactly this: safely having libssl.1.1.dylib and libssl.3.dylib in the same directory.

To summarize: please reconsider.

[thetrial (alabay)]

Regarding the isolation of openssl11 and rebuilding with new links: Ignored/not adjusted ports like #62220 break and will break again. Some ports are not clearly adapted to stucked OS and go on, ignoring the final version of a build that works under an old OS. One can only manually adapt things like that. Or any other solutions? Except kicking no-maintained stuff? I guess this is especially a problem with Qt-applications, but they rely on ssl, too.

[cjones051073 (Chris Jones)]

Replying to mouse07410:

The openssl11 port intentionally does not install anything in the primary prefix. It *only* installs to its isolated install area under libexec. Any build using openssl11 neds to be configured to use the libraries from that area

Respectfully disagree.

openssl11 and openssl3 must co-exist, at least for some time. It makes perfect sense to make both available in the "main" Macports library dir /opt/local/lib. Especially since there are plenty of (non-Macports) apps that cannot (yet?) switch to OpenSSL-3.0. E.g., I've been linking my stuff that uses OpenSSL against Macports-installed OpenSSL for ages.

The whole reason for "dotted" versions for .dylib is to support exactly this: safely having libssl.1.1.dylib and libssl.3.dylib in the same directory.

To summarize: please reconsider.

No. The only reason we are discussing this is because the OPs ports where not up to date.

Any port now requiring openssl11 must either use the openssl PG, which will (help to) configure the build correctly to link against the libexec install area, or (not really recommended) do this by hand. The openssl shim port make links into the primary prefix to make that version available. Thats all, the older versions are intentionally left isolated under libexec.

[cjones051073 (Chris Jones)]

Replying to thetrial:

Regarding the isolation of openssl11 and rebuilding with new links: Ignored/not adjusted ports like #62220 break and will break again. Some ports are not clearly adapted to stucked OS and go on, ignoring the final version of a build that works under an old OS. One can only manually adapt things like that. Or any other solutions? Except kicking no-maintained stuff? I guess this is especially a problem with Qt-applications, but they rely on ssl, too.

Sorry, I really cannot parse the above to work out what you are asking here...

[thetrial (alabay)]

In short: the libexec solution will make problems. I agree with Mouse.

[cjones051073 (Chris Jones)]

Another point. Just putting the 1.1 libraries into the primary prefix would not be enough. Any port wanting to build against openssl 1.1 would need to find the headers, and there is no way to have both these side by side under /opt/local/include. The *only* solution is to specifically configure the build to use an alternate install area for openssl 1.1, the one under libexec, where everything is available with the nominal directory structures.

[cjones051073 (Chris Jones)]

Replying to thetrial:

In short: the libexec solution will make problems. I agree with Mouse.

not when the user keeps their ports up to date, and when ports correctly use the openssl PG to build against specific older versions.

[mouse07410 (Mouse)]

To summarize: please reconsider.

No. The only reason we are discussing this is because the OPs ports where not up to date.

This is not only about Macports ports. It's about tons of user-installed apps that require OpenSSL, and chose to rely upon Macports-provided OpenSSL.

Just putting the 1.1 libraries into the primary prefix would not be enough. Any port wanting to build against openssl 1.1 would need to find the headers

Again, you're talking about ports (as if no other software existed). I'm talking about a pile of existing installed binaries - not ports - that suddenly stopped working. Some of them may not be easy to rebuild.

[mascguy (Christopher Nielsen)]

Folks, bear in mind that this is just a temporary blip, while we get everything working against OpenSSL 3. (Or revert ports to use OpenSSL 1.1, for the limited cases where that's necessary.)

So please try to be patient: Everyone is working hard to resolve all of the issues, and we'll get it straightened out quickly. But the more comments and such that we have to respond to, the less time we'll have for fix-related work!

[thetrial (alabay)]

Replying to cjones051073:

Replying to thetrial:

In short: the libexec solution will make problems. I agree with Mouse.

not when the user keeps their ports up to date, and when ports correctly use the openssl PG to build against specific older versions.

Yes, you are right … when. But #62220 is a decent example of breaking things.

[cjones051073 (Chris Jones)]

Replying to thetrial:

Replying to cjones051073:

Replying to thetrial:

In short: the libexec solution will make problems. I agree with Mouse.

not when the user keeps their ports up to date, and when ports correctly use the openssl PG to build against specific older versions.

Yes, you are right … when. But #62220 is a decent example of breaking things.

We expected the migration to openssl3 to throw up ports that would need to be pegged back to 1.1. I've just posted instructions there how to do this, its not hard.

[cjones051073 (Chris Jones)]

Replying to mouse07410:

To summarize: please reconsider.

No. The only reason we are discussing this is because the OPs ports where not up to date.

This is not only about Macports ports. It's about tons of user-installed apps that require OpenSSL, and chose to rely upon Macports-provided OpenSSL.

Just putting the 1.1 libraries into the primary prefix would not be enough. Any port wanting to build against openssl 1.1 would need to find the headers

Again, you're talking about ports (as if no other software existed). I'm talking about a pile of existing installed binaries - not ports - that suddenly stopped working. Some of them may not be easy to rebuild.

*all* existing binaries indeed need to be rebuilt. This is (was) expected. All ports where in fact rev-bump when openssl3 was made the default for the very reason. Most rebuilt just fine against openssl3. The handful that have not nbeed to be addressed. Again, this was totally expected. Fixing them all *before* the migration was not practical, so this was the pragmatic approach.

[mouse07410 (Mouse)]

The only reason we are discussing this is because the OPs ports where not up to date.

No! That's just one symptom - there are plenty more of these. For me, the majority (surprisingly!) are not Macports-installed.

We're discussing this because now with one quick change all of the binaries (including those not installed by Macports!) that were ever linked with OpenSSL-1.1.1, stopped working. It is impractical to expect a normal user to suddenly become a full-time sysadmin who goes through the apps one by one, reconfiguring and rebuilding them until everything starts working again.

To remind: one reason people turn to Macports is exactly to avoid having to do a lot of heavy lifting on their own.

not when the user keeps their ports up to date, and when ports correctly use the openssl PG to build against specific older versions.

Again and again - OpenSSL is a port that provides services to pretty much all the apps and libraries that require OpenSSL on Mac, partially because you can't even link against LibReSSL version that comes with MacOS. So, breaking it you not only (temporarily) break some ports - you break all the other user-installed software.

A lot of that software uses pkg-config to locate OpenSSL. So far, it was not able to locate OpenSSL-1.1.1 in libexec - special instructions are necessary.

*all* existing binaries indeed need to be rebuilt. This is (was) expected. All ports where in fact rev-bump when openssl3 was made the default for the very reason.

I'm afraid I'm not making myself clear enough. You seem to be talking exclusively about ports...?

Most rebuilt just fine against openssl3

Unfortunately, outside of ports themselves, this is wrong - some do, and some don't.

so this was the pragmatic approach.

I concur with everything except for not symlinking 1.1 libs into the primary prefix.

OK, I don't think I've explained the situation clearly enough to convince you. Fine. Thankfully, manual symlinking works, and if that approach contradicts your wisdom - I'd have to live with that.

[mascguy (Christopher Nielsen)]

Mouse, we appreciate your passion, as well as your various contributions to MacPorts. And we also appreciate your pain and frustration, which is inevitable with a big migration like this.

Just bear in mind that we're all feeling it too, including myself. And everyone's working really, really hard to fix this stuff! And that includes @cjones, who's been continually jumping in with prompt fixes, in numerous areas, since we started the process.

But the more time he wastes on replying to comments, the less time he has for real work. And that goes for all of us.

Again, we feel your pain too. But can we please try to tone down the comments and such, while we get all of this working?

Thanks,

-Chris

[thetrial (alabay)]

Let's be honest, that's the most exciting thread for a long time :-)

I appreciate the work of you all, folks. If someone fixes even this stuck #62220 thing, I’ll be happy and try again updating. Revbumping usually is no problem, unless …

Thanks for all! ;-)

[mouse07410 (Mouse)]

Just bear in mind that we're all feeling it too, including myself. And everyone's working really, really hard to fix this stuff!

I certainly do, and I appreciate the hard work you guys are putting into it. I'm one of those grateful for getting the benefits of those efforts.

My problem is that just one (and pretty small scale-wise, in my view) decision of several made in this process broke things far beyond the Macports itself.

And you can't do anything to fix at least some of this stuff, because it's not the ports. You certainly can (maybe, given time) fix whatever port that got broken. You may be able to affect upstream, when it needs fixing. But what power do you have over other applications?

My contention and frustration is that the workaround for all those apps and binaries is merely adding two symlinks. And voila! - users aren't blown out of the water until whatever apps they need are either ported to OSSL-3.0 (quite non-trivial in some cases) or rebuilt from source (also, non-trivial in some cases).

I feel that resistance against this symlinks-workaround (mainly useful and necessary for non-Macports binaries!) is almost on a religious level, considering how simple, standard, and harmless the proposed approach is.

Examples:

$ ll /opt/local/lib/libbotan*.dylib
-rwxr-xr-x  1 root  admin  15700184 Nov  1 10:14 /opt/local/lib/libbotan-2.18.18.2.dylib*
lrwxr-xr-x  1 root  admin        24 Nov  1 10:14 /opt/local/lib/libbotan-2.18.dylib@ -> libbotan-2.18.18.2.dylib
lrwxr-xr-x  1 root  admin        24 Nov  1 10:14 /opt/local/lib/libbotan-2.dylib@ -> libbotan-2.18.18.2.dylib
-rwxr-xr-x  1 root  admin   5194248 Nov  2 18:03 /opt/local/lib/libbotan-3.0.0.0.dylib*
lrwxr-xr-x  1 root  admin        22 Nov  2 18:03 /opt/local/lib/libbotan-3.0.dylib@ -> libbotan-3.0.0.0.dylib
lrwxr-xr-x  1 root  admin        22 Nov  2 18:03 /opt/local/lib/libbotan-3.dylib@ -> libbotan-3.0.0.0.dylib
$ ll /opt/local/lib/libcrypto*.dylib
lrwxr-xr-x  1 root  admin  52 Nov  8 10:43 /opt/local/lib/libcrypto.1.1.dylib@ -> /opt/local/libexec/openssl11/lib/libcrypto.1.1.dylib
lrwxr-xr-x  1 root  admin  49 Nov  7 15:31 /opt/local/lib/libcrypto.3.dylib@ -> /opt/local/libexec/openssl3/lib/libcrypto.3.dylib
lrwxr-xr-x  1 root  admin  17 Nov  7 15:31 /opt/local/lib/libcrypto.dylib@ -> libcrypto.3.dylib

and

$ ll /usr/local/lib/libyubihsm*.dylib
-rwxr-xr-x@   1 root  wheel   200352 Sep  6  2017 libyubihsm.0.1.0.dylib*
-rwxr-xr-x@   1 root  wheel   212132 Sep 14  2017 libyubihsm.0.2.0.dylib*
-rwxr-xr-x@   1 root  wheel   218848 Jul 26  2018 libyubihsm.1.dylib*
-rwxr-xr-x    1 root  wheel   136776 Mar 25  2019 libyubihsm.2.0.0.dylib*
-rwxr-xr-x    1 root  wheel   144880 Sep 24  2019 libyubihsm.2.0.1.dylib*
-rwxr-xr-x    1 root  wheel   148956 Sep  9  2020 libyubihsm.2.0.2.dylib*
-rwxr-xr-x    1 root  wheel   201904 Mar  1  2021 libyubihsm.2.0.3.dylib*
lrwxr-xr-x    1 root  wheel       22 Nov 26  2018 libyubihsm.2.0.dylib@ -> libyubihsm.2.0.0.dylib
-rwxr-xr-x    1 root  wheel   185736 Apr 14  2021 libyubihsm.2.1.0.dylib*
-rwxr-xr-x    1 root  wheel   186288 Nov  2 12:02 libyubihsm.2.2.0.dylib*
lrwxr-xr-x    1 root  wheel       22 Apr 29  2021 libyubihsm.2.dylib@ -> libyubihsm.2.2.0.dylib
lrwxr-xr-x    1 root  wheel       18 Nov 26  2018 libyubihsm.dylib@ -> libyubihsm.2.dylib

[mascguy (Christopher Nielsen)]

Mouse, we're presently so overwhelmed with trying to get everything working, that none of us have any time - or energy either, for that matter - to even digest what you're saying.

So can you kindly hold off on this discussion for a few weeks? Once the dust has settled - and we've all been able to take a much-needed break, and catch up on our sleep - we can revisit this.

At the moment though, you're simply adding to everyone's stress, not to mention our inboxes.

[thetrial (alabay)]

You guys made and make a great job! Now everything works again. Thanks.