python312 @3.12.6 cannot compile against libressl

[SaintBol]

Using Xcode clang compiler (Command Line Tools not present): cannot compile the port on macOS 15 Sequoia.

Attached main.log

Issues:

:info:build ./Modules/_ssl.c:4532:18: error: call to undeclared function 'X509_OBJECT_set1_X509'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
:info:build  4532 |             ok = X509_OBJECT_set1_X509(ret, X509_OBJECT_get0_X509(obj));
:info:build       |                  ^
:info:build ./Modules/_ssl.c:4532:18: note: did you mean 'X509_OBJECT_get0_X509'?
:info:build /opt/local/include/openssl/x509_vfy.h:285:7: note: 'X509_OBJECT_get0_X509' declared here
:info:build   285 | X509 *X509_OBJECT_get0_X509(const X509_OBJECT *xo);
:info:build       |       ^

:info:build ./Modules/_ssl.c:4536:18: error: call to undeclared function 'X509_OBJECT_set1_X509_CRL'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
:info:build  4536 |             ok = X509_OBJECT_set1_X509_CRL(
:info:build       |                  ^
:info:build ./Modules/_ssl.c:4536:18: note: did you mean 'X509_OBJECT_get0_X509_CRL'?
:info:build /opt/local/include/openssl/x509_vfy.h:286:11: note: 'X509_OBJECT_get0_X509_CRL' declared here
:info:build   286 | X509_CRL *X509_OBJECT_get0_X509_CRL(X509_OBJECT *xo);
:info:build       |           ^
:info:build ./Modules/_ssl.c:4553:1: error: static declaration of 'X509_STORE_get1_objects' follows non-static declaration
:info:build  4553 | X509_STORE_get1_objects(X509_STORE *store)
:info:build       | ^
:info:build /opt/local/include/openssl/x509_vfy.h:296:24: note: previous declaration is here
:info:build   296 | STACK_OF(X509_OBJECT) *X509_STORE_get1_objects(X509_STORE *xs);
:info:build       |                        ^
:info:build ./Modules/_ssl.c:4556:10: error: call to undeclared function 'X509_STORE_lock'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
:info:build  4556 |     if (!X509_STORE_lock(store)) {
:info:build       |          ^
:info:build ./Modules/_ssl.c:4559:11: error: call to undeclared function 'sk_X509_OBJECT_deep_copy'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
:info:build  4559 |     ret = sk_X509_OBJECT_deep_copy(X509_STORE_get0_objects(store),
:info:build       |           ^
:info:build ./Modules/_ssl.c:4559:9: error: incompatible integer to pointer conversion assigning to 'struct stack_st_X509_OBJECT *' from 'int' [-Wint-conversion]
:info:build  4559 |     ret = sk_X509_OBJECT_deep_copy(X509_STORE_get0_objects(store),
:info:build       |         ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:info:build  4560 |                                    x509_object_dup, X509_OBJECT_free);
:info:build       |                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:info:build ./Modules/_ssl.c:4561:5: error: call to undeclared function 'X509_STORE_unlock'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
:info:build  4561 |     X509_STORE_unlock(store);
:info:build       |     ^

[SaintBol]

main.log

[jmroot (Joshua Root)]

I'm guessing this is probably a case of libressl not being 100% compatible with openssl.

[SaintBol]

Actually libressl was auto-installed as dependancy of the following requested ports:

 ldns @1.8.3_1
 rsync @3.3.0_0
 libevent @2.1.12_2

[jmroot (Joshua Root)]

Yes, and libressl claims to satisfy the openssl dependency but clearly doesn't in some cases.

[SaintBol]

I can confirm, it compiles with libressl removed and openssl installed instead.

[ryandesign (Ryan Carsten Schmidt)]

Replying to SaintBol:

Actually libressl was auto-installed as dependancy of the following requested ports:

 ldns @1.8.3_1
 rsync @3.3.0_0
 libevent @2.1.12_2

No, those ports depend on openssl, unless you installed libressl first or (in the case of ldns) selected its libressl variant.

[SaintBol]

I didn't. First I

port uninstall installed

then I

port install csshX gnutar grepcidr inetutils ipcalc iperf3 ipv6calc ldns lft lftp minicom mtr nmap p7zip pv py-awscli rsync sipcalc smartmontools tmux tmux-pasteboard truncate unrar watch xz

Because of the various issues with commandlinetools it became some sequences of port install with some stuff removed, but no mention of libressl there...

[ryandesign (Ryan Carsten Schmidt)]

Ok, I've read the ldns portfile more closely and it does indeed say that if neither openssl nor libressl are installed, then it will default to the libressl variant. I don't know why. It is highly unusual for a port to vary its behavior based on what ports are already installed and my vote would be for eliminating that behavior and defaulting to openssl like all other ports do. rpki-client does the same thing.

[jmroot (Joshua Root)]

As far as python312 is concerned, I guess the dependency just needs to be port:openssl if it doesn't work with libressl.

[artkiver (グレェ)]

Replying to jmroot:

As far as python312 is concerned, I guess the dependency just needs to be port:openssl if it doesn't work with libressl.

I think, generally speaking, Portfiles would benefit from something such as:

depends_lib         path:lib/libssl.dylib:openssl \

Referencing the dylib facilitates a port to be more TLS library agnostic.

With regards to ldns, that was from a PR here: ​https://github.com/macports/macports-ports/pull/17568

As suggested by the commit message, the "extra smarts" were inspired by modifications made to the rpki-client Portfile, from suggestions provided by @neverpanic (at the moment I am not even sure why it is not CCing Clemens Lang, did I forget how to use Trac? I swear there used to be a means to provide a drop down username select.)

Lamentably, I have attempted, unsuccessfully, to try to apply similar "smarts" to the Got Portfile, but while I documented some of my failed attempts here: #69827 I never came up with a satisfactory refactoring which functioned as intended.

It's my personal observation, maybe something changed with how TLS is being handled by MacPorts, not just the Portfiles? It's plausible that rpki-client also does not function with those similar "smarts" as it did when it was originally tested and committed.

Anyway, while a smarter approach might be beneficial, I never was able to make it generically consistent and maybe something changed since that ldns PR was merged (which most certainly was thoroughly tested locally, as was the rpki-client change prior to being merged).

However, this kind of stuff would sincerely benefit from more attention than I can personally provide at present given my limited resources. I do not even have mpbb running locally despite multiple attempts at following those instructions, they haven't gelled.

I would also posit that rather than defaulting to OpenSSL, defaulting to LibreSSL, given that macOS has used that as its reference TLS library for approximately a decade at this point, might be a wiser strategy? IIRC that may touch over 800 Portfiles though and is more than a one person job, to understate it.

If MacPorts ever gets around to holding hackathons again, it's the sort of thing which could probably use its own. I know Bob Beck (one of the LibreSSL developers) personally and maybe could inspire him to visit too, but last I checked the last MacPorts in person gathering was before COVID and in Europe. I am in California, Bob Beck is in Canada and Clemens I think is in Germany?

[neverpanic (Clemens Lang)]

Replying to artkiver:

Replying to jmroot:

As far as python312 is concerned, I guess the dependency just needs to be port:openssl if it doesn't work with libressl.

I think, generally speaking, Portfiles would benefit from something such as:

depends_lib         path:lib/libssl.dylib:openssl \

Referencing the dylib facilitates a port to be more TLS library agnostic.

Sure – but in this case the alternate ports that provide lib/libssl.dylib do not actually work, as Python fails to compile. Either (a) libressl needs to commit to providing the same API as openssl does, or (b) somebody needs to do the work of making sure Python continues to work with libressl upstream by adding appropriate #ifdefs and alternative code paths, or (c) somebody needs to do this work downstream to patch Python to add this support, or (d) Python needs to depend on openssl, not on path:lib/libssl.dylib:openssl.

For this case, it seems like it's going to be (d).

(b) and (c) also wouldn't work correctly, since we build binaries of Python that we distribute to users, which would use X509_OBJECT_set1_X509 since the buildbot compiles against OpenSSL. Those binaries would not work when installed on a system where lib/libssl.dylib does not provide this symbol.

With regards to ldns, that was from a PR here: ​https://github.com/macports/macports-ports/pull/17568

As suggested by the commit message, the "extra smarts" were inspired by modifications made to the rpki-client Portfile, from suggestions provided by @neverpanic (at the moment I am not even sure why it is not CCing Clemens Lang, did I forget how to use Trac? I swear there used to be a means to provide a drop down username select.)

I suggested these "extra smarts" with the variant precisely so that prebuilt binaries compiled against openssl would not be used on systems that have libressl installed and vice-versa, since they are not binary compatible.

It's my personal observation, maybe something changed with how TLS is being handled by MacPorts, not just the Portfiles? It's plausible that rpki-client also does not function with those similar "smarts" as it did when it was originally tested and committed.

I don't think there's anything that changed in MacPorts. It's just that OpenSSL and LibreSSL are no longer compatible (enough), for this to work.

However, this kind of stuff would sincerely benefit from more attention than I can personally provide at present given my limited resources. I do not even have mpbb running locally despite multiple attempts at following those instructions, they haven't gelled.

I would also posit that rather than defaulting to OpenSSL, defaulting to LibreSSL, given that macOS has used that as its reference TLS library for approximately a decade at this point, might be a wiser strategy? IIRC that may touch over 800 Portfiles though and is more than a one person job, to understate it.

Most upstreams follow OpenSSL, attempting to use LibreSSL would lead to problems such as the one in Python reported here. I don't have time to make all those ports work with LibreSSL, and neither do I want to considering that we'll eventually want to do a post-quantum transition and OpenSSL has a working solution for post-quantum cryptography right now (using liboqs and oqsprovider) and will integrate one soon. Also, OpenSSL's support for smartcards will be vastly improved using pkcs11-provider. I don't see any of those improvements happening in LibreSSL (although I honestly haven't looked, either).

Personally, I'd go for OpenSSL and drop LibreSSL support completely.

[artkiver (グレェ)]

Thanks for chiming in Clemens!

"Personally, I'd go for OpenSSL and drop LibreSSL support completely."

Well, OK, but if I recall correctly, I think you once mentioned you get paid in your day job to work on OpenSSL? Not to disparage what Steve Jobs referred to as "vested interests" but I kind of expect that advocacy from you. ;)

Similarly, Python's creator, Guido van Rossum is currently employed as a "Distinguished Engineer" at Microsoft as one of his two jobs.

I'm not paid to work on any of this!

I've never really liked Python for a lot of other reasons; but I swear I had built it in MacPorts against LibreSSL multiple times within the past year. Not just python312, but also python313-devel and earlier versions of Python too. So maybe this is something that changed upstream recently?

Choice (d) is probably the "easiest" option for sure!

Choices c and d will require more work from someone, but that someone probably isn't me.

I was kind of hoping maybe there would be a way to refactor the "smarts" Clemens helpfully offered for rpki-client which I was unsuccessful at refactoring for Got, but that doesn't seem particularly plausible in this instance either, darn.

Would it be beneficial to simplify the ldns Portfile in this instance?

[ryandesign (Ryan Carsten Schmidt)]

Replying to artkiver:

I think, generally speaking, Portfiles would benefit from something such as:

depends_lib         path:lib/libssl.dylib:openssl \

Referencing the dylib facilitates a port to be more TLS library agnostic.

Referencing the dylib makes the port incompatible with non-macOS operating systems, so please choose any other file that the port installs rather than the dylib.

[jmroot (Joshua Root)]

In a7e01ca3ad01cdef1fcd595ec257e0525dac8396/macports-ports (master):

python312: disallow building against libressl

Closes: #70767

[neverpanic (Clemens Lang)]

Replying to artkiver:

Thanks for chiming in Clemens!

"Personally, I'd go for OpenSSL and drop LibreSSL support completely."

Well, OK, but if I recall correctly, I think you once mentioned you get paid in your day job to work on OpenSSL? Not to disparage what Steve Jobs referred to as "vested interests" but I kind of expect that advocacy from you. ;)

Just to put that into perspective, I (currently) work at Red Hat in the RHEL crypto team, and one of the components this team maintains is OpenSSL. If Red Hat would consider LibreSSL the better alternative, we'd have no problem in packaging that instead. It is both my team's and my opinion that LibreSSL isn't better. I do not work directly for OpenSSL.

Similarly, Python's creator, Guido van Rossum is currently employed as a "Distinguished Engineer" at Microsoft as one of his two jobs.

I'm not paid to work on any of this!

I'm also not being paid to maintain OpenSSL in MacPorts (or any MacPorts work, really). I do this in my free time, same as you.