python ports: update to use openssl 3, allowing dependent binaries to be redistributed

[mascguy (Christopher Nielsen)]

Utilizing OpenSSL 3 should help eliminate the current license conflicts, which prevent redistribution of binaries for dependent ports.

NOTE: This ticket covers our foundational Python ports only - python{26,27}, and python{32..39,310}. Assuming that can be done (?), without having to immediately do so for all Python libs/components, dependent upon those.

[cjones051073 (Chris Jones)]

For some(*) ports using the openssl PG, switching to openssl 3 is just a matter of adding

openssl.branch 3

to the portfile in question.

(*) I say some because if that works depends on how amenable the build in question is to using an openssl installation that is from a different install area (libexec) to the primary prefix (e.g. /opt/local). 'regular' builds, e.g. ./configure, cmake etc. usually have configuration options to set this (and the PG already tries to handle cmake internally). python builds are a bit different and I am not sure how this would work with them.

My gut feeling is doing this en-mass with the above (say by putting it into the python PG) will not work as enough ports won't like it, to make it a non-starter. For them, access via the shim openssl port, which currently points to 1.1 still, might be the way to go. So to do this we might well end up back at the discussion as to when would be the right time to make openssl 3 the default (which can now be done by just changing the default branch in the openssl PG).

[kencu (Ken)]

just change the python Portfiles to build with openssl3 ...nowhere else.

the license fix will trickle down to 5000 ports, none of which care how python is built, but are whacked by the python <-> openssl license anyway.

[cjones051073 (Chris Jones)]

Ah, I see. Mis-understood. You aren't talking about all the py-XYZ ports but just the core pythonX ports. Yes, changing those to use the openssl PG, and then v3, should be much simpler.

[reneeotten (Renee Otten)]

I've made these changes locally already and everything builds fine; I can push them in a PR to save Josh some time if you're interested in that.

[mascguy (Christopher Nielsen)]

Replying to reneeotten:

I've made these changes locally already and everything builds fine; I can push them in a PR to save Josh some time if you're interested in that.

Awesome! And yes, please!

[reneeotten (Renee Otten)]

well... it's still Josh his ticket and call whether he wants to move to openssl3 at all ;)

[mascguy (Christopher Nielsen)]

Sorry, my excitement surrounding this change, is getting the better of me! LOL

[cjones051073 (Chris Jones)]

Renee, please do create a PR with your changes, we can review there.

[jmroot (Joshua Root)]

Why not just update openssl to 3?

[cjones051073 (Chris Jones)]

Well yes, an option. But then We are then back at the old discussion as to if now is the best time to do this enmass, when we now there are ports out there that will not work with openssl 3 and will need pegging back to 1.1. Are we happy with making the change to 3 first, and then finding these ports as-and-when they turn up, or do we have to find them all before hand ?

[reneeotten (Renee Otten)]

Replying to jmroot:

Why not just update openssl to 3?

I agree, that would solve all of this licensing business at once and I'm a strong proponent of just doing that! We can fix whatever ports that do not build with openssl3 by specifying them to build with v1.1 using the openssl PG. There seems to be no consensus though whether or not to do this (with Ken a strong proponent of "yes" and Chris of "no, not yet"). The discussion on the mailinglist didn't really result in a conclusion, but if you feel comfortable with making that change Josh, I would be happy about it.

[reneeotten (Renee Otten)]

Replying to cjones051073:

Well yes, an option. But then We are then back at the old discussion as to if now is the best time to do this enmass, when we now there are ports out there that will not work with openssl 3 and will need pegging back to 1.1. Are we happy with making the change to 3 first, and then finding these ports as-and-when they turn up, or do we have to find them all before hand ?

do it now - fix things that don't work and if people are interested in those ports. If ports really don't build and the software is old and unmaintained upstream, that's would be great reason to purge them IMHO...

[cjones051073 (Chris Jones)]

I am also in favour of just updating to openssl 3 and letting the dust settle …..

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

I am also in favour of just updating to openssl 3 and letting the dust settle …..

Well, here's a different perspective: I'm presently neck-deep in rabbit holes, trying to fix buildbot issues, across umpteen different foundational ports. Failures which are blocking a huge number of downstream ports from building, with a tremendous number of total casualties.

And given that we're nowhere close to being in a good state, in terms of such things...

I'm definitely NOT in favor of a blanket switch to OpenSSL 3, unless you folks are willing to actively take on fixing everything that fails.

[cjones051073 (Chris Jones)]

Well then how do you propose such a switch happens ? If we insist on waiting until all openssl users are tested against 3 and if need be pegged back to 1.1 we will be waiting forever to make the change, as that is not going to happen.

[cjones051073 (Chris Jones)]

Just to note we are now back at precisely the same discussion that happens on the mailing lists a while back. As far as I see it there are only two options

1. never update
2. update, and fix the fallout as and when it is found.

I just don't see any other practical alternatives.

[mascguy (Christopher Nielsen)]

Well, I'm still a little bit unclear how we made the jump from the scope of this ticket - which is to ONLY change the version of OpenSSL used for our core PythonXX ports - to suddenly doing it for everything.

Unless I completely misunderstood what you folks are proposing. In which case, Never Mind... ;-)

[jmroot (Joshua Root)]

Most of the necessary fixing was already done with the transitions to 1.0 and 1.1. A lot of the breakage in those versions was due to changes that enable a more stable API and ABI going forward, like switching to opaque structs.

[cjones051073 (Chris Jones)]

Replying to mascguy:

Well, I'm still a little bit unclear how we made the jump from the scope of this ticket - which is to ONLY change the version of OpenSSL used for our core PythonXX ports - to suddenly doing it for everything.

Unless I completely misunderstood what you folks are proposing. In which case, Never Mind... ;-)

I was simply replying to Josh's query 'Why not just update openssl to 3?'

[cjones051073 (Chris Jones)]

Replying to jmroot:

Most of the necessary fixing was already done with the transitions to 1.0 and 1.1. A lot of the breakage in those versions was due to changes that enable a more stable API and ABI going forward, like switching to opaque structs.

AFAIK the openssl folks don't rule out some ABI breakage between 1.1 and 3. But, yes, I would expect it to be fairly minimal.

[mascguy (Christopher Nielsen)]

If we think the migration will be as painless as it was for libjpeg-turbo - and apart from a few minor Universal-related things, it was as close to flawless as a mass-migration can be - than I'm fine with it.

[cjones051073 (Chris Jones)]

​https://github.com/macports/macports-ports/pull/12807

just really don't want this to turn into another war-and-peace saga...

[reneeotten (Renee Otten)]

thanks Chris, shouldn't turn into a saga I would think... Maintainers anyway have 72 hours to respond / test the proposed change; so ideally they would make sure it builds with the new openssl and, if not, set it back to the 1.1 branch.

[cculianu (Calin Culianu)]

Welp. OpenSSL3 on-by-default broke Python3.x. This no longer works:

$ python3
Python 3.10.0 (default, Nov  7 2021, 21:08:03) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.algorithms_available
{'sha512_224', 'ripemd160', 'md5-sha1', 'sha1', 'shake_128', 'shake_256', 'sha512_256', 'sha256', 'whirlpool', 'blake2b', 'sha3_512', 'sha384', 'sm3', 'blake2s', 'sha3_224', 'sha224', 'mdc2', 'md4', 'md5', 'sha3_256', 'sha3_384', 'sha512'}
>>> hashlib.new('ripemd160')
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/hashlib.py", line 160, in __hash_new
    return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] initialization error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/hashlib.py", line 166, in __hash_new
    return __get_builtin_constructor(name)(data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type ripemd160

[reneeotten (Renee Otten)]

Replying to cculianu:

Welp. OpenSSL3 on-by-default broke Python3.x. This no longer works:

$ python3
Python 3.10.0 (default, Nov  7 2021, 21:08:03) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.algorithms_available
{'sha512_224', 'ripemd160', 'md5-sha1', 'sha1', 'shake_128', 'shake_256', 'sha512_256', 'sha256', 'whirlpool', 'blake2b', 'sha3_512', 'sha384', 'sm3', 'blake2s', 'sha3_224', 'sha224', 'mdc2', 'md4', 'md5', 'sha3_256', 'sha3_384', 'sha512'}
>>> hashlib.new('ripemd160')
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/hashlib.py", line 160, in __hash_new
    return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] initialization error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/hashlib.py", line 166, in __hash_new
    return __get_builtin_constructor(name)(data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type ripemd160

you already report this as 63857 Trac 63857; the update to openssl3 discussed here was completed. Whatever fallout happens because of that should be filed separately (as you did already, thank you!).

[reneeotten (Renee Otten)]

okay, so with this commit (thanks Chris!), it should all work again so we can move back to openssl3?!

[jmroot (Joshua Root)]

I will check if the tests pass for each version.

[jmroot (Joshua Root)]

In 1834f57c9b2df368be6b82ef819669174921856f/macports-ports (master):

python39, python310: switch to openssl 3

Hashlib and SSL tests confirmed passing.

See: #63801

[reneeotten (Renee Otten)]

thank you Joshua! Out of curiosity: does this mean that the earlier versions do not pass or you didn't have time to check them?

[jmroot (Joshua Root)]

In 9b310d616d289a59c4f9829ad5ae841488a24fba/macports-ports (master):

python38: switch to openssl 3

Hashlib and SSL tests confirmed passing.

See: #63801

[jmroot (Joshua Root)]

Replying to reneeotten:

Out of curiosity: does this mean that the earlier versions do not pass or you didn't have time to check them?

I hadn't tested them all yesterday, but now I have. For python 3.7 and older, test_ssl passes when built against openssl 1.1 but has some failures when built against openssl 3.