python27: OpenSSL error causing hashlib to malfunction

[Lord-Kamina (Gregorio Litenstein)]

So... I originally noticed these errors a couple days ago when trying to build qt5-qtwebengine. After some google, I found identical errors from people running Homebrew, which I've copied below,

ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)

And these were corrected by using OpenSSL-1.0. I tried that, everything went without a hitch and thus I made ​https://github.com/macports/macports-ports/pull/11868

@jmroot pointed out that he didn't have the same issue and thus, this was probably a symptom and not the actual issue.

I did some more digging and I think I found what's actually happening.

For context, I build most of my ports targetting older systems. In this case I was targetting 10.13 (although I also tried 10.12 and 10.14 and the error persisted).

Checking the python build log I saw the following:

usr/bin/clang -bundle -undefined dynamic_lookup -m64 -mmacosx-version-min=10.13 -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -arch x86_64 build/temp.macosx-10.13-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_sqlite/cache.o build/temp.macosx-10.13-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_sqlite/connection.o build/temp.macosx-10.13-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_sqlite/cursor.o build/temp.macosx-10.13-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_sqlite/microprotocols.o build/temp.macosx-10.13-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_sqlite/module.o build/temp.macosx-10.13-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_sqlite/prepare_protocol.o build/temp.macosx-10.13-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_sqlite/row.o build/temp.macosx-10.13-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_sqlite/statement.o build/temp.macosx-10.13-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_sqlite/util.o -L/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/lib -lsqlite3 -o build/lib.macosx-10.13-x86_64-2.7/_sqlite3.so -Wl,-search_paths_first
*** WARNING: renaming "_sqlite3" since importing it failed: dlopen(build/lib.macosx-10.13-x86_64-2.7/_sqlite3.so, 2): Symbol not found: _sqlite3_enable_load_extension
  Referenced from: build/lib.macosx-10.13-x86_64-2.7/_sqlite3.so
  Expected in: flat namespace
 in build/lib.macosx-10.13-x86_64-2.7/_sqlite3.so

Python build finished, but the necessary bits to build these modules were not found:
_ssl               bsddb185           dl              
gdbm               imageop            linuxaudiodev   
ossaudiodev        spwd               sunaudiodev     
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_sqlite3  

Running otool -L on the failed .so file showed me it was built against /usr/lib/libsqlite3.dylib instead of the macports version.

Adding a -L/opt/local/lib to the configure.ldflags when building python27 makes it link against the macports library instead, eliminating the above errors.

[kencu (Ken)]

I see in the PR that everything works properly for you now.

Can we close this?

[Lord-Kamina (Gregorio Litenstein)]

Replying to kencu:

I see in the PR that everything works properly for you now.

Can we close this?

No. My PR was the wrong fix to the issue that I'm reporting here, which is still a thing:

For some reason (I suspect to do with changing deployment target), the _sqlite3 module tries to link against the system version of libsqlite3.dylib instead of the one provided by macports, this causes the module to fail and the _ssl module to not be built at all.

This can be fixed by passing the proper library dir under the macports folder when compiling python27.

[jmroot (Joshua Root)]

Replying to Lord-Kamina:

Adding a -L/opt/local/lib to the configure.ldflags when building python27 makes it link against the macports library instead, eliminating the above errors.

The default value of configure.ldflags is -L${prefix}/lib, so it's hard to see how that would help. Are you seeing the problem if you install the prebuilt binary, or only with your local builds?

[Lord-Kamina (Gregorio Litenstein)]

Ok, ignore me. You can close this.

[jmroot (Joshua Root)]

I'm not denying you're seeing a problem, I just don't think it's fully understood given the available information. If you can identify what's different in the logs of working and non-working builds, that might help?

[Lord-Kamina (Gregorio Litenstein)]

Sorry. I wasn't trying to br pouty. You just made me realize I solved a problem I largely created myself by unwittingly overriding the default ldflags.

[jmroot (Joshua Root)]

Ah, OK, that would certainly do it. :) No worries.