py39-scrapy @2.7.1 apparently installs without problems, but produces errors when run

[typonaut]

py39-scrapy appears to install without issues, but when run produces errors (MacOS 10.6):

python39 -m scrapy
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/__main__.py", line 1, in <module>
    from scrapy.cmdline import execute
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/cmdline.py", line 9, in <module>
    from scrapy.crawler import CrawlerProcess
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/crawler.py", line 18, in <module>
    from scrapy.core.engine import ExecutionEngine
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/core/engine.py", line 17, in <module>
    from scrapy.core.scraper import Scraper
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/core/scraper.py", line 24, in <module>
    from scrapy.utils.log import failure_to_exc_info, logformatter_adapter
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/utils/log.py", line 12, in <module>
    from scrapy.utils.versions import scrapy_components_versions
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/utils/versions.py", line 12, in <module>
    from scrapy.utils.ssl import get_openssl_version
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/utils/ssl.py", line 1, in <module>
    import OpenSSL
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import SSL, crypto
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/OpenSSL/SSL.py", line 9, in <module>
    from OpenSSL._util import (
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 14, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so, 2): Symbol not found: _ERR_put_error
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so
  Expected in: flat namespace
 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so

or

scrapy
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 581, in _build_master
    ws.require(__requires__)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 909, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 800, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (cryptography 3.3.2 (/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages), Requirement.parse('cryptography<39,>=38.0.0'), {'pyOpenSSL'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/local/bin/scrapy", line 33, in <module>
    sys.exit(load_entry_point('Scrapy==2.7.1', 'console_scripts', 'scrapy')())
  File "/opt/local/bin/scrapy", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/__init__.py", line 12, in <module>
    from scrapy.spiders import Spider
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/spiders/__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/http/__init__.py", line 11, in <module>
    from scrapy.http.request.form import FormRequest
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scrapy/http/request/form.py", line 12, in <module>
    from parsel.selector import create_root_node
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/parsel/__init__.py", line 16, in <module>
    from parsel.selector import Selector, SelectorList  # NOQA
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/parsel/selector.py", line 22, in <module>
    from pkg_resources import parse_version
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3260, in <module>
    def _initialize_master_working_set():
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3234, in _call_aside
    f(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3272, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 596, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pkg_resources/__init__.py", line 800, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (cryptography 3.3.2 (/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages), Requirement.parse('cryptography<39,>=38.0.0'), {'pyOpenSSL'})

[kurthindenburg (Kurt Hindenburg)]

I just updated this to 2.8.1; can you try that version? It may be that 10.6 needs special code.

[jmroot (Joshua Root)]

Seems unlikely that that would fix the missing symbol in the .so installed by py39-cryptography. It's suspicious that the cryptography version found is 3.3.2.

[typonaut]

I think the py39-cryptography module is at the heart of this problem.

I tried installing the 38.0.3 version that is claimed to be the latest available, but get this:

sudo port install py39-cryptography @38.0.3_0
Error: py39-cryptography version 38.0.3_0 is not available (current version is 3.3.2_0)

The 3.3.2 version dates from 2021, the latest version available via PyPi is 39.0.0, but I get a cargo rustc error when attempting to install that.

[kurthindenburg (Kurt Hindenburg)]

It has to be hitting this, so 3.3.2 is the only version that you'll get.

set cryptography_darwin_min_ver 13


    # legacy support
    if {${python.version} eq 27
        || ${os.platform} eq "darwin" && ${os.major} < ${cryptography_darwin_min
_ver}} {
        PortGroup       openssl 1.0
        github.setup    pyca cryptography 3.3.2

[typonaut]

There seem to be a number of problems with this:

i) The indication seems to be that cryptography requires MacOS 13.

ii) The python version checking seems obsolete if one is running port install py39-cryptography and not clear whether it is checking for other than the default python (my Mac has at least seven different python versions on it).

iii) An obsolete version of cryptography is installed, without any meaningful warning.

iv) The scrapy install proceeds, after installing a cryptography version, with which it cannot run.

Perhaps a (obsolete) scrapy package should be paired up with a (obsolete) cryptography package with which it will run?

[kurthindenburg (Kurt Hindenburg)]

That says "python27 or "os.major < 13"

For you suggestion, you'd have to test all the previous version to see which one works. I don't have an old system.

[typonaut]

It seems to me that the test of Python version, or OS version, is not a lot of help, because, having a python >= 2.7 still results in a scrapy that does not run, because the cryptography version is not sufficient.

From my tests, it appears that cryptography > 3.3.2 cannot be installed on MacOS 10.6, with pip, even if you are running python >= 3.9. There may be some other way to install cryptography, but I do not know what it is.

So, upon running port install scrapy I think you need to ask the user into which Python version they wish to install the module. When installing the dependencies, the cryptography install should fail, if it is attempting to install a version that the scrapy install does not support. The setup.py for scrapy 2.8.0 says "cryptography>=3.4.6" – so the python 2.7 test than ends up installing cryptography 3.3.2 does not meet that requirement.

I have checked the versions of scrapy available on github, and the last version that appears to support cryptography 3.3.2 is scrapy 2.7. So, at the least, if bumping the cryptography down to 3.3.2 upon the install, the scapy should also be downgraded to 2.7 - with a notice/check to the user in both cases.

Thanks!

[kencu (Ken)]

newer versions of py-cryptography require rust to build.

rust used to only work on 10.13 and up, so older systems were pegged to an older version of py-cryptography.

then rust was fixed to build back to 10.5, but py-cryptography has not yet been changed to account for that.

Somebody needs to update that.

[typonaut]

I can confirm that I have rust 1.67.1. I have tried again to install py-cryptography, while the port install output reads:

---> Installing py-cryptography @39.0.1_0

---> Activating py-cryptography @39.0.1_0

---> Cleaning py-cryptography

I only appear to have 3.3.2.