Pandas Segmentation Fault on Import

[essandess (Steve Smith)]

The ports py*-pandas create a segfault on import. This happens on both arm64 and x86_64 boxes, installed from prebuilt or built from source, for both Python 310 and 311, and using the latest MacPorts version and the latest available version from ​https://github.com/macports/macports-ports/pull/19544.

python3 -c 'import pandas as pd'
Segmentation fault: 11

This may be associated with the port jemalloc. I wasn't able to create a core dump, but here's the maOS crash report:

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x00000000000023f8
Exception Codes:       0x0000000000000001, 0x00000000000023f8

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [16142]

VM Region Info: 0x23f8 is not in any region.  Bytes before following region: 105553518910472
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      MALLOC_NANO (reserved)   600018000000-600020000000 [128.0M] rw-/rwx SM=NUL  ...(unallocated)

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libjemalloc.2.dylib           	       0x144803d28 je_free_default + 152
1   libprotobuf.3.21.12.0.dylib   	       0x144ef71f0 google::protobuf::internal::ArenaStringPtr::Destroy() + 56

[essandess (Steve Smith)]

Further information:

I do not see this behavior for every version of MacPorts Python.

Segmentation faults occur for Python 310 and 311, but not Python 39, i.e. python3.9 -c 'import pandas as pd' works, but python3.10 -c 'import pandas as pd' and python3.11 -c 'import pandas as pd' both crash.

Additionally, though I observe this issue on multiple boxes—a Mac Studio M2 Ultra and a Mac Pro 2019 Intel—I do not observe it on a Mac Mini 2018 Intel.

This may be an underlying Python issue.

[essandess (Steve Smith)]

I found the issue using faulthandler: it's a pyarrow import compiled from ​https://github.com/macports/macports-ports/pull/19664, itself likely related to protobuf3-cpp per the crash dump above.

I've converted that PR to a draft and ask that this issue be closed.

PYTHONFAULTHANDLER=1 python3.11 -c 'import pandas as pd'
Fatal Python error: Segmentation fault

Current thread 0x00000001e83da080 (most recent call first):
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1233 in create_module
  File "<frozen importlib._bootstrap>", line 573 in module_from_spec
  File "<frozen importlib._bootstrap>", line 676 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pyarrow/__init__.py", line 65 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pandas/compat/pyarrow.py", line 8 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pandas/compat/__init__.py", line 29 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pandas/__init__.py", line 22 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<string>", line 1 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.hashing (total: 37)
Segmentation fault: 11

[essandess (Steve Smith)]

Superseded by ​https://trac.macports.org/ticket/67900

[essandess (Steve Smith)]

In a80c8a588c168815c6fdb5e776269f4859b84441/macports-ports (master):

apache-arrow: Update to version 13.0.0

• Update to version 13.0.0
• Add Python 311
• Delete Python 37
• Remove jemalloc dependency
  • See: ​https://github.com/jemalloc/jemalloc/issues/2511
• Fixes: #67900
• Fixes: #67896