Opened 2 months ago

Closed 2 months ago

Last modified 8 weeks ago

#70764 closed defect (fixed)

Segfault fetching ports from FTP URLs

Reported by: Ygar-Vanaxe-Tsy-xtledron (YVT) Owned by: jmroot (Joshua Root)
Priority: Normal Milestone: MacPorts 2.10.2
Component: base Version: 2.10.1
Keywords: monterey, ventura, sonoma, sequoia Cc: Ygar-Vanaxe-Tsy-xtledron (YVT), jmroot (Joshua Root), wmszeliga, ryandesign (Ryan Carsten Schmidt), hapaguy (Brian Kurt Fujikawa), Dave-Allured (Dave Allured)
Port:

Description (last modified by Ygar-Vanaxe-Tsy-xtledron (YVT))

When migrating to darwin 24, restore command failed due to ncurses seg fault.

$ sudo port install ncurses
--->  Fetching archive for ncurses
--->  Attempting to fetch ncurses-6.5_0.darwin_24.arm64.tbz2 from https://packages.macports.org/ncurses
--->  Attempting to fetch ncurses-6.5_0.darwin_24.arm64.tbz2 from https://cph.dk.packages.macports.org/ncurses
--->  Attempting to fetch ncurses-6.5_0.darwin_24.arm64.tbz2 from https://mse.uk.packages.macports.org/ncurses
--->  Fetching distfiles for ncurses
--->  Attempting to fetch ncurses-6.5.tar.gz from ftp://ftp.funet.fi/pub/gnu/prep/ncurses
[1]    29141 segmentation fault  sudo port install ncurses

As of earlier today, ncurses installs without issue. However, the same problem now occurs with psutils.

--->  Computing dependencies for psutils
--->  Fetching archive for psutils
--->  Attempting to fetch psutils-p17_1.darwin_24.arm64.tbz2 from https://packages.macports.org/psutils
--->  Attempting to fetch psutils-p17_1.darwin_24.arm64.tbz2 from https://cph.dk.packages.macports.org/psutils
--->  Attempting to fetch psutils-p17_1.darwin_24.arm64.tbz2 from https://mse.uk.packages.macports.org/psutils
--->  Fetching distfiles for psutils
--->  Attempting to fetch psutils-p17.tar.gz from ftp://mirror.macomnet.net/pub/CTAN/support/psutils
[1]    92420 segmentation fault  sudo port install psutils

Or sometimes due to bus error.

--->  Computing dependencies for psutils
--->  Fetching archive for psutils
--->  Attempting to fetch psutils-p17_1.darwin_24.arm64.tbz2 from https://packages.macports.org/psutils
--->  Attempting to fetch psutils-p17_1.darwin_24.arm64.tbz2 from https://cph.dk.packages.macports.org/psutils
--->  Attempting to fetch psutils-p17_1.darwin_24.arm64.tbz2 from https://mse.uk.packages.macports.org/psutils
--->  Fetching distfiles for psutils
--->  Attempting to fetch psutils-p17.tar.gz from ftp://mirror.macomnet.net/pub/CTAN/support/psutils
[1]    92223 bus error  sudo port install psutils

Attachments (1)

main.log (60.0 KB) - added by Ygar-Vanaxe-Tsy-xtledron (YVT) 2 months ago.
psutils_log

Download all attachments as: .zip

Change History (28)

comment:1 Changed 2 months ago by Ygar-Vanaxe-Tsy-xtledron (YVT)

Keywords: segfault, sequoia, ,migrationsegfault, sequoia, migration

comment:2 Changed 2 months ago by Ygar-Vanaxe-Tsy-xtledron (YVT)

Keywords: ncurses added
Priority: NormalHigh

comment:3 Changed 2 months ago by Ygar-Vanaxe-Tsy-xtledron (YVT)

Cc: Ygar-Vanaxe-Tsy-xtledron added

comment:4 Changed 2 months ago by jmroot (Joshua Root)

Description: modified (diff)
Keywords: segfault ncurses migration removed
Summary: ncurses Segfault on MacOS 15.0Segfault fetching ncurses on MacOS 15.0

Can you attach the crash report?

comment:5 Changed 2 months ago by Ygar-Vanaxe-Tsy-xtledron (YVT)

Nothing crashed despite the segfault, ncurses simply failed to install making its dependents (pandoc for example) also un-installable. As of writing, ncurses installs fine.

sudo port install ncurses
--->  Fetching archive for ncurses
--->  Attempting to fetch ncurses-6.5_0.darwin_24.arm64.tbz2 from https://packages.macports.org/ncurses
--->  Attempting to fetch ncurses-6.5_0.darwin_24.arm64.tbz2 from https://cph.dk.packages.macports.org/ncurses
--->  Attempting to fetch ncurses-6.5_0.darwin_24.arm64.tbz2 from https://mse.uk.packages.macports.org/ncurses
--->  Fetching distfiles for ncurses
--->  Attempting to fetch ncurses-6.5.tar.gz from https://fra.de.distfiles.macports.org/ncurses
--->  Verifying checksums for ncurses
--->  Extracting ncurses
--->  Applying patches to ncurses
--->  Configuring ncurses
--->  Building ncurses
--->  Staging ncurses into destroot
--->  Installing ncurses @6.5_0
--->  Activating ncurses @6.5_0
--->  Cleaning ncurses
--->  Updating database of binaries
--->  Scanning binaries for linking errors
--->  No broken files found.
--->  No broken ports found.
Last edited 2 months ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:6 Changed 2 months ago by Ygar-Vanaxe-Tsy-xtledron (YVT)

Description: modified (diff)
Priority: HighNormal
Summary: Segfault fetching ncurses on MacOS 15.0Segfault fetching ports on MacOS 15.0

Changed 2 months ago by Ygar-Vanaxe-Tsy-xtledron (YVT)

Attachment: main.log added

psutils_log

comment:7 Changed 2 months ago by Ygar-Vanaxe-Tsy-xtledron (YVT)

Latest attempt to install outputs this:

sudo port install psutils
--->  Computing dependencies for psutils
--->  Fetching archive for psutils
--->  Attempting to fetch psutils-p17_1.darwin_24.arm64.tbz2 from https://packages.macports.org/psutils
--->  Attempting to fetch psutils-p17_1.darwin_24.arm64.tbz2 from https://cph.dk.packages.macports.org/psutils
--->  Attempting to fetch psutils-p17_1.darwin_24.arm64.tbz2 from https://mse.uk.packages.macports.org/psutils
--->  Fetching distfiles for psutils
--->  Attempting to fetch psutils-p17.tar.gz from ftp://mirror.macomnet.net/pub/CTAN/support/psutils
tclsh8.6(93125,0x1f3f77240) malloc: *** error for object 0x600001384081: pointer being freed was not allocated
tclsh8.6(93125,0x1f3f77240) malloc: *** set a breakpoint in malloc_error_break to debug
[1]    93124 abort      sudo port install psutils

comment:8 in reply to:  5 Changed 2 months ago by ryandesign (Ryan Carsten Schmidt)

Replying to Ygar-Vanaxe-Tsy-xtledron:

Nothing crashed despite the segfault

A segfault is a crash. Please double-check whether macOS has recorded a crash log and if so please attach it; without it it's difficult for anyone to diagnose where in the code the problem needs to be fixed.

comment:9 Changed 2 months ago by jmroot (Joshua Root)

The common factor appears to be FTP sites.

comment:10 Changed 2 months ago by jmroot (Joshua Root)

Cc: jmroot added
Component: portsbase
Port: ncurses removed
Summary: Segfault fetching ports on MacOS 15.0Segfault fetching ports from FTP URLs on MacOS 15.0

comment:11 Changed 2 months ago by wmszeliga

I'm having the same issue with installing the psutils port. It appears as if tclch8.6 is the program crashing. Here's the relevant thread info from the crash log.

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [48902]

VM Region Info: 0x892513e3ad is not in any region.  Bytes after previous region: 107996242862  Bytes before following region: 104964083686483
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      commpage (reserved)        1000000000-7000000000   [384.0G] ---/--- SM=NUL  reserved VM address space (unallocated)
--->  GAP OF 0x5f9000000000 BYTES
      MALLOC_NANO              600000000000-600020000000 [512.0M] rw-/rwx SM=PRV  

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libcurl.4.dylib               	       0x1b53e27b4 Curl_bufq_free + 36
1   libcurl.4.dylib               	       0x1b5401134 http2_data_done + 216
2   libcurl.4.dylib               	       0x1b53fe2d4 cf_h2_cntrl + 108
3   libcurl.4.dylib               	       0x1b53dafb0 Curl_conn_ev_data_detach + 104
4   libcurl.4.dylib               	       0x1b540fc2c Curl_detach_connection + 40
5   libcurl.4.dylib               	       0x1b542d968 extract_if_dead + 376
6   libcurl.4.dylib               	       0x1b542d7d8 call_extract_if_dead + 36
7   libcurl.4.dylib               	       0x1b53dba98 Curl_conncache_foreach + 152
8   libcurl.4.dylib               	       0x1b542c324 prune_dead_connections + 152
9   libcurl.4.dylib               	       0x1b542b008 Curl_connect + 4420
10  libcurl.4.dylib               	       0x1b5410904 multi_runsingle + 476
11  libcurl.4.dylib               	       0x1b541067c curl_multi_perform + 208
12  Pextlib.dylib                 	       0x102b727d8 CurlFetchCmd + 2256
13  Pextlib.dylib                 	       0x102b73c60 CurlCmd + 212
14  libtcl8.6.dylib               	       0x102d752b8 TclNRRunCallbacks + 80
15  libtcl8.6.dylib               	       0x102e00778 ChildEval + 132
16  libtcl8.6.dylib               	       0x102e0211c NRChildCmd + 744
17  libtcl8.6.dylib               	       0x102d752b8 TclNRRunCallbacks + 80
18  libtcl8.6.dylib               	       0x102d7603c TclEvalEx + 1668
19  libtcl8.6.dylib               	       0x102e148cc Tcl_FSEvalFileEx + 512
20  libtcl8.6.dylib               	       0x102e1b428 Tcl_MainEx + 920
21  tclsh8.6                      	       0x102a83a4c main + 48
22  dyld                          	       0x19c304274 start + 2840

comment:12 Changed 2 months ago by jmroot (Joshua Root)

Cc: wmszeliga added

OK, so it's crashing pretty deep in libcurl calls, which means it's probably an Apple bug. The question is, how can we mitigate it?

@wmszeliga, was your crash also when fetching from an FTP site? If so, it's a bit odd that http2_data_done is involved.

comment:13 Changed 2 months ago by jmroot (Joshua Root)

The workarounds from ProblemHotlist#fetch-failures should work here as well. You could also try editing macports.conf and adding distfiles.macports.org or some other http mirror to preferred_hosts.

comment:14 in reply to:  12 Changed 2 months ago by ryandesign (Ryan Carsten Schmidt)

Replying to jmroot:

OK, so it's crashing pretty deep in libcurl calls, which means it's probably an Apple bug.

Not necessarily. We may be calling curl incorrectly in a way that causes the crash. However I see macOS 15.0 has curl 8.7.1 and this looks like https://github.com/curl/curl/issues/13731 which was fixed in curl 8.8.0.

I tried /usr/bin/curl -OL ftp://ftp.funet.fi/pub/gnu/prep/ncurses/ncurses-6.5.tar.gz on macOS 15 and it downloaded successfully without crashing so it is possible to download from ftp without crashing. The upstream bug report suggests the crash occurs when a curl easy handle that had been used for http2 is then reused for a different protocol. The fix would be not reusing curl easy handles.

comment:15 Changed 2 months ago by ryandesign (Ryan Carsten Schmidt)

Keywords: monterey ventura sonoma added
Summary: Segfault fetching ports from FTP URLs on MacOS 15.0Segfault fetching ports from FTP URLs

macOS 12.7.6, macOS 13.7, and macOS 14.7 also shipped with curl 8.7.1 and would also be affected but users probably notice it less or not at all since we have binaries for those OS versions. macOS 11.7.10 has curl 8.1.2 and is not affected.

comment:16 Changed 2 months ago by jmroot (Joshua Root)

Owner: set to jmroot
Resolution: fixed
Status: newclosed

In ff881a52faeb700213259c33163a552bd7325f05/macports-base (master):

Work around libcurl crash when reusing handles

Closes: #70764

comment:17 Changed 2 months ago by ryandesign (Ryan Carsten Schmidt)

Cc: ryandesign added
Milestone: MacPorts Future

Thanks Josh. Hopefully we can get Apple to ship fixed versions of curl. I was trying to create a minimal example that causes a crash that I could include in a bug report to Apple but so far my example does not crash.

comment:18 Changed 2 months ago by jmroot (Joshua Root)

It seems like you often need multiple reuses of a handle to get it to the point where it actually crashes. The most minimal example I found so far was port -v livecheck libsdl2_net libsdl2_ttf postfix.

comment:19 Changed 2 months ago by ryandesign (Ryan Carsten Schmidt)

I meant a minimal example not requiring MacPorts. I wrote a C program to do an http2 request and then an ftp request and it succeeded.

Your fix dumps and reinitializes the handle if the scheme changes, but the upstream bug report didn't talk about schemes; it talked about protocols. I suspect the crash would occur if accessing an http2 host and then accessing an http1 host but I would like to test that in a minimal example program.

comment:20 Changed 2 months ago by jmroot (Joshua Root)

Based on the above, this works as a reproducer using only /usr/bin/curl:

/usr/bin/curl -o 1.txt -o 2.txt -o 3.txt https://github.com/libsdl-org/SDL_net/tags https://github.com/libsdl-org/SDL_ttf/tags ftp://ftp.porcupine.org/mirrors/postfix-release/official/

comment:21 Changed 2 months ago by jmroot (Joshua Root)

Replacing the third URL in that example with http://www.iozone.org/, which is a HTTP 1.1 site, doesn't crash.

comment:22 Changed 2 months ago by jmroot (Joshua Root)

The fix I committed stopped the crash for livecheck, but I've since seen the same crash while fetching a port. Maybe making a new handle if either the scheme or the host change would work, and allow us to keep the performance benefits of reusing the handle for subsequent requests to the same host.

comment:23 Changed 2 months ago by jmroot (Joshua Root)

In 1ac551fcd29eb5f24b80a6b8e9198b262f1d4938/macports-base (master):

Better curl crash workaround

Clean up and reallocate the multi handle if either the URL scheme or
host changed from the previous transfer.

See: #70764

comment:24 Changed 2 months ago by jmroot (Joshua Root)

That seems to do the trick so far. Note that I found that reallocating the multi handle, not the easy handle, was necessary. That doesn't seem to be what the upstream discussion was implying, so maybe it would be a good idea to follow up there about whether we actually need to reallocate both handles.

comment:25 Changed 8 weeks ago by jmroot (Joshua Root)

Milestone: MacPorts FutureMacPorts 2.10.2

comment:26 Changed 8 weeks ago by hapaguy (Brian Kurt Fujikawa)

Cc: hapaguy added

comment:27 Changed 8 weeks ago by Dave-Allured (Dave Allured)

Cc: Dave-Allured added
Note: See TracTickets for help on using tickets.