remoted hangs on Big Sur when installing p5* ports, losing network and sometimes panicking kernel

[BishopBerkeley]

When trying to install

p5.28-package-stash            @0.390.0        perl/p5-package-stash

as part of an "upgrade outdated" command, port consistently causes a kernel panic. If someone can direct me to the appropriate log file, I can post the transcript.

I'm running ports on BS. I simply installed the latest ports for BS over the old one. Xcode and command line tools have all been updated, too.

[ryandesign (Ryan Carsten Schmidt)]

You're the third person to report this problem on Big Sur. Please see prior discussion at ​https://lists.macports.org/pipermail/macports-users/2020-November/049004.html and ​https://lists.macports.org/pipermail/macports-dev/2020-November/042573.html. I believe it is related to problems previously observed by one user on Lion and by me on 32-bit Snow Leopard; see #60509.

Do you have any third-party software installed that intercepts network traffic, such as Little Snitch or Carbon Black?

[BishopBerkeley]

I do, indeed, have Little Snitch running. However, even when I turn Little Snitch's network monitor off, the kernel panic still happens.

[ryandesign (Ryan Carsten Schmidt)]

Try my suggestions from those threads: use MacPorts master (2.6.99) instead. If that doesn't work, change the limit of 50 simultaneous pings to something lower.

[ryandesign (Ryan Carsten Schmidt)]

Also, try fully uninstalling Little Snitch. Maybe it still inserts itself into your network stack even if you turn its network monitoring off.

[jmroot (Joshua Root)]

Not saying we shouldn't work around this if possible, but kernel panics are always a bug in the kernel or a kernel extension (and I believe Big Sur no longer supports kernel extensions). So you should definitely report this to Apple.

[BishopBerkeley]

Replying to ryandesign:

Try my suggestions from those threads: use MacPorts master (2.6.99) instead. If that doesn't work, change the limit of 50 simultaneous pings to something lower.

Thank you. I will try to get to that. How do I change the ping limit?

[BishopBerkeley]

Replying to ryandesign:

You're the third person to report this problem on Big Sur. Please see prior discussion at ​https://lists.macports.org/pipermail/macports-users/2020-November/049004.html and ​https://lists.macports.org/pipermail/macports-dev/2020-November/042573.html. I believe it is related to problems previously observed by one user on Lion and by me on 32-bit Snow Leopard; see #60509.

Do you have any third-party software installed that intercepts network traffic, such as Little Snitch or Carbon Black?

Yes, those two threads describe what I'm experiencing as well. I have upgrade to little Snitch 5.0.3, which ditches the kext and implements its monitoring another way. I have activated LS's Endpoint security option, which pulls data from the Berkeley packet filter. This did not change the outcome. But, it seems as if the problem is not dependent on LS since the other two reports don't report having a network monitor installed.

[ryandesign (Ryan Carsten Schmidt)]

Replying to BishopBerkeley:

How do I change the ping limit?

See ​https://lists.macports.org/pipermail/macports-dev/2020-November/042592.html

Replying to BishopBerkeley:

it seems as if the problem is not dependent on LS since the other two reports don't report having a network monitor installed.

Perhaps not. But it is probably dependent on something nonstandard that you and those other users have on your systems, since most Big Sur users do not experience the problem. If you can think of what that might be, let us know.

[BishopBerkeley]

I removed Little Snitch and successfully installed all outdated ports. Upon uninstalling LS, the software informed me that it was "hosting" certain extensions. I guess this is the new way the process goes under BS. I will apprise Objective Software of the bug in LS that persists up to version 5.0.4.

Thank you for your help.

[BishopBerkeley]

Can you provide the full path to the folder that contains the number of simultaneous pings? I cannot follow the path provided in the thread below.

Replying to ryandesign:

Replying to BishopBerkeley:

How do I change the ping limit?

See ​https://lists.macports.org/pipermail/macports-dev/2020-November/042592.html

[BishopBerkeley]

It seems as if this bug is a kernel bug that will be fixed by Apple in BS 11.1. Details on Objective Development's web site. ​https://obdev.at/support/littlesnitch Expand the "troubleshooting" header.

[BishopBerkeley]

I upgraded to 11.1 last night, but the problem persisted. I usually install the latest command line tools by launching Xcode, but since they were not installed on launching xcode this time, I installed them manually with xcode-select. This solved the problem.

So, this problem is solved by ensuring that you have updated to MacOS 11.1 AND installed or rinstalled the latest command line tools. This resolves the kernel panic caused by Little Snitch.

[jtrtik (Jan Trtík)]

I have still this problem even after upgrading to MacOS 11.1 and reinstalling command line tools. The problem is less frequent, but still present... I sent report to Apple via Feedback Assistant and output of spindump command to Objective Development for further inspection.

[mf2k (Frank Schima)]

Works fine for me now. Are you running version 5.0.4 of Little Snitch?

[jtrtik (Jan Trtík)]

Yes, i am running the latest version of Little Snitch. The issue is less frequent, but still present. The system gets restarted because /usr/libexec/remoted hangs. It is clearly bug in MacOS.

Folks at Objective Development recommended to try to kill the hanging process as soon as i notice there will be a panic soon. Noz sure the system will fully recover, but it is worth a try.

[ryandesign (Ryan Carsten Schmidt)]

Replying to BishopBerkeley:

Replying to ryandesign:

See ​https://lists.macports.org/pipermail/macports-dev/2020-November/042592.html

Can you provide the full path to the folder that contains the number of simultaneous pings? I cannot follow the path provided in the thread below.

The default installation path of the file is /opt/local/libexec/macports/lib/port1.0/fetch_common.tcl

[ryandesign (Ryan Carsten Schmidt)]

Replying to BishopBerkeley:

I usually install the latest command line tools by launching Xcode, but since they were not installed on launching xcode this time, I installed them manually with xcode-select. This solved the problem.

Opening Xcode never installs or updates the command line tools. It does install or update "additional components" but that is a different thing (for mobile device development).

[thermalecology]

I get a Kernel panic when trying to upload the URI perl module. I'm using 11.2. It's been happening for weeks now.

[bosmacs (Craig Bosma)]

Ran into this problem after upgrading directly to 11.2 from Catalina -- installing/upgrading anything that hit the CPAN mirrors would stop local networking from working at all, applications would hang, etc. Uninstalled Little Snitch 5.0.4 and the problem is gone, at least for the moment.

[deusama (Amadeus)]

Same issue here after upgrading to 11.2 directly from Catalina. I have never installed Little Snitch tho. As soon as I hit the "Fetching distfiles ..." part it starts crashing.

[krissen (Kristian Niemi)]

Replying to deusama:

Same issue here after upgrading to 11.2 directly from Catalina. I have never installed Little Snitch tho. As soon as I hit the "Fetching distfiles ..." part it starts crashing.

Just echoing what deusama said. Was working fine on Catalina. Having upgraded to 11.2 the kernel started panicking when fetching distfiles for perl ports. I do not have Little Snitch or some such installed.

Is changing ping times as described ​here the recommended workaround? 

[krissen (Kristian Niemi)]

• Installed port from git master (port reports v. 2.6.99).
• (Now have the line set max_hosts_to_ping 50 in /opt/local/libexec/macports/lib/port1.0/fetch_common.tcl.)
• Set max_hosts_to_ping to 10.
• Can now fetch perl ports without kernel panics. 🎉

[nortcele]

Same issue here during installation of port @p5.28-mozilla-ca@.

Macports 2.6.4

macOS 11.2.2 20D80

Xcode 12.4 12D4e

[hakusaro (Lucas Nicodemus)]

I had the same problem, found after googling in resignation and landing at #62360. I'm on macOS 11.2.2 with Little Snitch 5.1.1. The interesting attributes I have are that I'm on an M1 system, so I figured it was a Rosetta issue or Apple Silicon issue. It didn't appear to be. While I couldn't get around the problem by disabling the network filter from the Little Snitch menu, completely removing the network filter interface in the Network prefpane worked.

I sent Apple Feedback a ticket with ID FB9031599. If you encounter the problem, it's worth running sudo sysdiagnose as the problem happens, then attach the CLI generated tar.gz that starts with sysdiagnose to a feedback report using applefeedback://. If you mention this thread and the feedback ID I used, it might help this get elevated up the chain at Apple so that someone can solve it.

My guess is that network filters exacerbate some kind of race condition or deadlock in the network stack, but the problem probably isn't directly related. Nevertheless, if you encounter the problem, it's worth reporting it, removing any network filter interfaces, and/or trying the max ping workaround.

Fundamentally, useful sysdiagnose dumps and feedback reports are going to be the way to get this to the right people at Apple.

[BishopBerkeley]

For the record, I haven’t had this problem since the bug was fixed by apple in macos 11.1. I use a 2017 macbook, latest xcode and little snitch 5.1.1

[BishopBerkeley]

I stand corrected. I got a kernel panic this morning at this stage of installing perl modules:

Attempting to fetch p5.28-date-manip-6.850.0_0.darwin_20.noarch.tbz2 from https://mse.uk.packages.macports.org/p5.28-date-manip
--->  Fetching distfiles for p5.28-date-manip

I have followed the above instructions and submitted the sysdiagnose file to the ticket ID above.

[jmroot (Joshua Root)]

Retitling since the issue doesn't always result in a kernel panic, sometimes just loss of network connectivity.

[BishopBerkeley]

I just managed to install everything without incident with BS 11.2.3.

[jakw0j]

Replying to krissen:

• Installed port from git master (port reports v. 2.6.99).
• (Now have the line set max_hosts_to_ping 50 in /opt/local/libexec/macports/lib/port1.0/fetch_common.tcl.)
• Set max_hosts_to_ping to 10.
• Can now fetch perl ports without kernel panics. 🎉

how to install port from git master? i can't find a solution

[krissen (Kristian Niemi)]

Replying to jakw0j:

how to install port from git master? i can't find a solution

Follow the instructions for Git install ​here but stay on master instead of checking out tag 2.6.4 (i.e. skip the line where the install instructions say "# skip this if you want to use the development version"). Good luck!

[jakw0j]

is it going to make all my ports to use 2.6.99? im confused now. so i am forced to use not latest version of macports?

Replying to krissen:

Replying to jakw0j:

how to install port from git master? i can't find a solution

Follow the instructions for Git install ​here but stay on master instead of checking out tag 2.6.4 (i.e. skip the line where the install instructions say "# skip this if you want to use the development version"). Good luck!

[krissen (Kristian Niemi)]

Replying to jakw0j:

is it going to make all my ports to use 2.6.99? im confused now. so i am forced to use not latest version of macports?

We’re talking about installing the development branch of port. If you’re unsure what that means or entails, I would suggest it it perhaps better not to proceed.

[jakw0j]

i see no other option as i cant install i.e. git because my machine crashes everytime on 2.6.4 :(

port: p5.28-mozilla-ca

[kencu (Ken)]

grand total time invested = 120 seconds

cd /tmp
git clone https://github.com/macports/macports-base.git
cd macports-base
./configure && make && sudo make install
cd ..
rm -rf macports-base

[jakw0j]

well, i've done that - no need to be rude

edit: anyway, thanks for help :)

[kencu (Ken)]

there are downsides to doing this. there are (AFAIK) no update prompts to tell you when MP has been updated, or to run a new "update" automatically. So you just "have to know". And some features are not fully debugged.

Good luck!

[minusf]

Replying to kencu:

grand total time invested = 120 seconds

cd /tmp
git clone https://github.com/macports/macports-base.git
cd macports-base
./configure && make && sudo make install
cd ..
rm -rf macports-base

my grand total time invested in this issue is in days already.

this issue is not new. it has been widely reported by many people. although it causes no kernel panic at the moment ("only" full networking loss until reboot), the going advice is "use master" and/or with some undocumented option overrides.

yes, it's not directly macports' fault per se, but it triggers it, and it makes running macports akin to playing russian roulette. i know macports is a volunteer effort, but i find it strange not to release a minor patch version with a temporary fix while the dust settles around the other software involved: macos, little snitch, etc. the others would be hardly hugely affected, 200+ mirrors for any package makes little sense to me, just to speed test them would be slower than grabbing a random mirror.

running master is not officially recommended by the project, otherwise what would be the point of releases. can we have a fix please even if only temporary?

[krissen (Kristian Niemi)]

Replying to BishopBerkeley:

I just managed to install everything without incident with BS 11.2.3.

The perl files weren't cached, were they? Anyway, I was curious, on BS 11.2.3. Although the release notes didn't seem to mention anything related, I reverted the changes to fetch_common.tcl and downgraded to the latest release of MacPorts (2.6.4). However, a port fetch -d p5.26-http-message brought the network down and eventually panicked the kernel as well.

Reinstalling MacPorts master (2.6.99), changing max_hosts_to_ping back to 10, and the port fetch command worked again, without kernel panics etc. *shrug*

[kencu (Ken)]

I'm sorry but I'm not on the chain of people that can help with a new release. I haven't seen or heard any rumblings about a new release, but that doesn't mean one might not happen later today either for all I know.

My only input here, not particularly appreciated it seems, was to try to point out that to install base from master is a really simple process that (IMHO) should not be considered a major obstacle to happiness. You can install the next release right over top of master whenever it comes.

[jakw0j]

i really appreciate your input. im sorry if i misunderstood your intentions and assumed that you were rude. im still learning, so that is quite an adventure for me to be honest

[jmroot (Joshua Root)]

If you would like a minimal workaround rather than installing from master, making this change to /opt/local/libexec/macports/lib/port1.0/fetch_common.tcl will prevent anything from being pinged (which means sites will be tried in the order they are listed):

/opt/local/libexec/macports/lib/port1.0/fetch_common.tcl

@@ -263 +263 @@
                 # first check the persistent cache
                 set pingtimes($host) [get_pingtime $host]
                 if {$pingtimes($host) eq {}} {
-                    if {[catch {set fds($host) [open "|ping -noq -c3 -t3 $host"]}]} {
+                    if {[catch {set fds($host) [open "|false -noq -c3 -t3 $host"]}]} {
                         ui_debug "Spawning ping for $host failed"
                         # will end up after all hosts that were pinged OK but before those that didn't respond
                         set pingtimes($host) 5000

[minusf]

Replying to jmroot:

thank you for the quick fix. i am in a back and forth with a little snitch dev but he cannot reproduce this no matter what. really sad state of the network stack in macos right now.

[conradwt (Conrad Taylor)]

At this time, I have 161 CPAN packages installed which are 22.2% of the total installed packages. Does any know which CPAN ports are required for MacPorts?

[minusf]

the latest info is that this particular bug is already fixed in 11.3 beta. let's see how it goes.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #62724.

[acheronfail]

I've also found that has been happening to me quite consistently when installing p5.28-sgmlspm. The above patch (changing ping to false) has resolved the problem for me.

Let's hope that 11.3 fixes this issue.

My system: 11.2.3, latest macports, Xcode SDK installed and also have Little Snitch 5.1.2 installed. If you'd like any more info hit me up. :)

[jkdufair (Jason Dufair)]

I will add that I had this issue as well and the culprit was Cisco AnyConnect VPN. Tried to install the git port and experienced the same problems described in this ticket. Uninstalling Cisco AnyConnect seems to have been a suitable workaround

[blair (Blair Zajac)]

Since upgrading to Big Sur 11.3 I haven't been able to reproduce the problem with multiple invocations of

$ rm /opt/local/var/macports/pingtimes
$ /opt/local/bin/port clean --all p5.30-libwww-perl
$ /opt/local/bin/port -v fetch p5.30-libwww-perl

[dsavransky (Dmitry Savransky)]

Another confirmation that 11.3 appears to fix this. Previously (on 11.2.1) doing port install git +svn (which pulls down multiple p5* ports) was a guaranteed kernel panic. Just updated to 11.3 (and updated xcode to 12.5) and was able to the install with zero issues. I also have Cisco AnyConnect installed (which has its own share of issues on big sur) and did not need to uninstall.

[jmroot (Joshua Root)]

OK, given the above comments saying 11.3 works, and no reports to the contrary, I think it's safe to close this.