After a time, MacPorts cannot determine the Xcode version, on Snow Leopard

[ryandesign (Ryan Carsten Schmidt)]

After a Snow Leopard machine has been up for a time, MacPorts becomes unable to determine what Xcode version is installed.

For example, most recently, ​build 7473 was able to determine the Xcode version:

DEBUG: Xcode 3.2.6

And the next build in the batch, ​build 7474, was unable to:

DEBUG: Xcode none

This in turn caused all subsequent builds to fail with:

Error: Port foo requires a full Xcode installation, which was not found on your system.
Error: You can install Xcode from the Mac App Store or https://developer.apple.com/xcode/

even for ports that don't specify use_xcode yes.

Restarting the machine fixes the problem.

When the problem is occurring, xcodebuild -version still returns the expected Xcode version information when run manually from the command line. So I don't think the problem is Xcode.

How long the system will work after a restart seems to vary. In this most recent case, the problem occurred after just 12 days of uptime. But the 10.6-10.8 workers have been unusually busy, working non-stop to rebuild packages after the switchover to libc++, so the number of builds or perhaps the number of times MacPorts tries to exec xcodebuild -version (or perhaps the number of times it tries to exec anything) may be a more relevant measure.

I am now wondering if the problem is that a tmp directory needed by MacPorts or Tcl to execute the xcodebuild -version command and process its results is getting deleted or its permissions are getting set incorrectly, which restarting the machine fixes, or if perhaps we are leaking file descriptors and at some point we just don't have any free descriptors left to use one to get the xcodebuild process's output. Perhaps we could log relevant tmp directory permissions and number of open files in mpbb somewhere.

I have not observed this on any other macOS version, only on Snow Leopard. Yet it has been a persistent problem for our x86_64 Snow Leopard buildbot worker for the three years the new buildbot system has been online. (I don't remember if we saw the problem on the prior buildbot system.) When the problem occurs, it causes builds to fail; someone has to notice that, restart the worker, and manually reschedule the failed builds. It would be good to identify and fix the cause of the problem, since aside from making the life of the buildbot administrator easier, it may turn out to be responsible for other as yet unexplained MacPorts misbehavior.

It's not a corruption of the OS on a specific macOS install. When we switched to libc++, we switched to a new 10.6 virtual machine, which exhibits this same problem that we saw on the old libstdc++ VM. I don't think I've seen the problem on the old or the new i386 Snow Leopard worker, but that needs to be restarted often for other reasons (kernel panic when building perl modules sometimes, due to i386 VMs not being truly supported by VMware ESXi) so we may just be avoiding the problem there by restarting the system before the problem hits.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

perhaps we are leaking file descriptors and at some point we just don't have any free descriptors left to use one to get the xcodebuild process's output.

the new i386 Snow Leopard worker [experiences] kernel panic when building perl modules sometimes

Those kernel panics could also be caused by a lack of available file descriptors; see #60509.