10.6-i386 buildbot worker is down

[ryandesign (Ryan Carsten Schmidt)]

One of our VMware hosts has suffered an SSD failure, as a result of which the 10.6-i386, 10.8, 10.11 and 10.13 workers are down and will need to be restored from backups. There may be separate issues involved in bringing each of these back online so I'll make separate tickets. I plan to restore them to spare hard disks temporarily until I can get a new SSD.

[ryandesign (Ryan Carsten Schmidt)]

I reinstalled 10.6 server from DVD, but it didn't want to restore from the Time Machine backup because "Migration is not supported from servers which are configured to use dynamic network addresses"; see ​https://discussions.apple.com/thread/2467711. The solution as it says there was to edit the file /Library/Preferences/SystemConfiguration/preferences.plist in the backup to indicate a manual IPv4 address instead of a DHCP one, but backups are protected from editing. As I found at ​https://apple.stackexchange.com/questions/31679/how-can-i-delete-a-file-marked-as-backup-item the solution was to use the bypass program inside the TMSafetyNet.kext.

That having been done, it restored the backup. Unlike on newer systems it did not ask me which users I wanted to restore, but also unlike on newer systems all the users appear to have been restored correctly including their original home directories.

OS updates are now being installed.

[ryandesign (Ryan Carsten Schmidt)]

I decided to start the process over on a separate machine: a 32-bit iMac. Having this builder on a dedicated machine will solve several problems:

The 10.6 i386 VM experienced frequent random kernel panics. (None of the 64-bit VMs do.) We are using a setting that lets us run a 32-bit VM on a 64-bit host. When I reported our problems to the VMware community, I was told that this setting is not supported. I predict that moving to a real 32-bit Mac will eliminate these kernel panics. Every time a kernel panic happened, first I would have to notice that it happened, and sometimes I would not notice for days, during which no builds could occur. I would then have to manually clean up the residue of the interrupted build before starting new builds again. Not having to do that cleanup anymore will save me time.

The 10.6 i386 VM experienced a kernel panic at every cold boot. The only way to boot it was to manually boot to safe mode first, then reboot into normal mode. This meant if a power outage occurred, the VM could not start back up automatically and I would have to intervene. Fixing this will save me more time.

Compared with the 10.6 i386 VM this is a slower machine with less RAM and has a hard drive rather than an SSD, but it increases the total CPU power and RAM we're building with and frees up Xserve CPU and RAM for the other VMs on that host.

This builder is now back online but will take days to work through the backlog of build requests.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

I decided to start the process over on a separate machine: a 32-bit iMac.

That was fun while it lasted, which was unfortunately less than 2 days. ​2 hours into building gcc8, the iMac suffered a hardware failure. We're now back to running this builder as a VM on the Xserve. Interestingly this problem:

The 10.6 i386 VM experienced a kernel panic at every cold boot.

is not happening now on the new VM.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

Interestingly this problem:

The 10.6 i386 VM experienced a kernel panic at every cold boot.

is not happening now on the new VM.

Yes it is.