10.8 buildbot worker is down

[ryandesign (Ryan Carsten Schmidt)]

One of our VMware hosts has suffered an SSD failure, as a result of which the 10.6-i386, 10.8, 10.11 and 10.13 workers are down and will need to be restored from backups. There may be separate issues involved in bringing each of these back online so I'll make separate tickets. I plan to restore them to spare hard disks temporarily until I can get a new SSD.

[ryandesign (Ryan Carsten Schmidt)]

I have tried to boot to the 10.8 installer and restore from Time Machine backup. It says it can't because the backup was made on a different Mac model. This is a lie; I have created the new virtual machine the same way as I created the other VMs before.

I tried to reinstall 10.8 which failed with an error message which suggested that the installer pkg certificate was invalid. Rather than wasting time downloading a fresh copy of the installer, I backdated the clock by a year, which allowed the installation to proceed. I restored from Time Machine backup, but apparently not everything gets backed up. Xcode got backed up, but the Xcode "additional components" didn't, because Xcode asked to install them when I opened it. Installing additional components failed. The dialog box told me to check install.log, but there was nothing there. system.log however pointed to the fact that it was accessing ​https://developer.apple.com but could not because the SSL certificate was issued by an unknown issuer. I opened that URL in Safari and told it always to trust the certificate. Then I retried installing additional components. This time it failed because it tried to access ​https://devimages.apple.com.edgekey.net but could not because it could not establish a secure connection. This suggests the server is configured to require newer TLS protocols than the networking libraries on Mountain Lion are capable of accommodating. I can't think of a good way of getting past this. I should report this to Apple but don't have high hopes that they will choose to resolve it by reenabling obsolete security protocols on their edge servers, but if they don't, it means that nobody can install a working Xcode on Mountain Lion anymore.

An alternative could be that I make a clone of the 10.8-legacy VM we decommissioned some months ago, excluding the contents of /opt/local and maybe deleting and recreating the buildworker directory. Or I could just copy the additional components from that VM, if I knew where they get installed.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

I should report this to Apple

Filed as ​https://feedbackassistant.apple.com/feedback/7594770.

[ryandesign (Ryan Carsten Schmidt)]

Apple actually replied:

Xcode versions earlier than Xcode 8.1 can no longer download simulators and watchOS debug symbols. Instead, use a newer version of Xcode to download these components, then re-launch the older version of Xcode, as described below.

Use the Component preferences in Xcode 8.1 or later to download and install iOS, tvOS, or watchOS simulator runtimes. Once downloaded, these simulators will be available to be used with any version of Xcode, including Xcode 8 and earlier.

Connect an iPhone paired with an Apple Watch to a Mac running Xcode 8.1 or later and allow Xcode 8.1 or later to download and install the watchOS debug symbols for the watchOS version on the Apple Watch. These debug symbols are then available to be used with any version of Xcode, including Xcode 8 and earlier.

[ryandesign (Ryan Carsten Schmidt)]

The list of pending builds was lost when the buildmaster's sqlite database became corrupted. When this builder is brought back online, builds for ports modified during the downtime should be rescheduled.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

Installing additional components failed.

This turned out to be because I was using the original version of Xcode 5.1.1 from 2014 and the embedded installer packages for the additional components were signed with a certificate that has expired. Using the new version of Xcode 5.1.1 from December 2019 would have worked. (I verified this on a different machine.) But before I realized that, I worked around the problem by manually running the installer packages in Xcode.app/Contents/Resources/Packages (MobileDevice.pkg and MobileDeviceDevelopment.pkg). Although the certificate was expired, the installer did not complain and allowed them to be installed. Thereafter Xcode then opened normally.

Maybe installing these packages wasn't necessary after all, since we don't do mobile device development. However when setting up buildbot workers it has been my habit to install Xcode and then open it and respond to any prompts until it will open normally and I didn't want to deviate from that practice here.

system.log however pointed to the fact that it was accessing ​https://developer.apple.com but could not because the SSL certificate was issued by an unknown issuer. I opened that URL in Safari and told it always to trust the certificate. Then I retried installing additional components. This time it failed because it tried to access ​https://devimages.apple.com.edgekey.net but could not because it could not establish a secure connection.

These issues remain but are unimportant because they relate to downloading documentation and simulators, neither of which we need.

[ryandesign (Ryan Carsten Schmidt)]

We're now ​building the ports that failed to build when the SSD failed.

Replying to ryandesign:

The list of pending builds was lost when the buildmaster's sqlite database became corrupted. When this builder is brought back online, builds for ports modified during the downtime should be rescheduled.

​The last build before the problem was for py-qtawesome due to [1f55aaedac3d0d850bdfac7fbbe7a2d96807ab0e/macports-ports].

​The first build when builds started again was for pihpsdr due to [8973b9f89f0acb28ae3efca3622ca59bd9a0bb51/macports-ports].

We will shortly be ​building the ports that were added or modified between those two commits.