Lion buildbot builder: No space left on device

[ryandesign (Ryan Carsten Schmidt)]

The Lion buildbot builder (tenseven-slave.macports.org) ran out of disk space trying to build gcc6 during this build:

​https://build.macports.org/builders/buildports-lion-x86_64/builds/29508

In file included from /opt/local/var/macports/build/_opt_mports_dports_lang_gcc6/libgcc-devel/work/gcc-6-20150621/libstdc++-v3/include/precompiled/stdc++.h:94:0:
/opt/local/var/macports/build/_opt_mports_dports_lang_gcc6/libgcc-devel/work/build/x86_64-apple-darwin11/libstdc++-v3/include/valarray:1233:1: error: while writing precompiled header: No space left on device
 } // namespace
 ^
/opt/local/var/macports/build/_opt_mports_dports_lang_gcc6/libgcc-devel/work/build/x86_64-apple-darwin11/libstdc++-v3/include/valarray:1233:1: fatal error: can't write PCH file: No space left on device
compilation terminated.

How much disk space is free? Can we get more? Or can we investigate if there's anything taking up space unnecessarily?

I imagine most of the disk space will be taken up by /opt/local/var/macports/software, because the way we have the buildbot builders set up is that all ports remain installed, but deactivated. This means that as more and more ports get added to MacPorts over time, more and more disk space is needed on the builders.

I'm not certain how the buildbot builder's disk volumes are set up... I thought a SAN or file share might be involved, but I'm not sure what parts are on that and what parts are on each builder's local disk volumes.

Is there a system in place that monitors our servers' disk space? It's important that we resolve low disk space issues before we run out completely, as happened here.

[ryandesign (Ryan Carsten Schmidt)]

This problem did not appear again for a few weeks and I thought maybe it was mysteriously resolved (as the disk space shortage on the packages server was mysteriously resolved in #48174), but Jeremy H reported that the issue ​resurfaced Sunday when building a large port (clang-3.7). Keith, can you please respond to this ticket in some way? Thanks.

[ryandesign (Ryan Carsten Schmidt)]

We are still waiting on a response from the Mac OS Forge admin to this issue.

Lack of disk space may have now caused the registry to become corrupted; see #48486.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign@…:

We are still waiting on a response from the Mac OS Forge admin to this issue.

Correction: There was some further communication on this issue which took place in private email, I believe unintentionally. I'll copy it here:

---

From: Keith Dart
Subject: Re: [MacPorts] #48173: Lion buildbot builder: No space left on device
Date: July 17, 2015 at 3:56:22 PM CDT
To: MacPorts Dev
Cc: Ryan Schmidt, Mac OS Forge Admin, MacPorts Tickets, Jeremy Huddleston Sequoia

I did look at it and there seems to be 20GB free. That still might not be enough for some builds, I guess. I did enlarge the virtual disk by another 20 GB. But so far I’m been unable to resize the file system in the guest. I tried the disk utility and it seems to see the extra size and let me expand it by dragging a pull handle. Then I apply it, and the log entries claim it was successful. However, it is not actually resized.

So I guess another option is to boot into another image, such as net booting. Then try to resize the offline partition. But there seems to be no way to do that with these VMs. So right now I’m stuck at this point. Any suggestions?

BTW, I’m curious, does anybody still use these older OS X?

---

From: Jeremy Huddleston Sequoia
Subject: Re: [MacPorts] #48173: Lion buildbot builder: No space left on device
Date: July 17, 2015 at 8:12:28 PM CDT
To: Keith Dart
Cc: Ryan Schmidt, MacPorts Dev, Mac OS Forge Admin, MacPorts Tickets

Yeah, we still have a bunch of users staying on Snow Leopard because of various reasons. Lion was the last OS that supported EFI32, so there are some users who can't upgrade past Lion.

That being said, you can probably netboot to a newer OS image and use Disk Utility there to resize it. Another way of doing that if the netboot approach fails is to shutdown the VM and add its disk to another VM (selecting "shared" when adding it). Then resize it in the other VM and remove it (selecting "don't delete" when removing).

---

From: Joshua Root
Subject: Re: [MacPorts] #48173: Lion buildbot builder: No space left on device
Date: August 1, 2015 at 11:53:16 AM CDT
To: Keith Dart
Cc: Jeremy Huddleston Sequoia, Ryan Schmidt, MacPorts Dev, Mac OS Forge Admin

Any updates on this? It looks like the lack of space may now be causing the registry db to break.

Worst case, you should be able to create a new larger disk image and clone the old one onto it, or make a Time Machine backup and restore it onto a bigger volume.

[ryandesign (Ryan Carsten Schmidt)]

Replying to keith_dart@…:

I did look at it and there seems to be 20GB free. That still might not be enough for some builds, I guess. I did enlarge the virtual disk by another 20 GB. But so far I’m been unable to resize the file system in the guest. I tried the disk utility and it seems to see the extra size and let me expand it by dragging a pull handle. Then I apply it, and the log entries claim it was successful. However, it is not actually resized.

What software is running the buildbot slave virtual machines? If it happens to be VMware ESXi, check if it's up to date: there was ​a bug in ESXi that prevents resizing disks of guests running OS X 10.6, 10.7, and 10.8 which was fixed in ESXi 5.1 update 3 and 5.5 update 2.

[mojca (Mojca Miklavec)]

Cc Me!

[mojca (Mojca Miklavec)]

Any news about this?

[jeremyhu (Jeremy Huddleston Sequoia)]

Do we care any more? Lion hasn't been a supported platform for a few years now. Maybe we should just leave the builder retired (and also retire 10.6 and 10.8 while we're at it).

[mojca (Mojca Miklavec)]

I do care. Unless we are short of resources, especially the 10.6 buildbot is very valuable in early discovery of problems that would otherwise have to be reported by users. I use some virtual machines with 10.6 to create binaries to support i386 & ppc all the way back to 10.5 (for others). And I still have two computers running 10.7 with basically no serious issues (other than perhaps support for C++14 and suboptimal support for C++11). One of the two is still my main machine and all the software I need is running perfectly fine on it. It would be perfectly fine if we would switch to libc++ instead of libstdc++ though and start supporting that configuration with a higher priority.

Getting the latest buildbot up and running is certainly our highest priority and should be taken care of before worrying about 10.7, but honestly I wouldn't mind having another buildbot for 10.5/ppc.

Unless the resources are tight and we need to minimize the usage & burden on webmaster, I would fully support having the full stack of versions available for the buildbots.

[danielluke (Daniel J. Luke)]

Replying to mojca@…:

And I still have two computers running 10.7 with basically no serious issues

not getting security updates from your OS vendor is a serious issue.

[mojca (Mojca Miklavec)]

After the transition to the new infrastructure on fresh build slaves this is no longer an issue.