El Capitan buildbot build server teneleven-slave.macports.org

[ryandesign (Ryan Carsten Schmidt)]

OS X 10.11 El Capitan is coming, and I'd like to request the necessary resources to get a new buildbot build server (presumably virtual machine) set up for El Capitan, to run alongside the existing build servers. I'm Cc'ing Joshua who I believe was involved with setting up our buildbot installation. Since El Capitan is not released yet, we probably can't/shouldn't set this up with a beta version of 10.11, and should wait until the final public release, but I wanted to make sure you're aware of the need for additional server resources for this so that we can get those allocated on time. To match the other servers, this should get a static IP in the MacPorts VLAN and be given the hostname teneleven-slave.macports.org. As I recall, there were also a number of firewall exceptions that had to be requested which, if you don't have a list of them, would be the same exceptions that are granted for the other build servers; maybe IS&T can look them up from that.

[fracai]

Cc Me!

[macports@…]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Just checking, Keith: has anything been done about this yet? The rumor sites speculate that El Capitan will be released September 30, which is only two weeks away. I know IS&T can sometimes take time to respond to requests, so if you haven't yet requested an IP address and hostname from them and spoken with them about the firewall exceptions, please begin that dialog with them now.

[ryandesign (Ryan Carsten Schmidt)]

El Capitan has been released. Is there any progress update on getting an El Capitan buildbot builder set up?

[ryandesign (Ryan Carsten Schmidt)]

Any news on this?

[danchr (Dan Villiom Podlaski Christiansen)]

Cc Me!

[mojca (Mojca Miklavec)]

Cc Me!

[casr+macports@…]

Is there anything that can be donated to help with this? Time, money, machines, etc.

[casr+macports@…]

Cc Me!

[casr+macports@…]

Cc Me!

[casr+macports@…]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

No, I will work on it, there's just other existing systems that are broken that I'm trying to fix before I add a new system which is merely a nice-to-have and not something that's actually required.

[casr+macports@…]

Okay and thank you for the update.

I wish you luck! I'll very much appreciate it :)

[ryandesign (Ryan Carsten Schmidt)]

Progress so far:

I cloned the Yosemite builder and gave it a new UUID and MAC address and IP address and computer name. I tried to create a new DNS name for it but I may not have restarted the DNS server properly because it's not resolving yet. I need to figure out how to change the buildbot configuration on it to give it a new buildbot identity, and on the buildbot master to let it know about this new builder. Obviously I still need to upgrade the OS, upgrade Xcode and the command line tools, reinstall MacPorts, and uninstall all ports and then have it build all ports.

[jmroot (Joshua Root)]

Replying to ryandesign@…:

I tried to create a new DNS name for it but I may not have restarted the DNS server properly because it's not resolving yet.

What's the TTL?

I need to figure out how to change the buildbot configuration on it to give it a new buildbot identity, and on the buildbot master to let it know about this new builder.

On the slave machine, there should be a directory with all the files for the buildbot-slave instance, including buildbot.tac. In that file, you want the slavename parameter.

On the master, you want master.cfg and slaves.json. You should probably sync up the former with svn, updating it with the changes from the repo and committing any local changes. Then you need to add the new platform to the build_platforms list. Also add an entry for the new slave to slaves.json.

[ryandesign (Ryan Carsten Schmidt)]

Thanks Joshua. I'll look into that.

I've upgraded the OS to El Capitan.

I've put in a request to open up the required network ports.

[ryandesign (Ryan Carsten Schmidt)]

Upgraded Xcode and command line tools. Reinstalled MacPorts. Uninstalled all ports prior to upgrading to El Capitan. Figured out how to properly reload the nameserver so the teneleven-slave hostname is resolving now. Changed the settings in the buildbot slave configuration. The ticket about opening the network ports has been escalated and I'm waiting to hear back.

[smbmacportstrac@…]

Cc Me!

[jan@…]

Cc Me!

[cjones051073 (Chris Jones)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign@…:

The ticket about opening the network ports has been escalated and I'm waiting to hear back.

The ticket has been escalated to a different group today, who are hopefully the ones who can resolve it.

[Raimondi (Israel Chauca Fuentes)]

Cc Me!

[dixiao@…]

Would it be possible to host the buildbot somewhere else? 4 month after the release is a long waiting time, and compiling packages like llvm on an aging macbook air is not a fun task. Even though to use macports does not require binaries, I think providing binaries is essential for usability of macports.

[anowacki (Andy Nowacki)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Replying to dixiao@…:

Would it be possible to host the buildbot somewhere else?

I am looking into all options.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign@…:

The ticket about opening the network ports has been escalated and I'm waiting to hear back.

The network ports were opened, except the port which enables the master and build slave to talk to each other. I've notified them of this problem.

I've upgraded Xcode and the command line tools to 7.2, also fixing ProblemHotlist#xcode7.2.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign@…:

The network ports were opened, except the port which enables the master and build slave to talk to each other. I've notified them of this problem.

This port is now open as well.

The next problem is that buildbot relies on the existence of /usr/bin/twistd, which Apple apparently removed from OS X in El Capitan.

[neverpanic (Clemens Lang)]

MacPorts has it in the py27-twisted port, but I'm not sure if you actually want to install that using MacPorts.

Otherwise a Python virtualenv could probably also install it using pip install twisted, but then virtualenvs are also not part of the OS X base install :/

[ryandesign (Ryan Carsten Schmidt)]

We can't install anything in the "main" MacPorts installation at /opt/local because that's controlled by the buildbot and everything in it gets deactivated after every build.

We could have a second MacPorts installation elsewhere for software used by the buildbot, but for now I'm trying to install twisted manually.

[ryandesign (Ryan Carsten Schmidt)]

I downloaded Twisted-13.2.0.tar.bz2 because that's version included with Yosemite. I installed it with sudo /usr/bin/python setup.py install. I edited org.macports.build.base-slave.plist and org.macports.build.ports-slave.plist, replacing /usr/bin/twistd with /usr/local/bin/twistd. It appears to be working: the first base build completed, and the first ports build is running. Let me monitor that, and then kick off a few builds for the more popular ports, before we start the long build of all ports.

[ryandesign (Ryan Carsten Schmidt)]

I built the most-requested ports, based on the web server's error_log. Now I've started the build of all ports:

​https://build.macports.org/builders/buildports-elcapitan-x86_64/builds/26

[dsedivec]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Hmm, it looks like the elcapitan ports builder stopped early:

2016-02-07 19:12:45-0800 [-] command timed out: 1200 seconds without output, attempting to kill
2016-02-07 19:12:45-0800 [-] trying to kill process group 71215
2016-02-07 19:12:45-0800 [-]  signal 9 sent successfully
2016-02-07 19:12:45-0800 [-] command finished with signal 9, exit code None, elapsedTime: 420716.093723
2016-02-07 19:12:45-0800 [-] SlaveBuilder.commandComplete <buildslave.commands.shell.SlaveShellCommand instance at 0x10a87d680>

This may have been because of #49759, which we really need to fix.

Then, it looks like the build master crashed. I think that's the order in which it happened. It could be that the master crashed while trying to do some processing on the builder's enormous log file or something.

The builder has apparently re-used build number 26 now to build the ports that had changed after I started the original build 26. I'll start a new build of all ports after that's done. It should go quicker and produce a smaller log this time since it got through ~5000 out of ~20000 ports on the last attempt.

[jmroot (Joshua Root)]

Somehow minivmac is hanging the build after failing to configure:

• ​https://build.macports.org/builders/buildports-elcapitan-x86_64/builds/38
• ​https://build.macports.org/builders/buildports-elcapitan-x86_64/builds/39

But why isn't it failing in pre-extract? Is there a ROM file on the buildslave?

[ryandesign (Ryan Carsten Schmidt)]

Noticed that... Yes, the ROM file is on the buildslaves. The build had been succeeding on the other builders. I had to make a change to make the port build on El Capitan. Maybe the buildbot isn't happy with that.

[ryandesign (Ryan Carsten Schmidt)]

Oh wait... minivmac. That's the old version. minivmac-devel is where I fixed it.

I moved the ROM file on the El Capitan builder so now it should fail in pre-extract and not hang.

[ryandesign (Ryan Carsten Schmidt)]

After that, Joshua started

• ​https://build.macports.org/builders/buildports-elcapitan-x86_64/builds/44

It finished today, after 177 hours or so, but again it crashed the build master and build number 44 has now been reused to build subsequently modified ports.

I think it got through all the ports though; it was a hundred or so ports shy of the end when I checked earlier today. One last attempt to build all ports should hopefully succeed, and upload the packages to the server.

[dixiao@…]

It seems the buildbot stopped uploading compiled packages. For example, free type 2.6.3 is shown to be successfully built, but it's not available at packages.macports.org/freetype. Log (​https://build.macports.org/builders/buildports-elcapitan-x86_64/builds/44/steps/deploy%20archives/logs/stdio) shows error at the very end. Same error occurred for build #45 too.

[ryandesign (Ryan Carsten Schmidt)]

Replying to dixiao@…:

It seems the buildbot stopped uploading compiled packages.

Sorry about that. It's an unrelated issue. I've filed #50668.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign@…:

One last attempt to build all ports should hopefully succeed, and upload the packages to the server.

Here it goes: ​https://build.macports.org/builders/buildports-elcapitan-x86_64/builds/51

[ryandesign (Ryan Carsten Schmidt)]

Finished. Made the buildbot master hang for about 13 minutes at the end, but it recovered on its own.