Travis build failure of xorg-libXau

[ryandesign (Ryan Carsten Schmidt)]

In the attached log, why is Travis trying to build xorg-libXau from source? It is distributable and an archive is available.

Why is it skipping all of the phases before the configure phase?

DEBUG: Skipping completed org.macports.archivefetch (xorg-libXau)
DEBUG: Privilege de-escalation not attempted as not running as root.
DEBUG: Skipping completed org.macports.fetch (xorg-libXau)
DEBUG: Privilege de-escalation not attempted as not running as root.
DEBUG: Skipping completed org.macports.checksum (xorg-libXau)
DEBUG: Privilege de-escalation not attempted as not running as root.
DEBUG: Skipping completed org.macports.extract (xorg-libXau)
DEBUG: Privilege de-escalation not attempted as not running as root.
DEBUG: Skipping completed org.macports.patch (xorg-libXau)
DEBUG: Privilege de-escalation not attempted as not running as root.

Travis is supposed to build each port in a clean environment. None of the phases of any port should be marked as completed at that point.

Why, having decided to build xorg-libXau from source for whatever reason, is xorg-libXau's dependency pkg-config not present and active, which results in its configure script failing?

[raimue (Rainer Müller)]

This was not the first build failure for xorg-libXau, as the port failed to fetch before. But the log was also lost, probably both caused by a temporary network problem: ​https://travis-ci.org/macports/macports-ports/jobs/365900385#L945

I noticed the runner is not using mpbb list-subports, but ​some other method. Therefore the list of ports to build is not sorted in dependency order. I am not sure if this could be the cause, but it should be fixed anyway.

[jmroot (Joshua Root)]

Backing up a step, if this is supposed to be a clean environment, how are a number of the dependencies already installed?

[raimue (Rainer Müller)]

See the link to the Travis build for the whole log of port install operations. After each subport, its build log is uploaded to paste.macports.org. The attached log is only the build log for one of the ports (rb-gstreamer), but other ports were installed before.

[mojca (Mojca Miklavec)]

See also #56110. We first need a minor change in list-subports to let Travis use it in a proper way.

[ryandesign (Ryan Carsten Schmidt)]

Same problem with libidn2. Why is it trying to build libidn2 from source, when it is distributable and an archive exists? Why is the program autoreconf not found, when libidn2 declares a dependency on the autoconf port?

[ryandesign (Ryan Carsten Schmidt)]

libidn2 failure

[ryandesign (Ryan Carsten Schmidt)]

Of the 11 builds attempted for ​this PR, 7 failed, 6 because of this problem.

[raimue (Rainer Müller)]

Unfortunately, the log of the first try of installing texinfo was once again ​lost. Could there be a correlation that the network outage causes it to fail like this later? I assume the port would be left in an unclean state...

@l2dy
If uploading the log file fails due to a network error, could we just back off for a bit and then try again?

[l2dy (Zero King)]

Replying to raimue:

@l2dy
If uploading the log file fails due to a network error, could we just back off for a bit and then try again?

Then we risk build timeout. How many times to retry? Should we specify a time limit for the HTTP requests?

[raimue (Rainer Müller)]

It is not about a timeout in the HTTP request, it looks like DNS failed.

If the upload failed, just wait for 30 seconds and then try again. Do that three times. As you can see in the logs, the next fetch worked again, so the network problem must have been quite short. That would not add that much to risk a build timeout.

[l2dy (Zero King)]

Replying to raimue:

It is not about a timeout in the HTTP request, it looks like DNS failed.

If the upload failed, just wait for 30 seconds and then try again. Do that three times. As you can see in the logs, the next fetch worked again, so the network problem must have been quite short. That would not add that much to risk a build timeout.

I've deployed a new version that uploads logs with ​a retryable HTTP client from HashiCorp, see ​https://github.com/macports/mpbot-github/commits/develop.

Closing this as it's probably a temporary network issue on Travis, and we can do nothing about that. The mpbb list-subports issue should be tracked in #56110.

[ryandesign (Ryan Carsten Schmidt)]

You "wontfix" the problem that Travis isn't using available binary archives, builds are skipping phases they shouldn't skip, dependencies that shouldn't already be installed are, and dependencies that are declared aren't installed, which results in error conditions that should be impossible? What Travis is doing for us is almost completely batty.

[l2dy (Zero King)]

Replying to ryandesign:

You "wontfix" the problem that Travis isn't using available binary archives, builds are skipping phases they shouldn't skip, dependencies that shouldn't already be installed are, and dependencies that are declared aren't installed, which results in error conditions that should be impossible? What Travis is doing for us is almost completely batty.

On second thought, cleaning /opt/local/var/macports/build/* after each build may fix some issues. And it's also probably the cause of skipped phases: build failed when installing dependencies of a previous port due to connectivity issue, and then another port also has a dependency on this port which hasn't been cleaned. Reopening then.

We could report this to Travis, but Travis depends on MacStadium for these VMs so I'm not sure if there can be a permanent fix soon.

I've been busy recently and didn't reread this ticket carefully. Sorry about that.

[l2dy (Zero King)]

Replying to l2dy:

Replying to ryandesign: On second thought, cleaning /opt/local/var/macports/build/* after each build may fix some issues. And it's also probably the cause of skipped phases: build failed when installing dependencies of a previous port due to connectivity issue, and then another port also has a dependency on this port which hasn't been cleaned. Reopening then.

I now recall that I deliberately skipped the clean phase to save some precious build time (If a port failed, why should I spend the time on cleaning it and then re-extract and rebuild it again?), but I didn't consider network failure by then. So "skipping phases they shouldn't skip" was somewhat working as intended.

Idea: find /opt/local/var/macports/build -type d -mindepth 4 -maxdepth 4 -print0 | sudo xargs -0 rm -rf would be ideal if network is stable and port is really failing (saves more time by making the next phase fail immediately). Use mpbb cleanup for unstable network, which also does other cleanups including deactivate ports.

dependencies that shouldn't already be installed are

Idea: Executing port -fp deactivate active and rleaves after mpbb install-dependencies may solve this. Executing port -fp deactivate active and mpbb install-dependencies again may solve this.

dependencies that are declared aren't installed

Maybe mpbb's fault? I'm just calling it to installing the dependencies...

Ultimately, if MacStadium's network is stable and CI builds can get undistributable binary archives, all these probably would not happen. I think when I closed the issue I was thinking that the ticket summary is for a specific port and the issue is caused by a network problem (which I consider temporary), so let's track actual issues in different tickets and close this one.

[ryandesign (Ryan Carsten Schmidt)]

Replying to l2dy:

dependencies that are declared aren't installed

Maybe mpbb's fault? I'm just calling it to installing the dependencies...

Ultimately, if MacStadium's network is stable

It sure doesn't seem to be, given how many times we've seen problems with DNS resolution for example.

and CI builds can get undistributable binary archives,

Yeah we need to do something about that. I see we have #54800 for this.

all these probably would not happen. I think when I closed the issue I was thinking that the ticket summary is for a specific port and the issue is caused by a network problem (which I consider temporary), so let's track actual issues in different tickets and close this one.

It is difficult for me to know which issues are related and which are separate. But I have now made a separate ticket for "dependencies that are declared aren't installed": #57457.

[l2dy (Zero King)]

Replying to ryandesign:

Travis is supposed to build each port in a clean environment. None of the phases of any port should be marked as completed at that point.

Environment cleanup is done in [18b6ba224c28e170584ae9007868c0a637867b16/mpbot-github]. We have more tickets for other issues, so closing this now.