'stack' port does not build solely from a MacPorts mirror

[greyhare]

I'm on a Macbook Pro running on an isolated network, using a mirror for MacPorts files.

I'm trying to update pandoc, which pulls in stack as a dependency, and stack won't build because it's trying to get stuff directly from GitHub:

$ sudo port upgrade pandoc
--->  Fetching archive for stack
--->  Attempting to fetch stack-2.1.3_0+bootstrap.darwin_18.x86_64.tbz2 from https://packages.macports.org/stack
--->  Attempting to fetch stack-2.1.3_0+bootstrap.darwin_18.x86_64.tbz2 from http://aus.us.packages.macports.org/macports/packages/stack
--->  Attempting to fetch stack-2.1.3_0+bootstrap.darwin_18.x86_64.tbz2 from http://fco.it.packages.macports.org/mirrors/macports-packages/stack
--->  Fetching distfiles for stack
--->  Verifying checksums for stack
--->  Extracting stack
--->  Configuring stack
--->  Building stack
Error: Failed to build stack: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_mirror.arlut.utexas.edu_macports_release_lang_stack/stack/main.log for details.
Error: Problem while installing stack
Error: Follow https://guide.macports.org/#project.tickets to report a bug.

main.log attached.

We have a limited mirror of GitHub (the limit is it only mirrors a whitelisted set of repos). Is there some way I can get a list of GitHub repos to mirror and point MacPorts at our mirror?

[greyhare]

main.log

[mf2k (Frank Schima)]

In the future, please add the port maintainer(s) to Cc (port info --maintainers stack), if any.

[essandess (Steve Smith)]

Replying to greyhare:

stack won't build because it's trying to get stuff directly from GitHub

That’s a feature, not a bug. Perhaps not the answer you’re looking for, but that’s how stack works.

Nevertheless, the MacPorts buildbot mirrored pandoc binary should just download and work—you shouldn’t need to perform a stack build or a network connection at all. Is your MacPorts mirror old-ish? These stack-based builds are relatively new, so perhaps the buildbot pandoc isn’t in your mirror yet? If that’s not it, please track down why pre-built pandoc binary isn’t in your mirror.

[essandess (Steve Smith)]

Replying to greyhare:

Is there some way I can get a list of GitHub repos to mirror and point MacPorts at our mirror?

Also, if you want to build behind your firewall, see this comment: ​https://github.com/macports/macports-ports/pull/4633#issuecomment-523418775

AFAIK, you need to whitelist:

• ​https://github.com/commercialhaskell/
• ​https://github.com/fpco/
• ​https://hackage.haskell.org/

I’d also search the issues at commercialhaskell‘s stack repo and see if anyone has posted a whitelist solution, or post an issue yourself.

[greyhare]

I'm not entirely sure what the "MacPorts buildbot mirrored pandoc binary" is. If you're referring to the prebuilt binary packages sometimes available that MacPorts will install instead of downloading source tarballs and running the whole build process for a port, it doesn't even look for binaries of pandoc before deciding it must have stack as a dependency, and doesn't look for binaries of stack either:

$ sudo port install pandoc
--->  Computing dependencies for pandoc
The following dependencies will be installed:  stack
Continue? [Y/n]: 
--->  Building stack
Error: Failed to build stack: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_mirror.arlut.utexas.edu_macports_release_lang_stack/stack/main.log for details.
Error: Problem while installing stack
Error: Follow https://guide.macports.org/#project.tickets to report a bug.
Error: processing of port pandoc failed

As for my mirror, it has a binary stack-2.1.3_0+bootstrap.darwin_18.x86_64.tbz2 (2019-08-25, 7.92MB) in /macports/packages/stack/ and pandoc-2.7.3_0.darwin_18.x86_64.tbz2 (2019-08-26, 11 MB) in /macports/packages/pandoc/.

After doing a sudo port clean pandoc stack, then sudo port install pandoc again tells me I need stack as a dependency, then I get the error messages in my original report.

It seems to be ignoring our mirror entirely: there's no attempt to fetch the binary from http://mirror.arlut.utexas.edu/macports/packages/stack despite url http://mirror.arlut.utexas.edu/macports/packages/ being set in my /opt/local/etc/macports/archive_sites.conf.

[ryandesign (Ryan Carsten Schmidt)]

Replying to essandess:

Replying to greyhare:

stack won't build because it's trying to get stuff directly from GitHub

That’s a feature, not a bug. Perhaps not the answer you’re looking for, but that’s how stack works.

As far as MacPorts is concerned, it is a bug in the port. Ports should fetch everything they need in the fetch phase, and ideally they should do so by downloading distfiles, since we can mirror those.

Because of this bug, the port ​cannot build on OS X 10.8 and earlier, because the git built into OS X 10.8 and earlier is too old to communicate with the GitHub servers' more advanced encryption needs.

Please fix the bug. :)

[ryandesign (Ryan Carsten Schmidt)]

Replying to greyhare:

I'm not entirely sure what the "MacPorts buildbot mirrored pandoc binary" is. If you're referring to the prebuilt binary packages sometimes available that MacPorts will install instead of downloading source tarballs and running the whole build process for a port, it doesn't even look for binaries of pandoc before deciding it must have stack as a dependency, and doesn't look for binaries of stack either:

$ sudo port install pandoc
--->  Computing dependencies for pandoc
The following dependencies will be installed:  stack
Continue? [Y/n]: 
--->  Building stack
Error: Failed to build stack: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_mirror.arlut.utexas.edu_macports_release_lang_stack/stack/main.log for details.
Error: Problem while installing stack
Error: Follow https://guide.macports.org/#project.tickets to report a bug.
Error: processing of port pandoc failed

As for my mirror, it has a binary stack-2.1.3_0+bootstrap.darwin_18.x86_64.tbz2 (2019-08-25, 7.92MB) in /macports/packages/stack/ and pandoc-2.7.3_0.darwin_18.x86_64.tbz2 (2019-08-26, 11 MB) in /macports/packages/pandoc/.

After doing a sudo port clean pandoc stack, then sudo port install pandoc again tells me I need stack as a dependency, then I get the error messages in my original report.

It seems to be ignoring our mirror entirely: there's no attempt to fetch the binary from http://mirror.arlut.utexas.edu/macports/packages/stack despite url http://mirror.arlut.utexas.edu/macports/packages/ being set in my /opt/local/etc/macports/archive_sites.conf.

I'm pleased to hear ARLUT is using MacPorts! You may already know that UT provides an official MacPorts mirror at ​http://aus.us.packages.macports.org/macports/ (that's an alias for ​http://marmot.tn.utexas.edu/macports/).

Your transcript above shows an unclean build attempt for stack: the first phase shown in the output is the build phase; all the earlier phases, including trying to fetch binary archives, were skipped because they were completed on a previous attempt. If you want MacPorts to try again to get a binary for stack, it should do so if you first run sudo port clean stack. You say above that you did run that, and that it still failed, and I see that in the main.log you attached.

Your log shows that MacPorts tried to get the binary from three different servers and failed on each of them. The reason why it failed was "Could not resolve host". This is not surprising since you're on an isolated network, but MacPorts doesn't know that and it doesn't distinguish between different types of failures. After three failed attempts, MacPorts gives up, assuming the binary is not available anywhere, and tries to build from source.

What is surprising though is that your ARLUT mirror server was not contacted first. MacPorts tries to reach the best server first, which is determined as the one with the lowest ping time. Is your server configured to respond to pings? If not, that would explain it, and if ping support can be enabled, that should fix it. If ping responses cannot be enabled on the server, then MacPorts will consider the server to have a very high ping time and will try it last.

Two changes to your settings will help alleviate that. First, you can set preferred_hosts mirror.arlut.utexas.edu in macports.conf. This overrides the ping time checks. Second, you can edit host_blacklist in macports.conf to specify hosts that should never be contacted. For example you might set host_blacklist *distfiles.macports.org *packages.macports.org—I think that setting should prevent MacPorts from trying to contact any of our official servers.

Is stack the only affected port? If MacPorts isn't checking for binaries of any other ports on your server either, then check what you've entered in archive_sites.conf. You mentioned entering the URL prefixed with url but the prefix should be urls, and you'll need a name entry before that.

[essandess (Steve Smith)]

Replying to ryandesign:

Replying to essandess:

Replying to greyhare:

stack won't build because it's trying to get stuff directly from GitHub

That’s a feature, not a bug. Perhaps not the answer you’re looking for, but that’s how stack works.

As far as MacPorts is concerned, it is a bug in the port … Please fix the bug. :)

Won't fix.

Rather the stack port is necessary to address years of broken and out-of-date Haskell-world MacPorts Haskell packages. The packages themselves are designed to be built with stack, and that's how it works. See the original PR ​https://github.com/macports/macports-ports/pull/4633 and related ports for technical details.

[ryandesign (Ryan Carsten Schmidt)]

Haskell is not my area of expertise. I've now read that very long PR. If you can't fix the problem, then you can't fix the problem. I merely meant to point out that ignoring MacPorts best practices, as stack does by downloading its own files, has real consequences for users, such as those reported in this ticket.

[ryandesign (Ryan Carsten Schmidt)]

But I am grateful that Haskell got updated. An up-to-date Haskell that works for most users is better than an out-of-date one.

[essandess (Steve Smith)]

Thanks. I'm grateful to all the work you've done and are doing to keep MacPorts such an outstandingly high quality project.

In a perfect world, stack wouldn't need to work like this (and would do DESTDIR and PREFIX properly), but they've found that robust, reproducible Haskell builds require an online managed approach like MacPorts has. Because it's effectively unobtanium to replicate what stack does within MacPorts, that implies just adopting stack.

[mojca (Mojca Miklavec)]

Steve, are you able to figure out how exactly stack fetches the sources?

For fetching anything else MacPorts does have a workaround (which needs to be enabled manually). The native curl doesn't work because the required certificates & encryption support are not present on the system. But if you install an auxiliary instance of MacPorts and build the main MacPorts installation against libcurl from that auxiliary location (else it's a chicken-and-egg problem), it works, and the machine can easily fetch from GitHub & others.

I'm pretty sure there must be a way to convince stack to download stuff using some library from MacPorts. We just need to figure it out.

[mojca (Mojca Miklavec)]

The error (from ​buildbot) is roughly:

Cloning 97e83ecc133cd60a99df8e1fa5a3c2739ad007dc from https://github.com/snoyberg/filelock.git
Received ExitFailure 128 when running
Raw command: /usr/bin/git clone https://github.com/snoyberg/filelock.git /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_stack/stack/work/.tmp/with-repo10589/cloned
Standard error:

Cloning into '/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_stack/stack/work/.tmp/with-repo10589/cloned'
fatal: unable to access 'https://github.com/snoyberg/filelock.git/': error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version

To start with it might help to use ${prefix}/bin/git instead of /usr/bin/git?

I tested on 10.6 and

/opt/local/bin/git clone https://github.com/snoyberg/filelock.git

seems to work fine there.

So it's probably just a matter of convincing stack to use the correct git.

[kencu (Ken)]

Potentially useful: Stack YAML Configuration ​additional directories to prepend to the PATH environment variable

[essandess (Steve Smith)]

Replying to mojca:

Steve, are you able to figure out how exactly stack fetches the sources? … I'm pretty sure there must be a way to convince stack to download stuff using some library from MacPorts. We just need to figure it out.

The Haskell code is in the http-download package, which is part of stack and relies on Conduit:

• ​http://hackage.haskell.org/package/http-download
• ​https://github.com/commercialhaskell/http-download
• ​https://github.com/commercialhaskell/http-download/blob/master/src/Network/HTTP/Download.hs

This isn’t going to be reproducible with a MacPorts library.

For the sources, see the variable {defaultSetupInfoYaml in ​https://github.com/commercialhaskell/stack/blob/master/src/Stack/Setup.hs . The default source yaml configuration is ​https://github.com/commercialhaskell/stackage-content/blob/master/stack/stack-setup-2.yaml .

Example ghc download configuration:

    macosx:
        7.8.4:
            url: "https://github.com/commercialhaskell/ghc/releases/download/ghc-7.8.4-release/ghc-7.8.4-x86_64-apple-darwin.tar.bz2"
            content-length: 129602990
            sha1: b7aff3983e9005b74d90c5a4fd7c837f9e752923
    …

I presume that source itself is downloaded from hackage, but I see links to fpcomplete’s S3 bucket fly by, so there’s some mirroring system for source lts and other resolvers.

I agree that in principal it should be possible to mirror everything where you want it, but before signing up for all that you should assess how much work it would be to duplicate effort to do what stack already does.

The stack issues page has some years-old requests for an offline mirror capability, but no one is working on it: ​https://github.com/commercialhaskell/stack/issues/1594

I’m honestly not sanguine about the availability of the necessary resources to build all this within MacPorts, much less maintain it.

[greyhare]

Basically, it just seems we're unlucky. stack doesn't support mirrors and hardcodes server names and root paths, and it's what the MacPorts maintainers chose to use.

I need to test my Gentoo setup at work next week. I know they use cabal, and I know that ghc-dependent package updates used to be a huge PITA there but aren't these days. If I can get it running on Gentoo, then I can just run jupyter-notebook in a Gentoo VM via Fusion when I need to generate PDF output, which needs pandoc, which is the only reason I need any of this.

My opinion of Haskell is pretty thoroughly in the gutter right now. Things need to not be useless without giving automated scripts unfettered access to an open Internet connection.

Thanks for trying.

[kencu (Ken)]

Replying to greyhare:

when I need to generate PDF output, which needs pandoc, which is the only reason I need any of this.

The old version of everything remains available by cloning this repo <​https://github.com/kencu/mp-haskell7-world> of the way thing used to be, and putting it in sources.conf. If you get really frustrated.

Best,

K

[essandess (Steve Smith)]

Replying to greyhare:

Basically, it just seems we're unlucky. stack doesn't support mirrors and hardcodes server names and root paths, and it's what the MacPorts maintainers chose to use.

No, it’s what pandoc and nearly the rest of Haskell world chose to use, and we’re just following a build plan that’s robust and works: ​https://github.com/jgm/pandoc/blob/master/stack.yaml

I need to test my Gentoo setup at work next week. I know they use cabal, and I know that ghc-dependent package updates used to be a huge PITA there but aren't these days. If I can get it running on Gentoo, then I can just run jupyter-notebook in a Gentoo VM via Fusion when I need to generate PDF output, which needs pandoc, which is the only reason I need any of this.

It’s still not clear why the pre-built pandoc binary isn’t in your mirror. You shouldn’t need stack at all.

My opinion of Haskell is pretty thoroughly in the gutter right now. Things need to not be useless without giving automated scripts unfettered access to an open Internet connection.

There’s an issues page to take that discussion: ​https://github.com/commercialhaskell/stack . Also, as mentioned above, the access isn’t unfettered, but laid out explicitly in ​https://github.com/commercialhaskell/stackage-content/blob/master/stack/stack-setup-2.yaml .

[kencu (Ken)]

stack is finding the wrong git on older systems because of this:

# the PATH environment must provide /usr/bin/gcc, not ${prefix}/bin/gcc
# note: this command does not change the destroot PATH environment, so export
# PATH explicitly in the necessary system command below
build.env \
    PATH=/usr/bin:$env(PATH) \
    PREFIX=${prefix} \
    CC=${system_gcc}

[essandess (Steve Smith)]

Replying to kencu:

stack is finding the wrong git on older systems because of this, I believe:

# the PATH environment must provide /usr/bin/gcc, not ${prefix}/bin/gcc
# note: this command does not change the destroot PATH environment, so export
# PATH explicitly in the necessary system command below
build.env \
    PATH=/usr/bin:$env(PATH) \
    PREFIX=${prefix} \
    CC=${system_gcc}

Thanks @kencu. If so, then PR ​https://github.com/macports/macports-ports/pull/5275 will fix this issue.

@mk2k @mojca @ ryandesign Would you please merge these Haskell-related PRs so this can be tested?

• ​https://github.com/macports/macports-ports/pull/5275
• ​https://github.com/macports/macports-ports/pull/5272
• ​https://github.com/macports/macports-ports/pull/5327

[ryandesign (Ryan Carsten Schmidt)]

Replying to essandess:

It’s still not clear why the pre-built pandoc binary isn’t in your mirror.

I explained that.

[essandess (Steve Smith)]

Replying to ryandesign:

Replying to essandess:

It’s still not clear why the pre-built pandoc binary isn’t in your mirror.

I explained that.

I see the explanation, but not the verification of the explanation.

[greyhare]

I'm not sure what's unclear.

Our mirror has the pandoc binary. Two versions, actually. But at no point (clean build or not) does MacPorts look for it; it goes directly to "stack is a dependency and I need to install it first." Then the observed behavior of ignoring the mirror and attempting to install stack from three outside sites.

It looks like MacPorts should try the mirror first, and I don't know why it doesn't. MacPorts has no trouble installing binary packages for other ports such as gcc8.

What needs verification?

[ryandesign (Ryan Carsten Schmidt)]

I proposed two possible explanations for why your mirror was not contacted when installing stack: it is not configured to respond to pings, or you've configured archive_sites.conf incorrectly. The fact that you're able to get a binary for gcc8 (which I didn't know before) seems to rule out both of those explanations.

However, MacPorts caches ping values for a short time, and it's possible that at the particular moment that MacPorts cached the ping values for your server when trying to download the stack binary, it was unable to reach the server. If that was the case, retrying again later should work, or to try again immediately you could delete the file /opt/local/var/macports/pingtimes.

Additionally, I suggested using preferred_hosts and host_blacklist in macports.conf. Did you try that?

You say trying to upgrade pandoc doesn't attempt to get a binary. Could you attach a main.log showing that behavior, please?

[greyhare]

Sorry, I forgot to respond earlier.

I tried again today after cleaning, uninstalling leaves, and so on, and it worked this time. pandoc still installed stack as a dependency, but this time stack looked at our mirror, found the binary, installed it, and after that pandoc installed from binary as well. Afterwards, stack was removed with the leaves.

Thanks for your efforts, and sorry for the trouble.

[mojca (Mojca Miklavec)]

If so, then PR ​https://github.com/macports/macports-ports/pull/5275 will fix this issue.

It didn't: ​https://build.macports.org/builders/ports-10.8_x86_64-builder/builds/1011/steps/install-port/logs/stdio

On 10.8 it still used the wrong git version:

Package index cache populated
Cloning 97e83ecc133cd60a99df8e1fa5a3c2739ad007dc from https://github.com/snoyberg/filelock.git
Received ExitFailure 128 when running
Raw command: /usr/bin/git clone https://github.com/snoyberg/filelock.git /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_stack/stack/work/.tmp/with-repo62737/cloned
Standard error:

Cloning into '/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_stack/stack/work/.tmp/with-repo62737/cloned'...
fatal: unable to access 'https://github.com/snoyberg/filelock.git/': error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version

Command failed:  cd "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_stack/stack/work/stack-2.1.3" && /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_stack/stack/work/bin/stack setup --with-gcc /usr/bin/clang --allow-different-user 
Exit code: 1

and on 10.7 and earlier it segfaulted (Ken bootstrapped an older version and kept updating it to work on 10.6, so that one could be used as a starting point).

[essandess (Steve Smith)]

Replying to mojca:

On 10.8 it still used the wrong git version:

My best guess is that this is a simple PATH issue. (I searched the Haskell code and couldn’t track down where the git command is issued).

For deprecated OS’s, one could:

• Take this upstream and ask how to specify the specific git executable. My expectation is that the answer is to set PATH.
• Modify the PATH in the appropriate .env phase, or adding an export PATH=… to a system command, and hop that it doesn’t break something else.

Related is the observation, shared by others, is that setting PATH in a .env doesn’t always work for some reason, so this approach will require double-checking and care.

Also I have to ask: those old SSL standards are quite insecure now. I personally wouldn’t have the confidence to use them, or recommend their general use. Has anyone documented the downside risks of taking these end runs around modern standards to continue to use unsupported OS’s, which themselves have numerous unpatched CVEs?

[kencu (Ken)]

Most people who have older systems can't use Safari for browsing, and likely don't use the mail client for anything except gmail. My world lives behind firewalls all the time.

The replacement browsers are all built against brand new openssl and curl. Any other software like octave or stellarium or pingus or pandoc or VLC or.. all comes from MacPorts and so all has current security. So, although I'm not a forensic network guy, the exposure would seem to be respectably low.

It's more a matter of keeping the landfills a little less full, and saving the planet :)

My stack build seems to have no issues using macports git on 10.7 and 10.6; I'll try it on 10.8 and see if I can see what's going on there.

[kencu (Ken)]

closing as a four-year-old unique situation resolved apparently by the OP and no further input expected or required