stack @2.3.1_0: Undefined symbols: _utimensat

[macdeport]

main.log <= stack-2.3.1_0.darwin_14.x86_64.tbz2

[ryandesign (Ryan Carsten Schmidt)]

It looks like the error in the log is:

:info:build Linking .stack-work/dist/x86_64-osx/Cabal-2.4.0.1/build/stack/stack ...
:info:build Undefined symbols for architecture x86_64:
:info:build   "_utimensat", referenced from:
:info:build       _cazW_info in libHSdirectory-1.3.3.0.a(Posix.o)
:info:build ld: symbol(s) not found for architecture x86_64
:info:build clang: error: linker command failed with exit code 1 (use -v to see invocation)

[ryandesign (Ryan Carsten Schmidt)]

I believe utimensat is only available on macOS 10.13 and later. You're on OS X 10.10 so I'm not sure why it's trying to use it.

[ryandesign (Ryan Carsten Schmidt)]

This problem affects macOS 10.12 and older. ​Here is the buildbot log from 10.12. Having the stack port unavailable on an OS as recent as Sierra is limiting. Please figure out how to make the port work on older systems.

I recall prior discussion about the fact that stack was made to work by using precompiled binaries of something. If that's still the case, then perhaps that precompiled binary was compiled on High Sierra or newer and thus makes use of symbols available on newer systems. This yet again exemplifies why we need to build from source on each OS version rather than repackage binaries.

[essandess (Steve Smith)]

Stack is self-compiling—the pre-compiled binaries are necessary to bootstrap. The situation is exactly the same as for ghc. IDK how upstream bootstraps stack (or ghc) for macOS, but I strongly suspect that they’re cross-compiled, which obvs. wouldn’t work within a MacPorts build.

Therefore, this is an upstream issue.

I’d suggest that anyone who needs stack on 10.12 or earlier drive the agenda by experimenting to see if older stack versions that run on these systems work to compile the latest version, or ask upstream to provide bootstrap instructions for macOS-only non-stack compilers.

[reneeotten (Renee Otten)]

In 8d76d66a4167ca3de6bd95ced08259605ece3ec5/macports-ports (master):

bali-phy: do not build docs on macOS 10.12 and older

See: #60477

[kencu (Ken)]

I have stack, ghc, pandoc, and cabal working back to 10.6, at the latest versions, for bootstrapping.

[ryandesign (Ryan Carsten Schmidt)]

Replying to kencu:

I have stack, ghc, pandoc, and cabal working back to 10.6, at the latest versions, for bootstrapping.

Why don't you contribute your fixes so everyone can benefit? :)

[kencu (Ken)]

Mostly, I've been looking for somewhere to host them other my personal google drive, and once that is done, integrating them into the existing bootstrap of our ghc/cabal/stack ports requires essandess on board.

I did send you the links to all of them on my Google drive previously, but I know it's not your issue to fix. Perhaps I can stick them up on GitHub someplace, as they seem willing to host just about anything these days...

[ryandesign (Ryan Carsten Schmidt)]

I had hoped that your fixes would involve changes to the portfile to allow it to build stack on older systems, not that you would provide binaries of something that you compiled outside of MacPorts.

[kencu (Ken)]

well the issue is that each of these, ghc, stack, and cabal, all require bootstrap binaries that the portfiles use to build newer versions.

And all the bootstrap binaries from upstream crash as they are not built with older systems in mind.

SO the next trick is to see how to use the ones I provided instead of the official upstream binaries (on older systems at least), and then the portfiles can build newer ones.

[iEFdev]

Replying to kencu:

well the issue that each of these, ghc, stack, and cabal, all require bootstrap binaries that the portfiles use to build newer versions.

Is it something that could be built/done i separate ports? and brought in as deps?

[kencu (Ken)]

It's a year since I last looked in detail -- I mostly wanted pandoc, and made that for myself.

As I recall, ghc/stack/cabal and their respective ports are all wired up to get their bootstrap compilers from the original sources.

We have to position the more compatible sources I made somewhere accessible, and then rewire to portfiles and the actual software to look whereever we put the sources instead of where they are wired to look.

Exactly how that is to be done is TBA.

If you are interested in digging in, I have the sources for 10.6.8+ working versions of all of those roots, that don't have any ofthe various utimensat etc errors.

And then we have to sort our how not to make a total mess of the (sometimes frighteningly complex) existing portfiles in the process.

[kencu (Ken)]

Yesterday I updated ghc for 10.6.8+ to 8.10.2, cabal for 10.6.8+ to 3.2.0, and pandoc for 10.6.8+ to the latest version. These are the actual binaries, not Portfiles, at present.

I'm just running into a little haskell-source-code hiccup in one dependency (casa-types) while updating stack to the current release.

Once I have that fixed, I'll make them all available for anyone who wants them.

[kencu (Ken)]

Got stack 2.3.3 sorted out. It needs legacysupport for the newer defines and symbols. Using an older stack and with the system ghc set to ghc-8.6.5 which is presently the one it needs, editing the stack.yaml in stack works.

Add this:

in the stack.yaml for stack itself, add this:

ghc-options:
  "$locals": -fhide-source-paths
+   "$targets": -isystem/opt/local/include/LegacySupport/ /opt/local/lib/libMacportsLegacySupport.a

and legacysupport links in statically, and seems to work as it should.

[kencu (Ken)]

I have posted current versions of ghc, cabal, and stack for 10.6.8 and newer here <​https://github.com/kencu/ghc-for-older-darwin-systems/releases/tag/8.10.2>.

These are the binaries, all built on 10.6.8, and all seem to be working as expected there. Of course, there are a few details -- you will have to install ghc and use the system-ghc switch for stack on older systems, otherwise you download the ghc version from the ghc website. There are going to be certain builds that need legacysupport added, most likely (see above post).

I'm going to make some Portfiles that use these on older systems, and see how well they integrate with the Portfiles already on MacPorts. No promises there.

If you need pandoc, which seems to be the most-commonly needed binary that ghc is used to build these days, you can build it quite easily using these roots -- took a few minutes, and I had a current pandoc for 10.6.8.

[kencu (Ken)]

stack downloads the info file for all the ghc and other software it might want to install from here <​https://raw.githubusercontent.com/fpco/stackage-content/master/stack/stack-setup-2.yaml>, as specified in this file:

stack-2.3.3/src/Stack/Setup.hs

So if we were going to override that to point towards our copies of ghc and stack etc, we'd have to replace that with a file that pointed to our software instead.

Not a trivial undertaking, but not overly hard either. At the moment, I think this is all hardcoded into stack, so you'd need to build stack with a patch that points to our yaml file instead. Perhaps there is some option in the 10,000 options that ghc / stack / cabal seem to have that lets you use a different URL without changing it in the stack build.

[Ionic (Mihai Moldovan)]

I finally have a working fix (on 10.9, at least) for this issue.

It requires utimensat support in legacy-support and a release (we're currently working on that part), though, but once everything is in place, we can get proper stack builds on older systems again.

[kencu (Ken)]

making stack build with legacysupport is --- tricky I found :>

[Ionic (Mihai Moldovan)]

It's tricky and almost impossible because of some bugs.

The general idea is to pass --ghc-options "-optc-I${prefix}/include/LegacySupport -optl${prefix}/lib/libMacportsLegacySupport.dylib" to the build process, which works surprisingly well.

Sadly, turning ${configure.cflags} and ${configure.ldflags} into ghc options (i.e., -optc... and -optl...) won't work, because this will also add -optl-L${prefix}/lib and -optc-I${prefix}/include, which will make packages within stack link against the GNU iconv implementation and fail to find the symbols later on. Not sure why, probably because Apple's iconv implementation is different, but that's definitely an issue to avoid, so we'll just the library and add the legacy-support-related include directory only manually.

That works... beautifully.

Up until the destroot phase, when everything blows up.

For some reason, stack does not pass down --ghc-options in the install command (which is really just an alias for build --copy-bins). I haven't found a way to make it do this, because I really can't wrap my head around Haskell well enough.

That's bad enough already, but here comes #62228: stack install will actually rebuild EVERYTHING a second time because it doesn't respect STACK_ROOT, which is where the stack database is written to. Instead, it seems to use ~/.stack in a hardcoded fashion, re-download GHC and re-build everything. Without any way to pass special GHC options.

Luckily, the workaround for that is an easy one: just define STACK_ROOT as ${workpath}/.home/.stack (which is really ${HOME}/.stack as far as the process is concerned) and this second rebuild, including its issues, will vanish completely.

[kencu (Ken)]

btw, many versions of ghc, stack, and cabal for systems back to 10.6.8 are here already:

<​https://github.com/kencu/ghc-for-older-darwin-systems/releases>

[kencu (Ken)]

the only thing needed to make those versions work for MacPorts is a way to tell stack to use those ghc versions as upstream ghc, rather than the ones on the official ghc website.

Unfortunately, that is not trivial. You have to somehow override stack's "portindex" (packaging manifest), and the mechanism to do that in a way that would work for a diffuse set of MacPorts users is not clear.

It's simple to do on a local, case-by-case basis though, as I explained above.

[essandess (Steve Smith)]

Please see ​https://github.com/macports/macports-ports/pull/10086

[harens <harensdeveloper@…>]

In 02369fddbd17602ac171f1b53e184a5808874fa3/macports-ports (master):

exa: update to 0.10.1, add git and doc variants

• Add harens as a co-maintainer
• Do not build docs on macOS 10.12 and older by default
• Declares libiconv as a dependency

See: #60477
Closes: #60604

[macdeport]

Feedback: Thank you for your persevering, sharp and efficient efforts: seems to be fixed, at least at this moment for me...

[kencu (Ken)]

Replying to Ionic:

It's tricky and almost impossible because of some bugs.

I actually don't see any bugs in this. Just the way it works.

The general idea is to pass --ghc-options "-optc-I${prefix}/include/LegacySupport -optl${prefix}/lib/libMacportsLegacySupport.dylib" to the build process, which works surprisingly well.

Yeah, that is the command-line equivalent to what I offered here 60477?replyto=21#comment:16 before.

I moved my fix as per the yaml file into my main config.yaml on my older systems, so now legacy support is always used when building with stack. No more troubles ;>

Sadly, turning ${configure.cflags} and ${configure.ldflags} into ghc options (i.e., -optc... and -optl...) won't work, because this will also add -optl-L${prefix}/lib and -optc-I${prefix}/include, which will make packages within stack link against the GNU iconv implementation and fail to find the symbols later on. Not sure why, probably because Apple's iconv implementation is different, but that's definitely an issue to avoid, so we'll just the library and add the legacy-support-related include directory only manually.

This iconv vs libiconv symbol-naming issue goes back many many years. libiconv does that on purpose -- different sympbols for system vs user installations of libiconv to find and flag misbuilt and/or misconfigured software that mixes up headers and libraries, which I guess they found was all too common (and apparently still is ;> ).

I have updated my builds of the haskell ecosystem for 10.6+, including ghc-9.x, stack 2.71, cabal 3.40, and the needed alex and happy versions. All this haskell stuff works great for me back to 10.6.8 still, and I can build anything I want, eg pandoc and shellcheck on Lion:

$ port -v installed pandoc
The following ports are currently installed:
  pandoc @2.13_0 (active) requested_variants='' platform='darwin 11' archs='x86_64' date='2021-05-27T07:31:22-0700'

$ port -v installed shellcheck
The following ports are currently installed:
  shellcheck @0.7.2_0 (active) requested_variants='' platform='darwin 11' archs='x86_64' date='2021-05-27T08:34:45-0700'

The stack that is downloaded from upstream is built on a system that is not friendly to older systems. I just replace that one in the port with my own, and everything is 100%.

I may update the stack port here on MacPorts to use the one I have which actually works on older systems, but perhaps it would be easier to just make the binaries of pandoc and shellcheck available because in the end, that is pretty much all that people want this for anyway.

[kencu (Ken)]

fixed by changes to the stack port over the past few years