ghc: needs a runtime dependency for its build compiler

[mojca (Mojca Miklavec)]

Full log is ​here:

--->  Configuring hs-bytestring-show
DEBUG: Preferred compilers: clang llvm-gcc-4.2 macports-clang-3.4 macports-llvm-gcc-4.2 apple-gcc-4.2
DEBUG: Using compiler 'Xcode Clang'
DEBUG: Executing proc-pre-org.macports.configure-configure-0
DEBUG: Listing installed haskell packages
DEBUG: system: /opt/local/bin/ghc-pkg list
/opt/local/lib/ghc-7.8.3/package.conf.d:
    Cabal-1.18.1.3
    array-0.5.0.0
    base-4.7.0.1
    bin-package-db-0.0.0.0
    binary-0.7.1.0
    rts-1.0
    bytestring-0.10.4.0
    containers-0.5.5.1
    deepseq-1.3.0.2
    directory-1.2.1.0
    filepath-1.3.0.2
    (ghc-7.8.3)
    ghc-prim-0.3.1.0
    haskeline-0.7.1.2
    (haskell2010-1.1.2.0)
    (haskell98-2.0.0.3)
    hoopl-3.10.0.1
    hpc-0.6.0.1
    integer-gmp-0.5.1.0
    old-locale-1.0.0.6
    old-time-1.1.0.2
    pretty-1.1.1.1
    process-1.2.0.0
    template-haskell-2.9.0.0
    terminfo-0.4.0.0
    time-1.4.2
    transformers-0.3.0.0
    unix-2.7.0.1
    xhtml-3000.2.1

DEBUG: Running ghc-pkg check
DEBUG: system: /opt/local/bin/ghc-pkg check
DEBUG: Executing org.macports.configure (hs-bytestring-show)
DEBUG: Environment: 
CC='/usr/bin/clang'
CC_PRINT_OPTIONS='YES'
CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_hs-bytestring-show/hs-bytestring-show/work/.CC_PRINT_OPTIONS'
CFLAGS='-pipe -Os -arch x86_64'
CPATH='/opt/local/include'
CPPFLAGS='-I/opt/local/include'
CXX='/usr/bin/clang++'
CXXFLAGS='-pipe -Os -stdlib=libstdc++ -arch x86_64'
F77FLAGS='-m64'
F90FLAGS='-pipe -Os -m64'
FCFLAGS='-pipe -Os -m64'
FFLAGS='-pipe -Os'
INSTALL='/usr/bin/install -c'
LDFLAGS='-L/opt/local/lib -Wl,-headerpad_max_install_names -arch x86_64'
LIBRARY_PATH='/opt/local/lib'
MACOSX_DEPLOYMENT_TARGET='10.7'
OBJC='/usr/bin/clang'
OBJCFLAGS='-pipe -Os -arch x86_64'
OBJCXX='/usr/bin/clang++'
OBJCXXFLAGS='-pipe -Os -stdlib=libstdc++ -arch x86_64'
Executing:  cd "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_hs-bytestring-show/hs-bytestring-show/work/bytestring-show-0.3.5.6" && runhaskell Setup configure --prefix=/opt/local --with-compiler=/opt/local/bin/ghc -v --enable-library-profiling --with-gcc=/usr/bin/clang 
DEBUG: system:  cd "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_hs-bytestring-show/hs-bytestring-show/work/bytestring-show-0.3.5.6" && runhaskell Setup configure --prefix=/opt/local --with-compiler=/opt/local/bin/ghc -v --enable-library-profiling --with-gcc=/usr/bin/clang 

Setup.lhs:2:3: Warning:
    Module ‘System.Cmd’ is deprecated: Use "System.Process" instead

Setup.lhs:4:50: Warning:
    In the use of ‘runTests’
    (imported from Distribution.Simple, but defined in Distribution.Simple.UserHooks):
    Deprecated: "Please use the new testing interface instead!"
Setup: Setup: could not execute: /opt/local/bin/clang-mp-3.4
Command failed:  cd "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_hs-bytestring-show/hs-bytestring-show/work/bytestring-show-0.3.5.6" && runhaskell Setup configure --prefix=/opt/local --with-compiler=/opt/local/bin/ghc -v --enable-library-profiling --with-gcc=/usr/bin/clang 
Exit code: 1
Error: Failed to configure hs-bytestring-show: configure failure: command execution failed

Note that the system clang was apparently too old to build ghc itself on 10.7, so maybe some haskell PortGroup could take care of blacklisting some compilers?

[mojca (Mojca Miklavec)]

build log of hs-bytestring-show on 10.7

[kencu (Ken)]

I see this, which looks like the error:

Setup: could not execute: /opt/local/bin/clang-mp-3.4

Which is a bit odd, as the compiler is set to /usr/bin/clang and /usr/bin/clang++. It comes to mind that haskell might possibly record the compiler used to build it somewhere, and then insist on finding that same compiler again. Maybe that is embedded in the ghc binary that is being installed? Is there a dependency on clang-3.4 needed here? Or possibly forcing a rebuild of ghc might find a new compiler.

I know I had to reinstall clang-3.8 one time when I tried to run ghc. I previously had clang-3.8 set as my default compiler, would have used that to build ghc, and then I moved on to clang-3.9 and uninstalled clang-3.8. But ghc complained when it couldn't find clang-3.8.

[mojca (Mojca Miklavec)]

Yes, it looks like the compiler is requested somewhere where it was recorded during build time.

[kencu (Ken)]

I suppose the only reliable way around this is to insist that ghc should always be installed from source, then. Or find a way to have the build compiler listed as a run-time dependency for it.

[mojca (Mojca Miklavec)]

The second. ROOT6 does that for example.

[kencu (Ken)]

Would you like to re-title this ticket as something like "ghc needs a runtime dependency for it's build compiler" and make clemens the owner?

[kencu (Ken)]

Ah, I think I might see what's going on - in the file clang-wrapper, in ghc-bootstrap, the exact path to clang is hard-coded into the script with a reinplace after installation. So that version of clang needs to be installed -- or the script overwritten with an installed version of clang that would also work. Usually it would be the system compiler, so there would be no problem. It's only these older systems that would find this issue. I guess it just hasn't happened very often that people have installed ghc but not the version of clang specified.

This is a script that Clemens wrote, so he knows all about it. I'm not exactly sure why a script from the bootstrap compiler would be run to install hs-bytestring-show, but it appears that it does - unless there is some other place that the same thing is done.

[mojca (Mojca Miklavec)]

How should the title be renamed now?

[kencu (Ken)]

ghc needs a runtime dependency for it's build compiler describes the situation.

It's not actually ghc, but the ghc port scripts that needs this, but same thing in the end. At least the error quite reasonably announces the compiler it is looking for.

The inelegant but easy workaround for systems that have this problem is to install the missing compiler manually, once you see which one it is looking for. Or force a rebuild of ghc from source, which fixes it (until the user someday uninstalls that compiler, like I did).

Or we could check each of the prebuilt ghc binaries for each buildbot, look at the clang-wrapper script in each one, see what compiler has been defaulted, and for each system, set that compiler as a runtime dep for ghc if it's not the default compiler already on the system.

This would fix the buildbots, and although it might force an unnecessary compiler install here and there (like on my system with libc++ / clang-3.9) that might not be such a bit deal.

The elegant solution would be to rewrite/overwrite/edit the clang-wrapper script with some suitable compiler the machine has installed already, if there is one, and if there is not, to request or require the installation of one. No idea how that might accomplished, though.

Clemens is likely more interested in working on ghc version 8 than going back to spend time on this, I imagine.

[kencu (Ken)]

I think this is a ghc thing rather than an hs-bytestring-show thing. Can we let Clemens own it?

[kencu (Ken)]

So, I can only see three ways through this ticket's issue.

1. Whitelist a compiler for ghc, like clang-4.0, and always build it with that on all systems older than 10.10, say. Then add this as a runtime dependency as well. Easy, simple. This is what I would likely suggest.

2. Check each build of ghc on the buildbots, and make a runtime dep on the compiler that built the binary if it's not a default compiler for that system. Also pretty easy. Wasteful if ghc is built on the system with a newer compiler (eg. clang-4.0 will build it on 10.6 setup with LibcxxOnOlderSystems, so there is no need to install clang-3.4 which might otherwise be the default that built the binary).

3. Some kind of complicated portfile logic to pick an installed compiler, or install one, and reset the clang-wrapper during activation. Most elegant, maybe, but ++ work.

[kencu (Ken)]

macports-clang-4.0 builds ghc successfully on all platforms I tested (10.6, 10.7, 10.8, 10.11, 10.12). It should most likely be whitelisted and listed as both a build and runtime dependency for 10.5 to 10.8, and 10.11 (where the system /usr/bin/clang does not work). This would need to be done in both ghc-bootstrap and later, after the buildbots build that, in ghc.

I believe this would solve the issues with compiler inconsistency in ghc causing problems. And all these systems have clang-4.0 installed anyway courtesy of the cxx11 1.1 PortGroup, so there is no point making them install clang-3.4.

For some reason, even doing all that, installing something like hs-bytestring-show indicates during it's configuration stages that it looks for /usr/bin/clang anyway, as in the build log in this initial ticket, where it looks like it's configured to use /usr/bin/clang but fails when clang-3.4 isn't found). This appears to be set in the haskell 1.0 portgroup, but setting a compiler whitelist for clang-4.0 there does not seem to influence that. Anyway, it works despite that, so long as the ghc-bootstrap && ghc build compiler is still installed.

[kencu (Ken)]

I have put together <​https://github.com/macports/macports-ports/pull/827> which I believe will fix this issue, along with a couple of other build failures for ghc on various systems like 10.11.

ghc is a tricky beast, though, so we'll see what Clemens thinks about it before committing.

[kencu (Ken)]

In b6060617db664a7ca586df1d8894da2054f7db75/macports-ports:

ghc and ghc-bootstrap: specify compiler to clang-4.0

on older systems and on 10.11
and add a build and runtime dependency for clang-4.0
as this compiler needs to be present when ghc is used.
closes: #54363
closes: #53709
closes: #50962
closes: #50480