pandoc is not available for arm64

[ryandesign (Ryan Carsten Schmidt)]

pandoc is only available for x86_64, which means any ports depending on it cannot build on arm64.

​https://build.macports.org/builders/ports-12_arm64-builder/builds/29743/steps/install-dependencies/logs/stdio

Error: Cannot install notcurses for the arch 'arm64' because
Error: its dependency pandoc only supports the arch 'x86_64'.
Error: mportdepends notcurses activate failed.
Calculating dependencies for 'notcurses' failed, aborting.
./mpbb/mpbb: error: `install-dependencies' failed to run successfully

[jmroot (Joshua Root)]

Everything that uses stack is in the same position. It looks like a lot of the dependents are just running the pandoc executable, and so could use depends_skip_archcheck. Pandoc's description says it is also a haskell library, so setting installs_libs no probably isn't appropriate?

[kencu (Ken)]

I see the description, but there is no pandoc library with separate command-line tool any more at least.

Just one (behemoth) binary, and a bunch of docs.

[judaew (Vadym-Valdis Yudaiev)]

pandoc cannot be built for arm64 due to the stack. In most cases, pandoc is not a mandatory dependency for building documentation. In such cases, I recommend using go-md2man, lowndown, or rb30-ronn-ng.

Sometimes projects in Markdown files use specific features of pandoc (which is not very cool), so it is a good idea to check that the documentation is displayed correctly before replacing the dependency.

[kencu (Ken)]

pandoc (x86_64) will work fine on arm64, though, so it would seem there is no need to start the project of looking for workarounds and testing them all out.

Someone just needs to sort out (again) how to get that x86_64 pandoc to automatically install onto the arm64 system.

A year ago, I just set pandoc up to install the x86_64 binary on arm64. That solved this problem for a few months, and everything was building, until that workaround was removed -- we've been broken on arm64 since then.

The proper fix should be to induce MacPorts to install the x86_64 binary by itself.

It may be adding installs_libs no to the pandoc Portfile is all that is needed now, as above.

[Vadim-Valdis Yudaev <judaew@…>]

In c7ee5bfd813a6f5c3d0c6b0b7c0aaa653926b174/macports-ports (master):

notcurses: Use prebuilt man-pages

See: #64063

[judaew (Vadym-Valdis Yudaiev)]

Replying to kencu:

... It may be adding installs_libs no to the pandoc Portfile is all that is needed now, as above.

pandoc still needs to be built, and it requires a stack to build, which is not available for arm64. So I don't think it will work.

[reneeotten (Renee Otten)]

Replying to judaew:

pandoc still needs to be built, and it requires a stack to build, which is not available for arm64. So I don't think it will work.

I guess Ken is referring to this commit where a binary is used for systems that cannot build pandoc for reasons mentioned above. In that case no building is required…

[kencu (Ken)]

macports is supposed to see that an x86_64 binary of pandoc is available and install it on arm64.

if that can be made to work, great.

otherwise a workaround like I did last year that Renee pointed to is a very trivial fix until ghc/stack etc can be made to work on arm.

[kencu (Ken)]

homebrew has an arm64-native pandoc built using this:

​https://github.com/haskell/cabal/issues/7433#issuecomment-858590474

they don't use stack to build it

[judaew (Vadym-Valdis Yudaiev)]

Replying to kencu:

homebrew has an arm64-native pandoc built using this:

​https://github.com/haskell/cabal/issues/7433#issuecomment-858590474

they don't use stack to build it

Currently, cabal in MacPorts does not support arm64 either, although the project itself has a prebuilt version and can be built under arm64. cabal requires ghc as build and run dependency.

cabal: The program 'ghc' version >=7.0.1 is required but it could not be found.

I think for a start we should add ghc-bootstrap for arm64 architecture, with which we can build dependencies for ghc (maybe alex, happy, hscolour) and build ghc itself directly.

This is a lot of work that someone will have to do one day.

[judaew (Vadym-Valdis Yudaiev)]

In the next PR I add binary pandoc x86_64 for arm64. But I cannot verify that these changes work for arm64:

​https://github.com/macports/macports-ports/pull/13165

[ryandesign (Ryan Carsten Schmidt)]

Replying to kencu:

macports is supposed to see that an x86_64 binary of pandoc is available and install it on arm64.

I believe that works fine if the port that isn't arm64 compatible is the one you asked to install, but I believe it doesn't work if the port that isn't arm64 compatible is a dependency of the one you asked to install. See #63092.

Replying to judaew:

In the next PR I add binary pandoc x86_64 for arm64.

That does not seem like the correct solution. You should not need to lie to MacPorts about what arch you're installing.

[judaew (Vadym-Valdis Yudaiev)]

Ryan, what is the best way to proceed in this case?

[kencu (Ken)]

Replying to ryandesign:

Replying to kencu:

macports is supposed to see that an x86_64 binary of pandoc is available and install it on arm64.

I believe that works fine if the port that isn't arm64 compatible is the one you asked to install, but I believe it doesn't work if the port that isn't arm64 compatible is a dependency of the one you asked to install. See #63092.

Josh suggests the standard fixes be tried: either put installs_libs no in the pandoc Portfile (best, if pandoc doesn't install any libs, which it doesn't, despite what the pandoc description says) or alternatively putting depends_skip_archcheck pandoc in everything that calls for pandoc.

I suppose we might get someone to try those, and see if it works.

I don't have an arm system, so I can't.

[judaew (Vadym-Valdis Yudaiev)]

What does installs_libs do? Does this apply to libraries in ${prefix}/lib?

[kencu (Ken)]

It is supposed to tell macports that the dependent port does not have to match the architecture of the port you are trying to build.

Because no libraries are installed, there is nothing to link against.

So MacPorts can feel free to install a working binary of a different architecture if it is available (eg an x86_64 pandoc on an arm64 system that can run it).

Ideally, this will "just work". Perhaps it does -- I can't test it at present.

[Vadim-Valdis Yudaev <judaew@…>]

In dd4f93b339f1dd82c7d1547e4da6cf03fb46f0c3/macports-ports (master):

pandoc: Add installs_libs no

See: #64063

[jmroot (Joshua Root)]

Replying to judaew:

What does installs_libs do? Does this apply to libraries in ${prefix}/lib?

From man portfile:

installs_libs
         By default, it is assumed that ports may install libraries or headers
         that can be incorporated into their dependents.  If this is not the
         case, set installs_libs to no.  This means that this port's depen-
         dents need not check that it is installed for the same architectures
         as them; that it is permissible to distribute binaries of the depen-
         dents even if their licenses conflict with the license of this port;
         and that updates to this port can never result in broken dynamic
         linking in its dependents.
         Type: optional
         Default: none
         Example:
               installs_libs no

[essandess (Steve Smith)]

There should be no issue either installing the x86_64 pandoc binary on arm64, or building the same x86_64 pandoc binary on arm64 using either stack or cabal. I have done all of these.

However, I have observed many MacPorts fragilities on an M1 box recognizing that it should be able to do these things. See, e.g. ​https://trac.macports.org/ticket/63943#comment:6

My belief is that these are issues with MacPorts, not the port pandoc itself or other ports, and that workarounds like ​https://github.com/macports/macports-ports/pull/13165 just hide the underlying issue, whatever it is.

[essandess (Steve Smith)]

Replying to kencu:

homebrew has an arm64-native pandoc built using this:

​https://github.com/haskell/cabal/issues/7433#issuecomment-858590474

they don't use stack to build it

That’s just using an x86_64 binary (cabal) to build another x86_64 binary (pandoc), all running on arm64. Native M1 stack or cabal binaries or builds do not exist.

[essandess (Steve Smith)]

Replying to judaew:

I think for a start we should add ghc-bootstrap for arm64 architecture, with which we can build dependencies for ghc (maybe alex, happy, hscolour) and build ghc itself directly.

See ​https://github.com/macports/macports-ports/pull/12730. There’s a native ghc for arm64 available right now, but bootstrapping is broken, so we’re waiting until upstream fixes it before we merge.

Theoretically, one could use the existing x64_64 cabal or stack with this arm64 ghc to build native binaries on the M1, but I’d recommend just waiting until these upstreams migrate to their own native builds and just use the x86_64 binaries on M1 in the meantime.

[kencu (Ken)]

Replying to essandess:

Replying to kencu: That’s just using an x86_64 binary (cabal)

Nope, it is 100% arm64:

% file cabal
cabal: Mach-O 64-bit executable arm64

[essandess (Steve Smith)]

Replying to kencu:

Replying to essandess:

Replying to kencu: That’s just using an x86_64 binary (cabal)

Nope, it is 100% arm64:

% file cabal
cabal: Mach-O 64-bit executable arm64

For some reason, the Homebrew cabal maintainer recently moved to the experimental tree that has a native arm64 cabal binary. But that binary still uses ghc version 8.10.7 (​https://formulae.brew.sh/formula/cabal-install#default), which is itself x86_64 and can only compile x86_64 things. Native arm64 ghc version 9.2.1 that can build native arm64 binaries just came out weeks ago.

Furthermore, Homebrew doesn’t bootstrap any of this stuff (neither cabal nor ghc nor stack) and simply relies on binary downloads. MacPorts already provides that basic option with the --prebuilt variant.

Again, I’d recommend just waiting until these upstreams migrate to their own native builds and just use the x86_64 binaries on M1 in the meantime.

[essandess (Steve Smith)]

In my view, this question about the fragility of recognizing the ability to install x86_64 binaries on M1 is the salient issue. What is breaking this?

essandess:

There should be no issue either installing the x86_64 pandoc binary on arm64, or building the same x86_64 pandoc binary on arm64 using either stack or cabal. I have done all of these.

However, I have observed many MacPorts fragilities on an M1 box recognizing that it should be able to do these things. See, e.g. ​https://trac.macports.org/ticket/63943#comment:6

My belief is that these are issues with MacPorts, not the port pandoc itself or other ports, and that workarounds like ​https://github.com/macports/macports-ports/pull/13165 just hide the underlying issue, whatever it is.

[judaew (Vadym-Valdis Yudaiev)]

Replying to essandess:

In my view, this question about the fragility of recognizing the ability to install x86_64 binaries on M1 is the salient issue. What is breaking this?

Herby Gillot has verified that installs_libs no build the x86_64 pandoc for arm64 correctly. I think this fix is enough for now until we have ghc for arm64.

Replying to ​herbygillot:

So I'm actually on an Apple Silicon machine at the moment, running macOS 12 (Monterey).

I synced my ports tree after this change, and installed pandoc, and it actually installed stack, and built pandoc from source without issue 🤷🏾‍♂️ Or at least I'm pretty sure that's what I saw.

I do see that my pandoc executable is an x86_64 one instead of arm64:

➜  ~ file `which pandoc`
/opt/local/bin/pandoc: Mach-O 64-bit executable x86_64

[kencu (Ken)]

what you want to see is whether pandoc x86_64 gets automatically pulled in as a dep when building an arm64 port.

that is what was not working.

still not sure if it works. Can anyone confirm/test?

[kencu (Ken)]

Steve, it looks to me that homebrew is using an arm64 ghc to bootstrap it's build of an arm64 ghc.

I think you are incorrect in your assertions.

see:

​https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/ghc.rb

[kencu (Ken)]

ie they build an arm64 ghc, bootstrapping with this:

​https://downloads.haskell.org/ghc/8.10.7/ghc-8.10.7-aarch64-apple-darwin.tar.xz

[essandess (Steve Smith)]

Replying to kencu:

ie they build an arm64 ghc, bootstrapping with this:

​https://downloads.haskell.org/ghc/8.10.7/ghc-8.10.7-aarch64-apple-darwin.tar.xz

I am out of date! Thanks for correcting me.

These issues may have a longer tail for us because we use stack for nearly all our builds, and stack is stable and reliable because it specifies and downloads a bunch of pre-determined LTS versions of ghc and Haskell libraries. I haven’t dug into the guts of stack recently but I’d be amazed if it’s downloading arm 64 versions of ghc.

[kencu (Ken)]

we could just do a quickie switch to building pandoc etc with cabal-install.

That is what I use to build pandoc, shellcheck, etc on 10.6 SnowLeopard, where stack is useless for me.

It might be a long time before stack catches up to support arm64.

[essandess (Steve Smith)]

Replying to essandess:

I haven’t dug into the guts of stack recently but I’d be amazed if it’s downloading arm64 versions of ghc.

It is not: ​https://github.com/commercialhaskell/stackage-content/blob/master/stack/stack-setup-2.yaml

[essandess (Steve Smith)]

Replying to kencu:

we could just do a quickie switch to building pandoc etc with cabal-install.

That is what I use to build pandoc, shellcheck, etc on 10.6 SnowLeopard, where stack is useless for me.

It might be a long time before stack catches up to support arm64.

There’s a haskell_cabal PG for that—I’ve used it myself. I’d suggest adding that as legacy support to the existing Portfiles if it’s necessary.

But I’m not eager to complicate Haskell world builds in general, which are finicky enough without trying to manage migration issues too.

I think we should focus on fixing our stuff and let Haskell world work out how to handle migration.

Non-native pandoc builds and runs right now on an M1. What is causing these install issues?

[kencu (Ken)]

Re: the install issues:

For all we know they have been fixed. Somebody just needs to test if the fix has worked or not -- it may work now.

[kencu (Ken)]

Just to follow-up here, we still have no arm version of pandoc, but the install issues appear to have been set right, and a pandoc x86_64 binary is transparently installed on an M1 Mac, built by using an x86_64 version of stack at present.

% uname -a
Darwin Kens-MBP 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:03:51 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T6000 arm64

% port -v installed stack
The following ports are currently installed:
  stack @2.9.1_0 (active) requested_variants='' platform='darwin 22' archs='x86_64' date='2022-12-30T14:11:58-0800'

% port -v installed pandoc
The following ports are currently installed:
  pandoc @2.19.2_0 (active) requested_variants='' platform='darwin 22' archs='x86_64' date='2022-12-30T14:12:55-0800'

[essandess (Steve Smith)]

I’ll go back and check if pandoc builds under the latest cabal/ghc. If so, we get arm64.

[essandess (Steve Smith)]

Please see ​https://github.com/macports/macports-ports/pull/17271.

[essandess (Steve Smith)]

In deda3fd1e70de298a34782f2401d72d75cd536e8/macports-ports (master):

pandoc: Add arm64 support

• Fixes: #64063