sbcl: build hangs on Monterey

[pmetzger (Perry E. Metzger)]

Attempting to build the SBCL port under macOS 12.2 (x86_64) prints this (nothing more) and then spins for hours at 100% CPU on a single processor without apparent progress:

--->  Building sbcl
This is SBCL 1.2.11, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.

I'm unsure how to debug this.

[ryandesign (Ryan Carsten Schmidt)]

On the macOS 12 x86_64 buildbot worker we got this output, after 32 minutes:

--->  Building sbcl
DEBUG: Executing org.macports.build (sbcl)
DEBUG: system -W /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_sbcl/sbcl/work/sbcl-2.2.1: SBCL_MACOSX_VERSION_MIN=12.0 CC=/usr/bin/clang CXX=/usr/bin/clang++ CPP=/usr/bin/cpp sh ./make.sh --fancy --prefix=/opt/local --xc-host="/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_sbcl/sbcl/work/sbcl-1.2.11-x86-64-darwin/src/runtime/sbcl --core /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_sbcl/sbcl/work/sbcl-1.2.11-x86-64-darwin/output/sbcl.core --disable-debugger --sysinit /dev/null --userinit /dev/null" 
This is SBCL 1.2.11, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
make-config.sh: line 233: 48392 Done                    echo '(lisp-implementation-type)'
     48393 Killed: 9               | $SBCL_TEST_HOST
No working host Common Lisp implementation.
See ./INSTALL, the "SOURCE DISTRIBUTION" section
Command failed: SBCL_MACOSX_VERSION_MIN=12.0 CC=/usr/bin/clang CXX=/usr/bin/clang++ CPP=/usr/bin/cpp sh ./make.sh --fancy --prefix=/opt/local --xc-host="/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_sbcl/sbcl/work/sbcl-1.2.11-x86-64-darwin/src/runtime/sbcl --core /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_sbcl/sbcl/work/sbcl-1.2.11-x86-64-darwin/output/sbcl.core --disable-debugger --sysinit /dev/null --userinit /dev/null" 
Exit code: 1

​https://build.macports.org/builders/ports-12_x86_64-builder/builds/20846/steps/install-port/logs/stdio

[dananau (Dana Nau)]

I can't build it either. After about 16 minutes of CPU time (I'm not sure how much clock time), the process's memory usage was more than 50 GB and it was doing no I/O at all, so I killed it.

I'm running MacOS 12.2 on a 2018 Macbook Pro.

[dancamper (Dan S. Camper)]

Bump, even if this is a duplicate of a few other recent tickets.

I have a project waiting for testing under SBCL on the latest Intel MBP running Monterey.

[pmetzger (Perry E. Metzger)]

Has anyone tried getting in touch with easye?

[pmetzger (Perry E. Metzger)]

So I'm assuming no one has gotten in touch with easye?

[easye]

Apologies, but I don't have access to an Intel MBP running Monterey, so am not currently able to contribute meaningfully.

[pmetzger (Perry E. Metzger)]

So you're basically saying you can't maintain the port?

[easye]

Replying to pmetzger:

So you're basically saying you can't maintain the port?

The port maintenance is open: all may contribute. I'm merely noting that I don't have access to any Intel macOS at the moment, as I am on an extended road trip. For what it is worth, I was able to built sbcl-2.1.2 successfully under x86_64-macos-10.15 ("Catalina") at the beginning of the month.

[chriterrier]

Hello, It seems that I have also some troubles with installing sbcl 2.2.2 with Monterrey 12.3.1 (Intel, iMac 2017). Thank you in advance!

[kencu (Ken)]

wisdom seems to be to use a newer sbcl to bootstrap from.

​https://bugs.launchpad.net/sbcl/+bug/1949580

[pmetzger (Perry E. Metzger)]

There is no newer SBCL available for x86_64 on mac to bootstrap from.

[easye]

Replying to pmetzger:

There is no newer SBCL available for x86_64 on mac to bootstrap from.

Correct. Someone needs to reach out to stasstats or krystof on IRC to ask for help in publishing one. Referencing this ticket in their <​https://bugs.launchpad.net/sbcl/> tracker would be a good start. I haven't had the time to pursue this request.

[pmetzger (Perry E. Metzger)]

I don't know the players involved or what exactly to say to them so it would probably be better if someone else who knows them were to reach out.

[kencu (Ken)]

It may be possible to use this one as the bootstrap compiler:

​https://packages.macports.org/sbcl/sbcl-2.2.2_0+fancy.darwin_20.x86_64.tbz2

and copy the parts into the build tree. It has no deps and links only against a couple of system libraries, so it appears 'defacto' relocatable.

[tomio-arisaka (Tomio Arisaka)]

Another way is to add some variants.

• a_ccl : active CCL is used to compile the source code of SBCL.
• a_sbcl : active SBCL is used to compile the source code of SBCL.

On Monterey, use the a_ccl variant when building SBCL at first.

$ sudo port install sbcl +a_ccl

After that, another SBCL can be build with the a_sbcl variant.

$ sudo port install sbcl +a_sbcl

My patch is the following:

$ diff -u Portfile.orig Portfile
--- Portfile.orig	2022-03-08 19:46:23.000000000 +0900
+++ Portfile	2022-02-27 17:00:03.000000000 +0900
@@ -64,6 +64,22 @@
         size    10038928
     global host_lisp
     set host_lisp "\"${workpath}/${name}-${bootversion}-x86-64-darwin/src/runtime/sbcl --core ${workpath}/${name}-${bootversion}-x86-64-darwin/output/sbcl.core --disable-debugger --sysinit /dev/null --userinit /dev/null\" "
+
+    if { [file isfile "${prefix}/bin/sbcl"] == 1 } then {
+        variant a_sbcl conflicts a_ccl description {Active SBCL is used to compile this.} {
+            set host_lisp "\"${prefix}/bin/sbcl --core ${prefix}/lib/sbcl/sbcl.core --disable-debugger --sysinit /dev/null --userinit /dev/null\" "
+            if { [lindex [exec file -F "" "${prefix}/bin/sbcl"] 4] ne "x86_64" } {
+                pre-fetch {
+                    return -code error "active SBCL is not an x86_64 binary"
+                }
+            }
+        }
+    }
+
+    variant a_ccl conflicts a_sbcl description {Active CCL is used to compile this.} {
+        set host_lisp "\"${prefix}/bin/ccl64 --no-init\" "
+        depends_build   path:bin/ccl64:ccl
+    }
 }
 if {${build_arch} eq "arm64"} {
     set bootversion 2.1.2

[chriterrier]

Hi, Personnaly, I'm unable to reproduce what Tomio Arisaka suggested. Is there any possibility that this problem with sbcl will be fixed? My goal is to install wxmaxima and it requests sbcl. Could wxmaxima be installed without sbcl? Best regards, Ch.

[dcamper (Dan S. Camper)]

I was able to get tomio-arisaka's patch to work, both with sbcl 2.2.2 and after attempting the 2.2.4 update.

The problem does not seem to lie with sbcl, exactly. The problem is with this port using an old -- and incompatible -- version of sbcl to bootstrap the compilation of the newer sbcl versions. The fix is to either use a compatible version of sbcl or to use ccl for the bootstrap.

I would like to see the default logic of the port changed to something like:

1. If sbcl is currently installed, bootstrap using installed sbcl; else
2. If ccl is current installed, bootstrap using installed ccl; else
3. Perform current action (download sbcl), hopefully with a newer, compatible version

For the record, here are my exact steps for reproducing tomio-arisaka's work (note that if you copy the patch from the comment text you will have to fix a bit of formatting; I will try to attach a cleaned-up version to this bug report):

# Grab the sbcl distribution
sudo port fetch sbcl

# Change directory to what was downloaded
cd $(port dir sbcl)

# Make a copy of the Portfile; the copy is
# what is used as the source for the patch
cp Portfile Portfile.orig

# Apply the patch (which assumes is in your
# Downloads folder and named sbcl_patch.diff)
patch -p0 < ~/Downloads/sbcl_patch.diff

# Build SBCL using a previously-installed CCL
sudo port build sbcl +a_ccl

# Install what you just built
sudo port -s install sbcl +a_ccl

EDIT: Noticed poor formatting, tried to fix.

[dcamper (Dan S. Camper)]

tomio-arisaka's patch

[jxy (Xiao-Yong)]

There are four issues related to this bootstrap problem since 6 months ago, #63731 #63752 #63806 #64621.

[jxy (Xiao-Yong)]

ping

[chriterrier]

pong

[kencu (Ken)]

Replying to jxy:

There are four issues related to this bootstrap problem since 6 months ago, #63731 #63752 #63806 #64621.

If you are interested in sbcl and would like this bootstrapping issue fixed, please go ahead and have a go at fixing it.

There is nobody here hired to fix ports. They are fixed by interested volunteers like yourself, and those fixes are shared to others less capable or with less interest.

pinging does nothing to further these fixes.

[kakuhen]

I am here to report that builds work as intended on Darwin/arm64, meaning this is likely an issue with the Lisp implementation used to bootstrap SBCL on Darwin/x86_64 (version 1.2.11). I have posted in #sbcl on LiberaChat asking the developers for updated bootstrap binaries. I do not expect this request to be fulfilled. In the mean time, tomio-arisaka's suggestion is the best way forward.

SBCL can be bootstrapped just fine with other Common Lisp implementations, such as CCL, ECL, and even CLISP (provided you use recent versions, which MacPorts indeed offers). In my experience, most people bootstrap SBCL with ECL (if they decide to not use an older SBCL), as ECL only requires a C99-compliant compiler. In absence of an updated bootstrap binary from upstream, we can fall back on CCL (or ECL) as a solution for i386 and x86_64. CCL is probably the way to go since we can pull in a binary blob rather than perform a full compilation of ECL just to begin bootstrapping SBCL.

[pmetzger (Perry E. Metzger)]

Would someone be willing to change the SBCL portfile to specify the use of CLISP (which compiles fast) to bootstrap? I'd prefer not to be beholden to future binary blob disasters.

[ryandesign (Ryan Carsten Schmidt)]

Replying to dcamper:

I would like to see the default logic of the port changed to something like:

1. If sbcl is currently installed, bootstrap using installed sbcl; else
2. If ccl is current installed, bootstrap using installed ccl; else
3. Perform current action (download sbcl), hopefully with a newer, compatible version

Having the port do something different depending on what may already exist on the user's system is contrary to the goals of ReproducibleBuilds.

[kakuhen]

Right now, I have a pending PR to finally update SBCL to version 2.2.8. I have attempted to ask upstream on their IRC for updated bootstrap binaries for Darwin/amd64, but nobody responded. I may try sending a message to their mailing lists, to see what it would take to get a newer, official bootstrap binary.

Otherwise, to retain reproducible builds, we must rework the entire port to begin with an alternative Common Lisp host, such as ECL. I have recently updated CLISP on MacPorts to 2.50.0, which also makes it a viable candidate for bootstrapping SBCL. But in my experience, SBCL's source code is a real stress test for conformance to the ANSI CL specification, so we need to be careful in our choice of host.

[pmetzger (Perry E. Metzger)]

If it builds at all (using CCL or CLISP or what have you) we can then rebuild it in itself. I think we should always bootstrap from a lisp interpreter written in C or C++ for permanent ease of maintenance.

[kakuhen]

CCL is no-go for me because it simply moves "future binary blob disasters" to another port and lacks Apple Silicon support. If I understood the ReproducibleBuilds guidelines, then any particular change to x86_64 requires a corresponding change to arm64.

CLISP has the issue of long pauses between releases, and Apple Silicon support is probably not as well tested some of the more actively maintained Common Lisp implementations. Any bugs in CLISP would take months to fix and even more months to appear in a release.

If we must switch to an alternative compiler, then either ABCL or ECL will be the sane options in my opinion. Neither of them are written entirely in C/C++ (ABCL is in fact built on top of the JVM), but again, we may be introducing possible bugs that arise from ABCL or ECL compiling SBCL. With that said, most package managers--if they wish to avoid starting an SBCL bootstrap from an older SBCL--will use ECL, so that particular compiler has, anecdotally, the most support. I would personally prefer to keep bootstrapping from SBCL for best results; I will start asking for updated bootstrap binaries on MLs to see if that helps the situation.

There really isn't a viable interpreter written entirely in C/C++. In fact, most Common Lisp implementations are compiler-only (i.e. no interpreted mode). I do know of one implementation that is indeed 100% C (npt), but its not on MacPorts, and its full of memory bugs, issues with CLOS, etc.

---

tl;dr: I recommend ECL, if any alternative host, but be aware of the tradeoffs; I prefer no alternative hosts.

[pmetzger (Perry E. Metzger)]

I have no strong opinion. ECL is probably fine. Remember that if you then build SBCL with the bootstrapped SBCL you should hit a steady state. Also remember we have access to conformance tests so they can be run on the bootstrapped binary. If SBCL is willing to help us get SBCL binaries that are kept up to date that's also fine for bootstrap.

[hraban (Hraban Luyat)]

FYI SBCL has released a darwin x86-64 binary for 2.2.9 today: ​https://sourceforge.net/projects/sbcl/files/sbcl/2.2.9/

This might help?

[kakuhen]

Replying to hraban:

FYI SBCL has released a darwin x86-64 binary for 2.2.9 today: ​https://sourceforge.net/projects/sbcl/files/sbcl/2.2.9/

This might help?

Thank you for notifying us about this. I have modified my PR to update SBCL using the latest bootstrap binaries. Unfortunately, I do not have access to an Intel Mac running Monterey. But I can report successful builds under trace mode on Catalina. Refer to ​https://github.com/macports/macports-ports/pull/15263 for more details.

[pmetzger (Perry E. Metzger)]

Fixed in commit 9e0c2c4961. Thanks to everyone who helped with this.