sbcl @2.1.10 +fancy: Cannot build on macOS 12.0.1 with Xcode 13.1

[foretspaisibles (Michaël Le Barbier)]

I cannot build sbcl@2.1.10 on macOS 12.0.1 (Monterey/i386). This is an installation migration following the migration guide.

A first examination of the attached log shows that the build system seems to download and run an older (1.2.11) binary version of SBCL, which results in a signal 9 (KILL).

Some rather odd detail is also that SYSINFO lines at the top of the log reports i386 as a building architecture, while

% uname -a
Darwin MacBook-Pro 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64

shows the expected x86_64 and the build system is also picking a x86_64 binary for downloading.

[foretspaisibles (Michaël Le Barbier)]

Build log

[ryandesign (Ryan Carsten Schmidt)]

Per the "Skipping completed" lines in the log, this was not a clean installation attempt; clean the port and try again. This may be a duplicate of #63752.

SYSINFO lines at the top of the log reports i386

In this context, "i386" means "any Intel Mac".

[foretspaisibles (Michaël Le Barbier)]

I retried a clean build, yielding the same result. See attached file. Since I saw other tickets hinting at strange patterns in memory consumption I also took a look at the activity and monitor and top.

It turns out that the first command in the build sequence

  PID TTY           TIME CMD
  712 ttys000    9:12.18 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_sbcl/sbcl/work/sbcl-1.2.11-x86-64-darwin/src/runtime/sbcl --core /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_sbcl/sbcl/work/sbcl-1.2.11-x86-64-darwin/output/sbcl.core --disable-debugger --sysinit /dev/null --userinit /dev/null

will run for almost 50 minutes on my system, slowly allocating memory until it is killed by the system after exhausting physical memory and swap. See attached screenshot.

[foretspaisibles (Michaël Le Barbier)]

When trying to run the command on the terminal, it does not display any logs besides the SBCL banner, as can be inferred from the logs. Interrupting the program with Control-C does not yield, which is usually the sign that the process is in a non interruptible system call. Rerunning again under dtruss shows the trail:

…
mmap(0x0, 0x80A004, 0x7, 0x41002, 0xFFFFFFFFFFFFFFFF, 0x0)		 = 0x5800000 0
mmap(0x0, 0x80, 0x7, 0x41002, 0xFFFFFFFFFFFFFFFF, 0x0)		 = 0x1F2000 0

[snowflake (Dave Evans)]

This may be a duplicate of my bug #63731

I am pleased to see a confirmation of my bug as I was wondering why ​https://ports.macports.org/port/sbcl was showing several users had successful builds on Monterey, so what is different about their builds?

The Big Sur build of sbcl does run on Monterey. The problem is the very old version of sbcl that is used to bootstrap the build.

[foretspaisibles (Michaël Le Barbier)]

I have been able to build and use SBCL by bootstrapping using CCL instead of SBCL. It can be done using the following variant definition:

variant ccl description {Uses CCL to boostrap SBCL.} {
    depends_build-append bin:ccl64:ccl
    set host_lisp "${prefix}/bin/ccl64"
}

However I am not sure how to cleanup or reorganise the Portfile so that we do not download the binary of SBCL if we are not going to use it.

[easye]

Unfortunately, I do not have access to a x86_64-macos-12 system to test.

As mentioned in <​https://trac.macports.org/ticket/63731#comment:7>, a possible solution would be to "move forward" the version of bootstrap SBCL image. I've asked for information on where we could get such an image in <​https://bugs.launchpad.net/sbcl/+bug/1949580/comments/2>.

[foretspaisibles (Michaël Le Barbier)]

Oh I see, but could not we use some older binary package produced by MacPorts itself? Or is there no backup available for these?

[easye]

Replying to foretspaisibles:

Oh I see, but could not we use some older binary package produced by MacPorts itself? Or is there no backup available for these?

Older MacPorts binary package would work with a little elbow grease to overcome the differing filesystem layout, but I don't know of any archiving mechanisms for MacPorts packages. Given that it isn't really possible to "go back in time" to a previous state of MacPorts, I don't think that such an archive is maintained within the MacPorts infrastructure but I could well be wrong.

Can anyone provide more information of whether MacPorts archives are retained at all and where?

[kencu (Ken)]

there are:

​https://packages.macports.org/sbcl/

This is not an archive like snapshot.debian.org that lasts forever, however. If you want one of these, better grab it. There are also issues regarding supporting ports being of the right vintage, and hardcoding to /opt/local to be aware of, so not a perfect bootstrap perhaps...

[foretspaisibles (Michaël Le Barbier)]

Unfortunately my SBCL 2.1.10 binary bootstrapped using CCL64 seems a bit buggy. It randomly exhausts heap when compiling some files. (It happens often but not reliably.)

The same codebase is very quickly and correctly processed by CCL64.

(Just documenting this here in case someone comes across the same problem before being very sure it is related to that issue.)

Heap exhausted during garbage collection: 0 bytes available, 64 requested.
Gen  Boxed   Code    Raw  LgBox LgCode  LgRaw  Pin       Alloc     Waste        Trig      WP GCs Mem-age
 2   12755      9   4311     34      0     15   59   555647216   5472016    40999370   17124   1  1.0702
 3   11309      7   3462     55      0     31    0   482359040   4704512     2000000   14864   0  0.0000
 4       0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
 5       0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
 6     492      2    220     55      0     10    0    24832016    694256     2000000     779   0  0.0000
           Total bytes allocated    =    1062838272
           Dynamic-space-size bytes =    1073741824
GC control variables:
   *GC-INHIBIT* = true
   *GC-PENDING* = true
   *STOP-FOR-GC-PENDING* = false
fatal error encountered in SBCL pid 98868 pthread 0xd819600:
Heap exhausted, game over.

   0: fp=0x96f5948 pc=0x52e0155f SB-REGALLOC::GROW-SC
   1: fp=0x96f59f8 pc=0x52b05d47 SB-REGALLOC::PACK-TN
   2: fp=0x96f5aa8 pc=0x52b061e3 (FLET SB-REGALLOC::PACK-TNS :IN SB-REGALLOC::PACK)
   3: fp=0x96f5b58 pc=0x52b051d3 SB-REGALLOC::PACK
   4: fp=0x96f5be8 pc=0x52c15d05 SB-C::%COMPILE-COMPONENT
   5: fp=0x96f5c50 pc=0x52b305a1 SB-C::COMPILE-COMPONENT
   6: fp=0x96f5cb0 pc=0x52c17b84 SB-C::%COMPILE
   7: fp=0x96f5ce0 pc=0x52db6379 SB-C::FOPCOMPILE-FUNCTION
   8: fp=0x96f5d50 pc=0x52b38734 SB-C::FOPCOMPILE
   9: fp=0x96f5d88 pc=0x52c16db5 SB-C::CONVERT-AND-MAYBE-COMPILE
  10: fp=0x96f5e00 pc=0x52c1937f SB-C::PROCESS-TOPLEVEL-FORM
  11: fp=0x96f5e30 pc=0x52c16f98 SB-C::PROCESS-TOPLEVEL-PROGN
  12: fp=0x96f5ea8 pc=0x52c1920a SB-C::PROCESS-TOPLEVEL-FORM
  13: fp=0x96f5f20 pc=0x52c192c4 SB-C::PROCESS-TOPLEVEL-FORM
  14: fp=0x96f5f98 pc=0x52c192c4 SB-C::PROCESS-TOPLEVEL-FORM
  15: fp=0x96f6078 pc=0x52c1bbf6 (LAMBDA (SB-KERNEL::FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-C::SUB-COMPILE-FILE)
  16: fp=0x96f6138 pc=0x52b7a2bd SB-C::%DO-FORMS-FROM-INFO
  17: fp=0x96f6208 pc=0x52c1a2d4 (FLET "LAMBDA0" :IN "SYS:SRC;COMPILER;MAIN.LISP")
  18: fp=0x96f62b0 pc=0x52b799bc (FLET SB-C::WITH-IT :IN SB-C::%WITH-COMPILATION-UNIT)
  19: fp=0x96f6428 pc=0x52c1b365 SB-C::SUB-COMPILE-FILE
  20: fp=0x96f64f0 pc=0x52c1c499 SB-C::%COMPILE-FILES
  21: fp=0x96f65b0 pc=0x531024fb COMPILE-FILE

[foretspaisibles (Michaël Le Barbier)]

I was able to bootstrap SBCL using an older SBCL version found on MacPorts archive. (Just took 2.1.9) It just requires the SBCL binary and the SBCL.CORE file, both moved out from the downloading quarantine with

xattr -d com.apple.quarantine tarball/opt/local/lib/sbcl/sbcl.core tarball/opt/local/bin/sbcl

Also, variants similar to the CCL64 bootstrapping method above with CLISP and ACBL just work fine too. (I did not test GCL yet).

[jxy (Xiao-Yong)]

I used ecl to bootstrap. I guess we could opt for other compatible common lisp too.

--- Portfile~	2022-01-12 11:38:01.000000000 -0600
+++ Portfile	2022-01-12 11:01:30.000000000 -0600
@@ -54,7 +54,7 @@
 # maintainer does not have systems on which to test such things.
 # However, if someone does come up with a way to support powerpc,
 # duplicating the "if" here would be the way to do it.
-if {${build_arch} eq "x86_64"} {
+if {${build_arch} eq "x86_64" && ![variant_isset ecl]} {
     set bootversion 1.2.11
     master_sites-append sourceforge:project/sbcl/sbcl/${bootversion}:sbcl_amd64
     distfiles-append    ${name}-${bootversion}-x86-64-darwin-binary${extract.suffix}:sbcl_amd64
@@ -144,6 +144,12 @@
     set make_sh_options --fancy
 }
 
+variant ecl description {Uses ECL to boostrap SBCL.} {
+    depends_build-append bin:ecl:ecl
+    global host_lisp
+    set host_lisp "${prefix}/bin/ecl"
+}
+
 test.run        yes
 test.dir        ${worksrcpath}/tests
 test.cmd        CC=${configure.cc} CXX=${configure.cxx} CPP=${configure.cpp} sh

[jxy (Xiao-Yong)]

It looks like ECL can no longer bootstrap SBCL 2.2.0. The previous contribution from @foretspaisibles using CCL still works.

[foretspaisibles (Michaël Le Barbier)]

@easye, this ticket seems to be history now. I performed several upgrades of SBCL without any issue… I can also build SBCL on a fresh 12.X install (GitHub Actions) using MacPorts, so we could probably close that!

[kencu (Ken)]

sbcl has been updated several times since this ticket, and the current version builds broadly, including on Monterey Intel

​https://build.macports.org/builders/ports-12_x86_64-builder/builds/61171