neovim @0.10.0_1: error loading module 'nlua0' from file 'libnlua0.so'

[wryfi]

Neovim is built around luajit, which is based on lua 5.1.

The neovim build process creates a C library, nlua0, which is subsequently used by luajit later in the build process.

In MacPorts, nlua0 gets built against the default lua, which is currently version 5.3. When luajit tries to then use this nlua0, the neovim build fails, because the built library includes symbols that don't exist in 5.1.

In the short term, port deactivate lua && port install neovim && port activate lua works around this issue.

The following ​GitHub PR updates the neovim port to add the proper include directory for lua 5.1 to cmake, and should resolve the issue.

Ultimately, some changes to the lua packaging and/or docs would help prevent future ports from breaking. See comments below and #46621 for additional background.

[wryfi]

build log

[wryfi]

Looks like a lua issue of some kind? Before uninstalling neovim, I had deactivated it, then ran port upgrade outdated successfully, then reactivated neovim and tried to upgrade it. All of my ports should be up to date.

:info:build cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/src/nvim && /opt/local/bin/luajit /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/neovim-0.10.0/src/nvim/generators/preload.lua /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/neovim-0.10.0 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/neovim-0.10.0/src/nvim/generators/gen_declarations.lua /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/neovim-0.10.0/src/nvim/api/extmark.c /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/src/nvim/auto/api/extmark.c.generated.h /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/include/api/extmark.h.generated.h /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/src/nvim/auto/api/extmark.i
:info:build /opt/local/bin/luajit/opt/local/bin/luajit::  error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0100794fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0100741bb4error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0100d00fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0100cadbb4/opt/local/bin/luajit: error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x01004d0fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x010047dbb4
:info:build /opt/local/bin/luajit: error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0100664fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0100611bb4
:info:build /opt/local/bin/luajit: error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0100aa4fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0100a51bb4
:info:build /opt/local/bin/luajit: error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0102ef4fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0102ea1bb4
:info:build make[2]: *** [src/nvim/auto/ui_events_call.generated.h] Error 1
:info:build make[2]: *** Waiting for unfinished jobs....
:info:build make[2]: *** [src/nvim/auto/api/extmark.c.generated.h] Error 1
:info:build make[2]: *** [src/nvim/auto/api/deprecated.c.generated.h] Error 1
:info:build make[2]: *** [src/nvim/auto/api/autocmd.c.generated.h] Error 1
:info:build make[2]: *** [src/nvim/auto/api/buffer.c.generated.h] Error 1
:info:build make[2]: *** [src/nvim/auto/api/command.c.generated.h] Error 1
:info:build make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build'
:info:build make[1]: *** [src/nvim/CMakeFiles/nvim_bin.dir/all] Error 2

[wryfi]

lua-related ports on my system:

❯ port installed | grep lua
  lua @5.3.6_3 (active)
  lua-luarocks @3.9.2_0 (active)
  lua51 @5.1.5_9 (active)
  lua51-lpeg @1.1.0_0 (active)
  lua51-mpack @1.0.9_0 (active)
  lua53 @5.3.6_3 (active)
  lua53-luarocks @3.9.2_0 (active)
  luajit @2.1.1700008891_0
  luajit @2.1.1703358377_0
  luajit @2.1.1710088188_0 (active)
  luametatex @2.10.08_0
  luametatex @2.11.02_0 (active)
  luv-luajit @1.45.0-0_1 (active)
  texlive-luatex @66513_0+doc
  texlive-luatex @70579_0+doc (active)

[wryfi]

Now I have no neovim.

I was able to revert back to 0.9.5.

[l2dy (Zero King)]

Part of the error message "symbol not found in flat namespace" is similar to #66077, which I have not been able to reproduce.

[wryfi]

I wonder why that would affect 0.10.0 but not 0.9.5?

Anyway I can try the lua patches in #66077 today and see if they change anything. Thanks!

[wryfi]

Something is definitely fishy with the way macports is managing lua and/or luarocks dependencies, but I don't know enough about either MacPorts or lua to pin it down precisely.

I uninstalled everything related to lua from my system, ran port reclaim, ran port clean --all on all of the lua and neovim related ports, and then tried port install neovim all over again. I got exactly the same errors as before, no change whatsoever.

Even though neovim uses exclusively lua-5.1, and I have no other lua dependent ports on my system, lua-5.3 packages always get installed — even though I have absolutely no use for them!

When building neovim, something gets compiled against lua-5.3, because (similar to #66077) _luaL_prepbuffsize does not exist in lua-5.1.

So then I uninstalled lua53-luarocks and lua53, and lua ports, despite a warning:

Note: It is not recommended to uninstall/deactivate a port that has dependents as it breaks the dependents.
The following ports will break: lua-luarocks @3.9.2_0

Et voila — port install neovim succeeds without a hitch!

I don't know if this actually breaks anything with lua-luarocks, but it fixes my problem.

This looks fishy to me:

The following ports are dependencies of neovim @0.10.1_0:
  ...
  luajit
  lua51-lpeg
    lua-luarocks
      lua
      lua53-luarocks
        lua53
    lua51
  lua51-mpack
  luv-luajit
  ...

Why do lua-5.1 packages depend on lua-5.3 packages? That can't be right, can it?!

[ryandesign (Ryan Carsten Schmidt)]

The lua-luarocks port has subports lua54-luarocks, lua53-luarocks, lua52-luarocks, and lua51-luarocks, yet no ports depend on them, other than lua-luarocks which depends on lua53-luarocks. All ports that depend on luarocks do so via the lua-luarocks port so they all depend on lua53-luarocks regardless what lua version they're actually using.

According to the last commit to the lua-luarocks port, "The luarock does not need to use the same Lua version as the module it is building so long as proper config file is installed" so I guess this is supposed to be working the way it is. Yet, what is the purpose of the versioned subports? Should all the ports that now depend on lua-luarocks be changed to use the subport that matches the lua version they use?

[wryfi]

Ok, I think I've got this figured out. There are a lot of things that are suboptimal about how macports is managing lua. This may not be the right place for detailing all of that, but I will do so anyway, and have faith that someone on the macports team will land them in the right place.

Starting with this bug report and #66077 ...

luajit is used for much of the build process, and the neovim Portfile contains configure.args-append -DLUA_PRG=${prefix}/bin/luajit. luajit is its own thing and adheres only to the lua-5.1 language spec. luajit works regardless of the state of other lua packages on my system. All good so far.

But neovim also uses some C, and builds a library, nlua0, from src/nlua0.c in the neovim source tree. The first line of this file is #include <lua.h>.

When the lua port is installed, it creates /opt/local/include/lua.h, which contains C headers for lua 5.3. Because this is the first lua.h file the compiler finds in the include directory, the C compiler uses this header and then builds nlua0 against lua 5.3.

This breaks the neovim build as described in this ticket and in #66077 because nlua0 pulls in symbols from lua 5.3 that luajit can't grok (because luajit is strictly 5.1).

The fix for this is to get the compiler to look in /opt/local/include/lua5.1 and use the lua.h header from the lua51 port. See ​GitHub PR.

diff --git a/editors/neovim/Portfile b/editors/neovim/Portfile
index 6c79bb4bd3e..1d0825f863e 100644
--- a/editors/neovim/Portfile
+++ b/editors/neovim/Portfile
@@ -48,7 +48,7 @@ cmake.build_type        Release
 configure.args-append   -DLUA_PRG=${prefix}/bin/luajit

 # Building parsers is normally an extra step, see https://github.com/neovim/neovim/issues/29042
-patchfiles              embed-parsers-build.diff
+patchfiles              embed-parsers-build.diff  patch-nlua0-includes.diff

 subport neovim-devel {
     github.setup    neovim neovim 0c2860d9e5ec5417a94db6e3edd237578b76d418
diff --git a/editors/neovim/files/patch-nlua0-includes.diff b/editors/neovim/files/patch-nlua0-includes.diff
new file mode 100644
index 00000000000..4e6f7c8faef
--- /dev/null
+++ b/editors/neovim/files/patch-nlua0-includes.diff
@@ -0,0 +1,12 @@
+diff --git src/nvim/CMakeLists.txt src/nvim/CMakeLists.txt
+index a100e733d..9c601819b 100644
+--- src/nvim/CMakeLists.txt
++++ src/nvim/CMakeLists.txt
+@@ -66,6 +66,7 @@ else()
+   target_include_directories(main_lib SYSTEM BEFORE INTERFACE ${LUAJIT_INCLUDE_DIR})
+   target_link_libraries(main_lib INTERFACE ${LUAJIT_LIBRARY})
+   target_include_directories(nlua0 SYSTEM BEFORE PUBLIC ${LUAJIT_INCLUDE_DIR})
++  target_include_directories(nlua0 AFTER PUBLIC /opt/local/include/lua5.1)
+   if(WIN32)
+     target_link_libraries(nlua0 PUBLIC ${LUAJIT_LIBRARY})
+   endif()

(Note that I'm not a CMake expert, and I'm not certain that target_include_directories(nlua0 AFTER PUBLIC /opt/local/include/lua5.1) is strictly correct; but it does the job.)

---

Now on to some of my other discoveries while investigating this issue ...

I don't think the build/install issues with luarocks are as simple as the commit referenced above suggests. There seem to be a few broken layers to unpack here (though to be clear this is unrelated to the neovim build).

At the top of the ​config file referenced in that git commit:

-- Run `luarocks` with no arguments to see the locations of
-- these files in your platform.

When I run luarocks on my system, it provides some interesting info at the end:

Configuration:
   Lua:
      Version    : 5.3
      Interpreter: /opt/local/bin/lua5.3 (ok)
      LUA_DIR    : /opt/local/libexec/lua53 (ok)
      LUA_BINDIR : /opt/local/bin (ok)
      LUA_INCDIR : /opt/local/libexec/lua53/include (ok)
      LUA_LIBDIR : /opt/local/libexec/lua53/lib (ok)

   Configuration files:
      System  : /opt/local/etc/luarocks/config-5.3.lua (ok)
      User    : /Users/wryfi/.luarocks/config-5.3.lua (not found)

   Rocks trees in use:
      /Users/wryfi/.luarocks ("user")
      /opt/local/share/luarocks ("system")

Whether I run luarocks, luarocks-5.1, luarocks-5.2, luarocks-5.3, or luarocks-5.4, they all return the same configuration. The versioned copies are all symlinks to the same /opt/local/bin/luarocks script, so this comes as no shock.

That luarocks script has lua-5.3 paths hard-coded into it:

package.path=[[/opt/local/share/lua/5.3/?.lua;]] .. package.path
local list = package.searchers or package.loaders; table.insert(list, 1, function(name) if name:match("^luarocks%.") then return loadfile([[/opt/local/share/lua/5.3/]] .. name:gsub([[%.]], [[/]]) .. [[.lua]]) end end)

It also has its shebang set to /opt/local/bin/lua, which is installed by the lua port, and seems to be (currently) version 5.3:

$ /opt/local/bin/lua
Lua 5.3.6  Copyright (C) 1994-2020 Lua.org, PUC-Rio
>

I find it interesting that luarocks reports Interpreter: /opt/local/bin/lua5.3 (ok) even though the shebang is /opt/local/bin/lua, especially since lua is not a symlink to lua5.3 but its own separate binary. (I also have discrete lua5.1 and lua5.3 binaries.) Not sure how this works.

The lua and lua5.3 binaries have different checksums. They are binaries, so I can't tell for sure if anything is different about them, but they both start a lua-5.3 interpreter, and running strings against them seems to return identical output.

I don't see a way to manage the version of /opt/local/bin/lua, other than mv lua5.1 lua. It seems like an antipattern that the version of the lua binary is arbitrarily set for me, with no input or obvious way to change it.

---

So in summary:

1. The lua port decides for you what version of lua (currently 5.3) will be the default lua on your system, and that all C libraries linking against lua.h will build against that version without extra care.
2. Any port that depends on lua installs and makes the lua port's version (currently 5.3) your default lua, even if your lua-based port actually depends on an older version of lua.
3. The lua port installs a 5.3 binary in /opt/local/bin/lua that appears to be similar, but not identical, to the /opt/local/bin/lua5.3 binary installed by the lua53 port.
4. There is no facility to manage which version of lua is used as the system default.
5. Uninstalling the lua port will break luarocks and potentially other lua ports, even though they could use a different version of lua.
6. neovim depends on lua51-lpeg, which depends on lua-luarocks, which depends on lua and lua53-luarocks, guaranteeing that lua 5.3 will be installed when trying to install neovim.
7. Building neovim is breaking because it links against the lua.h provided by the lua port (currently 5.3), which is incompatible with neovim and luajit (not for any reasons having to do with luarocks).
8. luarocks always runs against the version of lua installed by the lua port, because its shebang is /opt/local/bin/lua, which is (currently) lua 5.3 by design of the lua port.
   1. This remains true even if running luarocks-5.n because the versioned copies are all symlinks to the same luarocks script.
9. luarocks always manages packages in /opt/local/share/lua/5.3 because this value is hard-coded in the script.

[rbeyer (Ross Beyer)]

I recently had to re-install my system and macports on my macOS 14 / Apple Silicon system and ran into this exact problem with installing neovim (whereas earlier this spring on initial installation, it worked just fine). I was excited to see wryfi's explanation of performing a port uninstall of the lua53 ports that got neovim working, but that did not happen for me. My experience was identical, with the symbol not found in flat namespace '_luaL_prepbuffsize' error in the log. I did the same:

  $> sudo port reclaim
  $> sudo port clean --all (of the two remaining lua ports: luametatex and texlive-luatex
  $> sudo port install neovim (which errors, just as reported in the OP)
  $> sudo port uninstall lua53-luarocks (with the same warning as reported above)
  $> sudo port uninstall lua53
  $> sudo port install neovim

And I was expecting that last port install neovim to work as wryfi's experience did, but it did not, and these namespace errors are still in the new log file:

:info:build /opt/macports/bin/luajit: error loading module 'nlua0' from file '/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0100664fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0100611bb4

I'm not sure what wryfi did that I didn't.

[wryfi]

Replying to rbeyer:

I'm not sure what wryfi did that I didn't.

port uninstall lua

It's actually the lua port (no version in the port name) that is the issue, as it installs a lua.h header that neovim uses to build nlua0 against lua 5.3.

(Updated comment#7 above to add lua to the list of things I uninstalled.)

[wryfi]

And actually, port deactivate lua && port install neovim && port activate lua would probably be the better sequence.

[wryfi]

Reading #46621 provided some interesting background on the lua packaging, and even mentions the lua.h issue that spawned this ticket (comment 12). That conversation seems to have stalled out without a complete resolution or follow up.

I would personally favor eliminating the lua port and creating a lua_select port to manage the default version selection as is done with python. It looks like someone started creating a luarocks_select port to manage luarocks versioning but it appears to be incomplete.

It seems like a more discoverable place to document best practices and roadmaps for specific ports and languages would be useful. Or maybe I just don't know where to look?

[rbeyer (Ross Beyer)]

Perfect, port deactivate lua && port install neovim && port activate lua worked. Thanks!

[Chris Haumesser <5400416-wryfi@…>]

In 391c7804e8d2a5f359d9108d95bcdc304d1b9d62/macports-ports (master):

neovim: include lua51 headers to build properly

fixes #70420
fixes #66077