neovim @0.10.0_1: error loading module 'nlua0' from file 'libnlua0.so'

[wryfi]

Neovim is built around luajit, which is based on lua 5.1.

The neovim build process creates a C library, nlua0, which is subsequently used by luajit later in the build process.

In MacPorts, nlua0 gets built against the default lua, which is currently version 5.3. When luajit tries to then use this nlua0, the neovim build fails, because the built library includes symbols that don't exist in 5.1.

In the short term, port deactivate lua && port install neovim && port activate lua works around this issue.

The following ​GitHub PR updates the neovim port to add the proper include directory for lua 5.1 to cmake, and should resolve the issue.

Ultimately, some changes to the lua packaging and/or docs would help prevent future ports from breaking. See comments below and #46621 for additional background.

[wryfi]

build log

[wryfi]

Looks like a lua issue of some kind? Before uninstalling neovim, I had deactivated it, then ran port upgrade outdated successfully, then reactivated neovim and tried to upgrade it. All of my ports should be up to date.

:info:build cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/src/nvim && /opt/local/bin/luajit /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/neovim-0.10.0/src/nvim/generators/preload.lua /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/neovim-0.10.0 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/neovim-0.10.0/src/nvim/generators/gen_declarations.lua /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/neovim-0.10.0/src/nvim/api/extmark.c /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/src/nvim/auto/api/extmark.c.generated.h /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/include/api/extmark.h.generated.h /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/src/nvim/auto/api/extmark.i
:info:build /opt/local/bin/luajit/opt/local/bin/luajit::  error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0100794fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0100741bb4error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0100d00fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0100cadbb4/opt/local/bin/luajit: error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x01004d0fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x010047dbb4
:info:build /opt/local/bin/luajit: error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0100664fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0100611bb4
:info:build /opt/local/bin/luajit: error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0100aa4fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0100a51bb4
:info:build /opt/local/bin/luajit: error loading module 'nlua0' from file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0102ef4fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0102ea1bb4
:info:build make[2]: *** [src/nvim/auto/ui_events_call.generated.h] Error 1
:info:build make[2]: *** Waiting for unfinished jobs....
:info:build make[2]: *** [src/nvim/auto/api/extmark.c.generated.h] Error 1
:info:build make[2]: *** [src/nvim/auto/api/deprecated.c.generated.h] Error 1
:info:build make[2]: *** [src/nvim/auto/api/autocmd.c.generated.h] Error 1
:info:build make[2]: *** [src/nvim/auto/api/buffer.c.generated.h] Error 1
:info:build make[2]: *** [src/nvim/auto/api/command.c.generated.h] Error 1
:info:build make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build'
:info:build make[1]: *** [src/nvim/CMakeFiles/nvim_bin.dir/all] Error 2

[wryfi]

lua-related ports on my system:

❯ port installed | grep lua
  lua @5.3.6_3 (active)
  lua-luarocks @3.9.2_0 (active)
  lua51 @5.1.5_9 (active)
  lua51-lpeg @1.1.0_0 (active)
  lua51-mpack @1.0.9_0 (active)
  lua53 @5.3.6_3 (active)
  lua53-luarocks @3.9.2_0 (active)
  luajit @2.1.1700008891_0
  luajit @2.1.1703358377_0
  luajit @2.1.1710088188_0 (active)
  luametatex @2.10.08_0
  luametatex @2.11.02_0 (active)
  luv-luajit @1.45.0-0_1 (active)
  texlive-luatex @66513_0+doc
  texlive-luatex @70579_0+doc (active)

[wryfi]

Now I have no neovim.

I was able to revert back to 0.9.5.

[l2dy (Zero King)]

Part of the error message "symbol not found in flat namespace" is similar to #66077, which I have not been able to reproduce.

[wryfi]

I wonder why that would affect 0.10.0 but not 0.9.5?

Anyway I can try the lua patches in #66077 today and see if they change anything. Thanks!

[wryfi]

Something is definitely fishy with the way macports is managing lua and/or luarocks dependencies, but I don't know enough about either MacPorts or lua to pin it down precisely.

I uninstalled everything related to lua from my system, ran port reclaim, ran port clean --all on all of the lua and neovim related ports, and then tried port install neovim all over again. I got exactly the same errors as before, no change whatsoever.

Even though neovim uses exclusively lua-5.1, and I have no other lua dependent ports on my system, lua-5.3 packages always get installed — even though I have absolutely no use for them!

When building neovim, something gets compiled against lua-5.3, because (similar to #66077) _luaL_prepbuffsize does not exist in lua-5.1.

So then I uninstalled lua53-luarocks and lua53, and lua ports, despite a warning:

Note: It is not recommended to uninstall/deactivate a port that has dependents as it breaks the dependents.
The following ports will break: lua-luarocks @3.9.2_0

Et voila — port install neovim succeeds without a hitch!

I don't know if this actually breaks anything with lua-luarocks, but it fixes my problem.

This looks fishy to me:

The following ports are dependencies of neovim @0.10.1_0:
  ...
  luajit
  lua51-lpeg
    lua-luarocks
      lua
      lua53-luarocks
        lua53
    lua51
  lua51-mpack
  luv-luajit
  ...

Why do lua-5.1 packages depend on lua-5.3 packages? That can't be right, can it?!

[ryandesign (Ryan Carsten Schmidt)]

The lua-luarocks port has subports lua54-luarocks, lua53-luarocks, lua52-luarocks, and lua51-luarocks, yet no ports depend on them, other than lua-luarocks which depends on lua53-luarocks. All ports that depend on luarocks do so via the lua-luarocks port so they all depend on lua53-luarocks regardless what lua version they're actually using.

According to the last commit to the lua-luarocks port, "The luarock does not need to use the same Lua version as the module it is building so long as proper config file is installed" so I guess this is supposed to be working the way it is. Yet, what is the purpose of the versioned subports? Should all the ports that now depend on lua-luarocks be changed to use the subport that matches the lua version they use?

[wryfi]

Ok, I think I've got this figured out. There are a lot of things that are suboptimal about how macports is managing lua. This may not be the right place for detailing all of that, but I will do so anyway, and have faith that someone on the macports team will land them in the right place.

Starting with this bug report and #66077 ...

neovim depends on lua51-lpeg, which depends on lua-luarocks, which depends on lua and lua53-luarocks, guaranteeing that lua and lua53 ports will be installed when trying to install neovim.

luajit is used for much of the build process, and the neovim Portfile contains configure.args-append -DLUA_PRG=${prefix}/bin/luajit. luajit is its own thing and adheres only to the lua-5.1 language spec. luajit works regardless of the state of other lua packages on my system.

But neovim also uses some C, and builds a library, nlua0, from src/nlua0.c in its source tree, which luajit uses later in the build process. The first line of nlua0.c is #include <lua.h>.

The lua port creates /opt/local/include/lua.h, which contains C headers for lua 5.3. Because this is the first lua.h file the compiler finds in the include directory, the C compiler uses this header and then builds nlua0 against lua 5.3.

This breaks the neovim build as described in this ticket and in #66077 because nlua0 pulls in symbols from lua 5.3 that luajit can't grok (because luajit is strictly 5.1).

The fix for this is to get the compiler to look in /opt/local/include/lua5.1 so that it uses the lua.h header from the lua51 port. See ​GitHub PR.

(Note that I'm not a CMake expert, and I'm not certain that target_include_directories(nlua0 AFTER PUBLIC /opt/local/include/lua5.1) is strictly correct; but it does the job.)

---

Now on to some of my other discoveries while investigating this issue ...

I don't think the build/install issues with luarocks are as simple as the commit referenced ​above suggests. There seem to be a few broken layers to unpack here (though to be clear this is unrelated to the neovim build).

At the top of the ​config file referenced in that git commit:

-- Run `luarocks` with no arguments to see the locations of
-- these files in your platform.

When I run luarocks on my system, it provides some interesting info at the end:

Configuration:
   Lua:
      Version    : 5.3
      Interpreter: /opt/local/bin/lua5.3 (ok)
      LUA_DIR    : /opt/local/libexec/lua53 (ok)
      LUA_BINDIR : /opt/local/bin (ok)
      LUA_INCDIR : /opt/local/libexec/lua53/include (ok)
      LUA_LIBDIR : /opt/local/libexec/lua53/lib (ok)

   Configuration files:
      System  : /opt/local/etc/luarocks/config-5.3.lua (ok)
      User    : /Users/wryfi/.luarocks/config-5.3.lua (not found)

   Rocks trees in use:
      /Users/wryfi/.luarocks ("user")
      /opt/local/share/luarocks ("system")

Whether I run luarocks, luarocks-5.1, luarocks-5.2, luarocks-5.3, or luarocks-5.4, they all return the same info. The versioned copies are all symlinks to the same /opt/local/bin/luarocks script, so this comes as no shock.

That luarocks script also has lua 5.3 paths hard-coded into it:

package.path=[[/opt/local/share/lua/5.3/?.lua;]] .. package.path
local list = package.searchers or package.loaders; table.insert(list, 1, function(name) if name:match("^luarocks%.") then return loadfile([[/opt/local/share/lua/5.3/]] .. name:gsub([[%.]], [[/]]) .. [[.lua]]) end end)

It also has its shebang set to /opt/local/bin/lua, which is installed by the lua port, and seems to be (currently) version 5.3:

$ /opt/local/bin/lua
Lua 5.3.6  Copyright (C) 1994-2020 Lua.org, PUC-Rio
>

I find it interesting that luarocks reports Interpreter: /opt/local/bin/lua5.3 (ok) even though the shebang is /opt/local/bin/lua, especially since lua is not a symlink to lua5.3 but its own separate binary. (I also have discrete lua5.1 and lua5.3 binaries.) Not sure how this works.

The lua and lua5.3 binaries have different checksums. They are binaries, so I can't tell for sure if anything is different about them, but they both start a lua-5.3 interpreter, and running strings against them seems to return identical output.

I don't see a way to manage the version of /opt/local/bin/lua, other than mv lua5.1 lua. It seems like an antipattern that the version of the lua binary is arbitrarily set for me, with no input or obvious way to change it.

---

So in summary:

1. The lua port decides for you what version of lua (currently 5.3) will be the default lua on your system.
   1. Note that all C libraries linking against lua.h will build against that version without extra care.
2. There is no facility to manage which version of lua is used as the system default.
3. Any port that depends on lua installs and makes the lua port's version (currently 5.3) the default, even if the dependent port is incompatible with that version.
4. The lua port installs a 5.3 binary in /opt/local/bin/lua that appears to be similar, but not identical, to the /opt/local/bin/lua5.3 binary installed by the lua53 port.
5. Uninstalling the lua port will break luarocks and potentially other lua ports, even though they could use a different version of lua.
6. neovim depends on lua51-lpeg, which depends on lua-luarocks, which depends on lua and lua53-luarocks, guaranteeing that lua53 will be installed when trying to install neovim, even though neither lua51-lpeg nor neovim wants anything to do with lua 5.3.
7. Building neovim is breaking because it links against the lua.h provided by the lua port (currently 5.3), which is incompatible with neovim and luajit.
8. luarocks always runs against the version of lua installed by the lua port, because its shebang is /opt/local/bin/lua, which is (currently) lua 5.3 by design of the lua port.
   1. This remains true even if running luarocks-5.n because the versioned copies are all symlinks to the same luarocks script.
   2. The luarocks-5.n symlinks are therefore not likely useful and/or probably broken for their intended purpose.
9. luarocks always manages packages in /opt/local/share/lua/5.3 because this value is hard-coded in the script.
10. MacPorts probably needs to add a selection mechanism for setting and using the desired version of lua on the system.

[rbeyer (Ross Beyer)]

I recently had to re-install my system and macports on my macOS 14 / Apple Silicon system and ran into this exact problem with installing neovim (whereas earlier this spring on initial installation, it worked just fine). I was excited to see wryfi's explanation of performing a port uninstall of the lua53 ports that got neovim working, but that did not happen for me. My experience was identical, with the symbol not found in flat namespace '_luaL_prepbuffsize' error in the log. I did the same:

  $> sudo port reclaim
  $> sudo port clean --all (of the two remaining lua ports: luametatex and texlive-luatex
  $> sudo port install neovim (which errors, just as reported in the OP)
  $> sudo port uninstall lua53-luarocks (with the same warning as reported above)
  $> sudo port uninstall lua53
  $> sudo port install neovim

And I was expecting that last port install neovim to work as wryfi's experience did, but it did not, and these namespace errors are still in the new log file:

:info:build /opt/macports/bin/luajit: error loading module 'nlua0' from file '/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so':
:info:build     dlopen(/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_editors_neovim/neovim/work/build/lib/libnlua0.so, 0x0006): symbol not found in flat namespace '_luaL_prepbuffsize'
:info:build stack traceback:
:info:build     [C]: at 0x0100664fb0
:info:build     [C]: in function 'require'
:info:build     ...eovim/work/neovim-0.10.0/src/nvim/generators/preload.lua:9: in main chunk
:info:build     [C]: at 0x0100611bb4

I'm not sure what wryfi did that I didn't.

[wryfi]

Replying to rbeyer:

I'm not sure what wryfi did that I didn't.

port uninstall lua

It's actually the lua port (no version in the port name) that is the issue, as it installs a lua.h header that neovim uses to build nlua0 against lua 5.3.

(Updated comment#7 above to add lua to the list of things I uninstalled.)

[wryfi]

And actually, port deactivate lua && port install neovim && port activate lua would probably be the better sequence.

[wryfi]

Reading #46621 provided some interesting background on the lua packaging, and even mentions the lua.h issue that spawned this ticket (comment 12). That conversation seems to have stalled out without a complete resolution or follow up.

I would personally favor eliminating the lua port and creating a lua_select port to manage the default version selection as is done with python. It looks like someone started creating a luarocks_select port to manage luarocks versioning but it appears to be incomplete.

It seems like a more discoverable place to document best practices and roadmaps for specific ports and languages would be useful. Or maybe I just don't know where to look?

[rbeyer (Ross Beyer)]

Perfect, port deactivate lua && port install neovim && port activate lua worked. Thanks!

[Chris Haumesser <5400416-wryfi@…>]

In 391c7804e8d2a5f359d9108d95bcdc304d1b9d62/macports-ports (master):

neovim: include lua51 headers to build properly

fixes #70420
fixes #66077