notmuch @0.31 +emacs: variable rsti_dir is empty

[lafpark]

When building notmuch 0.31 +emacs the following error is reported

:info:build Configuration error:
8509 :info:build There is a programmable error in your configuration file:
8510 :info:build Traceback (most recent call last):
8511 :info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sphinx/config.py", line 319, in eval_config_file
8512 :info:build     execfile_(filename, namespace)
8513 :info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sphinx/util/pycompat.py", line 89, in execfile_
8514 :info:build     exec(code, _globals)
8515 :info:build   File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_mail_notmuch/notmuch/work/notmuch-0.31/doc/conf.py", line 49, in <modu
8516 :info:build     lines.extend(open(rsti_dir+'/'+file))
8517 :info:build FileNotFoundError: [Errno 2] No such file or directory: '/notmuch.rsti'

The error occurs in the file /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_mail_notmuch/notmuch/work/notmuch-0.31/doc/conf.py at line 49.

 44 if tags.has('WITH_EMACS'):
 45     # Hacky reimplementation of include to workaround limitations of
 46     # sphinx-doc
 47     lines = ['.. include:: /../emacs/rstdoc.rsti\n\n'] # in the source tree
 48     for file in ('notmuch.rsti', 'notmuch-lib.rsti', 'notmuch-show.rsti', 'notmuch-tag.rsti'):
 49         lines.extend(open(rsti_dir+'/'+file))
 50     rst_epilog = ''.join(lines)
 51     del lines
 52 else:

The variable rsti_dir at build time contains an empty string. The variable contents is read from the file

/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_mail_notmuch/notmuch/work/notmuch-0.31/sphinx.config

which contains:

  1 # Generated by configure, run from doc/conf.py
  2 tags.add('WITH_EMACS')
  3 tags.add('WITH_PYTHON')
  4 rsti_dir = ''

showing the problem (the empty string). The above file is generated using

/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_mail_notmuch/notmuch/work/notmuch-0.31/configure

which contains the line

1539     printf "rsti_dir = '%s'\n" $(realpath emacs)

showing that $(realpath emacs) is returning the empty string.

[lafpark]

Changing line 4 of /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_mail_notmuch/notmuch/work/notmuch-0.31/sphinx.config to

4 rsti_dir = '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_mail_notmuch/notmuch/work/notmuch-0.31/emacs'

allowed the build to complete without error.

[rseichter (Ralph Seichter)]

Thank you for the detailed error description, lafpark. However, I cannot (yet) reproduce the issue here. How exactly have you launched the build?

[lafpark]

The build was done through macports using sudo port install notmuch +emacs

[lafpark]

Also, I had previously installed

  notmuch @0.29.3_0+emacs
  notmuch @0.30_0+emacs

without issue (using macports). I was not able to install version 0.31+emacs since its release, until I modified the sphinx.config file shown above.

[rseichter (Ralph Seichter)]

Still no luck reproducing the issue here. I tried a fresh installation on both High Sierra and on Catalina, and both builds succeeded.

[lafpark]

I just uninstalled and installed to check that I still had the issue, but it still appears.

➜  ~ sudo port uninstall notmuch
Password:
The following versions of notmuch are currently installed:
 1) notmuch @0.29.3_0+emacs
 2) notmuch @0.30_0+emacs
 3) notmuch @0.31_0
 4) notmuch @0.31_0+emacs (active)
Enter option(s) [1-4/all]: 4
Note: It is not recommended to uninstall/deactivate a port that has dependents as it breaks the dependents.
The following ports will break:
 muchsync @5_0
 neomutt @20200501_0
 neomutt @20200807_0
 neomutt @20200814_0
Continue? [y/N]: y
Warning: Uninstall forced.  Proceeding despite dependencies.
--->  Deactivating notmuch @0.31_0+emacs
--->  Cleaning notmuch
--->  Uninstalling notmuch @0.31_0+emacs
➜  ~ sudo port install notmuch +emacs
--->  Computing dependencies for notmuch
--->  Fetching archive for notmuch
--->  Attempting to fetch notmuch-0.31_0+emacs.darwin_17.x86_64.tbz2 from https://packages.macports.org/notmuch
--->  Attempting to fetch notmuch-0.31_0+emacs.darwin_17.x86_64.tbz2 from https://kmq.jp.packages.macports.org/notmuch
--->  Attempting to fetch notmuch-0.31_0+emacs.darwin_17.x86_64.tbz2 from https://ywg.ca.packages.macports.org/mirror/macports/packages/notmuch/notmuch
--->  Fetching distfiles for notmuch
--->  Verifying checksums for notmuch
--->  Extracting notmuch
--->  Configuring notmuch
--->  Building notmuch
Error: Failed to build notmuch: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_mail_notmuch/notmuch/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets to report a bug.
Error: Processing of port notmuch failed

I then proceeded to make the change a documented above and the package installed.

➜  ~ sudo vi /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_mail_notmuch/notmuch/work/notmuch-0.31/sphinx.config
➜  ~ sudo port install notmuch +emacs
--->  Computing dependencies for notmuch
--->  Fetching archive for notmuch
--->  Attempting to fetch notmuch-0.31_0+emacs.darwin_17.x86_64.tbz2 from https://packages.macports.org/notmuch
--->  Attempting to fetch notmuch-0.31_0+emacs.darwin_17.x86_64.tbz2 from https://kmq.jp.packages.macports.org/notmuch
--->  Attempting to fetch notmuch-0.31_0+emacs.darwin_17.x86_64.tbz2 from https://ywg.ca.packages.macports.org/mirror/macports/packages/notmuch/notmuch
--->  Building notmuch
--->  Staging notmuch into destroot
--->  Installing notmuch @0.31_0+emacs
--->  Activating notmuch @0.31_0+emacs
--->  Cleaning notmuch
--->  Scanning binaries for linking errors
--->  No broken files found.
--->  No broken ports found.

Is there anything I can do that will help?

[rseichter (Ralph Seichter)]

What still surprises me is that, as you wrote, the realpath command returns an empty string on your machine. This is what happens here when the call fails:

→ which realpath
/opt/local/bin/realpath

→ realpath does_not_exist
realpath: /Users/ralph/does_not_exist: No such file or directory

I don't know what could cause realpath to return an empty string.

[lafpark]

My realpath behaves differently, if that helps.

➜  ~ realpath does_not_exist
/Users/lapark/does_not_exist

It provided the path to a nonexistent file, whereas when you ran it, it provided an error message.

[lafpark]

I need to provide -e to verify existence.

➜  ~ realpath -e does_not_exist
realpath: does_not_exist: No such file or directory

[rseichter (Ralph Seichter)]

If your version of realpath is broken in some way, there's nothing I can do about it. It should never return an empty string, but always a path (or an error). The notmuch port works on my machines and on the MacPorts build infrastructure, so I have little choice but to close this issue as non-reproducible.

[lafpark]

When I run realpath it does return a string. Is there a chance that the config file is not running realpath, but not showing the error?

[rseichter (Ralph Seichter)]

Only you can answer that, by looking at the full build log written by 'port install'. In any case, this looks like an upstream problem to me. I don't know how rsti_dir is passed into conf.py and what could go wrong in the process; that's for the notmuch authors to answer.

[lafpark]

Ok, thanks.

[ryandesign (Ryan Carsten Schmidt)]

Replying to lafpark:

My realpath behaves differently, if that helps.

Where is your realpath?

which realpath

What version is it?

realpath -v
port provides $(which realpath)

(I'm aware that MacPorts realpath 1.0.1 claims to be 1.0.0; see #61330.)

Replying to rseichter:

If your version of realpath is broken in some way, there's nothing I can do about it. It should never return an empty string, but always a path (or an error). The notmuch port works on my machines and on the MacPorts build infrastructure, so I have little choice but to close this issue as non-reproducible.

On MacPorts build infrastructure, ​you'll see that the configure script printed this warning:

./configure: line 1539: realpath: command not found

Since it seems evident that notmuch needs realpath at configure time, the port should add port:realpath to its depends_build. This will ensure* that MacPorts realpath, which behaves the way you expect, will be used, rather than whatever other realpath the user somehow has.

(* except if the user removes /opt/local/bin from binpath in macports.conf)

[ryandesign (Ryan Carsten Schmidt)]

Here's a PR to add the realpath build dep, among other fixes: ​https://github.com/macports/macports-ports/pull/8821

[rseichter (Ralph Seichter)]

lafpark indeed has not yet answered what realpath binary he is using and where it is located. Without the full build logs, one cannot tell if it is a standalone binary that is included in PATH, or perhaps a builtin function (FISH includes one, for example).

As for the pull request, the C++ stdlib and Python version fixes are helpful, of course. I am not quite sure why you called the newly introduced variable "python_branch", though? It Python lingua, 3.8 is a version, not a branch.

[ryandesign (Ryan Carsten Schmidt)]

To match the variable names used by the python 1.0 portgroup.

[rseichter (Ralph Seichter)]

Still seems odd to me, but that's just a detail. I have since asked upstream to document the dependency on "realpath", although I'd prefer if they found a different method altogether, and I am not sure if they really require an absolute path.

[ryandesign (Ryan Carsten Schmidt)]

The naming conventions used by the python portgroup seem odd to me as well, but that portgroup has been used by thousands of ports for over 9 years already so I decided to reuse its odd yet familiar naming conventions here rather than invent a new one.

[lafpark]

➜  ~ which realpath
/opt/local/libexec/gnubin//realpath
➜  ~ realpath -v
realpath: invalid option -- 'v'
Try 'realpath --help' for more information.
➜  ~ realpath --version
realpath (GNU coreutils) 8.32
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Padraig Brady.
➜  ~ port provides $(which realpath)
/opt/local/libexec/gnubin/realpath is provided by: coreutils

[lafpark]

Yes, the configure script does not find realpath. This is a portion of the log file when installing notmuch fails. Line 2853 reports that realpath is not found.

2850 :info:configure   -Wall -Wextra -Wwrite-strings
2851 :info:configure Checking for available C compiler warning flags...
2852 :info:configure   -Wall -Wextra -Wwrite-strings -Wmissing-declarations
2853 :info:configure ./configure: line 1539: realpath: command not found
2854 :info:configure All required packages were found. You may now run the following
2855 :info:configure commands to compile and install notmuch:
2856 :info:configure   make
2857 :info:configure   sudo make install

[lafpark]

Also, my realpath is a symlink to grealpath

➜  ~ ls -l /opt/local/libexec/gnubin//realpath
lrwxr-xr-x 1 root admin 24 Mar 20  2020 /opt/local/libexec/gnubin//realpath -> /opt/local/bin/grealpath

[lafpark]

Installing the realpath port has fixed the problem. So it turns out that the notmuch port requires the realpath port to be installed. Thanks for your help!

[ryandesign (Ryan Carsten Schmidt)]

In 40bc2310eb88b0c0d099af4ecbfc119d242249b3/macports-ports (master):

notmuch: Add patch to avoid the need for realpath

Closes: #61310