ncurses@6.1 : /opt/local/include/unctrl.h:60:63: error: unknown type name 'SCREEN'

[kencu (Ken)]

our ncurses port is 6.1, and appears to include some new functionality with respect to screen pointers that can cause build errors with software not expecting this.

Most MacOS systems are using ncurses < 6.0, and this can generate errors like this, when building software that calls in the SDK's headers, eg #53155:

In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/include/ncurses.h:141:
/opt/local/include/unctrl.h:60:63: error: unknown type name 'SCREEN'
NCURSES_EXPORT(NCURSES_CONST char *) NCURSES_SP_NAME(unctrl) (SCREEN*, chtype);
                                                              ^
/opt/local/include/unctrl.h:60:53: error: function cannot return function type 'char *(int *, chtype)' (aka 'char *(int *, unsigned int)')
NCURSES_EXPORT(NCURSES_CONST char *) NCURSES_SP_NAME(unctrl) (SCREEN*, chtype);
                                                    ^
/opt/local/include/unctrl.h:60:54: error: a parameter list without types is only allowed in a function definition
NCURSES_EXPORT(NCURSES_CONST char *) NCURSES_SP_NAME(unctrl) (SCREEN*, chtype);
                                                     ^
3 errors generated.

I recently ran into this trying to enable the LLVM test suite for libtapi on Darwin, which errors out with that error.

There are many reports of this issue on Google, and they all say to "uninstall MacPorts" or "get rid of that" etc. Google this :

"error: unknown type name 'SCREEN'"

for a number of examples.

It looks like this new SCREEN functionality is optional

Extensions:
  --disable-ext-funcs     disable function-extensions
  --enable-sp-funcs       enable SCREEN-extensions

however, when I tried this in ncurses:

configure.args-append --disable-sp-funcs

it failed due to :

../ncurses/./base/new_pair.c:280:25: error: use of undeclared identifier 'sp'
/usr/bin/clang -DHAVE_CONFIG_H -I../ncurses -I. -I../include -D_DARWIN_C_SOURCE -DNDEBUG -pipe -Os -arch x86_64 -arch i386 -no-cpp-precomp --param max-inline-insns-single=1200 -dynamic -c ../ncurses/./base/resizeterm.c -o ../obj_s/resizeterm.o
                if (_nc_reserve_pairs(sp, pair) == 0) {
                                      ^
1 error generated.
make[1]: *** [../obj_s/new_pair.o] Error 1
make[1]: *** Waiting for unfinished jobs....

On a case-by-case basis, you can sometimes block the troublesome MacPorts header like this:

configure.cppflags-append -DNCURSES_UNCTRL_H_incl

but of course that is less than ideal.

I'm not sure if we are finding the new SCREEN functionality useful, but perhaps if not, disabling it for now until all the various software catches up might be useful.

[jmroot (Joshua Root)]

Mixing the headers of two different versions of anything can't be expected to work. It looks like HandBrake should use the ncurses port instead of the system ncurses.

[kencu (Ken)]

This happens in multiple builds, not just handbrake.

I don't fully understand why this is happening, but this post <​https://stackoverflow.com/questions/44485899/xcode-unknown-type-name-screen-only-when-building-for-release-in-file-unct> claims to have sorted it out.

[jmroot (Joshua Root)]

It's managing to include /usr/include/ncurses.h despite -I/opt/local/include being used. The important part seems to be a line you didn't quote,

In file included from <module-includes>:337:

That seems to be an alternate way of finding headers used by Xcode. Again, using the MacPorts ncurses as per our general policy would solve the problem.

[kencu (Ken)]

You seem to have it understood. Do you have some insight into what is the fix?

MacPorts ncurses is installed. CMake is configured as usual. I"m sorry I don't see what to do.

[kencu (Ken)]

(I just edited our MacPorts' unctrl.h locally to disable the SCREEN usage, and that works just fine until we sort this out properly).

[jmroot (Joshua Root)]

Replying to kencu:

MacPorts ncurses is installed. CMake is configured as usual. I"m sorry I don't see what to do.

You're talking about libtapi? I haven't ever seen a log for that, so I don't know what it's even trying to do.

[kencu (Ken)]

it seems a systemic issue, for which HandBrake and the libtapi tests are examples.

The issue may well be with modules -- <​https://clang.llvm.org/docs/Modules.html> -- although how to fix it is not at present clear to me.

For now, disabling SCREEN manually in unctrl.h works as a workaround for anyone bitten by this, I guess. I don't know any other fix, other than building our ncurses port with SCREEN disabled.

[jmroot (Joshua Root)]

The fix is to include /opt/local/include/ncurses.h not /usr/include/ncurses.h. I don't know how to say that any more plainly.

[kencu (Ken)]

it is being included by the module map, over which we have no control. What you suggest is not possible, AFAICT.

[jmroot (Joshua Root)]

If you're having trouble understanding the issue, I'll try to explain it in more generic terms. Suppose you have a pair of headers foo.h and bar.h, and they exist in both /usr/include and /usr/local/include. The pair in each location is designed to work together, but using one header from one location and the other header from the other location won't work for whatever reason (the possibilities are endless really).

Now further suppose each foo.h contains a line #include <bar.h>. Now consider a C program baz.c that starts like this:

#include <foo.h>
...

When compiled, the preprocessor will looks for foo.h in its search paths. /usr/local/include comes first, so it will use /usr/local/include/foo.h. While processing that file, it will see #include <bar.h>, and go through the same process to find that file, again using the one in /usr/local/include. Everything works fine.

But consider if you change baz.c to this:

#include </usr/include/foo.h>
...

Now the preprocessor will load /usr/include/foo.h because it is specified with an absolute path. While processing foo.h, it will again see #include <bar.h>, and find bar.h in /usr/local/include because that location is first in the list. Oops, the headers aren't compatible and the program breaks.

[kencu (Ken)]

homebrew does not seem to have this issue <​https://github.com/Homebrew/homebrew-core/blob/master/Formula/ncurses.rb>, and appears to make no special compensations for it.

I could guess that is because it installs ncurses.h into /usr/local/include and so it is found automatically by the module map there ahead of /usr/include, but I'm not certain about that.

[jmroot (Joshua Root)]

Replying to kencu:

it is being included by the module map, over which we have no control. What you suggest is not possible, AFAICT.

If the module map doesn't understand that /usr/include/ncurses.h needs to include /usr/include/unctrl.h then it's pretty broken.

How do you know Homebrew doesn't have the issue? Have they built the libtapi test suite?

[kencu (Ken)]

Replying to jmroot:

If the module map doesn't understand that /usr/include/ncurses.h needs to include /usr/include/unctrl.h then it's pretty broken.

Yes, I guess so. I wish I knew what Jeremy knows -- where does this module map come from? Can we regenerate it? Disable it? Edit it to point to our ncurses.h?

I think we can set ${prefix}/include to be a system include path ahead of /usr/include to fix this, maybe... PITA, though... each build system does this differently...

How do you know Homebrew doesn't have the issue? Have they built the libtapi test suite?

Just a guess. The general solution for this error on the internet / SourceForge / etc is to uninstall MacPorts and use homebrew instead, so that's one clue. Also googlng this error shows only MacPorts errors with it. Homebrew does have a functioning HandBrake formulae -- but things have changed with the HandBrake build so can't compare, really.

[kencu (Ken)]

I continue to be completely unable to get past this error without hand-editing unctrl.h to disable the SCREEN usage that triggers the error.

No combination of trickery seems to fix it.

Module information is here <​https://clang.llvm.org/docs/Modules.html#using-modules>. I presume the module map that is causing the issue is right in the CoreFoundation Framework, but my knowledge of where this is going wrong is apparently incomplete as I can't (so far) fix it.

[kencu (Ken)]

This same issue now prevents building a current clang-devel, due to the same error with modules once again.

clang refuses to configure without module support, which cannot be made to work (so far, by me at least) with this version of ncurses enabling SCREEN.

This is with the current llvm / clang checkout from GitHub, using Apple's build scripts. So ... I'm open to ideas.

[kencu (Ken)]

OK -- so is there any fix or workaround to our ncurses port that would be considered acceptable?

I don't think it's realistic to ask Apple to fix their modules implementation, whatever is wrong with it that causes it to pull in our unctrl.h header instead of the one in the SDK. Leaving this broken can't really be a great plan.

in uncntrl.h there is

#include <curses.h>

#undef unctrl
NCURSES_EXPORT(NCURSES_CONST char *) unctrl (chtype);

#if 1
NCURSES_EXPORT(NCURSES_CONST char *) NCURSES_SP_NAME(unctrl) (SCREEN*, chtype);
#endif

I could patch in an extra guard in there to exclude Apple's curses 5.x curses.h from pulling in the SCREEN definition, like this:

#include <curses.h>

#undef unctrl
NCURSES_EXPORT(NCURSES_CONST char *) unctrl (chtype);

#if NCURSES_VERSION_MAJOR > 5
#if 1
NCURSES_EXPORT(NCURSES_CONST char *) NCURSES_SP_NAME(unctrl) (SCREEN*, chtype);
#endif
#endif

And then I think we'd change nothing in our port, but we'd no longer be broken with Apple's modules. I have confirmed on my systems here that fixes the problem with modules by blocking our header from defining SCREEN.

Otherwise I guess we get Jeremy or some similar Apple engineer in here to tell us how to make MacPorts' ncurses port play nice with Apple's clang modules implementation.

As mentioned above, the general fix on the internet / stack exchange, etc is to just "uninstall MacPorts".

[kencu (Ken)]

Here is the clang documentation on module support and search paths <​https://clang.llvm.org/docs/Modules.html>

[jmroot (Joshua Root)]

There is no bug in ncurses here. I think you need to work with the LLVM developers to find a solution. This can't possibly be working as intended on the LLVM side.

[kencu (Ken)]

I guess we'll copy Jeremy in on it and let him resolve it. Modules support will have to be disabled on clang until MacPorts / llvm come to an agreement about it.

Modules are part of the c++20 standard, but not much software is calling for it yet (other than Apple's software).

Let me know if you change your mind about ncurses -- it's a 10 second, no-harm-no-foul workaround. Not a pure fix, but a workaround.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Most of my research on this issue has focused on ​cargo since that ticket was assigned to me.
I hope the ideas are generally applicable.

Minimal reproducible error is

echo "#import <Foundation/Foundation.h>" | env CPATH=/opt/local/include clang -xobjective-c -c -fmodules -v -o test.o -

The odd part is that this is not an error

echo "#import <Foundation/Foundation.h>" | clang -xobjective-c -c -fmodules -v -o test.o -
echo "#import <Foundation/Foundation.h>" | env CPATH=/opt/local/include clang -xobjective-c -c -fmodules -v -o test.o -

The reason seems to be the flag -fmodules-cache-path=....

echo "#import <Foundation/Foundation.h>" | clang -xobjective-c -c -fmodules -v -o test.o -

generates a bunch of precompiled header files.
Once generated,

echo "#import <Foundation/Foundation.h>" | env CPATH=/opt/local/include clang -xobjective-c -c -fmodules -v -o test.o -

uses them with no issues.

If the cache files are deleted,

echo "#import <Foundation/Foundation.h>" | env CPATH=/opt/local/include clang -xobjective-c -c -fmodules -v -o test.o -

goes back to generating an error.

It seems that the problem is fundamentally that CPATH=${prefix}/include prevents the proper generation of precompiled header files.

The new port in #60150 actually seems to build with the simple addition of

compiler.cpath

Removing -fmodules also prevents the error

echo "#import <Foundation/Foundation.h>" | env CPATH=/opt/local/include clang -xobjective-c -c -v -o test.o -

I have not yet experimented with C++20 modules to see if similar issues arise.

[kencu (Ken)]

FYI I believe this is being looked into upstream at Apple, but I don't know of a radar, etc, for it, if there is one

[kencu (Ken)]

essentially, when clang is using modules, but a CPATH is set, instead of honouring the CPATH it first looks at the cached module map which sends it into the SDK, but then does honour the CPATH for further includes.

This situation is non-viable. Disabling modules of course works. disabling CPATH disables MacPorts, basically.

Some clang trickery to regenerate the module map automatically if CPATH is set would appear to be needed.

This can't be the only time this is noticed, but MacPorts is likely the first place this would show up, running a current clang with non-standard header include paths and setting CPATH.

[kencu (Ken)]

<​http://lists.llvm.org/pipermail/cfe-dev/2020-March/065021.html>

let's see what they say...

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Please forgive my ignorance, but where has this issue cropped up?
Just reading through this ticket:

• ​topgrade
• HandBrake? (other build issues prevent me from testing).
• libtapi? (builds fine for me)
• others?

[kencu (Ken)]

Anything that uses Xcode clang and modules is currently not functional in MacPorts by ncurses. Modules are a newer feature, so penetrance is evolving.

I manually delete the libtapi tests that use modules, so you wouldn't see it. Proper clang-devel builds now want modules, and fail in MacPorts because of ncurses.

google "error: unknown type name 'SCREEN'" for hits on other projects.

I have tweaked my MacPorts unctrl.h to work around this, as at present that is the only solution I know of that generally works --- it's not a pure fix, I know .

But neither is unsetting CPATH and the library path, or disabling module support, or the parts of projects that require modules. We'll have to go back and undo all those when clang is fixed too, and IMHO those are more impactful than tweaking unctrl.h until Apple figures this out.

Or perhaps we can fix the clang issue -- if we could do that, all this would go away in a proper fashion.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Just to refine my analysis:

• Minimal reproducible error
  • echo "#import <sys/types.h>" | env CPATH=/opt/local/include clang -xobjective-c -c -fmodules -v -o test.o -
  • echo "import Darwin" | env CPATH=/opt/local/include swiftc -v -o test.o -
• The problem happens only during the generation of the module cache.
  • Once the cache files are generated (see -fmodules-cache-path=...), subsequent compilations work fine.
  • The specific cache directory seems to depends on quite a few variables.
    • MACOSX_DEPLOYMENT_TARGET
    • DEVELOPER_DIR
    • SDKROOT
    • -O
• The problem happens with the reading of /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/module.modulemap.
  • module Darwin has a submodule explicit module ncurses, which has a submodule explicit module unctrl.
  • ncurses has a circular include dependency.
    • ncurses.h includes unctrl.h
    • unctrl.h includes ncurses.h
    • Newer versions of ncurses have changed exactly how these two files interact.
  • module.modulemap does not seem to have a mechanism to handle circular dependencies.

Here are some possible solutions/workarounds:

1. Patch Clang
   • If the circular dependencies are a fundamental issue. there is nothing to be done.
   • Any changes would take quite a while (if ever) to reach Xcode clang.
   • Almost certainly, Clang upstream help would be required.
2. Patch uncntrl.h to remove SCREEN.
   • We would be removing expected functionality to facilitate the generation of cache files.
   • We would forever be responsible for making sure uncntrl.h continued to work.
3. Patch uncntrl.h to minimize the effect of the circular dependency.
   • It is not clear yet how this could be accomplished.
   • We should have to work with ncurses upstream to get any changes merged.
4. Generate required cache files before building the port (e.g. during post-extract).
   • I was able to accomplish this with the port topgrade.
   • A PortGroup could accomplish this with some effort.
   • It is very fragile since the cache directory name is highly dependent on the compiler settings.
5. Attempt to get ncurses to work within the modules infrastructure.
   • There is a ​pull request to that effect.
   • I am not 100% sure why this works or how fragile it is.

[kencu (Ken)]

FYI, this is what I did in our unctrl.h

#include <curses.h>

#undef unctrl
NCURSES_EXPORT(NCURSES_CONST char *) unctrl (chtype);

+ #if NCURSES_VERSION_MAJOR > 5
#if 1
NCURSES_EXPORT(NCURSES_CONST char *) NCURSES_SP_NAME(unctrl) (SCREEN*, chtype);
#endif
+ #endif

[ryandesign (Ryan Carsten Schmidt)]

Replying to kencu:

<​http://lists.llvm.org/pipermail/cfe-dev/2020-March/065021.html>

let's see what they say...

I see no response.

[ryandesign (Ryan Carsten Schmidt)]

Replying to kencu:

disabling CPATH disables MacPorts, basically.

I've always considered CPATH and LIBRARY_PATH to be extra sugar to help build systems that weren't doing things quite right. MacPorts existed for a long time before it started setting CPATH and LIBRARY_PATH and worked just fine by using -I and -L flags.

[kencu (Ken)]

In the case of the specific port Marcus was working on, is was a cargo build, there were no FLAGS being used at all. It was being driven by CPATH and LIBRARY_PATH <​https://github.com/macports/macports-ports/pull/6484/commits/0b67f8df23624aa279846dc8279817d8f084091e> and that's why I said that disabling CPATH disabled MacPorts, basically, for that specific instance.

Which is why I considered it to be not a fix for this issue, although it did work around the broken build in that one specific port.

In general, for other kinds of builds, autotools, cmake, etc, then yes, the passing of include flags makes CPATH redundant, and likewise for -L and LIBRARY_PATH and what you say would be accurate for those ports.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

In d2c8da96efe60cc3a72c03c92aa2d4af8eb280d4/macports-ports (master):

ncurses: add support for modules infrastructure

See #59992

[jmroot (Joshua Root)]

This is working its way through the proper channels at Apple (DTS incident). Currently no workaround to offer apart from "don't use -fmodules" (which isn't actually that unreasonable, since modules are just a build speed optimisation after all.)

[kencu (Ken)]

Anyone encountering this error can use the new port macports-module-map into which the modules fix has been moved:

[93c93a6b4739d9754fa13b9df3a9677ee91dcbd6/macports-ports]

even if Apple figures out how to fix the SDK to not barf on this issue, we will presumably have to carry this port and it's fix along pretty much forever, as all the SDKs up to the point where it is perhaps fixed someday won't have the fix in it.

[jmroot (Joshua Root)]

DTS has suggested to try using the __building_module builtin in our headers. For example, our unctrl.h would look like:

#if __building_module(Darwin)
#include_next <unctrl.h>
#else
// normal content of file from ncurses 6
#endif

It would be a little more code in practice, e.g. we'd have to first check that the compiler has that builtin.

[saagarjha (Saagar Jha)]

It seems like you're working on a fix, but I do want to mention that "don't use -fmodules" is not going to work for any of our Swift ports.

[p-keller]

For info: the #if __building_module(Darwin) #include_next patch floated my boat. I didn't bother with any checks for availability of the builtin, since my problem was with a standard macOS build in Xcode.

[kencu (Ken)]

DTS's suggestion is not much of a fix, and also not very broadly compatible.

Marcus' module.map fix is a lot more elegant than what DTS suggested, not that I exactly understand why/how Marcus' fix actually works :>

Neither of these should be needed, but modules remain very new, and I don't think anyone really gets them yet, so "all hacks on board".

[jmroot (Joshua Root)]

In 703e76e8f90d7fb0837b315cf414b4175b61fbbb/macports-ports (master):

ncurses: add workaround for Apple modules bug

See: #59992

[jmroot (Joshua Root)]

I'm inclined to close this as "as fixed as it's going to get on our end", unless anyone knows of anything else that we can/should do?

[kencu (Ken)]

SGTM.

[jmroot (Joshua Root)]

In 55780aa0ef008b7f6845d01d5ea8d92c3d398061/macports-ports (master):

Remove macports-module-map dependencies

See: #59992

[jmroot (Joshua Root)]

Still broken on Apple's end, so please send them feedback.