Inconsistent behavior between ncurses 6.1.20180127 and ncurses 5.7

[LiamFry]

While working on an ncurses app, I discovered inconsistent behavior between ncurses 5.7 (bundled with macOS) and ncurses 6.1 (installed via MacPorts). I do not know what is causing the issue but I have isolated the relevant code. I have reproduced the issue on several machines running Mavericks, El Capitan, and High Sierra. The code to exercise this problem is very short and straightforward. I've attached a copy to the ticket.

The intent of the code is to "blink the screen" similar to flash() without using flash(). When compiled against the macOS-bundled ncurses library (5.7), behavior in Terminal, iTerm, and tmux is as expected. When compiled against MacPorts ncurses library (6.1), behavior in Terminal and tmux is as expected. However, when run in iTerm, only a column of characters appears on the left side of the screen.

I initially thought this was an iTerm issue as it only happens when using iTerm. However, it only happens when using ncurses 6.1 from MacPorts.

[jmroot (Joshua Root)]

I don't think the terminfo database shipped with macOS has a definition for iTerm. What version of iTerm are you using? What is TERM set to?

[LiamFry]

The default TERM for iTerm - which I am using - is xterm-256color

[jmroot (Joshua Root)]

Does the behaviour change if you set TERM=iterm2 or TERM=iterm as appropriate?

[LiamFry]

Yes; using TERM=iterm2 or TERM=iterm "works" but it breaks every other curses-based application I have installed. Vim, htop, top, nano, cmus, ... the list goes on. "Error opening terminal: iterm2." Using those TERM settings even breaks less and man: "WARNING: terminal is not fully functional"

[jmroot (Joshua Root)]

OK, so that confirms that the problem is that iTerm does not in fact have the same capabilities as xterm. See this FAQ (the questions immediately before and after are also relevant): ​https://invisible-island.net/ncurses/ncurses.faq.html#xterm_generic

The xterm definition in ncurses 6.1's terminfo has been updated to more accurately reflect the capabilities of xterm. You need to use a TERM value that accurately describes the capabilities of the terminal you are using.

Getting programs that use the system ncurses to use the new iterm2 terminfo entry takes more than just setting TERM, but it can be done. Try setting TERMINFO_DIRS=/opt/local/share/terminfo:. That should make the system ncurses find the MacPorts terminfo database, but fall back to its own.

[jmroot (Joshua Root)]

If the above doesn't work, you could also try installing the iTerm definition in /usr/share/terminfo, or simply using some other TERM value that exists in both versions of the terminfo database and specifies a subset of iTerm's actual capabilities.

I'm closing this ticket now in any case, as it appears that ncurses is just doing what it's told to do.

[LiamFry]

Thank you for the great info! I found the ncurses FAQ very eye-opening. (What a bloody mess we've made!) Unfortunately, the suggestion you offered didn't help, RE: setting TERMINFO_DIRS. This change resolved some issues and introduced others. It's proven impossible to find an optimal combination of TERM and TERMINFO_DIRS that results in consistent behavior for TUI apps vis-à-vis MacPorts-installed ncurses 6 and macOS-bundled ncurses 5, at least as it relates to iTerm2. I'll look elsewhere - and deeper - for an iTerm-specific solution and, failing that, drop iTerm for something else. Thanks again for the help!

[LiamFry]

After more testing, I re-opened the ticket. I downloaded and built the ncurses test suite (​https://invisible-island.net/ncurses/ncurses-examples.html) I built it against the macOS-bundled ncurses and the MacPorts-installed ncurses. I also built it on a Debian GNU/Linux system. I then ran "tclock" in macOS/iTerm2, macOS/iTerm2/tmux, macOS/XQuartz/xterm, Debian UXTerm, UXTerm/tmux. Every 5 seconds, the clock's background switches between black and blue.

I'm using the non-macOS system as the exemplar.

• Debian/UXTerm/ncurses6 (TERM=xterm-256color) --> blue-to-black background transition fully resets (OK)
• Debian/UXTerm/tmux/ncurses6 (TERM=screen) --> blue-to-black resets (OK)

• macOS/iTerm2/ncurses5 (TERM=xterm-256color) --> blue-to-black background transition fully resets (OK)
• macOS/iTerm2/tmux/ncurses5 (TERM=screen) --> blue-to-black resets (OK)
• macOS/XQuarts/xterm/ncurses5 (TERM=xterm-256color) --> blue-to-black resets (OK)
• macOS/XQuarts/xterm/tmux/ncurses5 (TERM=screen) --> blue-to-black resets (OK)

• macOS/iTerm2/ncurses6 (TERM=xterm-256color) --> blue-to-black background transition lingers on hour digits (ERR)
• macOS/iTerm2/tmux/ncurses6 (TERM=screen) --> blue-to-black lingers (ERR)
• macOS/XQuarts/xterm/ncurses6 (TERM=xterm-256color) --> blue-to-black lingers (ERR)
• macOS/XQuarts/xterm/tmux/ncurses6 (TERM=screen) --> blue-to-black lingers (ERR)

The common component in the misbehaving scenarios is the MacPorts-installed ncurses 6.1; iTerm2 and TERM are out of the equation for 1/2 of these failed tests. If this is not an iTerm issue - which I believe I've demonstrated, then it must either be a macOS issue or a MacPorts/ncurses-6.1 issue.

[jmroot (Joshua Root)]

I'm not sure which features tclock is using, but your initial screenshots are clearly demonstrating that iTerm2 lacks repeat character support, as mentioned in the ncurses FAQ. Older versions of terminfo mistakenly do not list xterm as supporting that feature, which is why iTerm2 works when using the older version. You don't list any iTerm2 results with TERM=iterm2 – if that works correctly, then it's definitely a matter of expected vs actual terminal capabilities.

Note also that Debian stable ships a version of ncurses from 2016 and still uses ABI 5.

It's of course possible that there are bugs, but if so they are likely to be from upstream since the MacPorts packaging makes very few modifications.

[jmroot (Joshua Root)]

AFAICT there's no bug here that MacPorts can fix.