libedit does not restore terminal state on exit (affects python)

[1st1 (Yury Selivanov)]

Here's what happens after you exit the repl:

yury@ysmac ~ $ python3.5
Python 3.5.0rc3 (default, Sep  7 2015, 23:03:25)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D>>>
yury@ysmac ~ $ yury@ysmac ~ $ yury@ysmac ~ $ yury@ysmac ~ $ yury@ysmac ~ $

[1st1 (Yury Selivanov)]

You also can't see what you type.

Typing 'reset[ENTER]' fixes the terminal. It works this way both in iTerm and in stock Terminal.app

Here's some output from python3.4:

yury@ysmac ~ $ python3
Python 3.4.3 (default, Aug 19 2015, 17:29:49)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
yury@ysmac ~ $
yury@ysmac ~ $
yury@ysmac ~ $
yury@ysmac ~ $
yury@ysmac ~ $

[1st1 (Yury Selivanov)]

Cc Me!

[mf2k (Frank Schima)]

In the future, please Cc the port maintainers (port info --maintainers python35), if any. As reporter, you do not need to Cc yourself.

[su-v]

On OS X 10.7.5, all python versions rebuilt with ncurses 6.0 are affected, see also:
​https://lists.macosforge.org/pipermail/macports-dev/2015-August/031308.html

[su-v]

Cc Me!

[Themanwithoutaplan]

Cc Me!

[Themanwithoutaplan]

I suspect it's the related to the same issue but within the Python shell I need to press Return a second time to get a command prompt. This is a serious inconvenience!

[1st1 (Yury Selivanov)]

Final Python 3.5 will be released in a few days. Is there a chance this will be fixed?

[musicant@…]

Cc Me!

[Themanwithoutaplan]

Replying to yselivanov@…:

Final Python 3.5 will be released in a few days. Is there a chance this will be fixed?

I don't think that really matters. This is affecting all versions of Python.

[ufdup]

reverting to ncurses @5.9_2 seems to fix the problem...

Replying to charlie.clark@…:

Replying to yselivanov@…:

Final Python 3.5 will be released in a few days. Is there a chance this will be fixed?

I don't think that really matters. This is affecting all versions of Python.

[Themanwithoutaplan]

Replying to ufdup89@…:

reverting to ncurses @5.9_2 seems to fix the problem...

Sure, but this isn't easily done because you can't downgrade ports. But it does suggest that there may be a bug in ncurses v6.

[eborisch (Eric A. Borisch)]

As noted in the mailing on the list, adjusting the port to use readline rather than libedit makes things work.

[eborisch (Eric A. Borisch)]

Cc Me!

[jambonrose (Andrew Pinkham)]

Cc Me!

[eborisch (Eric A. Borisch)]

Just copying in my original email about this (with a workaround) :

I've recently noticed (not sure when it changed) that when I enter and then exit() the python (using python27 in particular) interpreter built against libedit, the tty flags (as reported by stty -a) aren't getting reset when exiting python -- most noticeably the echo flag is getting turned off. Yes, yes, reset will fix it, but still.

If I build python27 against libreadline instead of libedit (by disabling the libedit patch file) it works as expected.

[eborisch (Eric A. Borisch)]

There is a stackoverflow thread as well: ​http://stackoverflow.com/questions/32578009/python-macports-and-buffer-problems - I've posted a link to this discussion there; completing the loop.

[eborisch (Eric A. Borisch)]

Does it work for those impacted if you install pyNN-readline?

[Themanwithoutaplan]

Replying to eborisch@…:

Does it work for those impacted if you install pyNN-readline?

I confirm that this works for Python 2.6, Python 2.7 and Python 3.4. But it doesn't work for Python 3.3 (port is obsolete) or Python 3.5 (no port py35-readline).

[su-v]

Replying to eborisch@…:

Does it work for those impacted if you install pyNN-readline?

On OS X 10.7.5:

• Confirmed for Python 2.6, 2.7 and 3.4
  
• Not confirmed for Python 2.5 (I reinstated py25-readline in a local portfile repository for testing)

[eborisch (Eric A. Borisch)]

As mentioned on the mailing list, this addresses the symptom, and has not identified the underlying issue. For users who just want it to work, that is likely OK, but we should still try to understand the root cause.

[1st1 (Yury Selivanov)]

It's nearly impossible to use python right now. Can this be fixed asap? If switching to readline from libedit helps -- can we do that?

[eborisch (Eric A. Borisch)]

Users can install pyNN-readline in the interim. The problem doesn't exist for everyone (see mailing list) so it's not clear what the best course of action is here...

[Themanwithoutaplan]

Replying to yselivanov@…:

It's nearly impossible to use python right now. Can this be fixed asap? If switching to readline from libedit helps -- can we do that?

Python 3.4 is usable once you install py34-readline. Unfortunately, there is no py35-readline but I'm not sure what you'd be using Python 3.5 for at the moment anyway (other than compatibility testing). If you really need it today then download the one from python.org until this bug is fixed.

Could you please add Python 2.6, 2.7, Python 3.3 and Python 3.4 to the ports affected by the bug?

[eborisch (Eric A. Borisch)]

I've added py35-readline, so if you 'port sync' you should be able to install that now.

I would propose adding the following variant (and likely making it default) on all python flavors until this is resolved.

+variant readline description {Use readline instead of libedit} {
+    patchfiles-delete       patch-libedit.diff
+    depends_lib-append      port:readline
+    depends_lib-delete      port:libedit
+}
+

[Themanwithoutaplan]

Replying to eborisch@…:

I've added py35-readline, so if you 'port sync' you should be able to install that now.

Thanks very much but not there yet. Could you add py33-readline for completeness? I tend to work in 3.4 or 2.7 and only use the other versions in tox.

Wouldn't another solution be to go back to the older version of ncurses? Or is that likely to cause conflicts with other ports that depend on the newer one?

[eborisch (Eric A. Borisch)]

Replying to charlie.clark@…:

Replying to eborisch@…:

I've added py35-readline, so if you 'port sync' you should be able to install that now.

Thanks very much but not there yet. Could you add py33-readline for completeness? I tend to work in 3.4 or 2.7 and only use the other versions in tox.

I'd prefer not; it's already in the python graveyard (use 3.4 instead).

Wouldn't another solution be to go back to the older version of ncurses? Or is that likely to cause conflicts with other ports that depend on the newer one?

I don't think that's the best course. I'd much rather see the readline variant above until the libedit/ncurses issue is resolved -- or identified, really. Still unclear as to why not everyone is having issues.

[ryandesign (Ryan Carsten Schmidt)]

#48911 may be related.

[Themanwithoutaplan]

Replying to ryandesign@…:

#48911 may be related.

Yep, that's the same issue.

[eborisch (Eric A. Borisch)]

Given there have been no insights or suggestions on how to fix python w/libedit issues, I've attached a proposed patch that links python against readline -- it is done as a default variant +readline, so it should be easy to un-do later once the underlying issues are identified and resolved.

Any objections to committing this?

[eborisch (Eric A. Borisch)]

Patches python 26,27,34,35 to have default +readline variant

[eborisch (Eric A. Borisch)]

I added +readline (not on by default) in r140531. Please 'port selfupdate' and then 'port install python(26|27|34|35) +readline' if you are experiencing these issues.

[icemac (Michael Howitz)]

Cc Me!

[dh@…]

This issue also affects the debugger (pdb), which often fails to print its prompt after a command.

[eborisch (Eric A. Borisch)]

Fixed when Python is installed +readline?

[icemac (Michael Howitz)]

Replying to eborisch@…:

Fixed when Python is installed +readline?

Yes, +readline helped for me for the python console and the debugger.

[smithsp (Sterling Smith)]

Cc Me!

[kurtjaeke@…]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Has duplicates #48911, #49479.

[eborisch (Eric A. Borisch)]

As I noted above, by not patching in libedit, this problem goes away. This is available via the +readline variant since r140531 (September 21, 2015).

Is there some objection to making +readline a default_variant, or perhaps going one step forward and making +libedit the variant, and make using libreadline the stock? (This is important wrt. what people get when using 'port upgrade outdated'.)

This is clearly continuing to bite people. Is there (potentially? haven't investigated) some licensing benefit to libedit vs. libreadline?

[Themanwithoutaplan]

Replying to eborisch@…:

Is there (potentially? haven't investigated) some licensing benefit to libedit vs. libreadline?

Yes, readline is GPL3 which a serious encumbrance. So we really need to fix what went wrong in libedit.

[eborisch (Eric A. Borisch)]

Really?

When I run a distributable check (/opt/mports/portmgr/jobs/port_binary_distributable.tcl -v <port> <variants>) on the ports I have installed that depend on python27 (I have over 200 installed that match; admittedly a subset) the only ones that lose distributability are gd2, graphviz[-gui], and quartz-wm. Having to build those from source doesn't seem (to me) to be too high a cost to pay for working python.

[Themanwithoutaplan]

Replying to eborisch@…:

When I run a distributable check (/opt/mports/portmgr/jobs/port_binary_distributable.tcl -v <port> <variants>) on the ports I have installed that depend on python27 (I have over 200 installed that match; admittedly a subset) the only ones that lose distributability are gd2, graphviz[-gui], and quartz-wm. Having to build those from source doesn't seem (to me) to be too high a cost to pay for working python.

Multiply by the number of Python versions you have installed (5 on my system…) And who has the time and energy to double-check this all the time?

Anyway, GPL3 isn't just about redistribution and linking but let's not go down that particular rabbit hole.

What I don't understand is why nothing else seems to have been so affected by the changes in libedit. But that is where we need to focus our efforts.

[steve+macports@…]

Cc Me!

[alexleigh (Alex Leigh)]

Cc Me!

[cdeil (Christoph Deil)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #49739.

[eborisch (Eric A. Borisch)]

Ok. Can we put a note in the python install that "if you experience issues with your terminal after exiting Python, please install pyNN-readline or 'pythonNN +readline' and report your issues on #48807." (With the actual URL.)

Or can we (for now) change to readline and make +libedit a variant. If the licensing is such an issue, perhaps this will motivate someone to dig into the underlying issue. Once the issue is resolved, switch back. It seems silly to put out a broken version because we like the licensing better. Any technical objections to doing this?

[mleinartas@…]

Cc Me!

[seanfarley (Sean Farley)]

Just had another report about this on the mercurial channel. What's the status with changing to readline and making +libedit a variant?

[seanfarley (Sean Farley)]

Anyone objected to me making the readline change?

[fhgwright (Fred Wright)]

Cc Me!

[bpanulla (Brian Panulla)]

Cc Me!

[xfacter@…]

Cc Me!

[piannucci (Peter Iannucci)]

Or we can just roll back. I just built and installed libedit from the portfile at rev. 127130, and it's working correctly with Python 3.4.

[seanfarley (Sean Farley)]

Replying to iannucci@…:

Or we can just roll back. I just built and installed libedit from the portfile at rev. 127130, and it's working correctly with Python 3.4.

Any objections to this?

[eborisch (Eric A. Borisch)]

This continues to cause problems:

• ​http://bugs.python.org/issue24961
• ​http://superuser.com/questions/983755/os-x-terminal-behaves-oddly-after-running-python-interactively
• ​http://apple.stackexchange.com/questions/206795/using-python-screws-up-the-shell-and-or-terminal
• ​http://stackoverflow.com/questions/32578009/python-macports-and-buffer-problems

Hearing no objections here in the past three months, I've committed the flip to readline in r145851 for python27 as a trial balloon. If we don't have an uproar (waiting for it) we can move over the others. You can install +libedit if you really want to, and once this is fixed, we can move the variant-less version back to libedit.

Note: I've left the port:libedit dependency to avoid the possibility of something that was built before and linked against libedit (indirectly "dependent" through python27) being broken by a 'port reclaim' operation.

[jmroot (Joshua Root)]

What was wrong with recommending use of py*-readline when readline support is required? A variant is a worse solution since it requires a reinstall for everyone (even those who don't use python interactively) and it makes python effectively GPLv3.

[eborisch (Eric A. Borisch)]

The problem is that we can't make python dependent (recursively) on py-readline, and it was installing broken software which prompted users to go find out what was going on and then try to fix it.

Yes, it requires a reinstall; for many users this will be a binary install (from packages.macports.org/python27). Still better than installing something that we've known has been broken for months and just figure users will search out the solution.

I'm all for moving it back to libedit once the combination is working. (Where working means not broken for interactive users.)

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #50794.

[piannucci (Peter Iannucci)]

Hm, perhaps I was too indirect. I object to flipping over to readline. I believe that there is already a good temporary solution -- roll back -- and a good long-term solution -- bisect and submit an upstream patch. In my view, flipping to readline would be akin to sweeping the problem under the rug.

[Russell-Jones-OxPhys (Russell Jones)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #51188.

[eborisch (Eric A. Borisch)]

This is still causing problems, clearly. How about we add "notes" to all Python versions to "Please install pyNN-readline if you will be using Python interactively (from the command line)." Would that make everyone happy -- and hopefully help people get working software. Delivering broken tools that we know are broken is just bad form.

[roger@…]

Replying to eborisch@…:

This is still causing problems, clearly. How about we add "notes" to all Python versions to "Please install pyNN-readline if you will be using Python interactively (from the command line)." Would that make everyone happy -- and hopefully help people get working software. Delivering broken tools that we know are broken is just bad form.

Would have made me happyier, more than hunting for this bug report.

[eborisch (Eric A. Borisch)]

I plan to commit the python-notes.patch changes tomorrow unless someone objects for some bizarre reason. (There is no change to the installs, only an addition to notes to inform console users of issue / suggesting they install pyNN-readline.)

[eborisch (Eric A. Borisch)]

Committed the notes patch (not a real solution, but at least a warning to users) in r148368.

[mojca (Mojca Miklavec)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Replying to iannucci@…:

Hm, perhaps I was too indirect. I object to flipping over to readline. I believe that there is already a good temporary solution -- roll back -- and a good long-term solution -- bisect and submit an upstream patch. In my view, flipping to readline would be akin to sweeping the problem under the rug.

And causes new problems. Specifically, this change caused graphviz to no longer be considered distributable, because it eventually depends on python27, and readline's GPL license conflicts with graphviz's EPL license.

What is the status of getting this issue resolved "properly"? Is there an upstream bug report? (The only upstream bug report linked in this ticket so far is closed.)

[eborisch (Eric A. Borisch)]

Patch against r148368; reverts to libedit but leaves notes (install py27-readline suggestion) intact.

[eborisch (Eric A. Borisch)]

Reverse of upstream 1.111 changes.

[eborisch (Eric A. Borisch)]

I've attached a patch​ that will make the python27 port act like the others, where it still installs and builds against libedit, but with the install time notes suggesting py27-readline be installed. I will commit it in a few days if there are no objections. This will "resolve" the licensing status, but means installing a broken (to my mind) python if the user runs sudo port install python27 and doesn't read/notice/act on the notes.

I've also attached a fix; but I'm not sure that it will be accepted upstream. Read on...

I believe I've found the (opposite) issue here: ​http://gnats.netbsd.org/48957. Arising from this PR, ​r1.111, undid a change in ​r1.85, which were from a set of Apple patches.

I can confirm that undoing the upstream (1.111) change (commenting out one line: readline_c.diff​) fixes the broken shell state on python exit under macOS. This change, however, was clearly viewed as incorrect upstream.

For reference, ​Apple's version based on 1.106 currently has this line commented out.

I suggest we match Apple's libedit behavior here, at least for this quirk. Remove the patch if/when Apple aligns with NetBSD. (Add readline_c.diff​ as a patchfile.) At that time, changes may be required in Python, and appropriate upstream reports can be filed.

Alternatively, we could see if python's __APPLE__ fenced code in its own readline.c can be removed safely with the latest upstream libedit; I haven't had time to do this, but maybe someone else will...

As to how this impacts anything else using libedit, I can't say as I haven't looked.

[zaxdo@…]

Cc Me!

[eborisch (Eric A. Borisch)]

Committed changes to python27 in r149730. Builds broken (IMHO) against libedit, but hey, it's distributable so at least there is that. Note for the user to install py27-readline if they want to use the terminal remains.

I've seen no comments on the libedit readline.c​ patch I posted here. Perhaps I should make that into a separate issue...

[jmroot (Joshua Root)]

The interesting part is that the problem didn't appear when using ncurses 5 with the same version of libedit. Ncurses 6 added its own buffering, whereas previously it shared buffers with stdio. The release notes mention that some programs may incorrectly assume that the buffers are the same. I wonder if python is doing some form of this.

I'd really prefer not to commit hacky workarounds that upstream says are incorrect, without understanding the problem. It might be instructive to file a radar asking Apple to adopt upstream's libedit changes?

[musicant@…]

Cc Me!

[musicant@…]

Cc Me!

[musicant@…]

Cc Me!

[eborisch (Eric A. Borisch)]

The new libedit (libedit-0160618-3.1_0) seems to have fixed this one. I can remove py27-readline, and things still work when I exit python in the terminal.

Can others confirm so we can finally put this out to pasture? I'll remove the notes from pythonXX after I see some "works for me" posts...

[fhgwright (Fred Wright)]

Replying to eborisch@…:

The new libedit (libedit-0160618-3.1_0) seems to have fixed this one. I can remove py27-readline, and things still work when I exit python in the terminal.

Can others confirm so we can finally put this out to pasture? I'll remove the notes from pythonXX after I see some "works for me" posts...

Still fails here (with python27 @2.7.12_1 and libedit @20160618-3.1_0+universal).

I wasn't sure of the relationship between py-readline and py27-readline, so I'd installed both, and uninstalled both to test this. I'd *also* previously installed python27 with the readline variant, which may have been belt-and-suspenders when combined with the separate package, but I switched to the default variants for the test.

Note that in the failing case, prompts aren't displayed properly in interactive use, as well as leaving the terminal screwed up on exit (though recoverable via "stty sane"). So the issue description is incomplete.

[Themanwithoutaplan]

Replying to eborisch@…:

The new libedit (libedit-0160618-3.1_0) seems to have fixed this one. I can remove py27-readline, and things still work when I exit python in the terminal.

Can others confirm so we can finally put this out to pasture? I'll remove the notes from pythonXX after I see some "works for me" posts...

Sorry, to be pedantic but can you please be specific about what we should test? For me the latest Python 2.7 (2.7.12_1) and libedit (20160618-3.1_0) does not resolve the problem and python-readline is still required. Should I be working with a checkout? Also, doesn't this have as much to do with ncurses as it does with libedit?

[eborisch (Eric A. Borisch)]

Hrmmm. Looks like python needs to be rebuilt against the new headers. I'll revbump pythonXX assuming this pans out.

(replace 27 with the version you would like to test)

sudo port uninstall -f python27 py27-readline
sudo port upgrade --no-rev-upgrade libedit
sudo port install -s python27  # -s : source-only mode

[Themanwithoutaplan]

Compiling from source does indeed solve the problem. Thanks!

[mf2k (Frank Schima)]

Replying to fw@…:

I wasn't sure of the relationship between py-readline and py27-readline

py-readline does nothing because it is a stub port that holds the real python modules such as py27-readline. This is true for all python modules in Macports. So py-* ports should never be installed.

[fhgwright (Fred Wright)]

Replying to mf2k@…:

Replying to fw@…:

I wasn't sure of the relationship between py-readline and py27-readline

py-readline does nothing because it is a stub port that holds the real python modules such as py27-readline. This is true for all python modules in Macports. So py-* ports should never be installed.

Yeah, I suspected as much, but the fact that it installed without error obfuscated the issue.

I also tried as many other versions of Python as possible, in some cases requiring the expansion of the version lists in py-readline and py-setuptools. In one case (python31), the failure mode is completely different but the py-readline fix still works. All other versions from 26 through 35 behave identically. The python25 build is currently broken for other reasons, so I couldn't test that. The python24 port was never patched to use libedit in the first place, so it doesn't have the issue.

[larryv (Lawrence Velázquez)]

Cc Me!

[eborisch (Eric A. Borisch)]

False alarm. The latest pythons are tripping during build if readline is installed, leading to the readline module not being built, but the build still "succeeds."

From the build log when port:readline is installed:

Failed to build these modules:
readline

The resulting python doesn't hose the terminal as it doesn't have the libedit-backed readline module.

Installing pyXX-readline is still the best option for most users, followed by pythonXX +readline, with the latter requiring a ~ 10x longer compile. Both require local compilation.

[icemac (Michael Howitz)]

Installing python36 (@3.6.0b1_0) suggests to install py36-readline. But this port does not exist.

Does this problem no longer exist for Python 3.6 or should I file a new ticket that the port py36-readline needs to be created?

[larryv (Lawrence Velázquez)]

We generally don’t add modules for new Python versions until they’re released.

However, do you observe the problem with 3.6.0 beta 1?

[icemac (Michael Howitz)]

Replying to larryv@…:

However, do you observe the problem with 3.6.0 beta 1?

Yes, 3.6.0b1 has this problem, too. (I should have tried it before, sorry.)

Installing it with +readline fixes the problem and does not show the suggestion to install py36-readline.

[cdeil (Christoph Deil)]

With sudo port install python36 I now see

##############################################################
# IF YOU ARE USING PYTHON FROM THE TERMINAL, PLEASE INSTALL:
#   py36-readline
# TO AVOID A LIBEDIT / PYTHON INTERACTION ISSUE.
# REF: https://trac.macports.org/ticket/48807
##############################################################

but py36-readline doesn't exist (yet?).

[fhgwright (Fred Wright)]

Replying to cdeil:

With sudo port install python36 I now see

##############################################################
# IF YOU ARE USING PYTHON FROM THE TERMINAL, PLEASE INSTALL:
#   py36-readline
# TO AVOID A LIBEDIT / PYTHON INTERACTION ISSUE.
# REF: https://trac.macports.org/ticket/48807
##############################################################

but py36-readline doesn't exist (yet?).

Note that Python 3.6 was only officially released two days ago :-)

And unfortunately it's not just a matter of adding the version to the py-readline port.

With a (locally generated) py36-readline installed:

MacPro:~ fw$ python3.6
Python 3.6.0 (default, Dec 23 2016, 12:59:30) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+1
Python(59962,0x7fff7a4d0310) malloc: *** error for object 0x103e7f640: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

Without it:

MacPro:~ fw$ python3.6
Python 3.6.0 (default, Dec 23 2016, 12:59:30) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+1
>>> 2
^D>>> 
MacPro:~ fw$ MacPro:~ fw$ MacPro:~ fw$ 

The latter shows the usual trouble until "stty sane" (unechoed).

[eborisch (Eric A. Borisch)]

You can install python36 +readline (python36 with the readline variant). I just tried it and it "worked for me."

[fhgwright (Fred Wright)]

Replying to eborisch:

You can install python36 +readline (python36 with the readline variant). I just tried it and it "worked for me."

Yeah, that works. I'd moved away from that approach once py-readline appeared, since it avoids the need to rebuild Python from source due to the non-default variant. What's weird about this (aside from how long libedit's been broken) is that upstream Python uses readline and has to be patched to use libedit, so apparently having a less restrictive license was prioritized over consistency with upstream.

It might be nice if the readline variant didn't suggest installing pyXX-readline. :-)

[cdeil (Christoph Deil)]

You can install python36 +readline (python36 with the readline variant). I just tried it and it "worked for me."

Yeah, that works. I'd moved away from that approach once py-readline appeared, since it avoids the need to rebuild Python from source due to the non-default variant.

+1 to add py36-readline, it's more convenient / faster than the +readline variant.

Most users (like me) will run into this issue after they already installed python36 and just being able to install py36-readline to fix it is the simplest solution (besides changing the install to not have this bug by default in the first place).

Please ...

[cdeil (Christoph Deil)]

py36-readline was now added: #53243

But it results in Python REPL crash on exit, so it's not a solution.

Installing py36-gnureadline doesn't solve this "messed up terminal state on exit" issue.

Sigh.

[raimue (Rainer Müller)]

Linking my findings from last April: ​https://lists.macosforge.org/pipermail/macports-dev/2016-April/032891.html

The problem appears to be that a bug in libedit that was fixed, but CPython actually relied on this bug to work properly. If this bug does not occur with CPython on other platforms using libedit, it might be worth a shot to scrub the whole #ifdef __APPLE__ parts for readline/libedit handling. This use of #ifdef __APPLE__ seems to be meant as "using libedit as shipped by Apple", but this is clearly wrong and a bad way to match this condition.

[mohd-akram (Mohamed Akram)]

I don't have this issue when I build python36 myself on macOS 10.12.3:

sudo port -s install python36

Packages need an update?

[eborisch (Eric A. Borisch)]

2 questions:

• Do you have py36-readline installed?
• What shell are you using?

[mohd-akram (Mohamed Akram)]

I don't have py36-readline installed and I'm using the default bash shell.

[eborisch (Eric A. Borisch)]

I'm guessing you had the readline port installed while you built python3.6. Try building in trace/sandbox mode. sudo port uninstall python36 && sudo port -ts install python36. By my brief test, this will yield a broken (as measured by having a bad terminal state on exit) python3.6.

Please confirm, and assuming you see the same results, we can try to track down why having readline present -- even just during configure; you can remove it between configure and the remainder of the install process -- on the system makes this problem go away.

[mohd-akram (Mohamed Akram)]

Indeed, it was because I had readline installed.

[mohd-akram (Mohamed Akram)]

Updated libedit patch

[raimue (Rainer Müller)]

Replying to eborisch:

Please confirm, and assuming you see the same results, we can try to track down why having readline present -- even just during configure; you can remove it between configure and the remainder of the install process -- on the system makes this problem go away.

The existing patch-libedit.diff replaces -lreadline with -ledit, which makes the check for readline to actually link with the readline compatibility layer of libedit. However, there are other checks still using -lreadline in the configure script.

@mohd-akram Regarding the attached patch, these hunks in your patch look bogus:

-#ifdef __APPLE__
+#ifndef __APPLE__

As I understand it, these #ifdef blocks were meant specifically for libedit as provided by Apple on macOS. Negating the conditional does not make sense to me, as that would mean they should be applied to all other systems. Just for additional background, most of this was added as ​a workaround to detect a possible bug in libedit as distributed by Apple at runtime. In order to eventually get such a fix upstream, the logic should be precise. Or just delete the conditional code with the patch, but then we will never get this upstream.

If you want to get rid of these blocks completely, the right thing would be to add an additional configure check defining a macro indicating whether it is using the broken Apple libedit or the fixed version from MacPorts.

[mohd-akram (Mohamed Akram)]

Yeah, the patch is no good, I'll have to look into this more. There are readline checks all over the place in configure and the readline.c file. Someone was working to push this upstream with a proper fix - ​https://bugs.python.org/issue13501 - which might be useful.

[lpsinger (Leo Singer)]

Replying to fhgwright:

Replying to cdeil: With a (locally generated) py36-readline installed:

MacPro:~ fw$ python3.6
Python 3.6.0 (default, Dec 23 2016, 12:59:30) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+1
Python(59962,0x7fff7a4d0310) malloc: *** error for object 0x103e7f640: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

I get this crash too.

[joshenders (Josh Enders)]

I get this crash as well:

$ sudo port install python36 +readline
--->  Computing dependencies for python36
--->  Fetching archive for python36
--->  Attempting to fetch python36-3.6.1_0+readline.darwin_16.x86_64.tbz2 from https://packages.macports.org/python36
--->  Attempting to fetch python36-3.6.1_0+readline.darwin_16.x86_64.tbz2 from http://sea.us.packages.macports.org/macports/packages/python36
--->  Attempting to fetch python36-3.6.1_0+readline.darwin_16.x86_64.tbz2 from http://kmq.jp.packages.macports.org/python36
--->  Fetching distfiles for python36
--->  Attempting to fetch Python-3.6.1.tar.xz from https://distfiles.macports.org/python36
--->  Attempting to fetch Python-3.6.1.tar.xz from http://www.python.org/ftp/python/3.6.1/
--->  Verifying checksums for python36
--->  Extracting python36
--->  Applying patches to python36
Warning: reinplace s|xargs -0 rm -r|/usr/bin/xargs -0 /bin/rm -r|g didn't change anything in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python36/python36/work/Python-3.6.1/Mac/PythonLauncher/Makefile.in
--->  Configuring python36
--->  Building python36
--->  Staging python36 into destroot
--->  Installing python36 @3.6.1_0+readline
--->  Deactivating python36 @3.6.1_0
--->  Cleaning python36
--->  Activating python36 @3.6.1_0+readline
--->  Cleaning python36
--->  Updating database of binaries
--->  Scanning binaries for linking errors
--->  No broken files found.
--->  Some of the ports you installed have notes:
  python36 has the following notes:
    To make this the default Python or Python 3 (i.e., the version run by the 'python' or 'python3' commands), run one or both of:

        sudo port select --set python python36
        sudo port select --set python3 python36

    ##############################################################
    # IF YOU ARE USING PYTHON FROM THE TERMINAL, PLEASE INSTALL:
    #   py36-readline
    # TO AVOID A LIBEDIT / PYTHON INTERACTION ISSUE.
    # REF: https://trac.macports.org/ticket/48807
    ##############################################################

$ python3
Python 3.6.1 (default, Mar 24 2017, 00:08:29)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello World")
Python(61973,0x7fffaf4293c0) malloc: *** error for object 0x10f92f300: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

[eborisch (Eric A. Borisch)]

Replying to joshenders:

I get this crash as well:

[snip]

---> Activating python36 @3.6.1_0+readline

Do you also have py36-readline installed? Having both python36 (itself) installed +readline (variant) and py36-readline (the readline python package) installed at the same time can cause that, if I recall. Use one or the other.

[aikchar (Hamza Sheikh)]

Replying to eborisch:

Replying to joshenders:

I get this crash as well:

[snip]

---> Activating python36 @3.6.1_0+readline

Do you also have py36-readline installed? Having both python36 (itself) installed +readline (variant) and py36-readline (the readline python package) installed at the same time can cause that, if I recall. Use one or the other.

I removed py36-readline and installed python36 +readline and I don't get this error anymore. Thanks, Eric!

$ port installed | grep readline
  py27-readline @6.2.4.1_1 (active)
  py34-readline @6.2.4.1_1 (active)
  py35-readline @6.2.4.1_1 (active)
  readline @6.3.003_1 (active)

$ sudo port install python36 +readline
<SNIP OUTPUT>

$ port installed | grep readline
  py27-readline @6.2.4.1_1 (active)
  py34-readline @6.2.4.1_1 (active)
  py35-readline @6.2.4.1_1 (active)
  python36 @3.6.1_0+readline (active)
  readline @6.3.003_1 (active)

$ /opt/local/bin/python3.6
Python 3.6.1 (default, Mar 24 2017, 15:43:12)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello, World!")
Hello, World!
>>> exit()

$ echo $?
0

$ sudo port install py36-readline
<SNIP OUTPUT>

$ port installed | grep readline
  py27-readline @6.2.4.1_1 (active)
  py34-readline @6.2.4.1_1 (active)
  py35-readline @6.2.4.1_1 (active)
  py36-readline @6.2.4.1_1 (active)
  python36 @3.6.1_0+readline (active)
  readline @6.3.003_1 (active)

$ /opt/local/bin/python3.6
Python 3.6.1 (default, Mar 24 2017, 15:43:12)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello, World!")
Python(58844,0x7fffd2b4a3c0) malloc: *** error for object 0x1041d6318: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

$ echo $?
134

$ sudo port uninstall py36-readline
<SNIP OUTPUT>

$ port installed | grep readline
  py27-readline @6.2.4.1_1 (active)
  py34-readline @6.2.4.1_1 (active)
  py35-readline @6.2.4.1_1 (active)
  python36 @3.6.1_0+readline (active)
  readline @6.3.003_1 (active)

$ /opt/local/bin/python3.6
Python 3.6.1 (default, Mar 24 2017, 15:43:12)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello, World!")
Hello, World!
>>> exit()

$ echo $?
0

[bs (Britt Selvitelle)]

Just installed python36 and the install recommend py36-readline and had malloc crashes along the lines of pointer being freed was not allocated python.

Found this thread. Uninstalled py36-readline and installed python36 +readline, which seems to have fixed this. Can this be pushed as a default flag, at the very least in place of the current misinformation that py36-readline works?

[bdbaddog]

Replying to bs:

Just installed python36 and the install recommend py36-readline and had malloc crashes along the lines of pointer being freed was not allocated python.

Found this thread. Uninstalled py36-readline and installed python36 +readline, which seems to have fixed this. Can this be pushed as a default flag, at the very least in place of the current misinformation that py36-readline works?

+1 on this suggestion..

[DavidBAEpstein]

Sequential Actions	Result
installed python36 with no variant	readline misbehaves in python interpreter
installed python36+readline	port notes python36 says "install py36-readline"
installed py36-readline	python interpreter crashes on receiving Carriage Return
deactivated py36-readline	readline in python interpreter appears to work perfectly
activated py36-readline	python interpreter crashes on receiving Carriage Return
deactivated py36-readline	readline appears to work perfectly

Maintainers: Please modify the note from port install python36 to remove the incorrect advice. The note should preferably say

Do not install py36-readline, or, if installed, deactivate.

[jmroot (Joshua Root)]

I suspect that the incorrect terminal state may be down to the fact that python never calls rl_deprep_terminal before exiting. However, I don't know why (a) I can't reproduce that issue, and (b) it doesn't happen with GNU readline.

[jmroot (Joshua Root)]

To clear up one part, zsh appears to be resistant to the issue, while bash is not.

[jmroot (Joshua Root)]

This seems to do the trick. Patch against python 3.6. Please test. (Note that it doesn't fix the display of the prompt, which is fortunately just a cosmetic issue, but I'll investigate further.)

[fhgwright (Fred Wright)]

Replying to jmroot:

This seems to do the trick. Patch against python 3.6. Please test. (Note that it doesn't fix the display of the prompt, which is fortunately just a cosmetic issue, but I'll investigate further.)

I was wondering whether the screwed-up prompt is a separate problem or another manifestation of the same problem. If the latter, it might provide a useful clue.

[1-61803]

Just a reminder to reset your terminal: stty sane.

[jmroot (Joshua Root)]

Further findings:

• I'm leaning towards the missing deprep being a bug in libedit's readline emulation. The readline docs say that rl_callback_handler_install will prep the terminal and rl_callback_handler_remove will deprep it. They are silent on whether an explicit call to rl_prep_terminal should need a matching call to rl_deprep_terminal, but the actual behaviour of readline is that it does not, so that's what libedit should emulate.
• What's going on with the prompt is that after a line is entered by the user, libedit prints a new prompt before python has a chance to print its output. Just using the readline() function doesn't have this issue, only the callback-based API. I'm attaching a reduced test program.

[jmroot (Joshua Root)]

Readline test case

[jmroot (Joshua Root)]

Looks like this was recently fixed upstream.

[jmroot (Joshua Root)]

Unfortunately the changes in upstream CVS seem to introduce a null pointer dereference.

[jmroot (Joshua Root)]

In 86fdc2922ed2b932b8c1d7a676e9ca3acc8ed85a/macports-ports:

libedit: fix misbehaviour with callback-based API

rl_callback_read_char does el_set(e, EL_UNBUFFERED, 1) after running the
client-supplied callback. However, if the callback ran
rl_callback_handler_remove, this undoes part of what that function did. So,
don't do it if the callback has been removed.

This fixes python's issues with not restoring terminal state on exit and
incorrect prompt display, and should avoid reintroducing the problem described
in <​http://gnats.netbsd.org/48957>.

Fixes: #48807

[1-61803]

Replying to jmroot:

In 86fdc2922ed2b932b8c1d7a676e9ca3acc8ed85a/macports-ports:

libedit: fix misbehaviour with callback-based API

Does it mean that we could/should install python36 without the readline variant?

[jmroot (Joshua Root)]

Replying to 1-61803:

Replying to jmroot:

In 86fdc2922ed2b932b8c1d7a676e9ca3acc8ed85a/macports-ports:

libedit: fix misbehaviour with callback-based API

Does it mean that we could/should install python36 without the readline variant?

Yes, python36 (and all the other python versions) works properly without the readline variant now. No real need to reinstall though.

[Themanwithoutaplan]

It's taken a while but the changes to libedit do indeed seem to have resolved the problems. Thanks very much!