Opened 5 years ago
Closed 5 years ago
#58831 closed defect (fixed)
select.poll() not available in Python 3.x but present in Python 2.7
| Reported by: | reneeotten (Renee Otten) | Owned by: | reneeotten (Renee Otten) |
|---|---|---|---|
| Priority: | Normal | Milestone: | |
| Component: | ports | Version: | |
| Keywords: | Cc: | jmroot (Joshua Root) | |
| Port: | python3 py-wurlitzer |
Description
When trying to update py-wurlitzer to its latest version (it is a dependency of py-spyder-devel), I noticed that it fails on Python 3.x because this version (1.0.3) now uses select.poll().
Google tells me that there appear to be issues with select.poll() on macOS, but on the other hand it works fine with the Conda installation and also using Python 2.7 installed through MacPorts.
Looking a bit further, the difference between the python27 and python3? ports that cause this difference is the absence/presence of the pyconfig.ed patchfile. It was perhaps added because of ticket 18376? It's still present for Python 3.x ports, but was removed in this commit for Python 2.7. Removing this patch for the python37 port locally and rebuilding it, makes spyder-devel start without errors because now select.poll() is indeed available.
Joshua, since you made that commit and have been maintaining the Python ports for quite some time - can you comment on this and do you know if there is a reason for keeping this patch still in the Python3 ports?
Change History (7)
comment:1 Changed 5 years ago by jmroot (Joshua Root)
comment:2 Changed 5 years ago by mf2k (Frank Schima)
| Port: | python3 added; python3? removed |
|---|
comment:3 Changed 5 years ago by jmroot (Joshua Root)
Hmm, so actually there was a genuine issue with waiting on 0 fds where poll() would return immediately even if a nonzero timeout was specified. That is the case on < 10.9 and also on 10.12-10.12.1. I don't know if python has any code to mitigate that or it's just a user-beware situation. I notice the python docs say that poll is not guaranteed to be available on all platforms.
comment:4 Changed 5 years ago by reneeotten (Renee Otten)
Yes, that indeed appears the case from browsing the web, at least from the blog of a curl developer. Compiling the test code in that link on macOS 10.14.6 suggests indeed that it works correctly now.
It also appears that the configure step performs some check for polll, but I am not sure if that's sufficient to predict these potential runtime errors. From the Python docs or bug tracker it doesn't become very clear to me.
On the other hand, this patch was removed from Python 2.7 more than 6 years ago now and it seems that nobody every filed a Ticket with an issue - perhaps that's enough to say that it's likely okay to remove it from Python 3.x as well? Or perhaps we would need to run the small test program on the different macOS versions and only apply the patch where needed (presumably only below 10.9; I think we can assume that users of 10.12 would have updated to 10.12.2 by now)?
comment:5 Changed 5 years ago by jmroot (Joshua Root)
The configure check seems to be only for a really severely broken poll. On < 10.9 it "works", it just burns a lot of CPU unnecessarily.
I agree it's probably OK to re-enable on 10.9+ at least. On < 10.9, the way forward is less clear: this one module would work (badly), but maybe other modules work better when they choose kqueue or select instead due to poll being unavailable. TBH, wurlitzer should be using the selectors module for its I/O multiplexing anyway; relying on poll alone really limits portability. (Even just within macOS, Apple's python still disables poll to this day.)
comment:6 Changed 5 years ago by reneeotten (Renee Otten)
I see that someone else opened a PR to move away from using select.poll() in wurlitzer. I checked upstream when the update came out, but clearly failed to look at it again more recently... If wurlitzer moves away from using poll, I have personally no immediate reason to change the status quo, but I am not sure that's going to happen soon. Of course we could always include that suggested change in MacPorts... (or just not update the package for the time being). Let me follow-up on the upstream Issue/PR as well and see what they think.
In short, I agree that there probably isn't a ideal solution to this: making the change only for one Python package seems unreasonable, especially if it's unclear whether/how this will affect other packages. On the other hand, if we are sure that more recent macOS versions have fixed the underlying issue then it might be worthwhile to do.
comment:7 Changed 5 years ago by reneeotten (Renee Otten)
| Owner: | set to reneeotten |
|---|---|
| Resolution: | → fixed |
| Status: | new → closed |
Poll was never "broken" as such, but it returned EAGAIN fairly often which was not handled well by python (apparently that is within spec but differs from Linux's behaviour). I don't know if (a) the behaviour of poll changed in later macOS versions, or (b) python started handling the behaviour better, or both. If either is the case, we need to find out exactly which macOS version and/or which python version this happened in, so we can remove the change where it's not needed without breaking other modules in cases where the change is needed.