Opened 5 years ago
Last modified 4 years ago
#60274 new defect
multiprocessing broken in Python38
Reported by: | mouse07410 (Mouse) | Owned by: | |
---|---|---|---|
Priority: | Normal | Milestone: | |
Component: | ports | Version: | |
Keywords: | Cc: | chrstphrchvz (Christopher Chavez) | |
Port: | python38 |
Description
This bug is from the upstream. Patch/solution to issue 33725 on https://bugs.python.org broke multiprocessing package. Now it crashes in spawn.py
with
Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 126, in _main self = reduction.pickle.load(from_parent) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/synchronize.py", line 110, in __setstate__ self._semlock = _multiprocessing.SemLock._rebuild(*state) FileNotFoundError: [Errno 2] No such file or directory
Here's the program that triggers the bug:
#!/usr/bin/env python3 # # Test "multiprocessing" package included with Python-3.6+ # # Usage: # ./mylti1.py [nElements [nProcesses [tSleep]]] # # nElements - total number of integers to put in the queue # default: 100 # nProcesses - total number of parallel processes/threads # default: number of physical cores available # tSleep - number of milliseconds for a thread to sleep # after it retrieved an element from the queue # default: 17 # # Algorithm: # 1. Creates a queue and adds nElements integers to it, # 2. Creates nProcesses threads # 3. Each thread extracts an element from the queue and sleeps for tSleep milliseconds # import sys, queue, time import multiprocessing as mp def getElements(q, tSleep, idx): l = [] # list of pulled numbers while True: try: l.append(q.get(True, .001)) time.sleep(tSleep) except queue.Empty: if q.empty(): print(f'worker {idx} done, got {len(l)} numbers') return if __name__ == '__main__': nElements = int(sys.argv[1]) if len(sys.argv) > 1 else 100 nProcesses = int(sys.argv[2]) if len(sys.argv) > 2 else mp.cpu_count() tSleep = float(sys.argv[3]) if len(sys.argv) > 3 else 17 # Uncomment the following line to revert to old 'fork' and make it work on 3.8+ #mp.set_start_method('fork') # Fill the queue with numbers from 0 to nElements q = mp.Queue() for k in range(nElements): q.put(k) # Start worker processes for m in range(nProcesses): p = mp.Process(target=getElements, args=(q, tSleep / 1000, m)) p.start()
Proposed solution is reverting the default of start
back to 'fork'
.
Change History (2)
comment:1 Changed 4 years ago by chrstphrchvz (Christopher Chavez)
comment:2 Changed 4 years ago by chrstphrchvz (Christopher Chavez)
Cc: | chrstphrchvz added |
---|
Note: See
TracTickets for help on using
tickets.
The reporter raised this issue on the original upstream ticket, but was told their suggested reversion is unlikely to be accepted, as well as some issues with their example script; but that upstream might consider a documentation change to help users avoid this issue. The reporter opened a new upstream ticket, but which hasn't been resolved.
As this appears to be either a user code or upstream issue, it might not be something MacPorts will address.