Opened 5 years ago

Last modified 4 years ago

#60274 new defect

multiprocessing broken in Python38

Reported by: mouse07410 (Mouse) Owned by:
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc: chrstphrchvz (Christopher Chavez)
Port: python38

Description

This bug is from the upstream. Patch/solution to issue 33725 on https://bugs.python.org broke multiprocessing package. Now it crashes in spawn.py with

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/synchronize.py", line 110, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory

Here's the program that triggers the bug:

#!/usr/bin/env python3
#
# Test "multiprocessing" package included with Python-3.6+
#
# Usage:
#    ./mylti1.py [nElements [nProcesses [tSleep]]]
#
#        nElements  - total number of integers to put in the queue
#                     default: 100
#        nProcesses - total number of parallel processes/threads
#                     default: number of physical cores available
#        tSleep     - number of milliseconds for a thread to sleep
#                     after it retrieved an element from the queue
#                     default: 17
#
# Algorithm:
#   1. Creates a queue and adds nElements integers to it,
#   2. Creates nProcesses threads
#   3. Each thread extracts an element from the queue and sleeps for tSleep milliseconds
#

import sys, queue, time
import multiprocessing as mp


def getElements(q, tSleep, idx):
    l = []  # list of pulled numbers
    while True:
        try:
            l.append(q.get(True, .001))
            time.sleep(tSleep)
        except queue.Empty:
            if q.empty():
                print(f'worker {idx} done, got {len(l)} numbers')
                return


if __name__ == '__main__':
    nElements = int(sys.argv[1]) if len(sys.argv) > 1 else 100
    nProcesses = int(sys.argv[2]) if len(sys.argv) > 2 else mp.cpu_count()
    tSleep = float(sys.argv[3]) if len(sys.argv) > 3 else 17

    # Uncomment the following line to revert to old 'fork' and make it work on 3.8+
    #mp.set_start_method('fork')

    # Fill the queue with numbers from 0 to nElements
    q = mp.Queue()
    for k in range(nElements):
        q.put(k)

    # Start worker processes
    for m in range(nProcesses):
        p = mp.Process(target=getElements, args=(q, tSleep / 1000, m))
        p.start()

Proposed solution is reverting the default of start back to 'fork'.

Change History (2)

comment:1 Changed 4 years ago by chrstphrchvz (Christopher Chavez)

The reporter raised this issue on the original upstream ticket, but was told their suggested reversion is unlikely to be accepted, as well as some issues with their example script; but that upstream might consider a documentation change to help users avoid this issue. The reporter opened a new upstream ticket, but which hasn't been resolved.

As this appears to be either a user code or upstream issue, it might not be something MacPorts will address.

comment:2 Changed 4 years ago by chrstphrchvz (Christopher Chavez)

Cc: chrstphrchvz added
Note: See TracTickets for help on using tickets.