python dies using multiprocessing/urllib2 from mac ports, but works with stock python

[vkuznet (Valentin Kuznetsov)]

Hi, I observed very tricky problem. I have application written in python which uses multiprocessing module. So using the python 2.6 from macports cause python die in a thread when urllib2 module is invoked. To test I installed stock python 2.6.5 from python.org and everything works. I also verify that the problem lies with urllib2/multiprocessing modules, since if I replace urllib2 to httplib, everything works too. I attach apple traceback which shows the problem.

My guess that macports uses either special patch or enable flags which didn't come with stock python source and that cause a problem.

Valentin.

[jmroot (Joshua Root)]

Please remember to fill in the Port field and cc the maintainers.

[jmroot (Joshua Root)]

Looks like _scproxy.so is passing a null pointer to SystemConfiguration. I don't see anything in the patches that looks like it would cause this.

[jmroot (Joshua Root)]

Except SCDynamicStoreCopyProxies is meant to work with a NULL argument, which raises the possibility of this being a bug in SystemConfiguration or CF.

[blb@…]

Do you have a simple example to reproduce this?

[vkuznet (Valentin Kuznetsov)]

I tried to make stand-alone version, but so far I can't reproduce the problem with it. Is it possible to get configuration/patches which went into mac-port. If I can re-apply all configuration steps into the one I download from python.org I can have a look if I can reproduce the problem with manual build of python.

[vkuznet (Valentin Kuznetsov)]

I don't know if it helps, I found that _scproxy.so is linked differently for python from macports and the stock one.

Here is the stock python _scproxy.so

otool -L /Users/vk/Work/Python/Python-2.6.5//install/lib/python2.6/lib-dynload/_scproxy.so
/Users/vk/Work/Python/Python-2.6.5//install/lib/python2.6/lib-dynload/_scproxy.so:
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 293.4.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.19.0)
	/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 246.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.1)

while the macports version is

otool -L $PWD/lib-dynload/_scproxy.so 
/opt/local/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/lib-dynload/_scproxy.so:
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 293.4.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.19.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.1)

Actually all _XXX.so from macport build are missing /usr/lib/libgcc_s.1.dylib link. Also I commented out the

if sys.platform == 'darwin':

...

in urllib.py in macport version and my code works just fine when there is no _scproxy.so objects are loaded, while it crash if _scproxy.so is there.

[blb@…]

The way MacPorts builds python26 (and the patches applied) can be see in the trac browser for the python26 port (patches are in the files/ subdir).

Note that not linking to libgcc_s.1.dylib should be fine since, at least on 10.6, it is just a symlink to libSystem anyway:

$ ls -l /usr/lib/libgcc_s.1.dylib 
lrwxr-xr-x  1 root  wheel  17 Apr 16 18:54 /usr/lib/libgcc_s.1.dylib -> libSystem.B.dylib

[blb@…]

Looking at your crash log, I see

...
       0x1007f2000 -        0x1007f6ff7 +_ssl.so ??? (???) <BC4937DC-E257-4B12-88BA-58678A48F931> /Users/vk/CMS/COMP/DAS/mylib/_ssl.so
...
       0x1012c9000 -        0x101308fff +libssl.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <98F9160D-1D42-9189-CE14-204FDB2B62E7>  /Users/vk/Work/openssl-0.9.8k/lib/libssl.0.9.8.dylib
       0x10131c000 -        0x10143bfe7 +libcrypto.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <4BB4F468-D80A-B3A1-AE43-DB544191E975> /Users/vk/Work/openssl-0.9.8k/lib/libcrypto.0.9.8.dylib

Not sure if that would be causing the crash since it's in SSL and not where the crash is happening but it might be an interesting experiment to see if using the MacPorts-provided SSL module crashes as well.

[vkuznet (Valentin Kuznetsov)]

The openssl module was compiled intentionally, since in our environment we still use renegotiation feature on front-end, which is disabled by default since 0.9.8m release of OpenSSL. In addition, both python (stock one and from mac ports) used my local module, so it should not be a problem.

[scott.moynes@…]

I am also experiencing this bug. Downgrading to python26 version 2.6.4 has corrected it for me.

I will work on a simple reproduction recipe.

[chawco (Craig Hawco)]

Cc Me!

[scott.moynes@…]

Cc Me!

[chawco (Craig Hawco)]

As a workaround setting any environment variable ending with "_proxy" to a non-empty value prevents urllib from calling SystemConfiguration, and thus prevents this particular crash. That does not resolve this central issue.

[jmroot (Joshua Root)]

Does 2.6.6 still have this problem? In any case, it's pretty hard to do anything about it without a test case.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

python26 has been revised several times since this bug was reported.
I was unable to reproduce even with the old version of python26.