Python statup time much slower then standard install (hardlinks?)

[nirs@…]

Python 2.4.1 installed by darwinports startup almost 3 times slower then Python built from source and installed at /usr/local.

$ time /usr/local/bin/python -c "print 'done'" done

real 0m0.046s user 0m0.050s sys 0m0.010s

$ time /opt/local/bin/python -c "print 'done'" done

real 0m0.114s user 0m0.020s sys 0m0.080s

This can have bad effect on CGI scripts. for example, this minimal CGI:

#!/path/to/python print """content-type: text/plain

hi """

Can run at 45 request per second with standard Python, but only 14 request per second when running with darwinports Python.

Here is a more "real" example. Most cgi need to import cgi and use a FieldStorage:

$ time /usr/local/bin/python -c "import cgi; args = cgi.FieldStorage(); print repr(args)" FieldStorage(None, None, [])

real 0m0.156s user 0m0.120s sys 0m0.050s

$ time /opt/local/bin/python -c "import cgi; args = cgi.FieldStorage(); print repr(args)" FieldStorage(None, None, [])

real 0m0.320s user 0m0.070s sys 0m0.260s

I suspect that this slowness is caused by using hardlinks.

[nirs@…]

More details: this python was installed by 'port install mod_python', which build python with some dynlib target.

[jberry@…]

Is this a Panther system? Hardlinks are truly broken on Panther, and can eventually lead you to a system that is completely unusable. I just spent the day yesterday replacing a server that had turned into a block of tar, due to a daemon that used hardlinks as a regular part of it's operation.

This system bug has been fixed, apparently, on Tiger.

[nirs@…]

Yea it is a Panther. The problem is Tiger has its own brokeness.

[nirs@…]

I installed python again after cleaning:

# port clean --all python24 # port install -u python24 ---> Fetching python24 ---> Attempting to fetch Python-2.4.1.tar.bz2 from ​http://www.python.org//ftp/python/2.4.1/ ---> Verifying checksum(s) for python24 ---> Extracting python24 ---> Applying patches to python24 ---> Configuring python24 ---> Building python24 with target all libpython.2.4.1.dylib ---> Staging python24 into destroot ---> Packaging tgz archive for python24 2.4.1_5 ---> Installing python24 2.4.1_5 ---> Activating python24 2.4.1_5 ---> Cleaning python24

And I get the same results.

[blb@…]

Nir, if this is truly a problem with hardlinks, simply reinstalling python alone won't help. You'll need to switch your DP installation from image to direct mode (by changing 'portinstalltype' in /opt/local/etc/ ports/ports.conf from 'image' to 'direct'), then redo all your installed ports. This switches DP to direct mode (you can no longer have active and inactive ports) but will get away from the hardlink issue.

[nirs@…]

Thanks! I did not know there is such option. It little work to remove and install back all, I wish there was recursive uninstall.

[mww@…]

I dont know if this is the problem - I'm using direct mode and have approximately the same startup-time as Nir. I've already tried to make a "real" copy of python2.4 and execute that, but that wont make a difference either. The only thing I can think of is that dp's python has a lot of auto-load modules available, while the self- installed has not (e. g. readline) and that loading those modules takes a short while.

[mww@…]

ok, I've just took a look at "python -v -v" to see what it is loading: # trying readline.so # trying readlinemodule.so # trying readline.py # trying readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python24.zip/readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python24.zip/ readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python24.zip/readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python24.zip/readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-darwin/ readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-darwin/ readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-darwin/ readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-darwin/ readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/ readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/ readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/ readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/ readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/lib- scriptpackages/readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/lib- scriptpackages/readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/lib- scriptpackages/readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/lib- scriptpackages/readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/lib-tk/ readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/lib-tk/ readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/lib-tk/ readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/lib-tk/ readline.pyc # trying /opt/local/lib/python2.4/lib-dynload/readline.so import readline # dynamically loaded from /opt/local/lib/python2.4/lib-dynload/readline.so --- I assume that the framework build looks into a lot more locations where to find its modules and takes a while to find readline. I suppose you didnt do a framework install but a plain unix one?

[nirs@…]

I did a plain ./cofigure; make install.

[mww@…]

btw. on 10.4: time /usr/bin/python -c "print 'done'" done

real 0m0.332s user 0m0.184s sys 0m0.084s

[nirs@…]

That time suck :) But it depends on the machine, mine timed on G5 Dual 2G.

[nirs@…]

It must be the hardlinks.

I uninstalled all ports, changed the install type to 'direct', and installed all back. Now there is only a tiny difference between standard python and dp python.

For example:

$ time /usr/local/bin/python2.4 -c 'import cgi; args = cgi.FieldStorage(); print repr(args)' FieldStorage(None, None, [])

real 0m0.154s user 0m0.080s sys 0m0.070s

$ time /opt/local/bin/python2.4 -c 'import cgi; args = cgi.FieldStorage(); print repr(args)' FieldStorage(None, None, [])

real 0m0.164s user 0m0.100s sys 0m0.070s

$ time /usr/local/bin/python2.4 -c 'print "done"' done

real 0m0.047s user 0m0.020s sys 0m0.020s

$ time /opt/local/bin/python2.4 -c 'print "done"' done

real 0m0.050s user 0m0.020s sys 0m0.040s

And in the minimal cgi test, 47 request per second with standard python vs 42 with dp python.

For most users this difference does not matter, maybe if you use it for to serve cgis, the little difference is worth the more trouble in installing python, apache 2, mod_python, mod_fastcgi etc manually. I never managed to install mod_python myself :)

[mww@…]

(In reply to comment #12)

It must be the hardlinks.

no, its the framework build - please compare your /usr/bin/python to dp's python and you'll see that dp's python is equally fast/slow (about factor 2 slower on my albook);

[nirs@…]

Look again at the results of same dp python using framework install, once when using 'image' install type, and then 'direct':

test image direct

---

minimal script 0.114s 0.050s script using cgi.FieldStorage 0.320s 0.164s minimal cgi (request per seconds) 14 4

Using direct give at least 2x improvement, practically same as non framework build of python.

[markd@…]

Any objections to marking this WONTFIX?