Dependencies fail to install because hostnames cannot be resolved because we've run out of file descriptors

[benedikt.hielscher@…]

I'm trying to install kdenlive, but the installation always fails because dependencies cannot be installed.

$ sudo port install kdenlive

Password:
--->  Computing dependencies for kdenlive
--->  Dependencies to be installed: automoc qt4-mac cdrtools smake cmake curl curl-ca-bundle libarchive libxml2 lzo2 dvdauthor ImageMagick autoconf automake djvulibre fftw-3 fontconfig freetype ghostscript jbig2dec libidn libpaper libtool perl5 perl5.22 gdbm xorg-libXext xorg-libX11 xorg-kbproto xorg-libXau xorg-xproto xorg-libXdmcp xorg-libxcb python27 db48 python2_select python_select readline xorg-libpthread-stubs xorg-xcb-proto xorg-xextproto xorg-libXt xorg-libsm xorg-libice jbigkit openexr ilmbase openjpeg urw-fonts webp libdvdread libdvdcss kde4-runtime attica phonon bison bison-runtime m4 exiv2 gpgme gnupg2 gpg-agent libassuan libgpg-error pth libgcrypt libksba pinentry-mac libusb-compat libusb openldap cyrus-sasl2 kerberos5 libcomerr db46 icu tcp_wrappers hicolor-icon-theme kactivities kdelibs4 aspell texinfo aspell-dict-en avahi dbus-python27 dbus-glib glib2 libffi gobject-introspection cairo libpixman xorg-xcb-util xrender xorg-renderproto py27-mako py27-beaker py27-setuptools py27-markupsafe gtk3 at-spi2-atk at-spi2-core xorg-libXi xorg-inputproto xorg-libXfixes xorg-fixesproto xorg-libXtst xorg-recordproto atk gdk-pixbuf2 jasper shared-mime-info libepoxy python34 python3_select mesa xorg-dri2proto xorg-glproto xorg-libXdamage xorg-damageproto xorg-libXmu xorg-libXxf86vm xorg-xf86vidmodeproto pango Xft2 harfbuzz graphite2 xorg-libXcomposite xorg-compositeproto xorg-libXcursor xorg-libXinerama xorg-xineramaproto xorg-libXrandr xorg-randrproto libdaemon libglade2 gtk2 py27-gdbm py27-pygtk py27-cairo py27-gobject py27-numpy libgcc cctools llvm-3.7 libcxx llvm_select gmp isl ld64 ld64-latest libmpc mpfr py27-cython cython_select py27-nose nosetests_select dbusmenu-qt qjson docbook-xml docbook-xml-4.1.2 docbook-xml-4.2 xmlcatmgr docbook-xml-4.3 docbook-xml-4.4 docbook-xml-4.5 docbook-xml-5.0 docbook-xsl enchant giflib grantlee libxslt qca shared-desktop-ontologies docbook-xsl-ns soprano libiodbc raptor2 redland rasqal mhash strigi boost clucene ffmpeg XviD fribidi gnutls libtasn1 nettle p11-kit desktop-file-utils popt libass libbluray libmodplug libsdl libtheora libvpx openjpeg15 schroedinger orc soxr speex x264 yasm virtuoso virtuoso-7 gawk nepomuk-core ebook-tools convertlit libtommath libzip kfilemetadata mobipocket poppler-qt4-mac git p5.22-authen-sasl p5.22-digest-hmac p5.22-digest-sha1 p5.22-gssapi p5.22-cgi p5.22-html-parser p5.22-html-tagset p5.22-error p5.22-net-smtp-ssl p5.22-io-socket-ssl p5.22-net-libidn p5.22-net-ssleay p5.22-test-exception p5.22-sub-uplevel p5.22-test-simple p5.22-test-nowarnings p5.22-test-warn p5.22-term-readkey rsync poppler poppler-data taglib kdepimlibs4 akonadi mariadb-server mariadb mysql_select qt4-mac-mariadb-plugin libical prison libdmtx qrencode libssh openslp oxygen-icons qimageblitz samba3 mlt doxygen flex frei0r-plugins libexif libsamplerate libsdl_image
--->  Fetching archive for qt4-mac
--->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from https://packages.macports.org/qt4-mac
--->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://fco.it.packages.macports.org/mirrors/macports-packages/qt4-mac/qt4-mac
--->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://her.gr.packages.macports.org/qt4-mac
--->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://jnb.za.packages.macports.org/packages/qt4-mac
--->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://lil.fr.packages.macports.org/qt4-mac
--->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://nou.nc.packages.macports.org/pub/macports/packages.macports.org/qt4-mac
--->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://nue.de.packages.macports.org/qt4-mac
--->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://mse.uk.packages.macports.org/sites/packages.macports.org/qt4-mac
--->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://osl.no.packages.macports.org/qt4-mac
--->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://sea.us.packages.macports.org/macports/packages/qt4-mac
--->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://jog.id.packages.macports.org/macports/packages/qt4-mac
Error: org.macports.archivefetch for port qt4-mac returned: archivefetch failed for qt4-mac @4.8.7_4
Error: Failed to install qt4-mac
Please see the log file for port qt4-mac for details:
    /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_aqua_qt4-mac/qt4-mac/main.log
Error: The following dependencies were not installed: automoc qt4-mac cdrtools smake cmake curl curl-ca-bundle libarchive libxml2 lzo2 dvdauthor ImageMagick autoconf automake djvulibre fftw-3 fontconfig freetype ghostscript jbig2dec libidn libpaper libtool perl5 perl5.22 gdbm xorg-libXext xorg-libX11 xorg-kbproto xorg-libXau xorg-xproto xorg-libXdmcp xorg-libxcb python27 db48 python2_select python_select readline xorg-libpthread-stubs xorg-xcb-proto xorg-xextproto xorg-libXt xorg-libsm xorg-libice jbigkit openexr ilmbase openjpeg urw-fonts webp libdvdread libdvdcss kde4-runtime attica phonon bison bison-runtime m4 exiv2 gpgme gnupg2 gpg-agent libassuan libgpg-error pth libgcrypt libksba pinentry-mac libusb-compat libusb openldap cyrus-sasl2 kerberos5 libcomerr db46 icu tcp_wrappers hicolor-icon-theme kactivities kdelibs4 aspell texinfo aspell-dict-en avahi dbus-python27 dbus-glib glib2 libffi gobject-introspection cairo libpixman xorg-xcb-util xrender xorg-renderproto py27-mako py27-beaker py27-setuptools py27-markupsafe gtk3 at-spi2-atk at-spi2-core xorg-libXi xorg-inputproto xorg-libXfixes xorg-fixesproto xorg-libXtst xorg-recordproto atk gdk-pixbuf2 jasper shared-mime-info libepoxy python34 python3_select mesa xorg-dri2proto xorg-glproto xorg-libXdamage xorg-damageproto xorg-libXmu xorg-libXxf86vm xorg-xf86vidmodeproto pango Xft2 harfbuzz graphite2 xorg-libXcomposite xorg-compositeproto xorg-libXcursor xorg-libXinerama xorg-xineramaproto xorg-libXrandr xorg-randrproto libdaemon libglade2 gtk2 py27-gdbm py27-pygtk py27-cairo py27-gobject py27-numpy libgcc cctools llvm-3.7 libcxx llvm_select gmp isl ld64 ld64-latest libmpc mpfr py27-cython cython_select py27-nose nosetests_select dbusmenu-qt qjson docbook-xml docbook-xml-4.1.2 docbook-xml-4.2 xmlcatmgr docbook-xml-4.3 docbook-xml-4.4 docbook-xml-4.5 docbook-xml-5.0 docbook-xsl enchant giflib grantlee libxslt qca shared-desktop-ontologies docbook-xsl-ns soprano libiodbc raptor2 redland rasqal mhash strigi boost clucene ffmpeg XviD fribidi gnutls libtasn1 nettle p11-kit desktop-file-utils popt libass libbluray libmodplug libsdl libtheora libvpx openjpeg15 schroedinger orc soxr speex x264 yasm virtuoso virtuoso-7 gawk nepomuk-core ebook-tools convertlit libtommath libzip kfilemetadata mobipocket poppler-qt4-mac git p5.22-authen-sasl p5.22-digest-hmac p5.22-digest-sha1 p5.22-gssapi p5.22-cgi p5.22-html-parser p5.22-html-tagset p5.22-error p5.22-net-smtp-ssl p5.22-io-socket-ssl p5.22-net-libidn p5.22-net-ssleay p5.22-test-exception p5.22-sub-uplevel p5.22-test-simple p5.22-test-nowarnings p5.22-test-warn p5.22-term-readkey rsync poppler poppler-data taglib kdepimlibs4 akonadi mariadb-server mariadb mysql_select qt4-mac-mariadb-plugin libical prison libdmtx qrencode libssh openslp oxygen-icons qimageblitz samba3 mlt doxygen flex frei0r-plugins libexif libsamplerate libsdl_image
To report a bug, follow the instructions in the guide:
    http://guide.macports.org/#project.tickets
Error: Processing of port kdenlive failed
watwatmac:~ Bene$ open /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_aqua_qt4-mac/qt4-mac/main.log

The log file is here:

version:1
:debug:main changing euid/egid - current euid: 0 - current egid: 0
:debug:main egid changed to: 20
:debug:main euid changed to: 502
:debug:main Executing org.macports.main (qt4-mac)
:debug:main Privilege de-escalation not attempted as not running as root.
:debug:archivefetch archivefetch phase started at Tue Mar 29 11:21:22 EEST 2016
:msg:archivefetch --->  Fetching archive for qt4-mac
:debug:archivefetch Executing org.macports.archivefetch (qt4-mac)
:debug:archivefetch euid/egid changed to: 0/0
:debug:archivefetch chowned /opt/local/var/macports/incoming to macports
:debug:archivefetch euid/egid changed to: 502/20
:info:archivefetch --->  qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 doesn't seem to exist in /opt/local/var/macports/incoming/verified
:debug:archivefetch Spawning ping for packages.macports.org failed
:debug:archivefetch Spawning ping for fco.it.packages.macports.org failed
:debug:archivefetch Spawning ping for her.gr.packages.macports.org failed
:debug:archivefetch Spawning ping for jnb.za.packages.macports.org failed
:debug:archivefetch Spawning ping for lil.fr.packages.macports.org failed
:debug:archivefetch Spawning ping for nou.nc.packages.macports.org failed
:debug:archivefetch Spawning ping for nue.de.packages.macports.org failed
:debug:archivefetch Spawning ping for mse.uk.packages.macports.org failed
:debug:archivefetch Spawning ping for osl.no.packages.macports.org failed
:debug:archivefetch Spawning ping for sea.us.packages.macports.org failed
:debug:archivefetch Spawning ping for jog.id.packages.macports.org failed
:msg:archivefetch --->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from https://packages.macports.org/qt4-mac
:debug:archivefetch Fetching archive failed:: Could not resolve host: packages.macports.org
:msg:archivefetch --->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://fco.it.packages.macports.org/mirrors/macports-packages/qt4-mac/qt4-mac
:debug:archivefetch Fetching archive failed:: Could not resolve host: fco.it.packages.macports.org
:msg:archivefetch --->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://her.gr.packages.macports.org/qt4-mac
:debug:archivefetch Fetching archive failed:: Could not resolve host: her.gr.packages.macports.org
:msg:archivefetch --->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://jnb.za.packages.macports.org/packages/qt4-mac
:debug:archivefetch Fetching archive failed:: Could not resolve host: jnb.za.packages.macports.org
:msg:archivefetch --->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://lil.fr.packages.macports.org/qt4-mac
:debug:archivefetch Fetching archive failed:: Could not resolve host: lil.fr.packages.macports.org
:msg:archivefetch --->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://nou.nc.packages.macports.org/pub/macports/packages.macports.org/qt4-mac
:debug:archivefetch Fetching archive failed:: Could not resolve host: nou.nc.packages.macports.org
:msg:archivefetch --->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://nue.de.packages.macports.org/qt4-mac
:debug:archivefetch Fetching archive failed:: Could not resolve host: nue.de.packages.macports.org
:msg:archivefetch --->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://mse.uk.packages.macports.org/sites/packages.macports.org/qt4-mac
:debug:archivefetch Fetching archive failed:: Could not resolve host: mse.uk.packages.macports.org
:msg:archivefetch --->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://osl.no.packages.macports.org/qt4-mac
:debug:archivefetch Fetching archive failed:: Could not resolve host: osl.no.packages.macports.org
:msg:archivefetch --->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://sea.us.packages.macports.org/macports/packages/qt4-mac
:debug:archivefetch Fetching archive failed:: Could not resolve host: sea.us.packages.macports.org
:msg:archivefetch --->  Attempting to fetch qt4-mac-4.8.7_4.darwin_14.x86_64.tbz2 from http://jog.id.packages.macports.org/macports/packages/qt4-mac
:debug:archivefetch Fetching archive failed:: Could not resolve host: jog.id.packages.macports.org
:error:archivefetch org.macports.archivefetch for port qt4-mac returned: archivefetch failed for qt4-mac @4.8.7_4
:debug:archivefetch Error code: NONE
:debug:archivefetch Backtrace: archivefetch failed for qt4-mac @4.8.7_4
    while executing
"portarchivefetch::fetchfiles"
    (procedure "portarchivefetch::archivefetch_main" line 5)
    invoked from within
"portarchivefetch::archivefetch_main org.macports.archivefetch"
    ("eval" body line 1)
    invoked from within
"eval $procedure $targetname"
:info:archivefetch Warning: targets not executed for qt4-mac: org.macports.activate org.macports.archivefetch org.macports.fetch org.macports.checksum org.macports.extract org.macports.patch org.macports.configure org.macports.build org.macports.destroot org.macports.install
:error:archivefetch Failed to install qt4-mac
:debug:archivefetch Registry error: libsdl_image not registered as installed & active.
    invoked from within
"registry_active $subport"
    invoked from within
"$workername eval registry_active \$subport"
:notice:archivefetch Please see the log file for port qt4-mac for details:
    /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_aqua_qt4-mac/qt4-mac/main.log
 

Now what is interesting is that I can easily install the port manually with

$ sudo port install qt4-mac

I had this problems with various other dependencies before, e.g. dubs, jpeg, tiff, sox, lcms2, vlmc etc pp. I alway was able to install the manually, but not automatically. I turned off the firewall, which didn't help.

[ryandesign (Ryan Carsten Schmidt)]

Replying to benedikt.hielscher@…:

:debug:archivefetch Spawning ping for packages.macports.org failed

Why did spawning ping fail? Are you able to use the ping command manually on the command line?

:debug:archivefetch Fetching archive failed:: Could not resolve host: packages.macports.org

Why did resolving the hostname fail? Are you able to access that hostname in a web browser?

[benedikt.hielscher@…]

Replying to ryandesign@…:

Replying to benedikt.hielscher@…:

:debug:archivefetch Spawning ping for packages.macports.org failed

Why did spawning ping fail? Are you able to use the ping command manually on the command line?

:debug:archivefetch Fetching archive failed:: Could not resolve host: packages.macports.org

Why did resolving the hostname fail? Are you able to access that hostname in a web browser?

I'm able to use the ping command on google.com and other hosts, I'm not able to use it on packages.macports.org

I'm able to access packages.macports.org in a web browser though.

[ryandesign (Ryan Carsten Schmidt)]

I can't explain this.

[ryandesign (Ryan Carsten Schmidt)]

A similar problem was reported on the mailing list:

​https://lists.macosforge.org/pipermail/macports-users/2016-April/040851.html

There, I suggested that we might be opening one (or more) file descriptor(s) for each dependency to be installed, and then by the time we want to fetch the files, we don't have any remaining file descriptors to do so.

The workaround suggested is to manually install one (or more) of the dependencies (listed on the "Dependencies to be installed " line) first.

[jmroot (Joshua Root)]

We could certainly add a setrlimit call to raise RLIMIT_NOFILE, if that's the problem. There's actually a code snippet doing this in tracelib already.

Reporter, have you tried fiddling with ulimit -n before running port? The default is 256, so maybe raise it to something like 1024.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #51320.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #51402.

[wlwood3@…]

I just wanted to say that the tips here worked for me in installing kmymoney4. It does not appear that I am able to increase the ulimit number, so I did have to break the large kmymoney list of dependencies down into smaller chunks. I thought that would mean an extremely tedious process of individually installing more than 100 packages, but the "port deps kmymoney" command gave me a list of only about a dozen. I installed each of those individually, then installed kmymoney4 and it works perfectly.

Thanks for the help.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #51511.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #51526.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #51573.

[mf2k (Frank Schima)]

Has duplicate #51672.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #51979.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #52139.

[larryv (Lawrence Velázquez)]

Cc Me!

[pmetzger (Perry E. Metzger)]

Cc Me!

[pmetzger (Perry E. Metzger)]

Just had this happen to me when trying to re-install all my ports from scratch for Sierra. Very, very annoying. It happened only for a port with a huge number of dependencies (claws-mail), and I got around it by manually installing dependencies until the file descriptor limit issue was gone.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #50791.

[aque (Allan Que)]

Cc Me!

[aque (Allan Que)]

I ran into this issue when restoring my ports under Sierra. ulimit -n 1024 got me going again.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #52487.

[grigutis@…]

Cc Me!

[nortcele]

Cc Me!

[mkae (Marko Käning)]

Cc Me!

[mkae (Marko Käning)]

I've never seen this error so far. Is this only for OSX >= El Capitan ?

[ryandesign (Ryan Carsten Schmidt)]

Yes.

[Russell-Jones-OxPhys (Russell Jones)]

Cc Me!

[mf2k (Frank Schima)]

Has duplicate #51294.

[jmroot (Joshua Root)]

In b856b495/macports-base:

Always maximise RLIMIT_NOFILE

This should at least work around problems with running out of fds.
Code calling setrlimit has been moved from one place in tracelib.c and
put in its own command, which is called from mportinit.

See: #50986

[jmroot (Joshua Root)]

As per the above, there's now a workaround in master that should hopefully do the trick. I'd still very much like to know what's using up all our fds.

[jmroot (Joshua Root)]

Doing a dry-run of a kdenlive install I see about 140 of these from lsof -c tclsh8.5:

tclsh8.5 13379 root  148u  IPv4 0xcb9ddeb2ae2c8ec3       0t0      TCP 10.0.0.4:49214->196-124-232-198.static.unitasglobal.net:https (CLOSE_WAIT)

198.232.124.196 is what packages.macports.org and distfiles.macports.org resolve to for me.

[jmroot (Joshua Root)]

In 0fd195f/macports-base:

Stop stale curl handles persisting

Local variables in 3 of the curl subcommands should have been removed in
99d934c which switched to reusing one global handle for all connections. As a
result, new handles were still created for each invocation but never cleaned
up.

Fixes: #50986