Dependency error in ffmpeg 2.1.3_1 on libogg 1.3.1_0

[Polyergic (Shad Sterling)]

After updating with $(sudo port selfupdate) and $(sudo port upgrade outdated), and repeating until "Nothing to upgrade.", $(ffmpeg) gives this output regardless of arguments:

dyld: Library not loaded: /opt/local/lib/libogg.0.dylib
  Referenced from: /opt/local/bin/ffmpeg
  Reason: Incompatible library version: ffmpeg requires version 9.0.0 or later, but libogg.0.dylib provides version 6.0.0
Trace/BPT trap: 5

The output is the same with the current ffmpeg (2.1.3_1) and after reactivating the oldest installed ffmpeg (2.1.1_0). I only have the current libogg installed (1.3.1_0), and the output was not affected by uninstalling libogg with $(port -f uninstall libogg) then reinstalling.

[neverpanic (Clemens Lang)]

What is the output of otool -L /opt/local/lib/libogg.0.dylib? What does env | grep DYLD_ print?

[dbevans (David B. Evans)]

It does appear that you have an older version of libogg installed. The current version 1.3.1_0 (the one ffmpeg was built with) says

$ otool -L /opt/local/lib/libogg.0.dylib
/opt/local/lib/libogg.0.dylib:
	/opt/local/lib/libogg.0.dylib (compatibility version 9.0.0, current version 9.1.0)

which is what ffmpeg is looking for (library version 9.0 or greater).

However, the error message indicates that the version of libogg installed on your machine is only library version 6.0.

Answering cal's questions above would be a start to understanding why that is.

You might also try

port installed libogg

to see what versions of libogg are installed and which is active.

[Polyergic (Shad Sterling)]

polyergic@laconic : ~ $ otool -L /opt/local/lib/libogg.0.dylib
/opt/local/lib/libogg.0.dylib:
	/opt/local/lib/libogg.0.dylib (compatibility version 9.0.0, current version 9.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
polyergic@laconic : ~ $ env | grep DYLD
DYLD_FALLBACK_LIBRARY_PATH=/opt/local/lib:
DYLD_LIBRARY_PATH=/usr/local/xuggler/lib
polyergic@laconic : ~ $ 

This makes no sense; ffmpeg says it has version 6.0.0, but otool says it has version 9.1.0. There is another libogg.0.dylib in $DYLD_LIBRARY_PATH:

polyergic@laconic : ~ $ otool -L $DYLD_LIBRARY_PATH/libogg.0.dylib
/usr/local/xuggler/lib/libogg.0.dylib:
	/usr/local/lib/libogg.0.dylib (compatibility version 6.0.0, current version 6.3.0)
	/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.4)
polyergic@laconic : ~ $ 

So is ffmpeg not falling back to the fallback, and reporting the wrong path for the one it tries to load?

DYLD_LIBRARY_PATH was changed by installing the OSP Tracker (​http://www.cabrillo.edu/~dbrown/tracker/), which doesn't seem to be available as a port. If the fallback path doesn't work, what configuration will be compatible with both at once?

[neverpanic (Clemens Lang)]

DYLD_LIBRARY_PATH on OS X does not behave like LD_LIBRARY_PATH on Linux. Despite this fact, several tools advise setting DYLD_LIBRARY_PATH on OS X in their documentation. This can lead to the problem you are seeing at hand – setting DYLD_LIBRARY_PATH makes the loader prefer any file with the same basename (in this case libogg.0.dylib) from the paths listed in DYLD_LIBRARY_PATH to those at the URL the binary linked to (which is what you see in the output of otool -L /opt/local/bin/ffmpeg).

As a consequence, just must not have DYLD_LIBRARY_PATH set when using ffmpeg. As a precaution, you should only set it when you need to run whatever software requires it. In fact, it is advisable to modify the software requiring the variable in a way that it no longer needs it. You may do so using install_name_tool, by changing the paths referenced by its binaries and libraries to absolute ones referencing the appropriate versions of the libraries in /usr/local/xuggler/lib. You may have to adjust this for the binaries in /usr/local/xuggler/bin, /usr/local/xuggler/sbin and /usr/local/xuggler/lib. If you can, please advise upstream of xuggler (or any other location that gave you the advice to set DYLD_LIBRARY_PATH) to avoid this and instead fix the paths in the binaries.

[Polyergic (Shad Sterling)]

I'm not sure this should be closed yet; ffmpeg is still misreporting the error, saying it's loading a different file than it's actually loading, and not using the fallback.

Do you have a link describing the behavior of DYLD_LIBRARY_PATH? From what you say it sounds like the "right way" is to never define it at all. Does install_name_tool work for all executables or just for macports executables? Unsetting DYLD_LIBRARY_PATH and DYLD_OVERRIDE_LIBRARY_PATH does resolve the error. I didn't set them by hand based on some advice, it was done automatically either by the Tracker installer or an installer for a dependency. If the variable acts as an override, why isn't it DYLD_OVERRIDE_LIBRARY_PATH?

[dbevans (David B. Evans)]

ffmpeg is just trying to load the first thing that it finds. The problem is that because of your system configuration, that is /usr/local/lib/libogg.0.dylib which is an older version than the Macports file at /opt/local/lib/libogg.0.dylib.

We advise not to try and mix MacPorts software with non-Macports software and particularly not software that is installed in /usr/local which can interfere (as in this case). MacPorts software is designed to not need DYLD_LIBRARY_PATH as cal has indicated. If you unset this variable or set it to /opt/local/lib ffmpeg should work fine.

[jeremyhu (Jeremy Huddleston Sequoia)]

I'm not sure this should be closed yet; ffmpeg is still misreporting the error, saying it's loading a different file than it's actually loading, and not using the fallback.

It's not reporting the *FILE* it's reporting the *DYLIB ID*. I understand your confusion, but it is correct.

Do you have a link describing the behavior of DYLD_LIBRARY_PATH

Read the dyld man page.

From what you say it sounds like the "right way" is to never define it at all.

YES! No user should ever define it. No released software should rely on it being set. If you *REALLY* need something like it, use DYLD_FALLBACK_LIBRARY_PATH

Does install_name_tool work for all executables or just for macports executables?

Anything mach-o

Unsetting DYLD_LIBRARY_PATH and DYLD_OVERRIDE_LIBRARY_PATH does resolve the error.

As expected

I didn't set them by hand based on some advice, it was done automatically either by the Tracker installer or an installer for a dependency.

Yes, likely xuggler. (​http://www.xuggle.com ?). Go yell at them.

If the variable acts as an override, why isn't it DYLD_OVERRIDE_LIBRARY_PATH?

It isn't an override. It does what it's documented to do.

[jeremyhu (Jeremy Huddleston Sequoia)]

Replying to devans@…:

ffmpeg is just trying to load the first thing that it finds. The problem is that because of your system configuration, that is /usr/local/lib/libogg.0.dylib which is an older version than the Macports file at /opt/local/lib/libogg.0.dylib.

No, the file is actually /usr/local/xuggler/lib/libogg.0.dylib which has a dylib id of /usr/local/lib/libogg.0.dylib ... yet another issue to report to xuggler (and likely the reason they tried to "fix" the issue with DYLD_LIBRARY_PATH).

[dbevans (David B. Evans)]

True. For discussion of why MacPorts uses /opt/local and /usr/local/whatever is discouraged see the corresponding items in the MacPorts FAQ.

[neverpanic (Clemens Lang)]

It's actually not ffmpeg misreporting this, but dyld, the dynamic loader on OS X. I'm not sure for what reasons it reports the incorrect path, but I suppose it just prints the path referenced from the executed binary (that is, ffmpeg in this case). The same applies to the naming of the variables. Since MacPorts is not in control of the loader, this is not our bug to fix, but Apple's, if any.

The behavior of the DYLD_* variables is explained in the manpage for dyld, i.e. man 1 dyld on your system. There it clearly states: "The dynamic linker searches these directories before it searches the default locations for libraries. It allows you to test new versions of existing libraries." This is appropriate and acceptable use of DYLD_LIBRARY_PATH. Since this is something only developers should be doing, it is an error to use DYLD_LIBRARY_PATH in any deployment situation – so yes, it is the right way to never define it at all, unless you absolutely know what you're doing.

install_name_tool is a tool shipped by Apple that works on all Mach-O (that is, Mac) binaries, so yes, it will also work for non-MacPorts binaries.

Setting DYLD_LIBRARY_PATH in an installer automatically certainly is a bug, considering the fallout this can have on other software (like you just saw).

I closed this bug as invalid, because there's nothing MacPorts can do at this point. We cannot avoid users (or software on behalf of users) setting these environment variables and we cannot detect this situation before the loader does.

[jeremyhu (Jeremy Huddleston Sequoia)]

Replying to cal@…:

It's actually not ffmpeg misreporting this, but dyld, the dynamic loader on OS X. I'm not sure for what reasons it reports the incorrect path, but I suppose it just prints the path referenced from the executed binary (that is, ffmpeg in this case). The same applies to the naming of the variables. Since MacPorts is not in control of the loader, this is not our bug to fix, but Apple's, if any.

It isn't a bug. It is *correctly* reporting the dylib id of the link being resolved. That is often the path name but not always. This is clear when you have a link that is using a relative path rather than an absolute on. For example:

Dyld Error Message:
  Library not loaded: @rpath/My.framework/Versions/A/My
  Referenced from: /Applications/My.app/Contents/MacOS/My
  Reason: image not found

[Polyergic (Shad Sterling)]

Filed Xuggler bug: ​http://code.google.com/p/xuggle/issues/detail?id=311

IME Apple bug reports are useless, so I won't bother.

MacPorts could give a warning when DYLD_LIBRARY_PATH is set. If the library's loaded at runtime, ffmpeg could make the error message less misleading by giving the path and id separately, and noting that dyld does not report the path of the library it loads. It would help if there were a reference for DYLD_LIBRARY_PATH that more clearly stated not just what it does but what it should and should not be used for.

[larryv (Lawrence Velázquez)]

Replying to me@…:

IME Apple bug reports are useless, so I won't bother.

That’s fine, because there is no Apple bug here.

MacPorts could give a warning when DYLD_LIBRARY_PATH is set.

MacPorts is not invoked in any way when you use a port; it only manages dependencies and installation.

It would help if there were a reference for DYLD_LIBRARY_PATH that more clearly stated not just what it does but what it should and should not be used for.

The dyld(1) man page does mention one use case, which is more or less the only use case.

DYLD_LIBRARY_PATH
       This  is  a  colon  separated  list  of directories that contain
       libraries. The dynamic linker searches these directories  before
       it  searches  the default locations for libraries. It allows you
       to test new versions of existing libraries.

Common sense suggests that prepending to the default library search path can easily cause shadowing problems. Imagine a program that added /usr/local/bin/foobar to the beginning of your $PATH; now imagine that that directory contained broken versions of bash, ls, ssh, and so on.

Setting DYLD_LIBRARY_PATH is analogous to prepending to $PATH; setting DYLD_FALLBACK_LIBRARY_PATH is kind of like appending to $PATH.

[Polyergic (Shad Sterling)]

Replying to larryv@…:

Replying to me@…:

IME Apple bug reports are useless, so I won't bother.

That’s fine, because there is no Apple bug here.

Apple's dyld is giving a misleading error message (failing to distinguish the dylib id - the same string as a file path - from an actual file path), which is a bug.

MacPorts could give a warning when DYLD_LIBRARY_PATH is set.

MacPorts is not invoked in any way when you use a port; it only manages dependencies and installation.

And when it does so, it could warn you that your configuration is broken so ports it installs can't be expected to work. When it scans for linking errors it could check against DYLD_LIBRARY_PATH, predict what the misleading error messages will be, and clarify them individually.

[larryv (Lawrence Velázquez)]

Replying to me@…:

Apple's dyld is giving a misleading error message (failing to distinguish the dylib id - the same string as a file path - from an actual file path), which is a bug.

It’s arguably intended behavior and thus would not be considered a bug. Confusing, but not a bug. Again, if the dylib ID is a complete file path that does not correspond to the actual library path or contain a special dyld variable like @rpath, then the library is broken.

And there happens to be an environment variable that prods dyld into exhibiting the behavior you’re expecting:

DYLD_PRINT_LIBRARIES
       When this is set, the dynamic linker writes to file descriptor 2
       (normally  standard  error)  the  filenames of the libraries the
       program is using.  This is useful to make sure that the  use  of
       DYLD_LIBRARY_PATH is getting what you want.

In any case, this is irrelevant because most of us don’t work at Apple and can’t change dyld.

MacPorts is not invoked in any way when you use a port; it only manages dependencies and installation.

And when it does so, it could warn you that your configuration is broken so ports it installs can't be expected to work. When it scans for linking errors it could check against DYLD_LIBRARY_PATH

No, it couldn’t. DYLD_* variables are scrubbed from MacPorts’ environment by sudo(8) as a security measure.

[jeremyhu (Jeremy Huddleston Sequoia)]

Replying to me@…:

Replying to larryv@…:

Replying to me@…:

IME Apple bug reports are useless, so I won't bother.

That’s fine, because there is no Apple bug here.

Apple's dyld is giving a misleading error message (failing to distinguish the dylib id - the same string as a file path - from an actual file path), which is a bug.

Yes, I actually filed an enhancement request yesterday to provide this extra information in dlerror(), but as Larry mentioned, you can use DYLD_PRINT_LIBRARIES to debug issues like this.

The "Library not loaded:" line is correct. The fact that you are confusing the dylib id and the path of the file actually loaded is unfortunate but not really a bug.

The behavior is not a bug.

And when it does so, it could warn you that your configuration is broken so ports it installs can't be expected to work. When it scans for linking errors it could check against DYLD_LIBRARY_PATH, predict what the misleading error messages will be, and clarify them individually.

This isn't a linking error. It's an issue at load time. ffmpeg linked successfully.

Replying to larryv@…:

In any case, this is irrelevant because most of us don’t work at Apple and can’t change dyld.

Note that dyld is OSS, and if you really care about fixing this error message, I'll pass along a patch for you. Otherwise, you'll have to wait until Nick (or I) get some cycles to make the change.

[larryv (Lawrence Velázquez)]

Replying to jeremyhu@…:

Note that dyld is OSS

Oops, I’d forgotten about that. :P

and if you really care about fixing this error message, I'll pass along a patch for you.

Sure, I could update ld64 (I presume? or would it be dyld-headers?) tomorrow.

[neverpanic (Clemens Lang)]

Replying to larryv@…:

Sure, I could update ld64 (I presume? or would it be dyld-headers?) tomorrow.

It's dyld: ​http://opensource.apple.com/source/dyld/dyld-239.3/. Btw, thanks to Apple for open sourcing this, it turned out to be a huge help while I was writing rev-upgrade.

[larryv (Lawrence Velázquez)]

Replying to cal@…:

It's dyld: ​http://opensource.apple.com/source/dyld/dyld-239.3/.

Right, but I assumed Jeremy was talking about updating some port. I guess he meant manually compiling a patched dyld.