py37-openslide @1.1.2_1: OSError: dlopen(libopenslide.0.dylib, 6): image not found

[DavidBAEpstein]

virtual environment created with

conda create -n openslide_bug python=3.7

python command import openslide fails

Attaching files with descriptive names

MacBook Pro (Retina, 15-inch, Mid 2015)
Macos 11.5.2

[ryandesign (Ryan Carsten Schmidt)]

The error shown in the first attached file is:

OSError: dlopen(libopenslide.0.dylib, 6): image not found

Does /opt/local/lib/libopenslide.0.dylib exist? If so, what's the output of otool -L /opt/local/lib/libopenslide.0.dylib? What I'm looking for is whether the second line of output lists the full library path as it should (/opt/local/lib/libopenslide.0.dylib) or just its name (libopenslide.0.dylib). Our binary packages of openslide look like they have the correct full library path.

dlopen probably needs a full path to the library file to work, but the error message only listed the library name. Looking closer at the error message, I see that the problem is probably here:

  File "/Users/dbae/opt/anaconda3/envs/openslide_bug/lib/python3.7/site-packages/openslide/lowlevel.py", line 46, in <module>
    _lib = cdll.LoadLibrary('libopenslide.0.dylib')

Looks like py-openslide's code explicitly calls cdll.LoadLibrary('libopenslide.0.dylib'). Presumably that will need to be changed to use the absolute library path, i.e. cdll.LoadLibrary('/opt/local/lib/libopenslide.0.dylib').

The code in lowlevel.py reads:

elif platform.system() == 'Darwin':
    try:
        _lib = cdll.LoadLibrary('libopenslide.0.dylib')
    except OSError:
        # MacPorts doesn't add itself to the dyld search path, but
        # does add itself to the find_library() search path
        # (DEFAULT_LIBRARY_FALLBACK in ctypes.macholib.dyld) on
        # Python 2.6 and 2.7.  Python 3 users on MacPorts should add
        # the MacPorts lib directory to DYLD_LIBRARY_PATH.
        import ctypes.util
        _lib = ctypes.util.find_library('openslide')
        if _lib is None:
            raise ImportError("Couldn't locate OpenSlide dylib.  " +
                    "Is OpenSlide installed?")
        _lib = cdll.LoadLibrary(_lib)

The advice in those comments seems terrible to me: users should most certainly not add the MacPorts lib directory to DYLD_LIBRARY_PATH; that will break immense numbers of things. The developers of py-openslide should fix their software so that that's not necessary.

[jmroot (Joshua Root)]

Disregarding the comment, the technique in the fallback case should normally work (we do add ${prefix}/lib to DEFAULT_LIBRARY_FALLBACK in python3x as well):

>>> import ctypes.util
>>> _lib = ctypes.util.find_library('png')
>>> print(_lib)
/opt/local/lib/libpng.dylib

Having DYLD_FALLBACK_LIBRARY_PATH set in the environment would be one way to make that not work.

[DavidBAEpstein]

Since I'm anxiously waiting for a fix, I hope the above discussion is homing in on the answer.

It would be great if Ryan Schmidt's suggestion works, perhaps as a strictly temporary fix, namely to replace

cdll.LoadLibrary('libopenslide.0.dylib')

by

cdll.LoadLibrary('/opt/local/lib/libopenslide.0.dylib')

I'm concerned that Ryan's rewriting suggestion might take much longer.

In my original ticket, I forgot to say that my Xcode is version 12.5.1, though I can't be sure that I ported py-openslide using that version. In any case the version of Xcode doesn't now seem to be relevant.

[reneeotten (Renee Otten)]

@DavidBAEpstein - why are you using conda and MacPorts at the same time, are you sure that that isn't causing the confusion here? What hapepens if you use only the MacPorts provided packages and try this with /opt/local/bin/python3.7?

[DavidBAEpstein]

@reneeotten Try reading what Ryan Schmidt and Joshua Root say about my ticket. They are impressive experts, not like you and me. Conda and Macports do not interfere with each other. As far as I know, they are not able to perform overlapping tasks, nor the same tasks.

[reneeotten (Renee Otten)]

Replying to DavidBAEpstein:

@reneeotten Try reading what Ryan Schmidt and Joshua Root say about my ticket. They are impressive experts, not like you and me. Conda and Macports do not interfere with each other. As far as I know, they are not able to perform overlapping tasks, nor the same tasks.

They certainly are, and I am perfectly fine if you don't want to take my advice. However, the Traceback you show tells you that the problem is with a Python package you have installed through conda and so this has likely nothing to do with MacPorts. We have seen many times that there sometimes are detrimental side-effects of using MacPorts / conda simultaneously, and the usual advice is to pick one package manager and stick with it.

So if you want to try my suggestion and show that the problem is actually with MacPorts please do - if that's the case we might be able to do something about it. If not, that's okay with me - it's you having a problem not me ;)

[DavidBAEpstein]

Replying to Renee Otten

I have learned some interesting things that I hadn't realized from your comments. So thanks for the comments.

Here is what happened: Outside any virtual environment Macports' python3.7 accepts "import openslide" without demur. This is python 3.7.12, rather than my virtual environment's python 3.7.11. But I don't suppose that makes any difference.

I ran recursive diff on the two site packages for openslide, one from the conda virtual environment, and one from /opt/local, that is from Macports. Yes, these do differ, but only in the binary files. I suppose this is caused by the use of different compilers.

The upshot is that I still don't understand what is going on. Benjamin Gilbert, the owner of the port on Macports, has written to me that the virtual environment may be the cause of the problem, which is also what you think. But I don't think he believes that the problem is the use of conda with Macports. I believe (without evidence) that, if I were to use Brew, the problem would go away.

Of course, this leaves me in exactly the same position I was in before, but with a little more knowledge. I am working as a small cog in a large software project, so a virtual environment is essential for me. The virtual environment I created to investigate the bug is the smallest possible to display the bug, and therefore (I thought) the least likely to cause a problem.

[reneeotten (Renee Otten)]

Replying to DavidBAEpstein:

Here is what happened: Outside any virtual environment Macports' python3.7 accepts "import openslide" without demur. This is python 3.7.12, rather than my virtual environment's python 3.7.11. But I don't suppose that makes any difference.

I ran recursive diff on the two site packages for openslide, one from the conda virtual environment, and one from /opt/local, that is from Macports. Yes, these do differ, but only in the binary files. I suppose this is caused by the use of different compilers.

I agree that the Python version does not make a difference and the fact that it works tells us that there isn't anything wrong with the MacPorts provided py-openslide port.

The upshot is that I still don't understand what is going on. Benjamin Gilbert, the owner of the port on Macports, has written to me that the virtual environment may be the cause of the problem, which is also what you think. But I don't think he believes that the problem is the use of conda with Macports. I believe (without evidence) that, if I were to use Brew, the problem would go away.

It's quite clear what is going on I think (based on your description and log files): you installed conda and created an (virtual) environment with it, then you installed openslide using pip, and then it tries to use the openslide library provided by MacPorts. So that's combining three different package managers, which is honestly asking for this kind of trouble. Of course, I am not saying that this can never work and it might be possible to work around the issues... but it's likely going to be hard. Additionally, I wouldn't consider really a MacPorts problem so don't know if anyone here would do it. It is possible that it would work out-of-the-box with brew since they install files in a different location and the library might be found there "automatically"; this is pure speculation of course and you don't seem to have investigated that either.

Of course, this leaves me in exactly the same position I was in before, but with a little more knowledge. I am working as a small cog in a large software project, so a virtual environment is essential for me. The virtual environment I created to investigate the bug is the smallest possible to display the bug, and therefore (I thought) the least likely to cause a problem.

I think the following options should work (please note, not tested):

(1) use MacPorts and pip:

• install openslide and pyXY-virtualenv with MacPorts (XY could be 37 for Python 3.7 or any other version you would like)
• create a virtual environment (virtualenv-X.Y /path_to_where_you_want_to_create_it && source /path_to_where_you_want_to_create_it/bin/activate)
• install the packages with pip: pip install python-openslide

(2) use only Anaconda/miniconda:

• create an environment with conda
• install the required packages with conda (and *not* pip), for example from the bioconda channel: conda install openslide-python openslide --channel bioconda

In both cases you should get a self-consistent set of packages and hopefully that will get you back in a working situation.

[DavidBAEpstein]

Replying to Renee Otten

That's extremely helpful, and looks promising. I will try that and report back, either way. I forget how I installed everything, but I certainly did use a mixture of conda, pip and port, without an inkling that this could cause problems.

...some time later...

My problem has now gone away. I found a conda requirements file on our Github repo, removed my old unsuccessful environment and created the new environment, using "conda create" with the -f flag for the requirements file. This installs a conda version of pip, so that pip in that environment works consistently with conda. I think that conda on its own cannot install the openslide binaries from C-code directly, and this has to be done by pip (but one must use the conda version of pip). Incidentally, although "import openslide" now works, the diff -r between the openslide packages on /opt/local and on ~/anaconda3, continues to give an identical answer to what I saw before, so my previous "test" using "diff -r" was misleading. I think all the binary files differ for both the failing environment and the new successful one. All the other files (python programs) are identical.

Your explanation above is excellent and I hope it will avoid others getting into the same pickle (at least in the case of openslide). You have pointed out that this has occurred from time to time with other ports. What do you think about you adding to the Macports guide(s) a short section with your wisdom about Macports and virtual environments, and associated hazards that must be avoided?

[reneeotten (Renee Otten)]

As I said above, I haven't tested it - but the "bioconda" channel provies both the openslide and ppenslide-python packages so that should install it just fine. If you prefer to stay with "conda" - which is a great software package btw - I would really recommend to install everything through conda install and stay away from using pip even if that is installed through conda. By only using conda install you can be sure that it knows about all Python packages and can make sure that it will result in a consistent installation. Last time I checked (but that might have changed) is that installing stuff through pip in your conda environment does not make conda "aware" of it and you run the risk of having similar issues you reported initially.

[bgilbert (Benjamin Gilbert)]

As background info, there are two components here: OpenSlide is a C library, and OpenSlide Python is a Python package that uses it. MacPorts has both, conda has both, and pip only has OpenSlide Python. import openslide invokes OpenSlide Python and causes it to look for OpenSlide, which must be in a directory where Python will find it. That's the problem you've encountered. MacPorts' Python will look in the MacPorts library directory for OpenSlide, while Conda's Python presumably won't.

I agree that it's safer to stick with one distribution, rather than mixing OpenSlide from MacPorts and OpenSlide Python from conda. A mix might be made to work if your Python program only uses OpenSlide, since OpenSlide and Python may not have any library dependencies in common. But the more libraries your Python program uses, the more likely it is that you'll inadvertently try to load, say, both the MacPorts and conda versions of the JPEG library into your program.

@DavidBAEpstein, I think it should be okay to install OpenSlide Python with pip, so long as that pip matches the distribution you got OpenSlide from -- either MacPorts or conda. I'm glad you were able to get it working.

@ryandesign, for the record, the comment text you quoted was dropped from that file ​a while ago. It was originally written before the Python 3 ports ​started setting DEFAULT_LIBRARY_FALLBACK.

[jmroot (Joshua Root)]

Replying to reneeotten:

However, the Traceback you show tells you that the problem is with a Python package you have installed through conda and so this has likely nothing to do with MacPorts.

Good catch BTW. I admit I didn't actually look at the backtrace and only skimmed the part of it that Ryan quoted.