Resolving libjpeg-turbo + jpeg Port conflicts

[davidfavor (David Favor)]

Since libjpeg-turbo released there are many Ports that could benefit from it's use, except dependencies in these Ports are for the jpeg Port which conflicts with libjpeg-turbo.

Currently there appears to be no way to specify an "OR dependency"... like...

depends_lib port:jpeg || port:libjpeg-turbo

So is the correct way to fix this to change...

depends_lib port:jpeg

to...

depends_lib path:lib/libjpeg.dylib

which would then pass the dependency check for any Port that installed this library.

Let me know if this is the correct fix.

Thanks.

[davidfavor (David Favor)]

Looks like this won't work for libjpeg-turbo, because jpeg9 breaks API/ABI compatibility with jpeg8 and libjpeg-turbo uses the jpeg8 API.

Grumble...

Let me know if this is the correct way to resolve this type of conflict, for possible future work.

Thanks.

[ryandesign (Ryan Carsten Schmidt)]

If libjpeg-turbo were a drop-in compatible replacement for jpeg (which I thought it was, but which you've pointed out as of jpeg 9 it isn't anymore) then yes the solution would be to change all port:jpeg dependencies in all ports to path:lib/libjpeg.dylib:jpeg.

[rmstonecipher@…]

Ryan, David,
The libjpeg-turbo ​commentary on jpeg 9, SmartScale, etc. gives me the impression that the changes to ijg jpeg are non-standard, academic in nature, and not widely-adopted.
Could we not modify the path dependency to depend on libjpeg-turbo by default and let jpeg 9, which seems to be drifting away from standards-compliance, be the alternative library?
Could Portfile voodoo be used to force systems upgrading jpeg-dependent packages to deactivate jpeg 9, install libjpeg-turbo, and rebuild against it?

[vergus@…]

Cc Me!

[tim.bruylants@…]

Please do not support libjpeg9! The comment and link that was posted in comment 4 is 100% correct.

Libjpeg9 not only has a changed ABI/API, but it also diverts from the official JPEG standard. Thus, it potentially generates conflicting, and non-standard JPEG files. As far as I know, most of the larger Linux distros also dropped libjpeg9 in favor of libjpeg-turbo.

I, myself, was surprised to find that MacPorts relies on libjpeg9. I am removing this lib from my system as we speak, and suggest everyone from not using it.

Kind regards, Tim.

[ryandesign (Ryan Carsten Schmidt)]

We are now discussing replacing jpeg with libjpeg-turbo on the macports-dev mailing list.

[ryandesign (Ryan Carsten Schmidt)]

Cc Me!

[devernay (Frédéric Devernay)]

Is there any update on this?

Ports that depend specifically on the new jpeg9 API (there is very probably none) could keep the dependency on port:jpeg, but most (if not all) ports should depend on path:lib/libjpeg.dylib:jpeg

Major Linux distributions now use libjpeg-turbo instead of libjpeg9 (Fedora, Debian, Mageia, openSUSE).

Official binaries for Mozilla and Chrome also use libjpeg-turbo.

Also note that libjpeg-turbo is now an Official ISO/ITU-T Reference Implementation (see ​https://jpeg.org/items/20190204_press.html ).

There should be no more discussion on changing depend_libs port:jpeg to depend_libs path:lib/libjpeg.dylib:jpeg.

The only remaining discussion should be whether MacPorts should default to libjpeg-turbo as its default JPEG library. I would vote YES for that, given the very strong support.

[devernay (Frédéric Devernay)]

In fact, one issue with that change is that ports compiled against jpeg9 may link to libjpeg.9.dylib (I just ran into this with the jpeg loading plugin in Qt), but libjpeg-turbo provides libjpeg.8.dylib, so it would also be a good thing to downgrade libjpeg to v8.

Once again, nobody uses the v9 API, which is described by the JPEG itself in ​https://jpeg.org/items/20190204_press.html as: "a complete implementation of JPEG coming from the Committee itself that also covers coding modes that were only known by a few experts."

[dclunie]

Any update on this?

I need to build something depends on libjpeg-turbo but also giflib, and the latter depends on jpeg.

This is a showstopper for me and I am going to have to switch to HomeBrew instead of MacPorts if it is not resolved.

[kencu (Ken)]

1. it's super-simple to switch a given port as you wish. I have dozens of such ports. Less than two minutes work. Let me know if you are interested.
2. what did homebrew decide? turbo for everything?

At the very least, seems to me we can change all the jpeg-requiring ports to a path dep, so they can all coexist....

[kencu (Ken)]

So there are 211 ports that depend on port:jpeg so that's why nobody wants to take this one. Every one will have to be individually tested against turbo, etc.

You say giflib depends on port:jpeg? I'm not seeing that. If you can find the one port that needs the jpeg spec changed, I can probably help with that:

$ port rdeps giflib
The following ports are dependencies of giflib @4.2.3_0+x11:
  xorg-libsm
    pkgconfig
      libiconv
        gperf
    xorg-xtrans
    xorg-libice
      xorg-xorgproto
  xorg-libX11
    xorg-util-macros
    xorg-libXdmcp
    xorg-libXau
    xorg-libxcb
      python37
        xz
          gettext
            ncurses
        bzip2
        expat
        libedit
        libffi
        openssl
          zlib
        sqlite3
        python_select
        python3_select
      xorg-xcb-proto
        autoconf
        automake
        libtool
          xattr
            unzip
        libxml2
          icu
      xorg-libpthread-stubs

[ryandesign (Ryan Carsten Schmidt)]

Replying to kencu:

At the very least, seems to me we can change all the jpeg-requiring ports to a path dep, so they can all coexist....

Read comment:3.

[kencu (Ken)]

Yes -- well I'm certainly not about to test 211 ports against a new jpeg library... I guess dclunie is off to homebrew, then, unless he can come back with the port that is really causing his trouble...

[ryandesign (Ryan Carsten Schmidt)]

It's not about testing, it's just that libjpeg and libjpeg-turbo are not binary compatible so they can't be listed as being so via a path:-style dependency. (It's the same problem with have with openssl: we list openssl and libressl as binary compatible by using a path:-style dep; they're not; we shouldn't do that.)

comment:7 suggests we should switch to libjpeg-turbo as our primary jpeg, making libjpeg9 an option only for ports that need the new features (and naturally moving jpeg9's files so that they don't conflict with libjpeg-turbo's.) We could certainly do that. I'm not going to work on that right now. If somebody else wants to, that's great.

[kencu (Ken)]

Oh, I see what you're saying .. a path dep should mean you could uninstall one and install the other and all would be hunky-dory without rebuilding anything...not that either could satisfy the need.

Yeah, by that standard, a path dep will never work -- actually would probably quite rarely work for anything, come to think of it.

I took the more liberal view of it, meaning one or the other could be installed.

Perhaps that more liberal view of a path dep is more in keeping with what people actually want.... for example, webkit2-gtk and webkit2-gtk-devel share a path dep, but are not definitely binary compatible --- but it is likely that software could build against one or the other (actually, even that is not particularly guaranteed, but ... ).

[ryandesign (Ryan Carsten Schmidt)]

Replying to kencu:

Oh, I see what you're saying .. a path dep should mean you could uninstall one and install the other and all would be hunky-dory without rebuilding anything...not that either could satisfy the need.

Yeah, by that standard, a path dep will never work -- actually would probably quite rarely work for anything, come to think of it.

It works in the original use case: a library, like libpixman or cairo or pango or glib2 or graphviz that maybe adds some functions in a new minor library version but does not remove functions and the major library version does not change. A user can install the -devel version of the port to get new features, and the programs that use the library still work without a rebuild.

I took the more liberal view of it, meaning one or the other could be installed.

Perhaps that more liberal view of a path dep is more in keeping with what people actually want.... for example, webkit2-gtk and webkit2-gtk-devel share a path dep, but are not definitely binary compatible --- but it is likely that software could build against one or the other (actually, even that is not particularly guaranteed, but ... ).

In that scenario, the binaries that the user gets of any programs linking with the library would be broken and would need to be rebuilt from source on the user's system. That's not the user experience we want.

[devernay (Frédéric Devernay)]

Replying to ryandesign:

It's not about testing, it's just that libjpeg and libjpeg-turbo are not binary compatible

jpeg v8 API is fully supported by libjpeg-turbo, and I don't know of any open source software that uses the jpeg v9 API, which is purely research and experimental.

Please read the full explanation here: ​https://libjpeg-turbo.org/About/Jpeg-9

Most Linux distributions already switched to libjpeg-turbo and it was painless since everyone uses the jpegv8 API.

I'm using the path: approach (depends_lib path:lib/libjpeg.dylib:jpeg) to patch all ports that depend on jpeg, and it works very well!

[dclunie]

Sorry for the delayed reply - I did not end up switching to Homebrew after all, but just worked around the problem with an Ubuntu Docker build environment that had the libjpeg-turbo and giflib dependencies installed.

That said, I would still like to be able to have both libjpeg (from the IJG) and libjpeg-turbo available so that packages that depended on them could co-exist (and I wouldn't have to mess with Docker).

For that matter there is also the ISO group's code from Tom Richter (​http://github.com/thorfdbg/libjpeg), which I understand is also called "libjpeg" even though it has nothing to do with the IJG libjpeg, but which includes a lot of other functionality.

[RJVB (René Bertin)]

I have been using libjpeg-turbo for about 3 years now myself, with a tweaked port:jpeg that can be co-installed but that acts mostly if not completely as a stub just to satisfy port dependencies (so I haven't kept it up to date). I'm now using libjpeg-turbo 2.0.4 (with a small patch so it registers a compatibility_version of 8 instead of 10, that feature mostly bites us in software from Linux, esp. if it switched from autoconf to cmake :-/) port:jpeg wouldn't be the first to be installed in subprefix, requiring a small patch to ports that really need it (or that want to provide an option to use the reference library).

As others above I have never run into any problems because of a missing v9 API, so I really don't understand what's the hold-up here. Even if port:jpeg provides the reference implementation I think that most users don't give a rat's ass about that while even more will probably appreciate any possible speed improvement in this department.

EDIT: the compatversion tweak above can easily be turned into a tweak that will make current binaries accept the turbo libjpeg dylib without needing to rebuild a single port. A rev-upgrade should then point out immediately if any of the installed ports do use the v9 API, in case anyone wants to be 100% certain about this question.

[jmroot (Joshua Root)]

As has already been stated by Ryan above, if someone were to make a PR switching all jpeg dependencies to libjpeg-turbo and rev bumping, there's no reason it wouldn't be accepted. It's just a matter of doing the work.

[devernay (Frédéric Devernay)]

What would be the prefered way, then?

depends_lib path:lib/libjpeg.dylib:jpeg

or

depends_lib path:lib/libjpeg.dylib:libjpeg-turbo

or

depends_lib libjpeg-turbo

?

The first and second solution allow switching between jpeg and jpeg-turbo, but it won't work if they have different dylib version numbers.

[jmroot (Joshua Root)]

depends_lib path:lib/libjpeg.dylib:libjpeg-turbo would be the right way. This would allow switching between libjpeg-turbo and mozjpeg. The jpeg port is not binary compatible with these, and should be changed such that it doesn't install the same files as them (different prefix for most files and maybe a libjpeg.9.dylib link in ${prefix}/lib).

[mascguy (Christopher Nielsen)]

To eliminate the conflict between opencv and opencv4, the latter now hosts *everything* in opencv4 subdirectories... including libraries. That way both ports can safely coexist, and dependent ports only need a configure flag to point to the new location.

And I'd propose that we use the same approach here, for libjpeg-turbo: Simply host everything in separate subdirs. And since libjpeg-turbo has no dependents at the moment, the work will be trivial.

This strategy will allow us to gradually migrate ports, with no mass-migration headaches. I'm even willing to make the changes, with Larry's blessing.

Thoughts?

[kencu (Ken)]

The idea was that a user could transparently install libjpeg-turbo instead of libjpeg if s/he preferred.

This plan would be breaking that, right?

[jmroot (Joshua Root)]

That has not been transparent when IJG jpeg 9 is involved; see comment:29. If you're going to work on this, please do it as per that comment. Gradual migration will be a bigger headache.

[mascguy (Christopher Nielsen)]

Well, we're certainly all passionate... and that's not a bad thing.

As for my proposal, one of the goals was to get past the ticket gridlock we're currently in... and finally allow libjpeg-turbo to simply be installed/used. Eliminating the conflict would finally allow that... eight years later.

Then there's the issue with API/ABI compatibility, regression testing of changed ports, etc: If we start transparently switching which library is used, without thorough testing of every single port... isn't that a potential nightmare?

[jmroot (Joshua Root)]

There's very little likelihood of API incompatibility. Most distros already switched to libjpeg-turbo. Everything using jpeg will need a rev bump due to the different ABI of course. It's not a complicated change, just one that will affect a lot of portfiles.

[mascguy (Christopher Nielsen)]

Replying to jmroot:

There's very little likelihood of API incompatibility. Most distros already switched to libjpeg-turbo. Everything using jpeg will need a rev bump due to the different ABI of course. It's not a complicated change, just one that will affect a lot of portfiles.

Okay, so we have consensus then... and we're just waiting for someone to write a script to apply the changes?

[mascguy (Christopher Nielsen)]

Rather than deal with changing all dependent ports, why don't we convert mozjpeg and libjpeg-turbo into subports of jpeg? And then have jpeg have a dependency on the default subport (something like jpeg-default), but allowing the user to select 'libjpeg-turbo' or 'mozjpeg' via some mechanism, like a variant?

I'm thinking in terms of how we handle 'ld64'.

That would eliminate the need to change any downstream ports, right?

Ultimately we'd need to rev-bump them when we finally decide to change the default, but not initially.

[jmroot (Joshua Root)]

In 0c3c5e770185f4c3577b94a49170a961b88d1b98/macports-ports (master):

switch jpeg dependencies to libjpeg-turbo

Change jpeg port to install in a location that does not conflict with
libjpeg-turbo or mozjpeg. Change all jpeg dependencies to use
libjpeg-turbo by default while allowing mozjpeg as a drop-in
replacement. Rev bump dependent ports due to differing library version.

Fixes: #38907

[mascguy (Christopher Nielsen)]

While there might be a minor issue or two, this update is a huge win for our users.

Thanks Josh!

[JDLH (Jim DeLaHunt)]

A huge win, unless libjpeg-turbo doesn't build. See #60018 .

[Lord-Kamina (Gregorio Litenstein)]

I'm being burned by this; apparently Poppler depends on libjpeg9.

[ryandesign (Ryan Carsten Schmidt)]

It does not. Rebuild poppler while libjpeg-turbo is installed and active and poppler should pick it up. Rev-upgrade, which automatically runs after any port install or upgrade, should have noticed this problem and rebuilt poppler for you automatically, unless you turned rev-upgrade off or told it not to rebuild.

[Lord-Kamina (Gregorio Litenstein)]

You're partially right. The problem was that poppler failed to build due to broken linking with jpeg9. What I had to do was deactivate the previous poppler before rebuilding it

[ryandesign (Ryan Carsten Schmidt)]

poppler has historically had problems building when a previous version is active, as presumably have other ports using gobject-introspection. This was thought to have been fixed. See #58574. If not, open a new ticket and/or comment in that one.