port:jpeg proposal to prepare for making port:libjpeg-turbo the default jpeg dependency

[RJVB (René Bertin)]

This is the draft proposal for a change in how port:jpeg installs, helping to prepare for a transition to using port:libjpeg-turbo (or mozjpeg) as the default "jpeg provider".

I propose

• to install to ${prefix}/libexec/jpeg
• to provide a +transitional variant (default for now) that hides this change from the installed ports
• to provide a grace period (during which +transitional exists and) during which even -transitional installs a symlink to just libjpeg.9.dylib so that existing ports can continue to use that library without requiring a rebuild.

I've tested both +transitional and -transitional, the latter with libjpeg-turbo installed and used by 2 ports (simply by rebuilding them).

The only anomaly I can think of is the fact that those 2 rebuilt ports are declaring to depend on port:jpeg while in fact they depend on a library provided by libjpeg-turbo. That's something that I think can get sorted out gradually during the grace period.

[RJVB (René Bertin)]

Cf. #47807 for an update to port:libjpeg-turbo

NB: this does not yet include the change to the versioning scheme that I suggested, for instance to 9:9a or 9.a

[ryandesign (Ryan Carsten Schmidt)]

I don't really understand the benefit of doing it this way. I propose to do it the way I ​described on the mailing list:

I'm open to replacing jpeg with libjpeg-turbo, with the option to use the mozjpeg fork of libjpeg-turbo instead. This would involve verifying that libjpeg-turbo works (or at least builds) correctly on PowerPC and Intel Macs running 10.4 and up. Then, assuming it does, identifying all ports that link with libjpeg; changing the dependency from port:jpeg to path:lib/libjpeg.dylib:libjpeg-turbo; increasing their revision; and marking jpeg as replaced_by libjpeg-turbo.

[RJVB (René Bertin)]

Replying to ryandesign@…:

I don't really understand the benefit of doing it this way. I propose to do it the way I ​described on the mailing list:

I have been participating in that thread... The way proposed by this patch is in no way incompatible with the way you propose, IMHO it complements it, and makes it less of a "one-shot that shouldn't be missed" venture.

verifying that libjpeg-turbo works (or at least builds) correctly

Is having an escape route ready if it only builds such an unimaginable idea, or even the idea of keeping the original/traditional/reference port around such that it can be co-installed with whatever it's going to be replaced with?

My professional as well as personal experience (as if they could be separated...) tells me that users (myself included) who rely on a central library like this rarely appreciate it if it's replaced behind their backs without as much as an option to opt-out and keep using what they've always been using. I'm quite certain that I've seen a statement in the libjpeg-turbo documentation (README or elsewhere) that there is no guarantee that the software renders to a result identical to what libjpeg would have given, despite the ABI compatibility. For me that is sufficient reason to keep the port around.

[small]Sometimes I wonder what MacPorts is supposed to be - a show-case what FOSS can be installed through it and that's catering to people who'd use Arch or Gentoo and spend their computer time keeping the system up to date, or something that's aiming to support productivity that's not MacPorts-related.small

[neverpanic (Clemens Lang)]

Replying to rjvbertin@…:

The way proposed by this patch is in no way incompatible with the way you propose, IMHO it complements it, and makes it less of a "one-shot that shouldn't be missed" venture.

But yet, it is much more complicated than our standard approach of 1. replace port, 2. fix dependencies, 3. revbump dependents.

Is having an escape route ready if it only builds such an unimaginable idea, or even the idea of keeping the original/traditional/reference port around such that it can be co-installed with whatever it's going to be replaced with?

No, but it adds significant complexity that we should avoid, IMO. Just in the spirit of keeping things simple, where they don't need to be complex.

Sometimes I wonder what MacPorts is supposed to be - a show-case what FOSS can be installed through it and that's catering to people who'd use Arch or Gentoo and spend their computer time keeping the system up to date, or something that's aiming to support productivity that's not MacPorts-related.

I believe that most of our users do not directly use libJPEG – essentially, most of them probably do not care which specific JPEG library is being used, as long as the end result (i.e. their use case of rendering or encoding JPEGs) continues to work. And, if due to a change we make, this use case becomes faster, that's probably even a welcome side-effect.

I would even argue that MacPorts should not be a show-case of what FOSS can be installed through it, and for that reason not keep the jpeg port around. We should just make a sane choice for most of our users that really don't care which JPEG library gets used, just as the big Linux distros have done by replacing jpeg with libjpeg-turbo.

[RJVB (René Bertin)]

Replying to cal@…:

But yet, it is much more complicated than our standard approach of 1. replace port, 2. fix dependencies, 3. revbump dependents. ... No, but it adds significant complexity that we should avoid, IMO. Just in the spirit of keeping things simple, where they don't need to be complex.

Much more, significant (to what level of confidence?) complexity? Now who's seeing problems where there really aren't? I've taken care that this change can be pushed without rebuilding a single port except for port:jpeg itself, which takes about 30s on my system (going on 4yo and with an HDD instead of an SSD). It also carries no inherent obligation at all to follow up on my suggestions on how to use it, and I'm not even thinking of obliging everyone to keep having the port installed. Plus I did all the work...

I believe that most of our users do not directly use libJPEG – essentially, most of them probably do not care which specific JPEG library is being used, as long as the end result (i.e. their use case of rendering or encoding JPEGs) continues to work. And, if due to a change we make, this use case becomes faster, that's probably even a welcome side-effect.

I'm not saying I care what libjpeg version is being used most of the time, except when dealing with photos when I want to be sure to get the correct/best rendering. That, and the fact that libjpeg-turbo v8 is a 2-releases wide emulation layer on top of something that was ABI compatible with libjpeg v6 (and even that version wasn't guaranteed to yield identical results) are reason enough for me to want to maintain the port.

---

For the most part, libjpeg-turbo should produce identical output to libjpeg v6b. The one exception to this is when using the floating point DCT/IDCT, in which case the outputs of libjpeg v6b and libjpeg-turbo can differ for the following reasons: <snip>

While libjpeg-turbo does emulate the libjpeg v8 API/ABI, under the hood, it is still using the same algorithms as libjpeg v6b, so there are several specific cases in which libjpeg-turbo cannot be expected to produce the same output as libjpeg v8:

-- When decompressing using scaling factors of 1/2 and 1/4, because libjpeg v8

implements those scaling algorithms differently than libjpeg v6b does, and libjpeg-turbo's SIMD extensions are based on the libjpeg v6b behavior.

-- When using chrominance subsampling, because libjpeg v8 implements this

with its DCT/IDCT scaling algorithms rather than with a separate downsampling/upsampling algorithm. In our testing, the subsampled/upsampled output of libjpeg v8 is less accurate than that of libjpeg v6b for this reason.

-- When decompressing using a scaling factor > 1 and merged (AKA "non-fancy" or

"non-smooth") chrominance upsampling, because libjpeg v8 does not support merged upsampling with scaling factors > 1.

---

Not so long ago when I was still in vision research this could have been a much bigger deal for me, now I don't even know for sure anymore how many of my former colleagues are still using OS X (and MacPorts as a resource).

Re: Performance: it was a comment in LibVNCServer's output that got me interested in libjpeg-turbo. This seemed the kind of application where a 2-4x decoding speed-up would be noticeable, but in the end I can't say I noticed anything. Other comparisons I ran on even the largest jpeg images I have around showed similar results: I suppose the decoding speed gain was insignificant compared to other bottlenecks, i.e. the overal process not decoding-constrained. So I'm tempted to think that you're preparing a change with a huge overhead for nothing much if a (mostly theoretical) performance gain is the sole reason.

What I do care about has been rehashed a couple of times already apparently without leaving much impression. I'm using a number of ports (including KDE and GTk-based apps) as alternatives for "native" applications, and not just from time to time, and without a backup OS X machine. I'm using "devel" versions of Qt4, Qt5 and kdelibs4, and there's just no way I'm going to get myself in a situation where I have to rebuild all of those plus the GTk ports at the same time because a basic library has been downgraded (which in turn was necessary because it was updated for updating's sake) without keeping the newer version around at least during a grace period.

I would even argue that MacPorts should not be a show-case of what FOSS can be installed through it, and for that reason not keep the jpeg port around. We should just make a sane choice for most of our users that really don't care which JPEG library gets used, just as the big Linux distros have done by replacing jpeg with libjpeg-turbo.

The situation is different with the big Linux distros and their users. Their packaging/distribution systems are usually much more powerful than MacPorts', and there are no such things as "variants" which oblige users to build from source instead of using the provided binary packages. But most of all, they've never been in the situation where they had to downgrade. I have a hunch that more than 1 distribution would have decided to roll their own v9 emulation for libjpeg-turbo because apparently that can be done easily enough.

Linux distributions also don't have activation/deactivation of older and/or alternative port versions. You did realise I hope that by doing an all-out replacement of port:jpeg by port:libjpeg-turbo, it becomes impossible to re-activate an older version of a port that was built before the change (and probably cannot be rebuilt anymore) without re- or de-activating *all* other ports depending on the jpeg replacement? The simple fact of providing libjpeg.9.dylib (which doesn't bite anything or anyone) makes that issue go away.

[RJVB (René Bertin)]

There's also this: ​http://www.libjpeg-turbo.org/About/Jpeg-9:

SmartScale files can currently only be decoded using jpeg-8, and Lossless SmartScale files that use the new color transform can only be decoded using jpeg-9. 

No idea how likely it is to encounter such images in the wild, but as far as I'm concerned the appropriate course of action is clear when the v9 (and v8!) changes also include things that lead to images that cannot be decoded by libjpeg-turbo or mozjpeg.

[larryv (Lawrence Velázquez)]

Replying to rjvbertin@…:

The only anomaly I can think of is the fact that those 2 rebuilt ports are declaring to depend on port:jpeg while in fact they depend on a library provided by libjpeg-turbo.

This is a non-starter.

[ryandesign (Ryan Carsten Schmidt)]

Sorry, I didn't mean to divert the discussion to this ticket. I just wanted to add a note for anyone reading the ticket that I did not recommend implementing your proposal as-is. Discussion can continue on the mailing list.

[RJVB (René Bertin)]

Replying to larryv@…:

Replying to rjvbertin@…:

The only anomaly I can think of is the fact that those 2 rebuilt ports are declaring to depend on port:jpeg while in fact they depend on a library provided by libjpeg-turbo.

This is a non-starter.

This is exactly the same kind of result one gets when qt5-mac-devel is installed instead of qt5-mac or any other of the -devel ports and yet that's accepted practice. You'd get the same result too when a path:libjpeg.8.dylib dependency is satisfied by port:mozjpeg instead of port:libjpeg-turbo, btw.

[larryv (Lawrence Velázquez)]

Replying to rjvbertin@…:

This is exactly the same kind of result one gets when qt5-mac-devel is installed instead of qt5-mac or any other of the -devel ports and yet that's accepted practice. You'd get the same result too when a path:libjpeg.8.dylib dependency is satisfied by port:mozjpeg instead of port:libjpeg-turbo, btw.

Oh, are you referring to the fact that if you install one of these ports against jpeg and then swap in libjpeg-turbo, MacPorts still thinks that the dependency is on jpeg? I wouldn’t call that “accepted practice”. That’s a bug in base, I believe.

[RJVB (René Bertin)]

Or more in general, if path:libjpeg.dylib:jpeg is resolved by any port other than port:jpeg, the dependency is still registered as being on port:jpeg. At least that's what I was told when I asked about this a while back, and we agree that's an oversight at least. The fact that a path: style dependency also allows to resolve a dependency with a binary that doesn't come from a port at all is iffy too (but sometimes comes in handy O:-) ).

Replacing a dependency by something ABI compatible post-installation is a different issue. I don't know if one could expect the registry to be rewritten in that case. After installation of a port, information about its dependencies is used only to prevent uninstallation of the dependencies (and to keep track of ports gone useless), right?

With accepted practice I referred to using a path: style dependency to provide a choice between a regular port and its -devel "variant".

[mascguy (Christopher Nielsen)]

Now that we've formally switched to libjpeg-turbo, can this be closed?

[ryandesign (Ryan Carsten Schmidt)]

[0c3c5e770185f4c3577b94a49170a961b88d1b98/macports-ports]