mkvtoolnix needs boost and icu built with same compiler

[brian@…]

Installing the latest version of mkvtoolnix fails on a fresh install of macports. The issue with the installation was that pkgconfig and curl are not defined as a dependency of mkvtoolnix. These are both required and I had to look in the log files to figure this out. After I installed them by hand mkvtoolnix install completed.

However, after installing mkvtoolnix the mkvextract tool doesn't work. Here is the runtime error I get:

mkvextract(48429) malloc: *** error for object 0x7fff7aac4860: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

This might be a compilation error since it appears that the mkvtoolnix package is shipped as source rather than binary.

[ryandesign (Ryan Carsten Schmidt)]

Virtually all ports build from source at one time or another.

I've added the missing curl and pkgconfig dependencies in r94091; thank you for reporting that.

As for the mkvextract malloc error message, I don't know how to reproduce that. When I run "mkvextract" with no arguments, it correctly prints the usage message. If you have a reproduction recipe, you can post it here, however the responsibility for fixing the problem ultimately lies with the developers of mkvtoolnix, so you should report the problem to them.

[Kona8lend@…]

however the responsibility for fixing the problem ultimately lies with the developers of mkvtoolnix

Not an upstream issue.

[brian@…]

You can reproduce this using any mkvextract command on a mkv file. Here's an example of what I used:

$ mkvextract tracks foo.mkv 0:video.h264 2:audio.aac

[ryandesign (Ryan Carsten Schmidt)]

Replying to brian@…:

You can reproduce this using any mkvextract command on a mkv file. Here's an example of what I used:

$ mkvextract tracks foo.mkv 0:video.h264 2:audio.aac

Ok yes, thank you, I do see the problem on Snow Leopard 10.6.8 x86_64 with Xcode 3.2.6.

Replying to Kona8lend@…:

however the responsibility for fixing the problem ultimately lies with the developers of mkvtoolnix

Not an upstream issue.

What makes you say that? You believe it is a MacPorts-specific issue? If so, what do we need to do to fix it?

[Kona8lend@…]

Not an upstream issue.

What makes you say that? You believe it is a MacPorts-specific issue? If so, what do we need to do to fix it?

To fix, mkvtoolnix must link against boost and icu built with a consistent compiler/version; ie: gcc46.

[ryandesign (Ryan Carsten Schmidt)]

Ah. I see. And that's not something that's likely to happen in MacPorts, unless we have completely separate boost-gcc46 and icu-gcc46 ports, which seems like a pain to maintain. Maybe we should downgrade mkvtoolnix back to the last version that could be compiled with the compilers that come with Xcode.

[ryandesign (Ryan Carsten Schmidt)]

Actually, boost-gcc46 and icu-gcc46 ports would be not too difficult to maintain if they were implemented as subports of the existing boost and icu ports.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #34821.

[machsna (J. ‘mach’ Wust)]

Cc Me!

[mojca (Mojca Miklavec)]

Cc Me!

[bgrupe27]

Cc Me!

[kevin.krouse@…]

Cc Me!

[evandrix (Lee Wei Yeong)]

Cc Me!

[evandrix (Lee Wei Yeong)]

How about trying ​http://www.macupdate.com/app/mac/16837/mkvtoolnix in the meantime?

Replying to brian@…:

Installing the latest version of mkvtoolnix fails on a fresh install of macports. The issue with the installation was that pkgconfig and curl are not defined as a dependency of mkvtoolnix. These are both required and I had to look in the log files to figure this out. After I installed them by hand mkvtoolnix install completed.

However, after installing mkvtoolnix the mkvextract tool doesn't work. Here is the runtime error I get:

mkvextract(48429) malloc: *** error for object 0x7fff7aac4860: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

This might be a compilation error since it appears that the mkvtoolnix package is shipped as source rather than binary.

[oliver.wagner@…]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Replying to evandrix@…:

How about trying ​http://www.macupdate.com/app/mac/16837/mkvtoolnix in the meantime?

I have deleted the Mkvtoolnix-5.8.0_2012-09-02-6920.dmg attachment from this ticket since it is available from the above URL.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign@…:

Actually, boost-gcc46 and icu-gcc46 ports would be not too difficult to maintain if they were implemented as subports of the existing boost and icu ports.

Well, it was more difficult than I anticipated and I gave up. The challenge is in making sure that the subports install their files to totally different locations than the main ports so that they do not conflict. Then you have the trouble of teaching mkvtoolnix how to find the files in those nonstandard locations.

Eric suggested in comment:ticket:37678:11 that the latest development version of clang (did you mean the clang-3.3 port?) would be able to build mkvtoolnix. But I presume that would still require boost and icu built with that same compiler. Or would that no longer be an issue because they'd both be using the same C++ library already?

[ecronin (Eric Cronin)]

I remember a bit more now... It's supposed to build with clang 3.1 and later. The stopper I ran into the one time I tried several months ago was that the libc++ that the Lion XCode/Lion itself (not sure which "owns" /usr/lib/libc++) provided was not new enough to implement the needed C++11 stdlib features, so it would need some macports-provided newer version of libc++. (I was confusing it with another program that needed an unreleased clang snapshot which is why I said clang-devel in the other ticket).

I don't know which boost libs mkvtoolnix is using these days, and how much there are differences in those libs when -std=c++11 is involved. But it would at least be ABI compatible I think, which mp-gcc won't be any time soon. You might get lucky and it will work well enough for mkvtoolnix, which is much better than today's mp-gcc which definitely will never work... It's still not a general solution to the boost versioning problem

[ecronin (Eric Cronin)]

and, if /usr/lib/libc++ is updated by XCode (or newer in ML than it is in Lion) it's possible that the Apple compiler works now if you pass -stdlib=libc++ -std=c++11 to it...

[kalin.f@…]

Cc Me!

[machsna (J. ‘mach’ Wust)]

Replying to ryandesign@…:

Replying to evandrix@…:

How about trying ​http://www.macupdate.com/app/mac/16837/mkvtoolnix in the meantime?

While macports' mkvtoolnix is broken (or have I just not understood how to make it work?), you can use Mkvtoonix.app from the command line once you add the following at the end of your ~/.profile file:

export PATH=/Applications/Mkvtoolnix.app/Contents/MacOS:$PATH

[Kona8lend@…]

I have custom ports for mkvtoolnix that work nicely. See attachment kona-mkvtoolnix-2fb9dfa.tar.bz2 (above).

It consists of 3 ports:

• mkvtoolnix 6.0.0
• mkvtoolnix.boost 1.53.0
• mkvtoolnix.icu 50.1.2

mkvtoolnix has some minor upstream patches to find private boost libraries before macports normal libraries. Depends on mkvtoolnix.boost .

mkvtoolnix.boost has a minor upstream patch to use -install_name properly. Depends on mkvtoolnix.icu .

mkvtoolnix.icu has no upstream patches.

All 3 ports are aware of and use /opt/local/private/mkvtoolnix/ as a private subtree for mkvtoolnix special needs. There are several good reasons for this. Firstly, we guarantee that all C++ and C++11 libraries for which mkvtoolnix depends on, either directly or indirectly, are built with the same compiler, and same c++11 mode flags. Second, this must co-exist with all other ports and not impact them negatively. Thirdly, mkvtoolnix tracks boost version bumps much faster than other apps, so tying them to the same version is not possible without halting mkvtoolnix version bumps.

The positive side is, port:mkvtoolnix.boost has been optimized to build on the necessary libraries and takes an order of magnitude less time to build than the port:boost .

There is support for 3 compiler variants, all tested on Mountain Lion. clang variant (default) is known to work with Xcode 4.6 . gcc47 and gcc48 variants also work. gcc46 support has been dropped by upstream. I recommend sticking with the clang.

If you want to try it out, add a custom directory in front of your default ports in /opt/local/etc/macports/sources.conf and extract the tarball. Run a port -d sync. Make sure old mkvtoolnix is deactivated. port info mkvtoolnix to verify it is picking up the new portfile. port install mkvtoolnix.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #38455.

[cooljeanius (Eric Gallager)]

Cc Me!

[root42]

Cc Me!

[root42]

@Kona8lend: I am trying to use your ports, but I don't succeed. I have extracted the tarball, and added a ​file:// URL to that directory to my sources.conf. However even after a port sync I only see the old 5.8.0 port. The generated PortIndex files are 0 bytes.

[raimue (Rainer Müller)]

Replying to arne.schmitz@…:

@Kona8lend: I am trying to use your ports, but I don't succeed. I have extracted the tarball, and added a ​file:// URL to that directory to my sources.conf. However even after a port sync I only see the old 5.8.0 port. The generated PortIndex files are 0 bytes.

Follow ​the guide. Make sure you also have directories for the categories at the path listed in your sources.conf.

[daniel@…]

Cc Me!

[raimue (Rainer Müller)]

Replying to Kona8lend@…:

All 3 ports are aware of and use /opt/local/private/mkvtoolnix/ as a private subtree for mkvtoolnix special needs. There are several good reasons for this. Firstly, we guarantee that all C++ and C++11 libraries for which mkvtoolnix depends on, either directly or indirectly, are built with the same compiler, and same c++11 mode flags. Second, this must co-exist with all other ports and not impact them negatively. Thirdly, mkvtoolnix tracks boost version bumps much faster than other apps, so tying them to the same version is not possible without halting mkvtoolnix version bumps.

I don't understand how you ensure these promised guarantees. The ports still have variants to select different compilers as explained below. Doesn't this still allow to use different compilers for each of the ports?

If mkvtoolnix is advancing too fast to allow a stable build with our toolchains we should just stick with an older, working release.

As far as I know no other port uses the ${prefix}/private namespace, which is outside the port hierarchy defined by porthier(7). I don't think "private" is exactly what we want here, the libraries should rather live in ${prefix}/lib/mkvtoolnix/lib*.dylib.

The positive side is, port:mkvtoolnix.boost has been optimized to build on the necessary libraries and takes an order of magnitude less time to build than the port:boost .

Except you need yet another port installed. boost is binary distributable and available as binary archives.

[Kona8lend@…]

Replying to raimue@…:

Replying to Kona8lend@…:

All 3 ports are aware of and use /opt/local/private/mkvtoolnix/ as a private subtree for mkvtoolnix special needs. There are several good reasons for this. Firstly, we guarantee that all C++ and C++11 libraries for which mkvtoolnix depends on, either directly or indirectly, are built with the same compiler, and same c++11 mode flags. Second, this must co-exist with all other ports and not impact them negatively. Thirdly, mkvtoolnix tracks boost version bumps much faster than other apps, so tying them to the same version is not possible without halting mkvtoolnix version bumps.

I don't understand how you ensure these promised guarantees. The ports still have variants to select different compilers as explained below. Doesn't this still allow to use different compilers for each of the ports?

The compilers that are known to work and tested are explicitly supported variants. What's the issue you have? Is it legalese? Sorry, but IANAL.

If mkvtoolnix is advancing too fast to allow a stable build with our toolchains we should just stick with an older, working release.

That is of course one choice. The reason for which these custom ports were created is because I made the other choice: to not wait an indeterminate number of years for tools and dependencies of "macports proper" to satisfy modern releases of mkvtoolnix.

As far as I know no other port uses the ${prefix}/private namespace, which is outside the port hierarchy defined by porthier(7). I don't think "private" is exactly what we want here, the libraries should rather live in ${prefix}/lib/mkvtoolnix/lib*.dylib.

There is also {bin,include,sbin,share} directories for which need to be unique, and for each port one or more of said directories. One choice is to go your route and mix-in such directories for each custom port into the main tree, which I think is too messy. Maybe another name besides "private" but if you actually try to convert these ports to your layout, it will become very intertwined. Remember, these ports must co-exist with existing macports proper.

The positive side is, port:mkvtoolnix.boost has been optimized to build on the necessary libraries and takes an order of magnitude less time to build than the port:boost .

Except you need yet another port installed. boost is binary distributable and available as binary archives.

Boost and ICU as built and distributed via binary archives of "macports proper" do not satisfy mkvtoolnix. The dependencies that are custom are that way for a reason: to satisfy mkvtoolnix. The c++ mode for which "macports proper" boost and icu are built are not likely to satisfy mkvtoolnix for quite sometime. This last part is an educated guess based on what I've seen with macports, its progress rate, and its complexity. No insults or negatives intended. Just the cold reality of incompatibilities.

Cheers.

[jeremyhu (Jeremy Huddleston Sequoia)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

This ticket explores the idea of getting new ports for boost and icu built with a newer gcc so that mkvtoolnix won't crash. Let's also explore the idea of getting mkvtoolnix to build with clang instead. Since this ticket is already very long, let's move this new discussion to #40231.

[temp@…]

Cc Me!

[jeremyhu (Jeremy Huddleston Sequoia)]

Replying to ryandesign@…:

This ticket explores the idea of getting new ports for boost and icu built with a newer gcc so that mkvtoolnix won't crash.

Newer versions of gcc won't fix the problem.

Let's also explore the idea of getting mkvtoolnix to build with clang instead.

That's the only thing that will fix the problem.

Since this ticket is already very long, let's move this new discussion to #40231.

Yeah, this should just be duped to #40231.

[mopihopi]

Cc Me!

[mojca (Mojca Miklavec)]

I'm reopening the ticket. While it is true that a successful build with clang/libc++ (#40231) solved the problem on 10.9 (and later), this is still a problem on 10.8 and earlier unless we officially declare mkvtoolnix as broken.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #44168.

[ryandesign (Ryan Carsten Schmidt)]

In r136336 I included the cxx11 portgroup, which declares that this port uses C++11 and therefore requires clang and libc++. This is how we're handling other ports that require C++11 as well.