rav1e: build fails due to git_allocator missing members

[lukaso (Lukas Oberhuber)]

First reported on #68026, via @macsguy:

This may require updating the version of the git2 crate [used by rav1e], now that libgit2 has been updated to 1.7.1:

libgit2/src/util/array.h:59:19: error: no member named 'greallocarray' in 'git_allocator'
        if ((new_array = git__reallocarray(a->ptr, new_size, item_size)) == NULL)
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

libgit2/src/libgit2/annotated_commit.c:34:21: error: no member named 'gcalloc' in 'git_allocator'
        annotated_commit = git__calloc(1, sizeof(git_annotated_commit));
                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

libgit2/src/libgit2/annotated_commit.c:48:34: error: no member named 'gstrdup' in 'git_allocator'
        annotated_commit->description = git__strdup(description);
                                        ^~~~~~~~~~~~~~~~~~~~~~~~

I gave a shot at trying to update the crates, but got errors after updating the crate list:

:info:extract error: failed to select a version for the requirement `git2 = "^0.15"`
:info:extract candidate versions found which didn't match: 0.18.0
:info:extract location searched: directory source `/Users/lukasoberhuber/macports-gimp3-arm64/var/macports/build/_Users_lukasoberhuber_project_ports_multimedia_rav1e/rav1e/work/.home/.cargo/macports` (which is replacing registry `crates-io`)
:info:extract required by package `built v0.5.2`
:info:extract     ... which satisfies dependency `built = "^0.5.2"` of package `rav1e v0.6.6 (/Users/lukasoberhuber/macports-gimp3-arm64/var/macports/build/_Users_lukasoberhuber_project_ports_multimedia_rav1e/rav1e/work/rav1e-0.6.6)`
:info:extract perhaps a crate was updated and forgotten to be re-vendored?
:info:extract As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.
:info:extract Command failed: /Users/lukasoberhuber/macports-gimp3-arm64/bin/cargo --offline update
:info:extract Exit code: 101

[lukaso (Lukas Oberhuber)]

Has multiple builds in it, so just focus on last one (they appear to be all the same, which might mean that macports tried to run it multiple times)

[rottegift (Sean Doran)]

FWIW, I just built successfully with

port -v install rav1e configure.ldflags="-ld_classic -force_load /usr/lib/libiconv.dylib" configure.cflags="-Wl,-ld_classic"

which is ugly, but works up to not-yet-found problems with differing header files

otool -L /opt/local/bin/rav1e
/opt/local/bin/rav1e:
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/opt/local/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.0.0)

the -force_load probably is portable to older macOS/xcode.

I don't think this is the right solution, but if one was, say, in the middle of building something with rav1e as a dependency, this might get the job done.

[mascguy (Christopher Nielsen)]

Lukas, my thoughts regarding updating the version of the git2 crate may be off in left field, given the following:

The source for libgit2 is included in the libgit2-sys crate so there's no need to pre-install the libgit2 library, the libgit2-sys crate will figure that and/or build that for you.

​https://github.com/rust-lang/git2-rs#version-of-libgit2

Given that - combined with the fact that we haven't changed the rav1e port - we should be fine as-is.

Can you try a clean build of rav1e, without any prior history/log, to ensure the same results occur?

[lukaso (Lukas Oberhuber)]

I did a forced uninstall and a port clean. If I should do any more clean than that, let me know how.

Attaching the new log.

[lukaso (Lukas Oberhuber)]

Post force uninstall and clean of rav1e

[lukaso (Lukas Oberhuber)]

OK, this actually happens on x86_64 as well. Attaching the log. Same problem.

[lukaso (Lukas Oberhuber)]

So it looks like these missing functions were removed in v1.7.1. So that's why we are stuffed. Also meaning they are not using semantic versioning (boo on them). ​https://github.com/libgit2/libgit2/commit/9dd1bfe81c3aba6ce35ed85bf62fa75d9c61e1a6.

The solution is probably to revert libgit2 in macports. Clearly the build is not pulling the embedded version when it is linking (or so it seems).

[mascguy (Christopher Nielsen)]

Replying to lukaso:

The solution is probably to revert libgit2 in macports. Clearly the build is not pulling the embedded version when it is linking (or so it seems).

No, that's probably not going to happen, unless this is more wide-spread. And given that other Rust-based ports build fine with the latest release of libgit2 - including cargo and rust itself - that strongly suggests this is a rav1e-specific issue.

[lukaso (Lukas Oberhuber)]

Replying to mascguy:

Replying to lukaso:

The solution is probably to revert libgit2 in macports. Clearly the build is not pulling the embedded version when it is linking (or so it seems).

No, that's probably not going to happen, unless this is more wide-spread. And given that other Rust-based ports build fine with the latest release of libgit2 - including cargo and rust itself - that strongly suggests this is a rav1e-specific issue.

Here's the thing to revert: ​https://github.com/macports/macports-ports/commit/c00e8c728a0458fcf0e1d2c555126d6bb642feaa and here's the PR in libgit2 which is labeled as sys api breaking: ​https://github.com/libgit2/libgit2/pull/6563.

This port has been broken for over a month now for one reason or another. It was also last broken in december 2022. Maybe it's time to cut rav1e loose? I tried to remove the dependency in the GIMP build by excluding it from libheif but it appears to be included in some other port that somehow I am including so that didn't help.

[mascguy (Christopher Nielsen)]

Replying to lukaso:

This port has been broken for over a month now for one reason or another. It was also last broken in december 2022. Maybe it's time to cut rav1e loose? I tried to remove the dependency in the GIMP build by excluding it from libheif but it appears to be included in some other port that somehow I am including so that didn't help.

Okay, just tried building rav1e from source. And sure enough, our port is currently broken.

But let me stress that's it's very likely our port that's broken, not upstream.

Why do I say that? Well, take a look at the portfile for rav1e, and specifically the crate for libgit2-sys:

libgit2-sys 0.14.2+1.5.1

That crate corresponds to libgit2 1.5.1. Whereas git2-rs upstream has released later releases, including the latest: libgit2-sys-0.16.1+1.7.1. And the latter corresponds to the latest libgit2 release. (Per the +1.7.1 suffix.) Which is what we should probably be used. Though perhaps one of the previous releases, corresponding to libgit2 1.6.x, might work too. Dunno without more research/testing.

In any case, updating that crate, likely involves updating others as well. Perhaps that's automated via a MacPorts script somewhere? (Not a Rust guru here, otherwise I would have fixed this by now.)

So I'm quite certain this is 100% our issue.

As for downgrading libgit2: Simply reverting one commit isn't going to cut it. We'd also need to downgrade py-pygit2, and a number of others. And then rebuild every dependent - including rust and cargo, along with numerous other dependents. (As the downgrade would break the ABI.) So no, not going to happen, when it seems very clear the rav1e port is the one that needs to be fixed.

[mascguy (Christopher Nielsen)]

Relative to eliminating rav1e from the dependency equation: Most of that's in-place already.

The ports that depend on it, are the following:

$ gfind . -type f -name "Portfile" -print0 | gxargs -0 -r ggrep -E --regexp=":rav1e"
./multimedia/av1an/Portfile:                    port:rav1e \
./multimedia/ffmpeg/Portfile:                    port:rav1e
./multimedia/ffmpeg-devel/Portfile:                    port:rav1e
./multimedia/ffmpeg6/Portfile:                    port:rav1e
./multimedia/libavif/Portfile:                        port:rav1e \
./multimedia/libheif/Portfile:                            port:rav1e \

You can ignore ffmpeg-devel and ffmpeg6 though, as you'll be using ffmpeg.

Regardless, all but two of those support variants that can eliminate rav1e from the equation:

• For ffmpeg, enable variant +slim.
• For av1an - assuming that's being pulled in for GIMP? - install with -av1.

As for libavif and libheif, I maintain both of those. And I'm happy to provide something similar there, if needed. Though it's trivial to overlay your own versions of those two, dropping rav1e from the deps:

• For libavif, remove the dep, and update -DAVIF_CODEC_RAV1E:BOOL=ON to -DAVIF_CODEC_RAV1E:BOOL=OFF.
• Likewise, for libheif, remove the dep, and add configure argument -DWITH_RAV1E:BOOL=OFF.

[lukaso (Lukas Oberhuber)]

Regardless, all but two of those support variants that can eliminate rav1e from the equation:

• For ffmpeg, enable variant +slim.
• For av1an - assuming that's being pulled in for GIMP? - install with -av1.

As for libavif and libheif, I maintain both of those. And I'm happy to provide something similar there, if needed. Though it's trivial to overlay your own versions of those two, dropping rav1e from the deps:

• For libavif, remove the dep, and update -DAVIF_CODEC_RAV1E:BOOL=ON to -DAVIF_CODEC_RAV1E:BOOL=OFF.
• Likewise, for libheif, remove the dep, and add configure argument -DWITH_RAV1E:BOOL=OFF.

I've done the overlay now for libheif and libavif. Should have done this ages ago! I think it would be worthwhile creating slim or removerav1e versions for all of them since this appears to be a persistent issue. But I can move forward for now.

On an aside: It would be great to have much simpler tools for determining dependency...I'm always struggling to find what the dependencies are. Especially the dependencies all the way down.

[mascguy (Christopher Nielsen)]

Replying to lukaso:

I've done the overlay now for libheif and libavif. Should have done this ages ago! I think it would be worthwhile creating slim or removerav1e versions for all of them since this appears to be a persistent issue. But I can move forward for now.

Sounds good, will try to tackle this over the next day or two.

On an aside: It would be great to have much simpler tools for determining dependency...I'm always struggling to find what the dependencies are. Especially the dependencies all the way down.

There definitely is, and macports provides a rich set of options to formally query for those. Take a look at man port, specifically the pseudo-portnames.

For example:

$ port info --index --line --name depends:rav1e
ffmpeg
ffmpeg-devel
ffmpeg6
libavif
libheif
av1an

Note: Option --index is optional, but it speeds things up tremendously. (Simply because it utilizes the portindex for queries.)

Along with various other ways to do it, such as port echo:

$ port echo depends:rav1e
ffmpeg
ffmpeg-devel
ffmpeg6
libavif
libheif
av1an

But take a thorough read through man port, for the various details. Cheers!

[mascguy (Christopher Nielsen)]

Lukas, I'm making progress, relative to updating rav1e to use the latest version of libgit2-sys, etc. (It simply required some time to review how dependencies are handled within the Rust/Cargo space.)

Presently dealing with compilation failures related to libssh2, but will figure it out with enough time.

So hopefully this port will be fixed in the next few days, and you won't need to remove it from the mix long-term. Stay tuned...

[lukaso (Lukas Oberhuber)]

Replying to mascguy:

Lukas, I'm making progress, relative to updating rav1e to use the latest version of libgit2-sys, etc. (It simply required some time to review how dependencies are handled within the Rust/Cargo space.)

Presently dealing with compilation failures related to libssh2, but will figure it out with enough time.

So hopefully this port will be fixed in the next few days, and you won't need to remove it from the mix long-term. Stay tuned...

Great. Thanks.

[Christopher Nielsen <mascguy@…>]

In 20dd1dc069d78e41cb7241c91d6f2ad39d0679b5/macports-ports (master):

lib{avif,heif}: add new variant rav1e; disabled by default

See: #68026
See: #68178

[Christopher Nielsen <mascguy@…>]

In c08417569d37bdbd835cbfe77c383c6028082bd8/macports-ports (master):

ffmpeg*: add variant rav1e, disabled by default

See: #68026
See: #68178

[lukaso (Lukas Oberhuber)]

Thanks! Trying out the rav1e free ports now.

[lukaso (Lukas Oberhuber)]

Worked great locally.

[mascguy (Christopher Nielsen)]

Observers: The work-in-progress fixes for rav1e - which may take some time to figure out - are being done via PR:

​PR 20532 - rav1e: update for libgit2 1.7.x

[lukaso (Lukas Oberhuber)]

Confirming that rav1e free ports has worked on my builds.

[mascguy (Christopher Nielsen)]

Has duplicate issue:68272

[mascguy (Christopher Nielsen)]

For new folks just added to this ticket: Ports which previously depended on rav1e, no longer do so by default:

• ffmpeg
• ffmpeg-devel
• ffmpeg6
• libavif
• libheif

The only port which still does, is av1an. And the latter is a standalone port, which isn't pulled in by anything.

So, what's the takeaway? Unless you have a definite need for rav1e, it's no longer in play by default. And it can be uninstalled, eliminating the errors.

Ultimately we do hope to fix it eventually, and we have a work-in-progress PR for that:

• ​PR 20532 - rav1e: update for libgit2 1.7.x

But at least for the near term, you needn't worry about rav1e any longer. (Again, unless you have a definite need for it.)

[Iritscen]

I see, thanks for explaining. I uninstalled rav1e, and nothing seems to miss it :-)

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #68696.

[kencu (Ken)]

Has duplicate #68948 and the solution in #68948 that worked for the OP was to deactivate libgit2.

[mascguy (Christopher Nielsen)]

Replying to kencu:

Has duplicate #68948 and the solution in #68948 that worked for the OP was to deactivate libgit2.

Bear in mind that rav1e is no longer in play by default, per comment:29.

Still, glad to hear that users have an easy workaround!