Opened 14 months ago

Last modified 10 months ago

#68178 assigned defect

rav1e: build fails due to git_allocator missing members

Reported by: lukaso (Lukas Oberhuber) Owned by: mascguy (Christopher Nielsen)
Priority: Normal Milestone:
Component: ports Version: 2.8.1
Keywords: Cc: MarcusCalhoun-Lopez (Marcus Calhoun-Lopez), Dave-Allured (Dave Allured), ddrum2000, Iritscen, pspanja (Petar Španja), ahrex, hnarayanan (Harish Narayanan), sudheerhebbale (Sudheer Hebbale), phloxic (Christian Ebert)
Port: rav1e

Description (last modified by ryandesign (Ryan Carsten Schmidt))

First reported on #68026, via @macsguy:

This may require updating the version of the git2 crate [used by rav1e], now that libgit2 has been updated to 1.7.1:

libgit2/src/util/array.h:59:19: error: no member named 'greallocarray' in 'git_allocator'
        if ((new_array = git__reallocarray(a->ptr, new_size, item_size)) == NULL)
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

libgit2/src/libgit2/annotated_commit.c:34:21: error: no member named 'gcalloc' in 'git_allocator'
        annotated_commit = git__calloc(1, sizeof(git_annotated_commit));
                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

libgit2/src/libgit2/annotated_commit.c:48:34: error: no member named 'gstrdup' in 'git_allocator'
        annotated_commit->description = git__strdup(description);
                                        ^~~~~~~~~~~~~~~~~~~~~~~~

I gave a shot at trying to update the crates, but got errors after updating the crate list:

:info:extract error: failed to select a version for the requirement `git2 = "^0.15"`
:info:extract candidate versions found which didn't match: 0.18.0
:info:extract location searched: directory source `/Users/lukasoberhuber/macports-gimp3-arm64/var/macports/build/_Users_lukasoberhuber_project_ports_multimedia_rav1e/rav1e/work/.home/.cargo/macports` (which is replacing registry `crates-io`)
:info:extract required by package `built v0.5.2`
:info:extract     ... which satisfies dependency `built = "^0.5.2"` of package `rav1e v0.6.6 (/Users/lukasoberhuber/macports-gimp3-arm64/var/macports/build/_Users_lukasoberhuber_project_ports_multimedia_rav1e/rav1e/work/rav1e-0.6.6)`
:info:extract perhaps a crate was updated and forgotten to be re-vendored?
:info:extract As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.
:info:extract Command failed: /Users/lukasoberhuber/macports-gimp3-arm64/bin/cargo --offline update
:info:extract Exit code: 101

Attachments (3)

rav1e_main.log (2.6 MB) - added by lukaso (Lukas Oberhuber) 14 months ago.
Has multiple builds in it, so just focus on last one (they appear to be all the same, which might mean that macports tried to run it multiple times)
rav1e_main.2.log (448.6 KB) - added by lukaso (Lukas Oberhuber) 14 months ago.
Post force uninstall and clean of rav1e
rav1e_x86_64_main.log (327.5 KB) - added by lukaso (Lukas Oberhuber) 14 months ago.

Change History (38)

Changed 14 months ago by lukaso (Lukas Oberhuber)

Attachment: rav1e_main.log added

Has multiple builds in it, so just focus on last one (they appear to be all the same, which might mean that macports tried to run it multiple times)

comment:1 Changed 14 months ago by mascguy (Christopher Nielsen)

Cc: MarcusCalhoun-Lopez added
Description: modified (diff)
Owner: set to mascguy
Status: newassigned
Summary: rav1e fails to build on arm64rav1e: build fails on arm, due to git_allocator missing members; git2 crate [used by rav1e] may need to be updated
Version: 2.8.1

comment:2 Changed 14 months ago by mascguy (Christopher Nielsen)

Keywords: arm64 added

comment:3 Changed 14 months ago by rottegift (Sean Doran)

FWIW, I just built successfully with

port -v install rav1e configure.ldflags="-ld_classic -force_load /usr/lib/libiconv.dylib" configure.cflags="-Wl,-ld_classic"

which is ugly, but works up to not-yet-found problems with differing header files

otool -L /opt/local/bin/rav1e
/opt/local/bin/rav1e:
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/opt/local/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.0.0)

the -force_load probably is portable to older macOS/xcode.

I don't think this is the right solution, but if one was, say, in the middle of building something with rav1e as a dependency, this might get the job done.

Last edited 14 months ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:4 Changed 14 months ago by mascguy (Christopher Nielsen)

Lukas, my thoughts regarding updating the version of the git2 crate may be off in left field, given the following:

The source for libgit2 is included in the libgit2-sys crate so there's no need to pre-install the libgit2 library, the libgit2-sys crate will figure that and/or build that for you.

https://github.com/rust-lang/git2-rs#version-of-libgit2

Given that - combined with the fact that we haven't changed the rav1e port - we should be fine as-is.

Can you try a clean build of rav1e, without any prior history/log, to ensure the same results occur?

comment:5 Changed 14 months ago by Dave-Allured (Dave Allured)

Cc: Dave-Allured added

comment:6 Changed 14 months ago by lukaso (Lukas Oberhuber)

I did a forced uninstall and a port clean. If I should do any more clean than that, let me know how.

Attaching the new log.

Changed 14 months ago by lukaso (Lukas Oberhuber)

Attachment: rav1e_main.2.log added

Post force uninstall and clean of rav1e

comment:7 Changed 14 months ago by lukaso (Lukas Oberhuber)

OK, this actually happens on x86_64 as well. Attaching the log. Same problem.

Changed 14 months ago by lukaso (Lukas Oberhuber)

Attachment: rav1e_x86_64_main.log added

comment:8 Changed 14 months ago by mascguy (Christopher Nielsen)

Summary: rav1e: build fails on arm, due to git_allocator missing members; git2 crate [used by rav1e] may need to be updatedrav1e: build fails due to git_allocator missing members

comment:9 Changed 14 months ago by mascguy (Christopher Nielsen)

Keywords: arm64 removed

comment:10 Changed 14 months ago by lukaso (Lukas Oberhuber)

So it looks like these missing functions were removed in v1.7.1. So that's why we are stuffed. Also meaning they are not using semantic versioning (boo on them). https://github.com/libgit2/libgit2/commit/9dd1bfe81c3aba6ce35ed85bf62fa75d9c61e1a6.

The solution is probably to revert libgit2 in macports. Clearly the build is not pulling the embedded version when it is linking (or so it seems).

comment:11 in reply to:  10 ; Changed 14 months ago by mascguy (Christopher Nielsen)

Replying to lukaso:

The solution is probably to revert libgit2 in macports. Clearly the build is not pulling the embedded version when it is linking (or so it seems).

No, that's probably not going to happen, unless this is more wide-spread. And given that other Rust-based ports build fine with the latest release of libgit2 - including cargo and rust itself - that strongly suggests this is a rav1e-specific issue.

comment:12 in reply to:  11 ; Changed 14 months ago by lukaso (Lukas Oberhuber)

Replying to mascguy:

Replying to lukaso:

The solution is probably to revert libgit2 in macports. Clearly the build is not pulling the embedded version when it is linking (or so it seems).

No, that's probably not going to happen, unless this is more wide-spread. And given that other Rust-based ports build fine with the latest release of libgit2 - including cargo and rust itself - that strongly suggests this is a rav1e-specific issue.

Here's the thing to revert: [c00e8c728a0458fcf0e1d2c555126d6bb642feaa/macports-ports] and here's the PR in libgit2 which is labeled as sys api breaking: https://github.com/libgit2/libgit2/pull/6563.

This port has been broken for over a month now for one reason or another. It was also last broken in december 2022. Maybe it's time to cut rav1e loose? I tried to remove the dependency in the GIMP build by excluding it from libheif but it appears to be included in some other port that somehow I am including so that didn't help.

Last edited 14 months ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:13 in reply to:  12 Changed 14 months ago by mascguy (Christopher Nielsen)

Replying to lukaso:

This port has been broken for over a month now for one reason or another. It was also last broken in december 2022. Maybe it's time to cut rav1e loose? I tried to remove the dependency in the GIMP build by excluding it from libheif but it appears to be included in some other port that somehow I am including so that didn't help.

Okay, just tried building rav1e from source. And sure enough, our port is currently broken.

But let me stress that's it's very likely our port that's broken, not upstream.

Why do I say that? Well, take a look at the portfile for rav1e, and specifically the crate for libgit2-sys:

libgit2-sys 0.14.2+1.5.1

That crate corresponds to libgit2 1.5.1. Whereas git2-rs upstream has released later releases, including the latest: libgit2-sys-0.16.1+1.7.1. And the latter corresponds to the latest libgit2 release. (Per the +1.7.1 suffix.) Which is what we should probably be used. Though perhaps one of the previous releases, corresponding to libgit2 1.6.x, might work too. Dunno without more research/testing.

In any case, updating that crate, likely involves updating others as well. Perhaps that's automated via a MacPorts script somewhere? (Not a Rust guru here, otherwise I would have fixed this by now.)

So I'm quite certain this is 100% our issue.

As for downgrading libgit2: Simply reverting one commit isn't going to cut it. We'd also need to downgrade py-pygit2, and a number of others. And then rebuild every dependent - including rust and cargo, along with numerous other dependents. (As the downgrade would break the ABI.) So no, not going to happen, when it seems very clear the rav1e port is the one that needs to be fixed.

comment:14 Changed 14 months ago by mascguy (Christopher Nielsen)

Relative to eliminating rav1e from the dependency equation: Most of that's in-place already.

The ports that depend on it, are the following:

$ gfind . -type f -name "Portfile" -print0 | gxargs -0 -r ggrep -E --regexp=":rav1e"
./multimedia/av1an/Portfile:                    port:rav1e \
./multimedia/ffmpeg/Portfile:                    port:rav1e
./multimedia/ffmpeg-devel/Portfile:                    port:rav1e
./multimedia/ffmpeg6/Portfile:                    port:rav1e
./multimedia/libavif/Portfile:                        port:rav1e \
./multimedia/libheif/Portfile:                            port:rav1e \

You can ignore ffmpeg-devel and ffmpeg6 though, as you'll be using ffmpeg.

Regardless, all but two of those support variants that can eliminate rav1e from the equation:

  • For ffmpeg, enable variant +slim.
  • For av1an - assuming that's being pulled in for GIMP? - install with -av1.

As for libavif and libheif, I maintain both of those. And I'm happy to provide something similar there, if needed. Though it's trivial to overlay your own versions of those two, dropping rav1e from the deps:

  • For libavif, remove the dep, and update -DAVIF_CODEC_RAV1E:BOOL=ON to -DAVIF_CODEC_RAV1E:BOOL=OFF.
  • Likewise, for libheif, remove the dep, and add configure argument -DWITH_RAV1E:BOOL=OFF.

comment:15 in reply to:  14 ; Changed 14 months ago by lukaso (Lukas Oberhuber)

Regardless, all but two of those support variants that can eliminate rav1e from the equation:

  • For ffmpeg, enable variant +slim.
  • For av1an - assuming that's being pulled in for GIMP? - install with -av1.

As for libavif and libheif, I maintain both of those. And I'm happy to provide something similar there, if needed. Though it's trivial to overlay your own versions of those two, dropping rav1e from the deps:

  • For libavif, remove the dep, and update -DAVIF_CODEC_RAV1E:BOOL=ON to -DAVIF_CODEC_RAV1E:BOOL=OFF.
  • Likewise, for libheif, remove the dep, and add configure argument -DWITH_RAV1E:BOOL=OFF.

I've done the overlay now for libheif and libavif. Should have done this ages ago! I think it would be worthwhile creating slim or removerav1e versions for all of them since this appears to be a persistent issue. But I can move forward for now.

On an aside: It would be great to have much simpler tools for determining dependency...I'm always struggling to find what the dependencies are. Especially the dependencies all the way down.

comment:16 in reply to:  15 Changed 14 months ago by mascguy (Christopher Nielsen)

Replying to lukaso:

I've done the overlay now for libheif and libavif. Should have done this ages ago! I think it would be worthwhile creating slim or removerav1e versions for all of them since this appears to be a persistent issue. But I can move forward for now.

Sounds good, will try to tackle this over the next day or two.

On an aside: It would be great to have much simpler tools for determining dependency...I'm always struggling to find what the dependencies are. Especially the dependencies all the way down.

There definitely is, and macports provides a rich set of options to formally query for those. Take a look at man port, specifically the pseudo-portnames.

For example:

$ port info --index --line --name depends:rav1e
ffmpeg
ffmpeg-devel
ffmpeg6
libavif
libheif
av1an

Note: Option --index is optional, but it speeds things up tremendously. (Simply because it utilizes the portindex for queries.)

Along with various other ways to do it, such as port echo:

$ port echo depends:rav1e
ffmpeg
ffmpeg-devel
ffmpeg6
libavif
libheif
av1an

But take a thorough read through man port, for the various details. Cheers!

comment:17 Changed 14 months ago by ryandesign (Ryan Carsten Schmidt)

Description: modified (diff)

comment:18 Changed 14 months ago by mascguy (Christopher Nielsen)

Lukas, I'm making progress, relative to updating rav1e to use the latest version of libgit2-sys, etc. (It simply required some time to review how dependencies are handled within the Rust/Cargo space.)

Presently dealing with compilation failures related to libssh2, but will figure it out with enough time.

So hopefully this port will be fixed in the next few days, and you won't need to remove it from the mix long-term. Stay tuned...

comment:19 in reply to:  18 Changed 14 months ago by lukaso (Lukas Oberhuber)

Replying to mascguy:

Lukas, I'm making progress, relative to updating rav1e to use the latest version of libgit2-sys, etc. (It simply required some time to review how dependencies are handled within the Rust/Cargo space.)

Presently dealing with compilation failures related to libssh2, but will figure it out with enough time.

So hopefully this port will be fixed in the next few days, and you won't need to remove it from the mix long-term. Stay tuned...

Great. Thanks.

comment:20 Changed 14 months ago by Christopher Nielsen <mascguy@…>

In 20dd1dc069d78e41cb7241c91d6f2ad39d0679b5/macports-ports (master):

lib{avif,heif}: add new variant rav1e; disabled by default

See: #68026
See: #68178

comment:21 Changed 14 months ago by Christopher Nielsen <mascguy@…>

In c08417569d37bdbd835cbfe77c383c6028082bd8/macports-ports (master):

ffmpeg*: add variant rav1e, disabled by default

See: #68026
See: #68178

comment:22 Changed 14 months ago by lukaso (Lukas Oberhuber)

Thanks! Trying out the rav1e free ports now.

comment:23 Changed 14 months ago by lukaso (Lukas Oberhuber)

Worked great locally.

comment:24 Changed 14 months ago by mascguy (Christopher Nielsen)

Observers: The work-in-progress fixes for rav1e - which may take some time to figure out - are being done via PR:

PR 20532 - rav1e: update for libgit2 1.7.x

comment:25 Changed 14 months ago by lukaso (Lukas Oberhuber)

Confirming that rav1e free ports has worked on my builds.

comment:26 Changed 14 months ago by ddrum2000

Cc: ddrum2000 added

comment:27 Changed 13 months ago by Iritscen

Cc: Iritscen added

comment:28 Changed 13 months ago by mascguy (Christopher Nielsen)

Cc: pspanja ahrex added

Has duplicate issue:68272

comment:29 Changed 13 months ago by mascguy (Christopher Nielsen)

For new folks just added to this ticket: Ports which previously depended on rav1e, no longer do so by default:

  • ffmpeg
  • ffmpeg-devel
  • ffmpeg6
  • libavif
  • libheif

The only port which still does, is av1an. And the latter is a standalone port, which isn't pulled in by anything.

So, what's the takeaway? Unless you have a definite need for rav1e, it's no longer in play by default. And it can be uninstalled, eliminating the errors.

Ultimately we do hope to fix it eventually, and we have a work-in-progress PR for that:

But at least for the near term, you needn't worry about rav1e any longer. (Again, unless you have a definite need for it.)

comment:30 Changed 13 months ago by Iritscen

I see, thanks for explaining. I uninstalled rav1e, and nothing seems to miss it :-)

comment:31 Changed 13 months ago by hnarayanan (Harish Narayanan)

Cc: hnarayanan added

comment:32 Changed 12 months ago by ryandesign (Ryan Carsten Schmidt)

Cc: sudheerhebbale added

Has duplicate #68696.

comment:33 Changed 11 months ago by kencu (Ken)

Has duplicate #68948 and the solution in #68948 that worked for the OP was to deactivate libgit2.

comment:34 in reply to:  33 Changed 11 months ago by mascguy (Christopher Nielsen)

Replying to kencu:

Has duplicate #68948 and the solution in #68948 that worked for the OP was to deactivate libgit2.

Bear in mind that rav1e is no longer in play by default, per comment:29.

Still, glad to hear that users have an easy workaround!

comment:35 Changed 10 months ago by phloxic (Christian Ebert)

Cc: phloxic added
Note: See TracTickets for help on using tickets.