#63443 closed defect (fixed)
ffmpeg-4.4_3+gpl2: build failures: libffi dylib version bump breaks core ports like LLVM
Reported by: | Tommaso93 | Owned by: | mascguy (Christopher Nielsen) |
---|---|---|---|
Priority: | Normal | Milestone: | |
Component: | ports | Version: | 2.7.1 |
Keywords: | Cc: | dbevans (David B. Evans), hapaguy (Brian Kurt Fujikawa), splaisan (Stephane Plaisance), JDLH (Jim DeLaHunt), ATL-Flaneur (Andreas Yankopolus), dsavransky (Dmitry Savransky), amagela (Anthony M. Agelastos), jeremyhu (Jeremy Huddleston Sequoia) | |
Port: | ffmpeg libffi |
Description
Dear experts,
I am having some issues trying to upgrade ffmpeg to the last version. In attachment you can find the main log from the build.
Best, Tommaso
Attachments (2)
Change History (32)
Changed 3 years ago by Tommaso93
comment:1 Changed 3 years ago by sambthompson (Sam Thompson)
Cc: | sambthompson added |
---|
comment:2 Changed 3 years ago by sambthompson (Sam Thompson)
Changed 3 years ago by sambthompson (Sam Thompson)
Attachment: | main.2.log added |
---|
xcode 8.2.1 on 10.11.6 main.log
comment:3 Changed 3 years ago by ATL-Flaneur (Andreas Yankopolus)
I'm getting the same slew of Undefined symbols for architecture x86_64
error messages on my system (MacOS 11.5.2 Intel, Xcode 12.5.1).
comment:4 Changed 3 years ago by hapaguy (Brian Kurt Fujikawa)
Cc: | hapaguy added |
---|
comment:5 follow-up: 6 Changed 3 years ago by JDLH (Jim DeLaHunt)
Build failed for me. ffmpeg @4.4_4+gpl2+x11 on macOS 10.14 Mojave.
It is interesting that the log says arch i386
. Doesn't that mean 32-bit, and isn't 32-bit unsupported on macOS 10.14 Mojave?
Excerpt of the interesting parts of the main.log:
ffmpeg @4.4_4+gpl2+x11 :debug:sysinfo macOS 10.14.6 (darwin/18.7.0) arch i386 :debug:sysinfo MacPorts 2.7.1 :debug:sysinfo Xcode 11.2.1 :debug:sysinfo SDK 10.14 :debug:sysinfo MACOSX_DEPLOYMENT_TARGET: 10.14 … [elided] … :info:build Undefined symbols for architecture x86_64: :info:build "_ff_butterflies_fixed_sse2", referenced from: :info:build _ff_fixed_dsp_init_x86 in fixed_dsp_init.o :info:build "_ff_butterflies_float_sse", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_cpu_cpuid", referenced from: :info:build _ff_get_cpu_flags_x86 in cpu.o :info:build "_ff_cpu_xgetbv", referenced from: :info:build _ff_get_cpu_flags_x86 in cpu.o :info:build "_ff_evaluate_lls_sse2", referenced from: :info:build _ff_init_lls_x86 in lls_init.o :info:build "_ff_image_copy_plane_uc_from_sse4", referenced from: :info:build _ff_image_copy_plane_uc_from_x86 in imgutils_init.o :info:build "_ff_pixelutils_sad_16x16_mmxext", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_pixelutils_sad_16x16_sse2", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_pixelutils_sad_32x32_avx2", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_pixelutils_sad_32x32_sse2", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_pixelutils_sad_8x8_mmx", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_pixelutils_sad_8x8_mmxext", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_pixelutils_sad_a_16x16_sse2", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_pixelutils_sad_a_32x32_avx2", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_pixelutils_sad_a_32x32_sse2", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_pixelutils_sad_u_16x16_sse2", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_pixelutils_sad_u_32x32_avx2", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_pixelutils_sad_u_32x32_sse2", referenced from: :info:build _ff_pixelutils_sad_init_x86 in pixelutils_init.o :info:build "_ff_scalarproduct_float_sse", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_update_lls_avx", referenced from: :info:build _ff_init_lls_x86 in lls_init.o :info:build "_ff_update_lls_fma3", referenced from: :info:build _ff_init_lls_x86 in lls_init.o :info:build "_ff_update_lls_sse2", referenced from: :info:build _ff_init_lls_x86 in lls_init.o :info:build "_ff_vector_dmac_scalar_avx", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_dmac_scalar_fma3", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_dmac_scalar_sse2", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_dmul_avx", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_dmul_scalar_avx", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_dmul_scalar_sse2", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_dmul_sse2", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmac_scalar_avx", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmac_scalar_fma3", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmac_scalar_sse", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmul_add_avx", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmul_add_fma3", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmul_add_sse", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmul_avx", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmul_reverse_avx", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmul_reverse_avx2", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmul_reverse_sse", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmul_scalar_sse", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmul_sse", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmul_window_3dnowext", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build "_ff_vector_fmul_window_sse", referenced from: :info:build _ff_float_dsp_init_x86 in float_dsp_init.o :info:build ld: symbol(s) not found for architecture x86_64 :info:build clang: error: linker command failed with exit code 1 (use -v to see invocation) :info:build gmake: *** [ffbuild/library.mak:103: libavutil/libavutil.56.dylib] Error 1 … [elided] …
comment:6 follow-up: 11 Changed 3 years ago by sambthompson (Sam Thompson)
Replying to JDLH:
Build failed for me. ffmpeg @4.4_4+gpl2+x11 on macOS 10.14 Mojave.
It is interesting that the log says
arch i386
. Doesn't that mean 32-bit, and isn't 32-bit unsupported on macOS 10.14 Mojave?
It could be a mismatch in arch, anyway; I'm on El Capitan which still supports 32-bit.
Same message re: i386 in my build log.
comment:8 Changed 3 years ago by JDLH (Jim DeLaHunt)
Cc: | JDLH added |
---|
comment:10 Changed 3 years ago by ATL-Flaneur (Andreas Yankopolus)
Cc: | ATL-Flaneur added |
---|
comment:11 Changed 3 years ago by sambthompson (Sam Thompson)
Replying to sambthompson:
Replying to JDLH:
Build failed for me. ffmpeg @4.4_4+gpl2+x11 on macOS 10.14 Mojave.
It is interesting that the log says
arch i386
. Doesn't that mean 32-bit, and isn't 32-bit unsupported on macOS 10.14 Mojave?It could be a mismatch in arch, anyway; I'm on El Capitan which still supports 32-bit.
Same message re: i386 in my build log.
Hmm. 4.4_4 just built OK for me during a port upgrade outdated
.
comment:12 Changed 3 years ago by sambthompson (Sam Thompson)
Cc: | sambthompson removed |
---|
comment:13 Changed 3 years ago by dsavransky (Dmitry Savransky)
Cc: | dsavransky added |
---|
comment:14 follow-up: 22 Changed 3 years ago by msbit (Tom Sullivan)
I had the same issue while running port upgrade outdated
, and while I am not claiming it's the same for every other report, the cause for mine may be relevant.
When troubleshooting the issue, I had tried to run nm
on some of the output .o
files, but that had failed with:
dyld: Library not loaded: /opt/local/lib/libffi.7.dylib Referenced from: /opt/local/libexec/llvm-10/lib/libLLVM.dylib Reason: image not found
which seems to be due to a big migration from libffi.7.dylib
to libffi.8.dylib
that happened while I'd been neglecting this machine? Anyhow, I didn't think much of it, reinstalling with libffi
, then later on performing a rebuild of ffmpeg directly (which worked fine). Poring over the output files under /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_ffmpeg/ffmpeg/work
I discovered one big difference in the config.log
files, namely:
-check_inline_asm inline_asm_direct_symbol_refs "movl _test, %eax" - 1 void foo(void){ __asm__ volatile("movl _test, %eax"); } -void foo(void){ __asm__ volatile("movl _test, %eax"); } +check_inline_asm inline_asm_direct_symbol_refs "movl test, %eax" + 1 void foo(void){ __asm__ volatile("movl test, %eax"); } +void foo(void){ __asm__ volatile("movl test, %eax"); } -check_inline_asm inline_asm_direct_symbol_refs "movl _test(%rip), %eax" +check_inline_asm inline_asm_direct_symbol_refs "movl test(%rip), %eax" - 1 void foo(void){ __asm__ volatile("movl _test(%rip), %eax"); } + 1 void foo(void){ __asm__ volatile("movl test(%rip), %eax"); }
The big giveaway is the presence of the _
prefix for the symbol on the separate build (post libffi
installation).
In the configure script, nm
is used to determine what the external prefix should be for name mangling. If nm
fails like it did for me, the prefix is empty, but if it succeeds the prefix is _
. This is also used to toggle defining PREFIX
when invoking nasm
, which ensures that the mangled symbols have the same _
prefix.
So, finally, looking over just _ff_butterflies_fixed_sse2
, I've got:
nm ./libavutil/x86/fixed_dsp{,_init}.o | grep ff_butterflies_fixed_sse2$ 0000000000000000 T ff_butterflies_fixed_sse2 U _ff_butterflies_fixed_sse2
in the MacPorts ffmpeg work directory and:
nm ./libavutil/x86/fixed_dsp{,_init}.o | grep ff_butterflies_fixed_sse2$ 0000000000000000 T _ff_butterflies_fixed_sse2 U _ff_butterflies_fixed_sse2
in the standalone ffmpeg build.
I'm currently re-running port upgrade outdated
and the built object file has the correct symbol, with mangling:
nm ./libavutil/x86/fixed_dsp{,_init}.o | grep ff_butterflies_fixed_sse2$ 0000000000000000 T _ff_butterflies_fixed_sse2 U _ff_butterflies_fixed_sse2
comment:15 Changed 3 years ago by ATL-Flaneur (Andreas Yankopolus)
Same set of build errors after updating to Xcode 13.
comment:16 Changed 3 years ago by amagela (Anthony M. Agelastos)
Cc: | amagela added |
---|
comment:17 Changed 3 years ago by mascguy (Christopher Nielsen)
Cc: | mascguy added |
---|
comment:18 Changed 3 years ago by mascguy (Christopher Nielsen)
Owner: | set to jeremyhu |
---|---|
Status: | new → assigned |
comment:19 Changed 3 years ago by mascguy (Christopher Nielsen)
Cc: | jeremyhu removed |
---|
comment:20 Changed 3 years ago by mascguy (Christopher Nielsen)
Summary: | ffmpeg-4.4_3+gpl2: Build Failed → ffmpeg-4.4_3+gpl2: build failures: possibly related to libffi |
---|
Updated summary, based on the investigative work in comment:14
comment:21 Changed 3 years ago by amagela (Anthony M. Agelastos)
FWIW, I did the following and was able to upgrade ffmpeg. This may help point to libffi
and its recent update as the item to focus upon.
$ sudo port deactivate libffi @3.4.2_0 $ sudo port activate libffi @3.3_1 $ sudo port upgrade ffmpeg ---> Computing dependencies for ffmpeg ---> Fetching archive for ffmpeg ---> Attempting to fetch ffmpeg-4.4_4+gpl2.darwin_19.x86_64.tbz2 from https://packages.macports.org/ffmpeg ---> Attempting to fetch ffmpeg-4.4_4+gpl2.darwin_19.x86_64.tbz2 from https://nue.de.packages.macports.org/ffmpeg ---> Attempting to fetch ffmpeg-4.4_4+gpl2.darwin_19.x86_64.tbz2 from http://atl.us.packages.macports.org/ffmpeg ---> Fetching distfiles for ffmpeg ---> Attempting to fetch ffmpeg-4.4.tar.xz from https://ffmpeg.org/releases/ ---> Verifying checksums for ffmpeg ---> Extracting ffmpeg ---> Applying patches to ffmpeg ---> Configuring ffmpeg ---> Building ffmpeg ---> Staging ffmpeg into destroot ---> Installing ffmpeg @4.4_4+gpl2 ---> Cleaning ffmpeg ---> Computing dependencies for ffmpeg ---> Deactivating ffmpeg @4.4_2+gpl2 ---> Cleaning ffmpeg ---> Activating ffmpeg @4.4_4+gpl2 ---> Cleaning ffmpeg
comment:22 follow-up: 26 Changed 3 years ago by mascguy (Christopher Nielsen)
Summary: | ffmpeg-4.4_3+gpl2: build failures: possibly related to libffi → ffmpeg-4.4_3+gpl2: build failures: libffi dylib version bump breaks core ports like LLVM |
---|
Replying to msbit:
When troubleshooting the issue, I had tried to run
nm
on some of the output.o
files, but that had failed with:dyld: Library not loaded: /opt/local/lib/libffi.7.dylib Referenced from: /opt/local/libexec/llvm-10/lib/libLLVM.dylib Reason: image not foundwhich seems to be due to a big migration from
libffi.7.dylib
tolibffi.8.dylib
that happened while I'd been neglecting this machine?
Just experienced a similar situation, with some macOS VMs that were at least a month out-of-date [in terms of MacPorts].
What's clear is that a libffi upgrade temporarily breaks foundational ports like LLVM, when the dylib major version changes. And that seems like a big problem.
To avoid this situation, I'd like to update libffi to create a dylib symlink, with the previous major version. So when it's updated to libffi.8.dylib
, for example, symlink libffi.7.dylib
to it. That will prevent breakage of LLVM, along with these related headaches. (The port would determine the previous relative major version, from the current built version. So nothing would be hard-coded.)
Thoughts?
Longer-term, perhaps LLVM components should utilize a separate libffi. I'm thinking in terms of how we avoid circular dependencies among toolchain components, by utilizing separate "bootstrap" ports. (So perhaps utilize a libffi-bootstrap
for all LLVM/toolchain components.) Or alternatively, statically link with libffi... assuming that doesn't cause issues?
comment:23 Changed 3 years ago by mascguy (Christopher Nielsen)
Cc: | jeremyhu added; mascguy removed |
---|---|
Owner: | changed from jeremyhu to mascguy |
Port: | libffi added |
comment:25 Changed 3 years ago by Christopher Nielsen <mascguy@…>
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
comment:26 Changed 3 years ago by jmroot (Joshua Root)
Replying to mascguy:
What's clear is that a libffi upgrade temporarily breaks foundational ports like LLVM, when the dylib major version changes. And that seems like a big problem.
All that's clear from this ticket is that ffmpeg is using cctools' nm without declaring a dependency. It should either ensure that the Xcode nm is used, or depend on cctools. In the latter case, llvm would then be upgraded to the revision linked with the new libffi, before ffmpeg is built.
comment:27 follow-up: 28 Changed 3 years ago by mascguy (Christopher Nielsen)
Great point, and perhaps that's enough to fix ffmpeg
.
But are you certain that we don't have a fair number of other ports, which opportunistically utilize cctools
components?
comment:28 Changed 3 years ago by jmroot (Joshua Root)
Replying to mascguy:
Great point, and perhaps that's enough to fix
ffmpeg
.But are you certain that we don't have a fair number of other ports, which opportunistically utilize
cctools
components?
If you want to look at installing cctools differently to prevent accidental usage, that's fine.
comment:29 Changed 3 years ago by Christopher Nielsen <mascguy@…>
comment:30 Changed 3 years ago by mascguy (Christopher Nielsen)
In retrospect, perhaps we all need to utilize trace mode more often, particularly when tackling major upgrades from upstream projects. (I realize that trace mode occasionally causes issues, and won't work 100% of the time. But still not a bad thing to try.)
More recently, I've been handling this by closely scrutinizing output from configure
(or the equivalent, for the build system used by a given project), and/or running configure --help
. It's more time-consuming to be sure, but pays off longer-term.
Seeing the same errors on 10.11.6 building variant +gpl2+nonfree; message of interest is:
while linking symbols with _ff_ prefixes.