Opened 4 years ago
Closed 4 years ago
#60987 closed defect (fixed)
meson: not finding pkg-config when building universal when using cross-files
Reported by: | kencu (Ken) | Owned by: | kencu (Ken) |
---|---|---|---|
Priority: | Normal | Milestone: | |
Component: | ports | Version: | |
Keywords: | Cc: | SoapZA, Gcenx, dbevans (David B. Evans) | |
Port: | meson graphene |
Description
To improve building universal with meson
we recently added machine cross-files. These are needed when the build configures itself differently (eg adding asm files) with different architectures.
This works, but there is an issue with pkg-config
, it appears, first noticed when rebuilding graphene +universal
.
meson
does not appear to be searching the PATH for pkg-config
when cross building, but it wants pkg-config
to be specified. meson
actually has a rather complicated (no surprise there) algorithm for searching for pkg-confg that will take some sorting out.
One thing that worked is to add the PKG_CONFIG
spec to the configure environment like this:
configure.env-append PKG_CONFIG=${prefix}/bin/pkg-config
We could do that in the meson
PortGroup, I suppose, or each port that uses meson and pkg-config could do it individually.
Attachments (1)
Change History (31)
Changed 4 years ago by kencu (Ken)
Attachment: | graphene-build-universal.txt added |
---|
comment:1 Changed 4 years ago by kencu (Ken)
Cc: | Gcenx added |
---|
comment:2 Changed 4 years ago by kencu (Ken)
comment:3 Changed 4 years ago by kencu (Ken)
Cc: | dbevans added |
---|
adding you in, Dave, as you're all over meson builds these days and you'll need to know what is going on here.
comment:4 Changed 4 years ago by Gcenx
I believe this should be handled by macports-base, when pkgconfig is set as a build dependence base should then add PKG_CONFIG=${prefix}/bin/pkg-config
to the environment.
comment:5 Changed 4 years ago by kencu (Ken)
This is a potentially useful bit, maybe:
allow_default_for_cross=False
perhaps that is a meson setting that allows the default pkg-config to be used in cross builds ? If so, perhaps we can set that in the meson portgroup...
comment:6 Changed 4 years ago by Gcenx
That could be the answer that’s within mesonbuild/cmake/executor.py and twice within mesonbuild/dependencies/base.py
Possibly setting those to true will resolve the pkg-config and also stop a potential issue with cmake
comment:7 Changed 4 years ago by Gcenx
Looks like we have a winner!, added the following into post-patch
reinplace "s|allow_default_for_cross=False|allow_default_for_cross=True|g" ${worksrcpath}/mesonbuild/dependencies/base.py \ ${worksrcpath}/mesonbuild/cmake/executor.py
Now meson finds pkg-config & cmake when using cross file
comment:8 follow-up: 9 Changed 4 years ago by kencu (Ken)
That indeed seems pretty neat and tidy.
Now, thinking caps on, is there any downside that we can anticipate?
Whenever we change software to work differently like this, it will probably, eventually, bite someone somehow. I think this is a minor patch, and serves our purposes quite well, with minimal risk for causing unexpected trouble. Certainly much easier than spec'ing pkg-config (and maybe cmake) all the time...
comment:9 Changed 4 years ago by Gcenx
Replying to kencu:
That indeed seems pretty neat and tidy.
Now, thinking caps on, is there any downside that we can anticipate?
Whenever we change software to work differently like this, it will probably, eventually, bite someone somehow. I think this is a minor patch, and serves our purposes quite well, with minimal risk for causing unexpected trouble. Certainly much easier than spec'ing pkg-config (and maybe cmake) all the time...
Only thing I can think of is if meson is being used to cross-compile for another system, but then again they will provide their own cross file and that can include the needed pkgconfig/cmake wrappers to use
Also tested building dav1d
+universal and no issues
comment:10 follow-up: 14 Changed 4 years ago by kencu (Ken)
Thanks!
So we now have several options, it appears:
- We could add non-volatile binaries to the existing cross-file:
[binaries] pkg-config = '/opt/local/bin/pkg-config' cmake = '/opt/local/bin/cmake'
- We could add PKG_CONFIG to the environment.
- We could write a cross-file specing all the binaries, including CXX and CC etc for each build as an overlay.
- We could patch meson to
allow_default_for_cross=True
.
All of those would appear to solve the problem -- which one is easiest and least likely to cause damage?
comment:11 follow-ups: 12 16 Changed 4 years ago by kencu (Ken)
I will admit I am leaning towards patching meson, as that looks most like what all the other build systems would be expected to work like.
comment:12 Changed 4 years ago by Gcenx
Replying to kencu:
I will admit I am leaning towards patching meson, as that looks most like what all the other build systems would be expected to work like.
As macports is using meson with the expectation to compile for the current system I think the above simply patching meson is ideal, cross files can still be configured to override this behavior by and end user if desired.
comment:13 Changed 4 years ago by kencu (Ken)
there is some easier-to-read-than-average info about meson here.
comment:14 Changed 4 years ago by Gcenx
Replying to kencu:
Thanks!
So we now have several options, it appears:
- We could add PKG_CONFIG to the environment.
This won't fully resolve meson it still won't find cmake, I believe macports-base should already be doing adding this to the environment when pkgconfig as listed as a built dependency
Replying to kencu:
there is some easier-to-read-than-average info about meson here <https://docs.mesa3d.org/meson.html>.
This seems similar to the example I had followed for my janky workaround, I'm leaning towards patching meson
comment:16 follow-up: 18 Changed 4 years ago by ryandesign (Ryan Carsten Schmidt)
Replying to Gcenx:
I believe this should be handled by macports-base, when pkgconfig is set as a build dependence base should then add
PKG_CONFIG=${prefix}/bin/pkg-config
to the environment.
I don't think there is any other situation where MacPorts base modifies the environment in response to a dependency being added or removed, so this would be an entirely new feature to add to MacPorts base, and I'm not convinced it's necessary or the correct solution, given that MacPorts has survived for 18 years without it.
If meson uniquely among all build systems needs to be told where pkg-config can be found, then the meson portgroup can do that.
Replying to kencu:
I will admit I am leaning towards patching meson, as that looks most like what all the other build systems would be expected to work like.
I would just urge extreme caution with making such changes. MacPorts provides ports not just for its own use but also for users to use, and users expect programs to work the way their developers intended. It's not our place to change how the tool works just to suit the purposes of MacPorts builds.
It is reasonable to open a discussion with the developers of meson, explain why this setting is causing us problems, and ask if there is any recommended way for us to override the default when doing MacPorts builds (it seems likely to me that there would be such a way), or maybe even if they want to change the default.
comment:17 Changed 4 years ago by kencu (Ken)
sage advice.
upstream will undoubtedly say to use their designed mechanism of adding the binaries to the cross files.
this is quite low-risk, as it is rare we would want to use a different pkg-config or cmake than our default one when building with meson.
dave is revbumping many ports at present, so we'd better do something soon, otherwise we risk having sucessfully-built but broken builds in the wild.
comment:18 Changed 4 years ago by Gcenx
Replying to ryandesign:
I don't think there is any other situation where MacPorts base modifies the environment in response to a dependency being added or removed, so this would be an entirely new feature to add to MacPorts base, and I'm not convinced it's necessary or the correct solution, given that MacPorts has survived for 18 years without it.
You have a point, under normal conditions this won’t be needed.
Replying to ryandesign:
If meson uniquely among all build systems needs to be told where pkg-config can be found, then the meson portgroup can do that.
The reason meson is now requiring to know where pkg-config/cmake/ect is due to it being used as a cross compiler not as a native compiler when building +universal, I’d guess that will be expanded later to cover all compilations.
Replying to ryandesign:
I would just urge extreme caution with making such changes. MacPorts provides ports not just for its own use but also for users to use, and users expect programs to work the way their developers intended. It's not our place to change how the tool works just to suit the purposes of MacPorts builds.
I don’t disagree with the stance, patching meson could potentially cause an issue when used for real cross compilation of the crossfile doesn’t provide its own wrappers for pkg-config/cmake/ect for the intended target.
Replying to ryandesign:
It is reasonable to open a discussion with the developers of meson, explain why this setting is causing us problems, and ask if there is any recommended way for us to override the default when doing MacPorts builds (it seems likely to me that there would be such a way), or maybe even if they want to change the default.
There stance has been to use the cross files (check DevKitPro as an example), however reaching out to them for a possible way to make use of allow_default_for_cross=True
or similar as an environment variable would be more than ideal as the meson portgroup could just set this and then not need to worry about listing all possible binaries within the crossfile macports uses & also won’t affect an end users crossfiles in anyway.
comment:19 Changed 4 years ago by Gcenx
Here is the issue that resulted in the current behavior https://github.com/mesonbuild/meson/issues/7276
Meson will find build machine binaries for host if the build machine and host machine are the same.
comment:20 Changed 4 years ago by kencu (Ken)
Right. Thanks for finding that. I'm a bit unsettled by this:
This alters cmake, pkg-config, and all config-tool based dependencies.
Just how many binaries are we going to have to start spec'ing if we go that route?
comment:21 Changed 4 years ago by Gcenx
@kencu of your near your system could you patch meson as I explained above and then make a cross file that targets Linux and see if it picks up the native copy of pkg-config?
Edit; Dam auto correct ugh
comment:22 follow-up: 23 Changed 4 years ago by Gcenx
It seems that currently the solution will be adding the needed binaries into the cross-files or getting meson to resolve the issue that caused this.
[95/98] /opt/local/bin/nasm -f macho64 -I /opt/local/var/macports/build/_Users_gcenx_ports_multimedia_dav1d/dav1d/work/dav1d_0.7.1-i386/src/ -I /opt/local/var/macports/build/_Users_gcenx_ports_multimedia_dav1d/dav1d/work/build-i386/ -MQ src/libdav1d.4.dylib.p/mc_sse.obj -MF src/libdav1d.4.dylib.p/mc_sse.obj.ndep ../dav1d_0.7.1-i386/src/x86/mc_sse.asm -o src/libdav1d.4.dylib.p/mc_sse.obj
Here is something I noticed while doing a native compile, meson is passing macho64 into the 32Bit build when it should be passing macho32. If we or meson can resolve this issue then possibly the crossfiles won't be required.
comment:23 follow-up: 24 Changed 4 years ago by kencu (Ken)
Replying to Gcenx:
Here is something I noticed while doing a native compile, meson is passing macho64 into the 32Bit build when it should be passing macho32. If we or meson can resolve this issue then possibly the crossfiles won't be required.
That is exactly why the cross files are needed; they fix that issue, and many similar issues in many other ports.
We'll keep the cross files in MacPorts (many other builds are fixes by them too) and sort out the (small) issue with the binaries by one of the above methods.
comment:24 Changed 4 years ago by Gcenx
Replying to kencu:
That is exactly why the cross files are needed; they fix that issue, and many similar issues in many other ports.
We'll keep the cross files in MacPorts (many other builds are fixes by them too) and sort out the (small) issue with the binaries by one of the above methods.
I was testing out using the --native-file=
in some cases that’s enough but of course that’s not the case for macOS.
I’ll give another go with --cross-file=
tomorrow with a properly configured target build for Linux to see if the patching affects these builds, else will be needing to add [binaries] section
comment:25 Changed 4 years ago by kencu (Ken)
The problem with adding a binaries section is not cmake and pkg-config, it's the 100 other restricted binaries we don't know about, like curl-config, and llvm-config, and blah blah blah blah blah.
And - to make it worse -- things like llvm-config are not static, and they are set on-the-fly by the Portfile as the port needs.
So that path -- a static set of binaries listed in the cross file -- is not really viable, beyond pkg-config and cmake.
SO -- we either write full cross file with all the binaries spec'd (which is a huge PITA) -- or we patch out that meson change, and revert back to it finding the default binaries on the PATH -- which works perfectly well for every application we need it to work for in macports, I think.
comment:26 Changed 4 years ago by Gcenx
Yes the crossfiles are undesirable due to this but that's meson.
I decided to properly test out meson when patched with allow_default_for_cross=True
[target_machine] system = 'linux' cpu_family = 'x86' cpu = 'i386' endian = 'little' [binaries] c = '/usr/bin/clang'
MacBook-Pro:~ gcenx$ meson /Users/gcenx/Downloads/graphene /Users/gcenx/Downloads/graphene_build --cross-file i386-linux The Meson build system Version: 0.55.0 Source dir: /Users/gcenx/Downloads/graphene Build dir: /Users/gcenx/Downloads/graphene_build Build type: cross build Project name: graphene Project version: 1.10.2 C compiler for the build machine: cc (clang 11.0.3 "Apple clang version 11.0.3 (clang-1103.0.32.62)") C linker for the build machine: cc ld64 556.6 C compiler for the host machine: /usr/bin/clang (clang 11.0.3 "Apple clang version 11.0.3 (clang-1103.0.32.62)") C linker for the host machine: /usr/bin/clang ld64 556.6 Build machine cpu family: x86_64 Build machine cpu: x86_64 Host machine cpu family: x86_64 Host machine cpu: x86_64 Target machine cpu family: x86 Target machine cpu: i386 Compiler for C supports arguments -ffast-math: YES Compiler for C supports arguments -fstrict-aliasing: YES Compiler for C supports arguments -Wpointer-arith: YES Compiler for C supports arguments -Wstrict-prototypes: YES Compiler for C supports arguments -Wnested-externs: YES Compiler for C supports arguments -Wold-style-definition: YES Compiler for C supports arguments -Wunused: YES Compiler for C supports arguments -Wmissing-noreturn: YES Compiler for C supports arguments -Wmissing-format-attribute: YES Compiler for C supports arguments -Wlogical-op: NO Compiler for C supports arguments -Wcast-align: YES Compiler for C supports arguments -Wno-unused-local-typedefs: YES Compiler for C supports arguments -Werror=float-conversion: YES Compiler for C supports arguments -Werror=float-equal: YES Compiler for C supports arguments -Werror=redundant-decls: YES Compiler for C supports arguments -Werror=missing-prototypes: YES Compiler for C supports arguments -Werror=missing-declarations: YES Compiler for C supports arguments -Werror=format=2: YES Compiler for C supports arguments -Werror=uninitialized: YES Compiler for C supports arguments -Werror=shadow: YES Compiler for C supports arguments -Werror=implicit: YES Compiler for C supports arguments -Werror=init-self: YES Compiler for C supports arguments -Werror=main: YES Compiler for C supports arguments -Werror=missing-braces: YES Compiler for C supports arguments -Werror=return-type: YES Compiler for C supports arguments -Werror=array-bounds: YES Compiler for C supports arguments -Werror=write-strings: YES Compiler for C supports arguments -Werror=undef: YES Library m found: YES Run-time dependency threads found: YES Has header "stdlib.h" : YES Has header "stdint.h" : YES Has header "stdbool.h" : YES Has header "memory.h" : YES Has header "pthread.h" with dependency threads: YES Checking for function "memalign" : NO Checking for function "_aligned_malloc" : NO Checking for function "aligned_alloc" : NO Checking for function "posix_memalign" : YES Checking for function "sincosf" with dependency -lm: NO Checking for function "isinff" with dependency -lm: NO Checking for function "isnanf" with dependency -lm: NO Message: Enabling various debug infrastructure Did not find pkg-config by name 'pkg-config' Found Pkg-config: NO Did not find CMake 'cmake' Found CMake: NO Run-time dependency gobject-2.0 found: NO (tried pkgconfig, framework and cmake) Looking for a fallback subproject for the dependency gobject-2.0 Subproject directory not found and glib.wrap file not found Subproject subprojects/glib is buildable: NO (disabling) Dependency gobject-2.0 from subproject subprojects/glib found: NO Checking if "SSE intrinsics" compiles: YES Checking if "GCC vector intrinsics" compiles: NO Checking if "ARM NEON intrinsics" compiles: NO WARNING: Project targeting '>= 0.50.1' but tried to use feature deprecated since '0.48.0': python3 module. Configuring graphene-config.h using configuration Configuring graphene-version.h using configuration Configuring config.h using configuration Run-time dependency mutest-1 found: NO (tried pkgconfig, framework and cmake) Looking for a fallback subproject for the dependency mutest-1 Subproject exists but has no meson.build file Subproject subprojects/mutest is buildable: NO (disabling) Dependency mutest-1 from subproject subprojects/mutest found: NO Build targets in project: 1 WARNING: Deprecated features used: * 0.48.0: {'python3 module'} graphene 1.10.2 Subprojects glib: NO Subproject directory not found and glib.wrap file not found mutest: NO Subproject exists but has no meson.build file Found ninja-1.10.0 at /opt/local/bin/ninja WARNING: Cross file does not specify strip binary, result will not be stripped. WARNING: Cross file does not specify strip binary, result will not be stripped.
As can be seen from these results pkg-config is not found when cross-compiling for a different target, it's only found if both host and target match.
comment:27 Changed 4 years ago by kencu (Ken)
If using a cross file, I think pkg-config
and friends are never found unless specified, so long as allow_default_for_cross=False
.
For MacPorts' purposes, this is just not what we want, and none of our other build systems work like that.
If we change to allow_default_for_cross=True
then things will work like everyone would expect them to work for MacPorts, and you can still override them with a crossfile of your own construction if you are building for ChromeOS on Apple Silicon with and x86_64 toolchain, or some other bastardization of a build plan.
comment:28 Changed 4 years ago by Gcenx
Yes
When allow_default_for_cross=False
is set, pkg-config and friends are never found when using a crossfile, setting allow_default_for_cross=True
pkg-config and friends are found as long as host and target match. If target and host don’t match then pkg-config and friends won’t be found.
To clarify setting allow_default_for_cross=True
restore the prior behavior from meson.
comment:29 Changed 4 years ago by kencu (Ken)
NB: There have been several ports updated recently that use meson and pkg-config. Some of these may now be broken in various ways.
comment:30 Changed 4 years ago by kencu (Ken)
Owner: | set to kencu |
---|---|
Resolution: | → fixed |
Status: | new → closed |
In the case of
graphene
, it built, but because it doesn't findpkg-config
it doesn't add the neededgobject-2.0
references, and later builds that usegraphene
fail for that reason: