meson: not finding pkg-config when building universal when using cross-files

[kencu (Ken)]

To improve building universal with meson we recently added machine cross-files. These are needed when the build configures itself differently (eg adding asm files) with different architectures.

This works, but there is an issue with pkg-config, it appears, first noticed when rebuilding graphene +universal.

meson does not appear to be searching the PATH for pkg-config when cross building, but it wants pkg-config to be specified. meson actually has a rather complicated (no surprise there) algorithm for ​searching for pkg-confg that will take some sorting out.

One thing that worked is to add the PKG_CONFIG spec to the configure environment like this:

configure.env-append PKG_CONFIG=${prefix}/bin/pkg-config

We could do that in the meson PortGroup, I suppose, or each port that uses meson and pkg-config could do it individually.

[kencu (Ken)]

In the case of graphene, it built, but because it doesn't find pkg-config it doesn't add the needed gobject-2.0 references, and later builds that use graphene fail for that reason:

Found Pkg-config: NO
Found CMake: NO
Run-time dependency gobject-2.0 found: NO (tried pkgconfig, framework and cmake)

[kencu (Ken)]

adding you in, Dave, as you're all over meson builds these days and you'll need to know what is going on here.

[Gcenx]

I believe this should be handled by macports-base, when pkgconfig is set as a build dependence base should then add PKG_CONFIG=${prefix}/bin/pkg-config to the environment.

[kencu (Ken)]

This is a potentially useful bit, maybe:

allow_default_for_cross=False

perhaps that is a meson setting that allows the default pkg-config to be used in cross builds ? If so, perhaps we can set that in the meson portgroup...

[Gcenx]

That could be the answer that’s within mesonbuild/cmake/executor.py and twice within mesonbuild/dependencies/base.py

Possibly setting those to true will resolve the pkg-config and also stop a potential issue with cmake

[Gcenx]

Looks like we have a winner!, added the following into post-patch

    reinplace "s|allow_default_for_cross=False|allow_default_for_cross=True|g" ${worksrcpath}/mesonbuild/dependencies/base.py \
                                                                               ${worksrcpath}/mesonbuild/cmake/executor.py

Now meson finds pkg-config & cmake when using cross file

[kencu (Ken)]

That indeed seems pretty neat and tidy.

Now, thinking caps on, is there any downside that we can anticipate?

Whenever we change software to work differently like this, it will probably, eventually, bite someone somehow. I think this is a minor patch, and serves our purposes quite well, with minimal risk for causing unexpected trouble. Certainly much easier than spec'ing pkg-config (and maybe cmake) all the time...

[Gcenx]

Replying to kencu:

That indeed seems pretty neat and tidy.

Now, thinking caps on, is there any downside that we can anticipate?

Whenever we change software to work differently like this, it will probably, eventually, bite someone somehow. I think this is a minor patch, and serves our purposes quite well, with minimal risk for causing unexpected trouble. Certainly much easier than spec'ing pkg-config (and maybe cmake) all the time...

Only thing I can think of is if meson is being used to cross-compile for another system, but then again they will provide their own cross file and that can include the needed pkgconfig/cmake wrappers to use

Also tested building dav1d +universal and no issues

[kencu (Ken)]

Thanks!

So we now have several options, it appears:

1. We could add non-volatile binaries to the existing cross-file:

   [binaries]
   pkg-config = '/opt/local/bin/pkg-config'
   cmake = '/opt/local/bin/cmake'
   

2. We could add PKG_CONFIG to the environment.
3. We could write a cross-file specing all the binaries, including CXX and CC etc for each build as an overlay.
4. We could patch meson to allow_default_for_cross=True.

All of those would appear to solve the problem -- which one is easiest and least likely to cause damage?

[kencu (Ken)]

I will admit I am leaning towards patching meson, as that looks most like what all the other build systems would be expected to work like.

[Gcenx]

Replying to kencu:

I will admit I am leaning towards patching meson, as that looks most like what all the other build systems would be expected to work like.

As macports is using meson with the expectation to compile for the current system I think the above simply patching meson is ideal, cross files can still be configured to override this behavior by and end user if desired.

[kencu (Ken)]

there is some easier-to-read-than-average info about meson here <​https://docs.mesa3d.org/meson.html#:~:text=Meson%20supports%20cross-compilation%20by%20specifying%20a%20number%20of,if%20you%20put%20it%20in%20%24XDG_DATA_HOME%2Fmeson%2Fcross%20or%20~%2F.local%2Fshare%2Fmeson%2Fcross>.

[Gcenx]

Replying to kencu:

Thanks!

So we now have several options, it appears:

2. We could add PKG_CONFIG to the environment.

This won't fully resolve meson it still won't find cmake, I believe macports-base should already be doing adding this to the environment when pkgconfig as listed as a built dependency

Replying to kencu:

there is some easier-to-read-than-average info about meson here <​https://docs.mesa3d.org/meson.html>.

This seems similar to the example I had followed for my janky workaround, I'm leaning towards patching meson

[kencu (Ken)]

I had to go look up ​janky in the Urban Dictionary :>

[ryandesign (Ryan Carsten Schmidt)]

Replying to Gcenx:

I believe this should be handled by macports-base, when pkgconfig is set as a build dependence base should then add PKG_CONFIG=${prefix}/bin/pkg-config to the environment.

I don't think there is any other situation where MacPorts base modifies the environment in response to a dependency being added or removed, so this would be an entirely new feature to add to MacPorts base, and I'm not convinced it's necessary or the correct solution, given that MacPorts has survived for 18 years without it.

If meson uniquely among all build systems needs to be told where pkg-config can be found, then the meson portgroup can do that.

Replying to kencu:

I will admit I am leaning towards patching meson, as that looks most like what all the other build systems would be expected to work like.

I would just urge extreme caution with making such changes. MacPorts provides ports not just for its own use but also for users to use, and users expect programs to work the way their developers intended. It's not our place to change how the tool works just to suit the purposes of MacPorts builds.

It is reasonable to open a discussion with the developers of meson, explain why this setting is causing us problems, and ask if there is any recommended way for us to override the default when doing MacPorts builds (it seems likely to me that there would be such a way), or maybe even if they want to change the default.

[kencu (Ken)]

sage advice.

upstream will undoubtedly say to use their designed mechanism of adding the binaries to the cross files.

this is quite low-risk, as it is rare we would want to use a different pkg-config or cmake than our default one when building with meson.

dave is revbumping many ports at present, so we'd better do something soon, otherwise we risk having sucessfully-built but broken builds in the wild.

[Gcenx]

Replying to ryandesign:

I don't think there is any other situation where MacPorts base modifies the environment in response to a dependency being added or removed, so this would be an entirely new feature to add to MacPorts base, and I'm not convinced it's necessary or the correct solution, given that MacPorts has survived for 18 years without it.

You have a point, under normal conditions this won’t be needed.

Replying to ryandesign:

If meson uniquely among all build systems needs to be told where pkg-config can be found, then the meson portgroup can do that.

The reason meson is now requiring to know where pkg-config/cmake/ect is due to it being used as a cross compiler not as a native compiler when building +universal, I’d guess that will be expanded later to cover all compilations.

Replying to ryandesign:

I would just urge extreme caution with making such changes. MacPorts provides ports not just for its own use but also for users to use, and users expect programs to work the way their developers intended. It's not our place to change how the tool works just to suit the purposes of MacPorts builds.

I don’t disagree with the stance, patching meson could potentially cause an issue when used for real cross compilation of the crossfile doesn’t provide its own wrappers for pkg-config/cmake/ect for the intended target.

Replying to ryandesign:

It is reasonable to open a discussion with the developers of meson, explain why this setting is causing us problems, and ask if there is any recommended way for us to override the default when doing MacPorts builds (it seems likely to me that there would be such a way), or maybe even if they want to change the default.

There stance has been to use the cross files (check DevKitPro as an example), however reaching out to them for a possible way to make use of allow_default_for_cross=True or similar as an environment variable would be more than ideal as the meson portgroup could just set this and then not need to worry about listing all possible binaries within the crossfile macports uses & also won’t affect an end users crossfiles in anyway.

[Gcenx]

Here is the issue that resulted in the current behavior ​https://github.com/mesonbuild/meson/issues/7276

Meson will find build machine binaries for host if the build machine and host machine are the same.

[kencu (Ken)]

Right. Thanks for finding that. I'm a bit unsettled by this:

This alters cmake, pkg-config, and all config-tool based dependencies.

Just how many binaries are we going to have to start spec'ing if we go that route?

[Gcenx]

@kencu of your near your system could you patch meson as I explained above and then make a cross file that targets Linux and see if it picks up the native copy of pkg-config?

Edit; Dam auto correct ugh

[Gcenx]

It seems that currently the solution will be adding the needed binaries into the cross-files or getting meson to resolve the issue that caused this.

[95/98] /opt/local/bin/nasm -f macho64 -I /opt/local/var/macports/build/_Users_gcenx_ports_multimedia_dav1d/dav1d/work/dav1d_0.7.1-i386/src/ -I /opt/local/var/macports/build/_Users_gcenx_ports_multimedia_dav1d/dav1d/work/build-i386/ -MQ src/libdav1d.4.dylib.p/mc_sse.obj -MF src/libdav1d.4.dylib.p/mc_sse.obj.ndep ../dav1d_0.7.1-i386/src/x86/mc_sse.asm -o src/libdav1d.4.dylib.p/mc_sse.obj

Here is something I noticed while doing a native compile, meson is passing macho64 into the 32Bit build when it should be passing macho32. If we or meson can resolve this issue then possibly the crossfiles won't be required.

[kencu (Ken)]

Replying to Gcenx:

Here is something I noticed while doing a native compile, meson is passing macho64 into the 32Bit build when it should be passing macho32. If we or meson can resolve this issue then possibly the crossfiles won't be required.

That is exactly why the cross files are needed; they fix that issue, and many similar issues in many other ports.

We'll keep the cross files in MacPorts (many other builds are fixes by them too) and sort out the (small) issue with the binaries by one of the above methods.

[Gcenx]

Replying to kencu:

That is exactly why the cross files are needed; they fix that issue, and many similar issues in many other ports.

We'll keep the cross files in MacPorts (many other builds are fixes by them too) and sort out the (small) issue with the binaries by one of the above methods.

I was testing out using the --native-file= in some cases that’s enough but of course that’s not the case for macOS.

I’ll give another go with --cross-file= tomorrow with a properly configured target build for Linux to see if the patching affects these builds, else will be needing to add [binaries] section

[kencu (Ken)]

The problem with adding a binaries section is not cmake and pkg-config, it's the 100 other restricted binaries we don't know about, like curl-config, and llvm-config, and blah blah blah blah blah.

And - to make it worse -- things like llvm-config are not static, and they are set on-the-fly by the Portfile as the port needs.

So that path -- a static set of binaries listed in the cross file -- is not really viable, beyond pkg-config and cmake.

SO -- we either write full cross file with all the binaries spec'd (which is a huge PITA) -- or we patch out that meson change, and revert back to it finding the default binaries on the PATH -- which works perfectly well for every application we need it to work for in macports, I think.

[Gcenx]

Yes the crossfiles are undesirable due to this but that's meson.

I decided to properly test out meson when patched with allow_default_for_cross=True

[target_machine]
system = 'linux'
cpu_family = 'x86'
cpu = 'i386'
endian = 'little'

[binaries]
c = '/usr/bin/clang'

MacBook-Pro:~ gcenx$ meson /Users/gcenx/Downloads/graphene /Users/gcenx/Downloads/graphene_build --cross-file i386-linux
The Meson build system
Version: 0.55.0
Source dir: /Users/gcenx/Downloads/graphene
Build dir: /Users/gcenx/Downloads/graphene_build
Build type: cross build
Project name: graphene
Project version: 1.10.2
C compiler for the build machine: cc (clang 11.0.3 "Apple clang version 11.0.3 (clang-1103.0.32.62)")
C linker for the build machine: cc ld64 556.6
C compiler for the host machine: /usr/bin/clang (clang 11.0.3 "Apple clang version 11.0.3 (clang-1103.0.32.62)")
C linker for the host machine: /usr/bin/clang ld64 556.6
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: x86_64
Host machine cpu: x86_64
Target machine cpu family: x86
Target machine cpu: i386
Compiler for C supports arguments -ffast-math: YES 
Compiler for C supports arguments -fstrict-aliasing: YES 
Compiler for C supports arguments -Wpointer-arith: YES 
Compiler for C supports arguments -Wstrict-prototypes: YES 
Compiler for C supports arguments -Wnested-externs: YES 
Compiler for C supports arguments -Wold-style-definition: YES 
Compiler for C supports arguments -Wunused: YES 
Compiler for C supports arguments -Wmissing-noreturn: YES 
Compiler for C supports arguments -Wmissing-format-attribute: YES 
Compiler for C supports arguments -Wlogical-op: NO 
Compiler for C supports arguments -Wcast-align: YES 
Compiler for C supports arguments -Wno-unused-local-typedefs: YES 
Compiler for C supports arguments -Werror=float-conversion: YES 
Compiler for C supports arguments -Werror=float-equal: YES 
Compiler for C supports arguments -Werror=redundant-decls: YES 
Compiler for C supports arguments -Werror=missing-prototypes: YES 
Compiler for C supports arguments -Werror=missing-declarations: YES 
Compiler for C supports arguments -Werror=format=2: YES 
Compiler for C supports arguments -Werror=uninitialized: YES 
Compiler for C supports arguments -Werror=shadow: YES 
Compiler for C supports arguments -Werror=implicit: YES 
Compiler for C supports arguments -Werror=init-self: YES 
Compiler for C supports arguments -Werror=main: YES 
Compiler for C supports arguments -Werror=missing-braces: YES 
Compiler for C supports arguments -Werror=return-type: YES 
Compiler for C supports arguments -Werror=array-bounds: YES 
Compiler for C supports arguments -Werror=write-strings: YES 
Compiler for C supports arguments -Werror=undef: YES 
Library m found: YES
Run-time dependency threads found: YES
Has header "stdlib.h" : YES 
Has header "stdint.h" : YES 
Has header "stdbool.h" : YES 
Has header "memory.h" : YES 
Has header "pthread.h" with dependency threads: YES 
Checking for function "memalign" : NO 
Checking for function "_aligned_malloc" : NO 
Checking for function "aligned_alloc" : NO 
Checking for function "posix_memalign" : YES 
Checking for function "sincosf" with dependency -lm: NO 
Checking for function "isinff" with dependency -lm: NO 
Checking for function "isnanf" with dependency -lm: NO 
Message: Enabling various debug infrastructure
Did not find pkg-config by name 'pkg-config'
Found Pkg-config: NO
Did not find CMake 'cmake'
Found CMake: NO
Run-time dependency gobject-2.0 found: NO (tried pkgconfig, framework and cmake)
Looking for a fallback subproject for the dependency gobject-2.0
Subproject directory not found and glib.wrap file not found
Subproject  subprojects/glib is buildable: NO (disabling)
Dependency gobject-2.0 from subproject subprojects/glib found: NO
Checking if "SSE intrinsics" compiles: YES 
Checking if "GCC vector intrinsics" compiles: NO 
Checking if "ARM NEON intrinsics" compiles: NO 
WARNING: Project targeting '>= 0.50.1' but tried to use feature deprecated since '0.48.0': python3 module.
Configuring graphene-config.h using configuration
Configuring graphene-version.h using configuration
Configuring config.h using configuration
Run-time dependency mutest-1 found: NO (tried pkgconfig, framework and cmake)
Looking for a fallback subproject for the dependency mutest-1
Subproject exists but has no meson.build file
Subproject  subprojects/mutest is buildable: NO (disabling)
Dependency mutest-1 from subproject subprojects/mutest found: NO
Build targets in project: 1
WARNING: Deprecated features used:
 * 0.48.0: {'python3 module'}

graphene 1.10.2

  Subprojects
      glib: NO Subproject directory not found and glib.wrap file not found
    mutest: NO Subproject exists but has no meson.build file

Found ninja-1.10.0 at /opt/local/bin/ninja
WARNING: Cross file does not specify strip binary, result will not be stripped.
WARNING: Cross file does not specify strip binary, result will not be stripped.

As can be seen from these results pkg-config is not found when cross-compiling for a different target, it's only found if both host and target match.

[kencu (Ken)]

If using a cross file, I think pkg-config and friends are never found unless specified, so long as allow_default_for_cross=False.

For MacPorts' purposes, this is just not what we want, and none of our other build systems work like that.

If we change to allow_default_for_cross=True then things will work like everyone would expect them to work for MacPorts, and you can still override them with a crossfile of your own construction if you are building for ChromeOS on Apple Silicon with and x86_64 toolchain, or some other bastardization of a build plan.

[Gcenx]

Yes

When allow_default_for_cross=False is set, pkg-config and friends are never found when using a crossfile, setting allow_default_for_cross=True pkg-config and friends are found as long as host and target match. If target and host don’t match then pkg-config and friends won’t be found.

To clarify setting allow_default_for_cross=True restore the prior behavior from meson.

[kencu (Ken)]

NB: There have been several ports updated recently that use meson and pkg-config. Some of these may now be broken in various ways.

[kencu (Ken)]

In fc6d6f4b5d98731545a0d463613fbdcff4074276/macports-ports (master):

meson: specify pkg-config and cmake when cross building

meson requires pkg-config and cmake to be specified
in the cross files when cross building.

see: ​https://github.com/mesonbuild/meson/issues/7276
closes: #60987