libgcrypt +universal Failed to destroot: libgcrypt.pc differs in .../destroot-arm64/... and .../destroot-ppc-intel/... and cannot be merged

[ShadSterling (Shad Sterling)]

If I understand right, the universal build is building separate binaries for arm64 and "ppc-intel" (not x86_64?), then failing to merge them into a universal binary because lipo doesn't understand the arm64 architecture

I think these are the key log lines:

:debug:destroot universal: merge: merging /opt/local/lib/pkgconfig/libgcrypt.pc from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-arm64 and /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-ppc-intel
 ⋮
:info:destroot fatal error: /Library/Developer/CommandLineTools/usr/bin/lipo: can't figure out the architecture type of: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-arm64//opt/local/lib/pkgconfig/libgcrypt.pc
 ⋮
:info:destroot error: /Library/Developer/CommandLineTools/usr/bin/libtool: file: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-arm64//opt/local/lib/pkgconfig/libgcrypt.pc is not an object file (not allowed in a library) ⋮
 ⋮
:info:destroot error: /Library/Developer/CommandLineTools/usr/bin/libtool: file: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-ppc-intel//opt/local/lib/pkgconfig/libgcrypt.pc is not an object file (not allowed in a library) ⋮
 ⋮
:error:destroot Failed to destroot libgcrypt: libgcrypt.pc differs in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-arm64//opt/local/lib/pkgconfig and /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-ppc-intel//opt/local/lib/pkgconfig and cannot be merged

I recently (maybe a week ago) reinstalled the command line tools after a port install message recommended doing so; I'm not sure if this is a problem with the port or with the command line tools. I was able to port upgrade outdated without error, this came up trying to manually update from git +perl5_30 to git +perl5_32

[Schamschula (Marius Schamschula)]

First of all port info git doesn't list a perl5_32 variant. It will build the default perl5_28 variant instead.

I don't think Perl has anything to do with libgcrypt:

% port rdeps libgcrypt
The following ports are dependencies of libgcrypt @1.9.4_0:
  libgpg-error
    libiconv
      gperf
    gettext
      ncurses

I have no experience with universal builds. I avoid them like the plague.

However, there is no such thing as the ppc-intel architecture. I have no idea what lipo is talking about.

[ShadSterling (Shad Sterling)]

On my mac, port variants git says

git has the variants:
[+]credential_osxkeychain: Install git credential-osxkeychain utility from contrib
[+]diff_highlight: Install git diff-highlight utility from contrib
[+]doc: Install HTML and plaintext documentation
   gitweb: Install gitweb.cgi
[+]pcre: Use pcre
[+]perl5_28: Use MacPorts perl5.28
     * conflicts with perl5_30 perl5_32
   perl5_30: Use MacPorts perl5.30
     * conflicts with perl5_28 perl5_32
   perl5_32: Use MacPorts perl5.32
     * conflicts with perl5_28 perl5_30
   python: Build with Python support
   svn: Bi-directional subversion repository support
(+)universal: Build for multiple architectures

port install git @2.33.1_0+perl5_32+universal says

--->  Computing dependencies for libgcrypt
--->  Fetching archive for libgcrypt
--->  Attempting to fetch libgcrypt-1.9.4_0+universal.darwin_20.arm64-x86_64.tbz2 from https://packages.macports.org/libgcrypt
--->  Attempting to fetch libgcrypt-1.9.4_0+universal.darwin_20.arm64-x86_64.tbz2 from https://ywg.ca.packages.macports.org/mirror/macports/packages/libgcrypt
--->  Attempting to fetch libgcrypt-1.9.4_0+universal.darwin_20.arm64-x86_64.tbz2 from https://mse.uk.packages.macports.org/libgcrypt
--->  Staging libgcrypt into destroot
Error: Failed to destroot libgcrypt: error copying "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot" to "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-arm64/destroot": file already exists
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/main.log for details.
Error: Unable to execute port: upgrade curl failed

... it's trying to upgrade curl, which is a library dependency of git. I can't find the connection from curl to libgcrypt but it doesn't seem surprising that an HTTPS client would use it.

Regardless, the libgcrypt build is failing. I mentioned how it came up because I didn't know whether or not the context was relevant; it sounds like it's not, except in that I can't install git with those variants

[Schamschula (Marius Schamschula)]

The machine I checked port info git on had an obsolete local branch Portfile for git.

curl has a gnutls variant, not default.

[Schamschula (Marius Schamschula)]

Nope. I was wrong.

You seem to have enabled curl +gsasl. This is the only variant that does require libgcrypt.

[ShadSterling (Shad Sterling)]

Yes, I do have curl +gsasl. Does that affect the build of libgcrypt?

After installing macos updates the logs are slightly different, but it looks like mostly timestamps and some messages appearing in a different order.

It looks like all of the references to ppc-intel are in paths in the section of the log beginning with :info:destroot ---> Staging libgcrypt into destroot for architecture x86_64, as if libgcrypt is using ppc-intel internally for x86_64

It also looks like libgcrypt.pc is invalid in both architectures:

:info:destroot error: /Library/Developer/CommandLineTools/usr/bin/libtool: file: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-arm64//opt/local/lib/pkgconfig/libgcrypt.pc is not an object file (not allowed in a library)
:info:destroot error: /Library/Developer/CommandLineTools/usr/bin/libtool: file: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-ppc-intel//opt/local/lib/pkgconfig/libgcrypt.pc is not an object file (not allowed in a library)
:info:destroot Command failed: /usr/bin/libtool "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-arm64//opt/local/lib/pkgconfig/libgcrypt.pc" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-ppc-intel//opt/local/lib/pkgconfig/libgcrypt.pc" -o "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot//opt/local/lib/pkgconfig/libgcrypt.pc"

There are 4 versions created

-rw-r--r-- 1 root wheel 644 Oct 16 21:00 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-arm64/opt/local/lib/pkgconfig/libgcrypt.pc
-rw-r--r-- 1 root wheel 643 Oct 16 21:00 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-intel/opt/local/lib/pkgconfig/libgcrypt.pc
-rw-r--r-- 1 root wheel 643 Oct 16 21:00 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-ppc-intel/opt/local/lib/pkgconfig/libgcrypt.pc
-rw-r--r-- 1 root wheel 643 Oct 16 21:00 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_libgcrypt/libgcrypt/work/destroot-x86_64/opt/local/lib/pkgconfig/libgcrypt.pc

Which are indeed not object files; file says they're ASCII text. The latter 3 are identical, the arm differs by one line, and they look like some sort of port info or maybe configure results; diff -y says

prefix=/opt/local						prefix=/opt/local
exec_prefix=${prefix}						exec_prefix=${prefix}
includedir=${prefix}/include					includedir=${prefix}/include
libdir=${exec_prefix}/lib					libdir=${exec_prefix}/lib
host=aarch64-apple-darwin20.6.0				      |	host=x86_64-apple-darwin20.6.0
api_version=1							api_version=1
symmetric_ciphers="arcfour blowfish cast5 des aes twofish ser	symmetric_ciphers="arcfour blowfish cast5 des aes twofish ser
asymmetric_ciphers="dsa elgamal rsa ecc"			asymmetric_ciphers="dsa elgamal rsa ecc"
digests="crc gostr3411-94  md4 md5 rmd160 sha1 sha256 sha512 	digests="crc gostr3411-94  md4 md5 rmd160 sha1 sha256 sha512 

Name: libgcrypt							Name: libgcrypt
Description: General purpose cryptographic library		Description: General purpose cryptographic library
Requires.private: gpg-error					Requires.private: gpg-error
Version: 1.9.4							Version: 1.9.4
Cflags: -I${includedir} 					Cflags: -I${includedir} 
Libs: -L${libdir} -lgcrypt					Libs: -L${libdir} -lgcrypt
Libs.private: 							Libs.private: 
URL: https://www.gnupg.org/software/libgcrypt/index.html	URL: https://www.gnupg.org/software/libgcrypt/index.html

So I'd guess the problem is that the universal build is trying to link a file it shouldn't

[ShadSterling (Shad Sterling)]

after installing macOS updates

[Schamschula (Marius Schamschula)]

Yes. curl +gsasl is the only variant that requires libgcrypt to be installed.

You are also correct that a .pc file should not be touched. These are pkg-config files and only contain text data.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ShadSterling:

If I understand right, the universal build is building separate binaries for arm64 and "ppc-intel" (not x86_64?), then failing to merge them into a universal binary

Yes.

because lipo doesn't understand the arm64 architecture

No. It is failing because the pkg-config files differ. The muniversal portgroup which is used for such build-then-merge builds understands how to merge a variety of files, but the pkg-config file format does not accommodate any form of merging, therefore the muniversal portgroup has no choice but to require that all the generated pkg-config files are identical, and in this case they are not, therefore an error message must be emitted.

This could indicate a bug in the way that the upstream build system creates the pkg-config files. For example, in this case, the difference appears to be in the host lines, but I'm not aware of any reason why a pkg-config file should contain a host line, so the solution may be to remove that line from the files.

On my system I see other pkg-config files from other ports that also have a host line, so all of those ports seem to have the same bug.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ShadSterling:

arm64 and "ppc-intel" (not x86_64?)

Yes. Before Apple Silicon existed, the muniversal process accommodated builds for up to four architectures. For any of them that were requested, the ppc and ppc64 directories would be merged into a powerpc directory, the i386 and x86_64 directories would be merged into an intel directory, and then the powerpc and intel directories would be merged into the final directory. Now that we have Apple Silicon we accommodate up to five architectures. The previously final directory is now called ppc-intel and it is merged with the arm64 directory. If you are building for fewer than five architectures (and you undoubtedly are since there is no OS version on which all five architectures can actually be built) this may seem somewhat roundabout but it is simpler to have a single codepath that accommodates all possibilities.

[ShadSterling (Shad Sterling)]

As quoted in comment:5, the host line includes the architecture, so naturally differs between architectures. That didn't immediately read as a bug, but looking up pkg-config nothing mentions the host line, so maybe that's nonstandard? Seems potentially useful for cross-compiling; maybe muniversal should remove it?

Thanks for the explanation of "ppc-intel", I had no idea ppc/ppc64 was still supported, and I guess I forgot i386 ever was

[ryandesign (Ryan Carsten Schmidt)]

Replying to ShadSterling:

As quoted in comment:5, the host line includes the architecture, so naturally differs between architectures.

Right, which is why it has no business being in a pkg-config .pc file which is supposed to be an architecture-independent file. Placing such information in a pkg-config file demonstrates an unawareness of the existence of systems that include support for multiple architectures within the same file, such as macOS; this is not entirely surprising since much of the software in MacPorts is developed by people who have never used Macs.

That didn't immediately read as a bug, but looking up pkg-config nothing mentions the host line, so maybe that's nonstandard? Seems potentially useful for cross-compiling; maybe muniversal should remove it?

pkg-config includes the ability to put arbitrary data into .pc files; the host line comes under that heading. We don't know in what ways, if at all, each program might use the host information in the pkg-config file. My expectation is that the developers put it in there on the assumption that it would be useful to someone but that they don't use it at all, but I don't know if that's true.

Instead of having the muniversal portgroup indiscriminately remove that nonstandard line at the risk of breaking some software that relies on it being there, it would be a better idea to educate each software developer about how adding architecture-specific information to pkg-config files is contrary to the purpose of pkg-config files and have them fix their code to not do that. That way everyone benefits, not just MacPorts users. (We can also patch the problem in each affected Portfile until the upstream software is fixed.)

[kencu (Ken)]

​https://github.com/macports/macports-ports/pull/17136

[kencu (Ken)]

In 3fd961ba60c083fefd0d6f054acb306ff5730307/macports-ports (master):

libgcrypt: allow cross-arch universal building

strip host= from pc file
closes: #63635