grpc: py-grpcio port build failures, when compiler wrapping enabled

[mascguy (Christopher Nielsen)]

Some rather interesting build errors, are occurring across all of our buildbots:

[...]
clang: error: no such file or directory: '"C"'
clang: error: no such file or directory: '__attribute__((visibility'
clang: error: no such file or directory: '("default")))'
clang: error: no such file or directory: 'PyObject*'
[...etc...]

That's from this example, for py39-grpcio: ​https://build.macports.org/builders/ports-11_arm64-builder/builds/20741/steps/install-port/logs/stdio

Full log also attached; filename: py39-grpcio-buildbot-failure-arm.txt.gz

UPDATE: Per discussion with @CJones in the comments, this occurs when compiler wrapping is enabled. For now, the port has been updated to disable said wrapping, to fix build failures. But keeping this ticket open to track the wrapping issue.

[mascguy (Christopher Nielsen)]

I'm not sure this is related, but I'm also seeing odd failures for py-pyarrow:

Command failed: ninja all -j8 -v
Exit code: 1
Error: Failed to configure py37-pyarrow: command execution failed
DEBUG: Error code: NONE
DEBUG: Backtrace: command execution failed
    while executing
"$post $targetname"
Error: See /opt/local/var/macports/logs/_opt_bblocal_var_buildworker_ports_build_ports_devel_apache-arrow/py37-pyarrow/main.log for details.
DEBUG: process_cmd failed: can't read "string": no such variable
    while executing
"return $string"
    (procedure "wrapline" line 7)
    invoked from within
"wrapline $line $maxlen $indent $indentline"
    (procedure "wrap" line 15)
    invoked from within
"wrap "Follow https://guide.macports.org/#project.tickets if you believe there is a bug." -${len}"
    (procedure "print_tickets_url" line 4)
    invoked from within
"print_tickets_url"
    ("uplevel" body line 82)
    invoked from within
"uplevel 1 $block"
    (procedure "foreachport" line 20)
    invoked from within
"foreachport $portlist {
        array unset portinfo
        # If we have a url, use that, since it's most specific
        # otherwise try to map the..."
    (procedure "action_target" line 10)
    invoked from within
"$action_proc $action $portlist [array get global_options]"
    (procedure "process_cmd" line 110)
    invoked from within
"process_cmd $remaining_args"
    ("uplevel" body line 3)
    invoked from within
"uplevel 1 $body"
Error: process_cmd failed: can't read "string": no such variable

[mascguy (Christopher Nielsen)]

Josh, is there anything suspicious in the stack trace for comment:3? This almost looks like it could be coming from MacPorts base, but perhaps it's just in response to the build error...?

[jmroot (Joshua Root)]

The stack trace is #62935; can be ignored as it only happens when something already failed (and COLUMNS is not set).

[mascguy (Christopher Nielsen)]

This looks like an argument quoting issue, as I noticed the following is being passed to Clang:

-DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject*

That explains the errors. CJones, could this be caused by recent portgroup changes...?

[mascguy (Christopher Nielsen)]

Of note, these failures started occurring when py-grpcio was updated to 1.37.1 on 5/1/2021:

​https://github.com/macports/macports-ports/commit/74364feae8140cde33def11adb7d7b92543a19ae

Since then, builds have been failing across-the-board:

​https://ports.macports.org/port/py39-grpcio/builds

​https://ports.macports.org/port/py38-grpcio/builds

​https://ports.macports.org/port/py37-grpcio/builds

So perhaps this is a change/regression in upstream's build scripts. Regardless, this either needs to be fixed ASAP, or we need to revert to the previous port version.

[mascguy (Christopher Nielsen)]

Actually, looking at the build history, 1.37.1 rev 0 built successfully. So it's not upstream, it's us. And something was broken between 5/2/2021 and today.

CJones, can you please take a look?

[mascguy (Christopher Nielsen)]

Reviewing the commit history for grpc, portgroup legacy-support-1.1 was added as a supplemental commit. However, I've tried building locally with that commented out, and the same failures occur.

I'm quite certain this is a bug in one of our portgroups, given the number of changes since 5/1.

[mascguy (Christopher Nielsen)]

Since py-setuptools has been updated twice since 5/1, I tried reverting to the March 2021 version (54.2.0), just to rule out the later 56.x releases. And sure enough, it's not that port.

I even tried testing with MacPorts 2.6.4, to rule out a base regression (as highly unlikely as that is). And still seeing the same issue.

That points to a regression in our portgroups...

[cjones051073 (Chris Jones)]

Looks to me like a lack of quoting around one of the flags being passed.

/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_grpc/py39-grpcio/work/compwrap/cc/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk -arch arm64 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -D_WIN32_WINNT=1536 -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -I/opt/local/include -I/opt/local/include/re2 -I/opt/local/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/opt/local/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/core/ext/upb-generated/envoy/service/discovery/v3/ads.upb.c -o python_build/temp.macosx-11.3-arm64-3.9/src/core/ext/upb-generated/envoy/service/discovery/v3/ads.upb.o -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread

I am not sure I really see how the wrapper is causing this, as it looks like the issue is in what the build is passing.

[mascguy (Christopher Nielsen)]

Yes, the quoting issue was brought up in comment:7.

We've ruled out upstream's build file, as 1.37.1 built successfully, per comment:9. We've even ruled out long shots like py-setuptools and MacBorts base, per comment:12.

Whether it's the wrapper that's causing this or not, I don't know. But what else could it be...?

[cjones051073 (Chris Jones)]

Its odd, but if in the above command I quoted if you replaced the wrapped compiler with the original unwrapped one, I still do not see how it should work as it still looks wrong to my eye. But, if you want to test if it is the wrapper you can disable it by adding

compwrap.compilers_to_wrap

To the subport after the python PG is includec. That wiill effectively turn off the wrapping.

[mascguy (Christopher Nielsen)]

Thanks. So... with wrapping disabled, the port builds fine.

[cjones051073 (Chris Jones)]

Can you post the equivalent compiler command to the one I quoted above ?

[Christopher Nielsen <mascguy@…>]

In e3c85ebe35bb57c88a65f44be8cf39275f4ac81d/macports-ports (master):

grpc: python subports: disable compiler wrapping to fix build failures

See: #62944

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

Can you post the equivalent compiler command to the one I quoted above ?

There aren't quotes in the output. But clearly the arguments are being sent to the subprocess correctly.

So if your wrapper is counting on quotes, that's a problem...

[cjones051073 (Chris Jones)]

The wrapper should be passing exactly the arguments it is passed directly to the underlying compiler, it makes no assumptions nor places any requirements on them… But anyway i will take a look when i can, but it will not be for a few days.

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

The wrapper should be passing exactly the arguments it is passed directly to the underlying compiler, it makes no assumptions nor places any requirements on them… But anyway i will take a look when i can, but it will not be for a few days.

No worries. Now that we have a fix, this isn't an emergency.

Thanks as always Chris!

[josephsacco]

Just when you thought it was safe to go back into the water...

System:

• Late 2015 iMac
• MacOS 11.3.1
• CommandLineTools SDK: MacOSX11.3.sdk

Build error:

:info:build distutils.errors.DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "11.0" but "11.2" during configure

-Joseph

[josephsacco]

I got py37-grpcio to build by applying the "hammer and tongs" approach to the Portfile, manually setting MACOSX_DEPLOYMENT_TARGET to 11.2

cavandish:grpc jsacco$ diff -u Portfile- Portfile
--- Portfile-	2021-05-21 15:06:24.000000000 -0400
+++ Portfile	2021-05-23 12:00:23.000000000 -0400
@@ -28,6 +28,9 @@

 set name_io         ${name}io

+# JES
+macosx_deployment_target 11.2
+
 # error: constexpr function never produces a constant expression
 # Requires c++17 support
 compiler.cxx_standard  2017

So... Something is amiss. Comparing the log files with and without the Portfile patch, I see that whatever is amiss occurs during the configuration phase.

Without patch:

:notice:configure --->  Configuring py37-grpcio
:debug:configure Preferred compilers: clang macports-clang-9.0 macports-clang-11 macports-clang-10
:debug:configure Using compiler 'Xcode Clang'
:debug:configure Executing proc-pre-org.macports.configure-configure-0
:debug:configure CPPFLAGS="-I/opt/local/include" inserted into CFLAGS="-Os -DNDEBUG -I/opt/local/include" CXXFLAGS="-Os -DNDEBUG -I/opt/local/include -stdlib=libc++"
:debug:configure Executing proc-pre-org.macports.configure-configure-1
:debug:configure Executing org.macports.configure (py37-grpcio)
:debug:configure Environment:
:debug:configure CC='/usr/bin/clang'
:debug:configure CC_PRINT_OPTIONS='YES'
:debug:configure CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_grpc/py37-grpcio/work/.CC_PRINT_OPTIONS'
:debug:configure CFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure CPATH='/opt/local/include'
:debug:configure CPPFLAGS='-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure CXX='/usr/bin/clang++'
:debug:configure CXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure DEVELOPER_DIR='/Library/Developer/CommandLineTools'
:debug:configure F90FLAGS='-pipe -Os -m64'
:debug:configure FCFLAGS='-pipe -Os -m64'
:debug:configure FFLAGS='-pipe -Os -m64'
:debug:configure INSTALL='/usr/bin/install -c'
:debug:configure LDFLAGS='-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure LIBRARY_PATH='/opt/local/lib'
:debug:configure MACOSX_DEPLOYMENT_TARGET='11.0'
:debug:configure MACPORTS_LEGACY_SUPPORT_DISABLED='1'
:debug:configure OBJC='/usr/bin/clang'
:debug:configure OBJCFLAGS='-pipe -Os -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure OBJCXX='/usr/bin/clang++'
:debug:configure OBJCXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure SDKROOT='/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'

With patch:

--->  Configuring py37-grpcio
DEBUG: Preferred compilers: clang macports-clang-9.0 macports-clang-11 macports-clang-10
DEBUG: Using compiler 'Xcode Clang'
DEBUG: Executing proc-pre-org.macports.configure-configure-0
DEBUG: CPPFLAGS="-I/opt/local/include" inserted into CFLAGS="-Os -DNDEBUG -I/opt/local/include" CXXFLAGS="-Os -DNDEBUG -I/opt/local/include -stdlib=libc++"
DEBUG: Executing proc-pre-org.macports.configure-configure-1
DEBUG: Executing org.macports.configure (py37-grpcio)
DEBUG: Environment:
DEBUG: CC='/usr/bin/clang'
DEBUG: CC_PRINT_OPTIONS='YES'
DEBUG: CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_grpc/py37-grpcio
/work/.CC_PRINT_OPTIONS'
DEBUG: CFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
DEBUG: CPATH='/opt/local/include'
DEBUG: CPPFLAGS='-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
DEBUG: CXX='/usr/bin/clang++'
DEBUG: CXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
DEBUG: DEVELOPER_DIR='/Library/Developer/CommandLineTools'
DEBUG: F90FLAGS='-pipe -Os -m64'
DEBUG: FCFLAGS='-pipe -Os -m64'
DEBUG: FFLAGS='-pipe -Os -m64'
DEBUG: INSTALL='/usr/bin/install -c'
DEBUG: LDFLAGS='-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
DEBUG: LIBRARY_PATH='/opt/local/lib'
DEBUG: MACOSX_DEPLOYMENT_TARGET='11.2'
DEBUG: MACPORTS_LEGACY_SUPPORT_DISABLED='1'
DEBUG: OBJC='/usr/bin/clang'
DEBUG: OBJCFLAGS='-pipe -Os -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
DEBUG: OBJCXX='/usr/bin/clang++'
DEBUG: OBJCXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.s
dk'
DEBUG: SDKROOT='/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'

-Joseph

[Chris Jones <jonesc@…>]

In 0d92bdecdf30fd4372232ede2350a9f79fe4d9c1/macports-ports (master):

compiler_wrapper PG: Fix argument quoting
Inadvertently broken by commit 97af063d25290d1abacd10c3583d5aad0a663569/macports-ports#diff-c4bcf7fad84fe74afb58060187858c995735190cf587676675e77d9f111f94c5
Closes: #62944