Opened 3 years ago

Closed 3 years ago

#62944 closed defect (fixed)

grpc: py-grpcio port build failures, when compiler wrapping enabled

Reported by: mascguy (Christopher Nielsen) Owned by: cjones051073 (Chris Jones)
Priority: Normal Milestone:
Component: ports Version:
Keywords: portgroup compilerwrap Cc: jmroot (Joshua Root), emcrisostomo (Enrico Maria Crisostomo)
Port: grpc py-grpcio

Description (last modified by mascguy (Christopher Nielsen))

Some rather interesting build errors, are occurring across all of our buildbots:

[...]
clang: error: no such file or directory: '"C"'
clang: error: no such file or directory: '__attribute__((visibility'
clang: error: no such file or directory: '("default")))'
clang: error: no such file or directory: 'PyObject*'
[...etc...]

That's from this example, for py39-grpcio: https://build.macports.org/builders/ports-11_arm64-builder/builds/20741/steps/install-port/logs/stdio

Full log also attached; filename: py39-grpcio-buildbot-failure-arm.txt.gz

UPDATE: Per discussion with @CJones in the comments, this occurs when compiler wrapping is enabled. For now, the port has been updated to disable said wrapping, to fix build failures. But keeping this ticket open to track the wrapping issue.

Attachments (1)

py39-grpcio-buildbot-failure-arm.txt.gz (28.5 KB) - added by mascguy (Christopher Nielsen) 3 years ago.

Download all attachments as: .zip

Change History (27)

Changed 3 years ago by mascguy (Christopher Nielsen)

comment:1 Changed 3 years ago by mascguy (Christopher Nielsen)

Description: modified (diff)
Keywords: arm removed
Summary: grpc: py-grpcio port build failures for armgrpc: py-grpcio port build failures

comment:2 Changed 3 years ago by mascguy (Christopher Nielsen)

Description: modified (diff)

comment:3 Changed 3 years ago by mascguy (Christopher Nielsen)

I'm not sure this is related, but I'm also seeing odd failures for py-pyarrow:

Command failed: ninja all -j8 -v
Exit code: 1
Error: Failed to configure py37-pyarrow: command execution failed
DEBUG: Error code: NONE
DEBUG: Backtrace: command execution failed
    while executing
"$post $targetname"
Error: See /opt/local/var/macports/logs/_opt_bblocal_var_buildworker_ports_build_ports_devel_apache-arrow/py37-pyarrow/main.log for details.
DEBUG: process_cmd failed: can't read "string": no such variable
    while executing
"return $string"
    (procedure "wrapline" line 7)
    invoked from within
"wrapline $line $maxlen $indent $indentline"
    (procedure "wrap" line 15)
    invoked from within
"wrap "Follow https://guide.macports.org/#project.tickets if you believe there is a bug." -${len}"
    (procedure "print_tickets_url" line 4)
    invoked from within
"print_tickets_url"
    ("uplevel" body line 82)
    invoked from within
"uplevel 1 $block"
    (procedure "foreachport" line 20)
    invoked from within
"foreachport $portlist {
        array unset portinfo
        # If we have a url, use that, since it's most specific
        # otherwise try to map the..."
    (procedure "action_target" line 10)
    invoked from within
"$action_proc $action $portlist [array get global_options]"
    (procedure "process_cmd" line 110)
    invoked from within
"process_cmd $remaining_args"
    ("uplevel" body line 3)
    invoked from within
"uplevel 1 $body"
Error: process_cmd failed: can't read "string": no such variable

comment:4 Changed 3 years ago by mascguy (Christopher Nielsen)

Cc: jmroot added

Josh, is there anything suspicious in the stack trace for comment:3? This almost looks like it could be coming from MacPorts base, but perhaps it's just in response to the build error...?

comment:5 Changed 3 years ago by mascguy (Christopher Nielsen)

Priority: NormalHigh

comment:6 Changed 3 years ago by jmroot (Joshua Root)

The stack trace is #62935; can be ignored as it only happens when something already failed (and COLUMNS is not set).

comment:7 Changed 3 years ago by mascguy (Christopher Nielsen)

Cc: cjones051073 removed
Owner: set to cjones051073
Status: newassigned

This looks like an argument quoting issue, as I noticed the following is being passed to Clang:

-DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject*

That explains the errors. CJones, could this be caused by recent portgroup changes...?

comment:8 Changed 3 years ago by mascguy (Christopher Nielsen)

Of note, these failures started occurring when py-grpcio was updated to 1.37.1 on 5/1/2021:

https://github.com/macports/macports-ports/commit/74364feae8140cde33def11adb7d7b92543a19ae

Since then, builds have been failing across-the-board:

https://ports.macports.org/port/py39-grpcio/builds

https://ports.macports.org/port/py38-grpcio/builds

https://ports.macports.org/port/py37-grpcio/builds

So perhaps this is a change/regression in upstream's build scripts. Regardless, this either needs to be fixed ASAP, or we need to revert to the previous port version.

comment:9 Changed 3 years ago by mascguy (Christopher Nielsen)

Actually, looking at the build history, 1.37.1 rev 0 built successfully. So it's not upstream, it's us. And something was broken between 5/2/2021 and today.

CJones, can you please take a look?

comment:10 Changed 3 years ago by mascguy (Christopher Nielsen)

Reviewing the commit history for grpc, portgroup legacy-support-1.1 was added as a supplemental commit. However, I've tried building locally with that commented out, and the same failures occur.

I'm quite certain this is a bug in one of our portgroups, given the number of changes since 5/1.

comment:11 Changed 3 years ago by reneeotten (Renee Otten)

Priority: HighNormal

comment:12 Changed 3 years ago by mascguy (Christopher Nielsen)

Since py-setuptools has been updated twice since 5/1, I tried reverting to the March 2021 version (54.2.0), just to rule out the later 56.x releases. And sure enough, it's not that port.

I even tried testing with MacPorts 2.6.4, to rule out a base regression (as highly unlikely as that is). And still seeing the same issue.

That points to a regression in our portgroups...

comment:13 Changed 3 years ago by cjones051073 (Chris Jones)

Looks to me like a lack of quoting around one of the flags being passed.

/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_grpc/py39-grpcio/work/compwrap/cc/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk -arch arm64 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -D_WIN32_WINNT=1536 -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -I/opt/local/include -I/opt/local/include/re2 -I/opt/local/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/opt/local/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/core/ext/upb-generated/envoy/service/discovery/v3/ads.upb.c -o python_build/temp.macosx-11.3-arm64-3.9/src/core/ext/upb-generated/envoy/service/discovery/v3/ads.upb.o -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread

I am not sure I really see how the wrapper is causing this, as it looks like the issue is in what the build is passing.

Last edited 3 years ago by cjones051073 (Chris Jones) (previous) (diff)

comment:14 Changed 3 years ago by mascguy (Christopher Nielsen)

Yes, the quoting issue was brought up in comment:7.

We've ruled out upstream's build file, as 1.37.1 built successfully, per comment:9. We've even ruled out long shots like py-setuptools and MacBorts base, per comment:12.

Whether it's the wrapper that's causing this or not, I don't know. But what else could it be...?

comment:15 Changed 3 years ago by cjones051073 (Chris Jones)

Its odd, but if in the above command I quoted if you replaced the wrapped compiler with the original unwrapped one, I still do not see how it should work as it still looks wrong to my eye. But, if you want to test if it is the wrapper you can disable it by adding

compwrap.compilers_to_wrap

To the subport after the python PG is includec. That wiill effectively turn off the wrapping.

comment:16 Changed 3 years ago by mascguy (Christopher Nielsen)

Thanks. So... with wrapping disabled, the port builds fine.

comment:17 Changed 3 years ago by cjones051073 (Chris Jones)

Can you post the equivalent compiler command to the one I quoted above ?

comment:18 Changed 3 years ago by Christopher Nielsen <mascguy@…>

In e3c85ebe35bb57c88a65f44be8cf39275f4ac81d/macports-ports (master):

grpc: python subports: disable compiler wrapping to fix build failures

See: #62944

comment:19 in reply to:  17 Changed 3 years ago by mascguy (Christopher Nielsen)

Replying to cjones051073:

Can you post the equivalent compiler command to the one I quoted above ?

There aren't quotes in the output. But clearly the arguments are being sent to the subprocess correctly.

So if your wrapper is counting on quotes, that's a problem...

comment:20 Changed 3 years ago by cjones051073 (Chris Jones)

The wrapper should be passing exactly the arguments it is passed directly to the underlying compiler, it makes no assumptions nor places any requirements on them… But anyway i will take a look when i can, but it will not be for a few days.

Last edited 3 years ago by cjones051073 (Chris Jones) (previous) (diff)

comment:21 in reply to:  20 Changed 3 years ago by mascguy (Christopher Nielsen)

Replying to cjones051073:

The wrapper should be passing exactly the arguments it is passed directly to the underlying compiler, it makes no assumptions nor places any requirements on them… But anyway i will take a look when i can, but it will not be for a few days.

No worries. Now that we have a fix, this isn't an emergency.

Thanks as always Chris!

comment:22 Changed 3 years ago by mascguy (Christopher Nielsen)

Description: modified (diff)
Keywords: portgroup compilerwrap added
Summary: grpc: py-grpcio port build failuresgrpc: py-grpcio port build failures, when compiler wrapping enabled

comment:23 Changed 3 years ago by mascguy (Christopher Nielsen)

Cc: emcrisostomo added

comment:24 Changed 3 years ago by josephsacco

Just when you thought it was safe to go back into the water...

System:

  • Late 2015 iMac
  • MacOS 11.3.1
  • CommandLineTools SDK: MacOSX11.3.sdk
  • MacPorts 2.7.0

Build error:

:info:build distutils.errors.DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "11.0" but "11.2" during configure

-Joseph

Last edited 3 years ago by josephsacco (previous) (diff)

comment:25 Changed 3 years ago by josephsacco

I got py37-grpcio to build by applying the "hammer and tongs" approach to the Portfile, manually setting MACOSX_DEPLOYMENT_TARGET to 11.2

cavandish:grpc jsacco$ diff -u Portfile- Portfile
--- Portfile-	2021-05-21 15:06:24.000000000 -0400
+++ Portfile	2021-05-23 12:00:23.000000000 -0400
@@ -28,6 +28,9 @@

 set name_io         ${name}io

+# JES
+macosx_deployment_target 11.2
+
 # error: constexpr function never produces a constant expression
 # Requires c++17 support
 compiler.cxx_standard  2017

So... Something is amiss. Comparing the log files with and without the Portfile patch, I see that whatever is amiss occurs during the configuration phase.

Without patch:

:notice:configure --->  Configuring py37-grpcio
:debug:configure Preferred compilers: clang macports-clang-9.0 macports-clang-11 macports-clang-10
:debug:configure Using compiler 'Xcode Clang'
:debug:configure Executing proc-pre-org.macports.configure-configure-0
:debug:configure CPPFLAGS="-I/opt/local/include" inserted into CFLAGS="-Os -DNDEBUG -I/opt/local/include" CXXFLAGS="-Os -DNDEBUG -I/opt/local/include -stdlib=libc++"
:debug:configure Executing proc-pre-org.macports.configure-configure-1
:debug:configure Executing org.macports.configure (py37-grpcio)
:debug:configure Environment:
:debug:configure CC='/usr/bin/clang'
:debug:configure CC_PRINT_OPTIONS='YES'
:debug:configure CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_grpc/py37-grpcio/work/.CC_PRINT_OPTIONS'
:debug:configure CFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure CPATH='/opt/local/include'
:debug:configure CPPFLAGS='-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure CXX='/usr/bin/clang++'
:debug:configure CXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure DEVELOPER_DIR='/Library/Developer/CommandLineTools'
:debug:configure F90FLAGS='-pipe -Os -m64'
:debug:configure FCFLAGS='-pipe -Os -m64'
:debug:configure FFLAGS='-pipe -Os -m64'
:debug:configure INSTALL='/usr/bin/install -c'
:debug:configure LDFLAGS='-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure LIBRARY_PATH='/opt/local/lib'
:debug:configure MACOSX_DEPLOYMENT_TARGET='11.0'
:debug:configure MACPORTS_LEGACY_SUPPORT_DISABLED='1'
:debug:configure OBJC='/usr/bin/clang'
:debug:configure OBJCFLAGS='-pipe -Os -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure OBJCXX='/usr/bin/clang++'
:debug:configure OBJCXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
:debug:configure SDKROOT='/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'

With patch:

--->  Configuring py37-grpcio
DEBUG: Preferred compilers: clang macports-clang-9.0 macports-clang-11 macports-clang-10
DEBUG: Using compiler 'Xcode Clang'
DEBUG: Executing proc-pre-org.macports.configure-configure-0
DEBUG: CPPFLAGS="-I/opt/local/include" inserted into CFLAGS="-Os -DNDEBUG -I/opt/local/include" CXXFLAGS="-Os -DNDEBUG -I/opt/local/include -stdlib=libc++"
DEBUG: Executing proc-pre-org.macports.configure-configure-1
DEBUG: Executing org.macports.configure (py37-grpcio)
DEBUG: Environment:
DEBUG: CC='/usr/bin/clang'
DEBUG: CC_PRINT_OPTIONS='YES'
DEBUG: CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_grpc/py37-grpcio
/work/.CC_PRINT_OPTIONS'
DEBUG: CFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
DEBUG: CPATH='/opt/local/include'
DEBUG: CPPFLAGS='-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
DEBUG: CXX='/usr/bin/clang++'
DEBUG: CXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
DEBUG: DEVELOPER_DIR='/Library/Developer/CommandLineTools'
DEBUG: F90FLAGS='-pipe -Os -m64'
DEBUG: FCFLAGS='-pipe -Os -m64'
DEBUG: FFLAGS='-pipe -Os -m64'
DEBUG: INSTALL='/usr/bin/install -c'
DEBUG: LDFLAGS='-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
DEBUG: LIBRARY_PATH='/opt/local/lib'
DEBUG: MACOSX_DEPLOYMENT_TARGET='11.2'
DEBUG: MACPORTS_LEGACY_SUPPORT_DISABLED='1'
DEBUG: OBJC='/usr/bin/clang'
DEBUG: OBJCFLAGS='-pipe -Os -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
DEBUG: OBJCXX='/usr/bin/clang++'
DEBUG: OBJCXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.s
dk'
DEBUG: SDKROOT='/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'

-Joseph

Last edited 3 years ago by josephsacco (previous) (diff)

comment:26 Changed 3 years ago by Chris Jones <jonesc@…>

Resolution: fixed
Status: assignedclosed

In 0d92bdecdf30fd4372232ede2350a9f79fe4d9c1/macports-ports (master):

compiler_wrapper PG: Fix argument quoting
Inadvertently broken by commit 97af063d25290d1abacd10c3583d5aad0a663569/macports-ports#diff-c4bcf7fad84fe74afb58060187858c995735190cf587676675e77d9f111f94c5
Closes: #62944

Note: See TracTickets for help on using tickets.