Opened 3 years ago
Closed 3 years ago
#62944 closed defect (fixed)
grpc: py-grpcio port build failures, when compiler wrapping enabled
Reported by: | mascguy (Christopher Nielsen) | Owned by: | cjones051073 (Chris Jones) |
---|---|---|---|
Priority: | Normal | Milestone: | |
Component: | ports | Version: | |
Keywords: | portgroup compilerwrap | Cc: | jmroot (Joshua Root), emcrisostomo (Enrico Maria Crisostomo) |
Port: | grpc py-grpcio |
Description (last modified by mascguy (Christopher Nielsen))
Some rather interesting build errors, are occurring across all of our buildbots:
[...] clang: error: no such file or directory: '"C"' clang: error: no such file or directory: '__attribute__((visibility' clang: error: no such file or directory: '("default")))' clang: error: no such file or directory: 'PyObject*' [...etc...]
That's from this example, for py39-grpcio
:
https://build.macports.org/builders/ports-11_arm64-builder/builds/20741/steps/install-port/logs/stdio
Full log also attached; filename: py39-grpcio-buildbot-failure-arm.txt.gz
UPDATE: Per discussion with @CJones in the comments, this occurs when compiler wrapping is enabled. For now, the port has been updated to disable said wrapping, to fix build failures. But keeping this ticket open to track the wrapping issue.
Attachments (1)
Change History (27)
Changed 3 years ago by mascguy (Christopher Nielsen)
Attachment: | py39-grpcio-buildbot-failure-arm.txt.gz added |
---|
comment:1 Changed 3 years ago by mascguy (Christopher Nielsen)
Description: | modified (diff) |
---|---|
Keywords: | arm removed |
Summary: | grpc: py-grpcio port build failures for arm → grpc: py-grpcio port build failures |
comment:2 Changed 3 years ago by mascguy (Christopher Nielsen)
Description: | modified (diff) |
---|
comment:3 Changed 3 years ago by mascguy (Christopher Nielsen)
comment:4 Changed 3 years ago by mascguy (Christopher Nielsen)
Cc: | jmroot added |
---|
Josh, is there anything suspicious in the stack trace for comment:3? This almost looks like it could be coming from MacPorts base, but perhaps it's just in response to the build error...?
comment:5 Changed 3 years ago by mascguy (Christopher Nielsen)
Priority: | Normal → High |
---|
comment:6 Changed 3 years ago by jmroot (Joshua Root)
The stack trace is #62935; can be ignored as it only happens when something already failed (and COLUMNS is not set).
comment:7 Changed 3 years ago by mascguy (Christopher Nielsen)
Cc: | cjones051073 removed |
---|---|
Owner: | set to cjones051073 |
Status: | new → assigned |
This looks like an argument quoting issue, as I noticed the following is being passed to Clang:
-DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject*
That explains the errors. CJones, could this be caused by recent portgroup changes...?
comment:8 Changed 3 years ago by mascguy (Christopher Nielsen)
Of note, these failures started occurring when py-grpcio
was updated to 1.37.1 on 5/1/2021:
https://github.com/macports/macports-ports/commit/74364feae8140cde33def11adb7d7b92543a19ae
Since then, builds have been failing across-the-board:
https://ports.macports.org/port/py39-grpcio/builds
https://ports.macports.org/port/py38-grpcio/builds
https://ports.macports.org/port/py37-grpcio/builds
So perhaps this is a change/regression in upstream's build scripts. Regardless, this either needs to be fixed ASAP, or we need to revert to the previous port version.
comment:9 Changed 3 years ago by mascguy (Christopher Nielsen)
Actually, looking at the build history, 1.37.1 rev 0 built successfully. So it's not upstream, it's us. And something was broken between 5/2/2021 and today.
CJones, can you please take a look?
comment:10 Changed 3 years ago by mascguy (Christopher Nielsen)
Reviewing the commit history for grpc
, portgroup legacy-support-1.1
was added as a supplemental commit. However, I've tried building locally with that commented out, and the same failures occur.
I'm quite certain this is a bug in one of our portgroups, given the number of changes since 5/1.
comment:11 Changed 3 years ago by reneeotten (Renee Otten)
Priority: | High → Normal |
---|
comment:12 Changed 3 years ago by mascguy (Christopher Nielsen)
Since py-setuptools
has been updated twice since 5/1, I tried reverting to the March 2021 version (54.2.0), just to rule out the later 56.x releases. And sure enough, it's not that port.
I even tried testing with MacPorts 2.6.4, to rule out a base regression (as highly unlikely as that is). And still seeing the same issue.
That points to a regression in our portgroups...
comment:13 Changed 3 years ago by cjones051073 (Chris Jones)
Looks to me like a lack of quoting around one of the flags being passed.
/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_grpc/py39-grpcio/work/compwrap/cc/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk -arch arm64 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -D_WIN32_WINNT=1536 -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 -DPyMODINIT_FUNC=extern "C" __attribute__((visibility ("default"))) PyObject* -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -I/opt/local/include -I/opt/local/include/re2 -I/opt/local/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -I/opt/local/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/core/ext/upb-generated/envoy/service/discovery/v3/ads.upb.c -o python_build/temp.macosx-11.3-arm64-3.9/src/core/ext/upb-generated/envoy/service/discovery/v3/ads.upb.o -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread
I am not sure I really see how the wrapper is causing this, as it looks like the issue is in what the build is passing.
comment:14 Changed 3 years ago by mascguy (Christopher Nielsen)
Yes, the quoting issue was brought up in comment:7.
We've ruled out upstream's build file, as 1.37.1 built successfully, per comment:9. We've even ruled out long shots like py-setuptools
and MacBorts base, per comment:12.
Whether it's the wrapper that's causing this or not, I don't know. But what else could it be...?
comment:15 Changed 3 years ago by cjones051073 (Chris Jones)
Its odd, but if in the above command I quoted if you replaced the wrapped compiler with the original unwrapped one, I still do not see how it should work as it still looks wrong to my eye. But, if you want to test if it is the wrapper you can disable it by adding
compwrap.compilers_to_wrap
To the subport after the python PG is includec. That wiill effectively turn off the wrapping.
comment:16 Changed 3 years ago by mascguy (Christopher Nielsen)
Thanks. So... with wrapping disabled, the port builds fine.
comment:17 follow-up: 19 Changed 3 years ago by cjones051073 (Chris Jones)
Can you post the equivalent compiler command to the one I quoted above ?
comment:18 Changed 3 years ago by Christopher Nielsen <mascguy@…>
comment:19 Changed 3 years ago by mascguy (Christopher Nielsen)
Replying to cjones051073:
Can you post the equivalent compiler command to the one I quoted above ?
There aren't quotes in the output. But clearly the arguments are being sent to the subprocess correctly.
So if your wrapper is counting on quotes, that's a problem...
comment:20 follow-up: 21 Changed 3 years ago by cjones051073 (Chris Jones)
The wrapper should be passing exactly the arguments it is passed directly to the underlying compiler, it makes no assumptions nor places any requirements on them… But anyway i will take a look when i can, but it will not be for a few days.
comment:21 Changed 3 years ago by mascguy (Christopher Nielsen)
Replying to cjones051073:
The wrapper should be passing exactly the arguments it is passed directly to the underlying compiler, it makes no assumptions nor places any requirements on them… But anyway i will take a look when i can, but it will not be for a few days.
No worries. Now that we have a fix, this isn't an emergency.
Thanks as always Chris!
comment:22 Changed 3 years ago by mascguy (Christopher Nielsen)
Description: | modified (diff) |
---|---|
Keywords: | portgroup compilerwrap added |
Summary: | grpc: py-grpcio port build failures → grpc: py-grpcio port build failures, when compiler wrapping enabled |
comment:23 Changed 3 years ago by mascguy (Christopher Nielsen)
Cc: | emcrisostomo added |
---|
comment:24 Changed 3 years ago by josephsacco
Just when you thought it was safe to go back into the water...
System:
- Late 2015 iMac
- MacOS 11.3.1
- CommandLineTools SDK: MacOSX11.3.sdk
- MacPorts 2.7.0
Build error:
:info:build distutils.errors.DistutilsPlatformError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "11.0" but "11.2" during configure
-Joseph
comment:25 Changed 3 years ago by josephsacco
I got py37-grpcio to build by applying the "hammer and tongs" approach to the Portfile, manually setting MACOSX_DEPLOYMENT_TARGET to 11.2
cavandish:grpc jsacco$ diff -u Portfile- Portfile --- Portfile- 2021-05-21 15:06:24.000000000 -0400 +++ Portfile 2021-05-23 12:00:23.000000000 -0400 @@ -28,6 +28,9 @@ set name_io ${name}io +# JES +macosx_deployment_target 11.2 + # error: constexpr function never produces a constant expression # Requires c++17 support compiler.cxx_standard 2017
So... Something is amiss. Comparing the log files with and without the Portfile patch, I see that whatever is amiss occurs during the configuration phase.
Without patch:
:notice:configure ---> Configuring py37-grpcio :debug:configure Preferred compilers: clang macports-clang-9.0 macports-clang-11 macports-clang-10 :debug:configure Using compiler 'Xcode Clang' :debug:configure Executing proc-pre-org.macports.configure-configure-0 :debug:configure CPPFLAGS="-I/opt/local/include" inserted into CFLAGS="-Os -DNDEBUG -I/opt/local/include" CXXFLAGS="-Os -DNDEBUG -I/opt/local/include -stdlib=libc++" :debug:configure Executing proc-pre-org.macports.configure-configure-1 :debug:configure Executing org.macports.configure (py37-grpcio) :debug:configure Environment: :debug:configure CC='/usr/bin/clang' :debug:configure CC_PRINT_OPTIONS='YES' :debug:configure CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_grpc/py37-grpcio/work/.CC_PRINT_OPTIONS' :debug:configure CFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk' :debug:configure CPATH='/opt/local/include' :debug:configure CPPFLAGS='-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk' :debug:configure CXX='/usr/bin/clang++' :debug:configure CXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk' :debug:configure DEVELOPER_DIR='/Library/Developer/CommandLineTools' :debug:configure F90FLAGS='-pipe -Os -m64' :debug:configure FCFLAGS='-pipe -Os -m64' :debug:configure FFLAGS='-pipe -Os -m64' :debug:configure INSTALL='/usr/bin/install -c' :debug:configure LDFLAGS='-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk' :debug:configure LIBRARY_PATH='/opt/local/lib' :debug:configure MACOSX_DEPLOYMENT_TARGET='11.0' :debug:configure MACPORTS_LEGACY_SUPPORT_DISABLED='1' :debug:configure OBJC='/usr/bin/clang' :debug:configure OBJCFLAGS='-pipe -Os -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk' :debug:configure OBJCXX='/usr/bin/clang++' :debug:configure OBJCXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk' :debug:configure SDKROOT='/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
With patch:
---> Configuring py37-grpcio DEBUG: Preferred compilers: clang macports-clang-9.0 macports-clang-11 macports-clang-10 DEBUG: Using compiler 'Xcode Clang' DEBUG: Executing proc-pre-org.macports.configure-configure-0 DEBUG: CPPFLAGS="-I/opt/local/include" inserted into CFLAGS="-Os -DNDEBUG -I/opt/local/include" CXXFLAGS="-Os -DNDEBUG -I/opt/local/include -stdlib=libc++" DEBUG: Executing proc-pre-org.macports.configure-configure-1 DEBUG: Executing org.macports.configure (py37-grpcio) DEBUG: Environment: DEBUG: CC='/usr/bin/clang' DEBUG: CC_PRINT_OPTIONS='YES' DEBUG: CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_devel_grpc/py37-grpcio /work/.CC_PRINT_OPTIONS' DEBUG: CFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk' DEBUG: CPATH='/opt/local/include' DEBUG: CPPFLAGS='-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk' DEBUG: CXX='/usr/bin/clang++' DEBUG: CXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk' DEBUG: DEVELOPER_DIR='/Library/Developer/CommandLineTools' DEBUG: F90FLAGS='-pipe -Os -m64' DEBUG: FCFLAGS='-pipe -Os -m64' DEBUG: FFLAGS='-pipe -Os -m64' DEBUG: INSTALL='/usr/bin/install -c' DEBUG: LDFLAGS='-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk' DEBUG: LIBRARY_PATH='/opt/local/lib' DEBUG: MACOSX_DEPLOYMENT_TARGET='11.2' DEBUG: MACPORTS_LEGACY_SUPPORT_DISABLED='1' DEBUG: OBJC='/usr/bin/clang' DEBUG: OBJCFLAGS='-pipe -Os -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk' DEBUG: OBJCXX='/usr/bin/clang++' DEBUG: OBJCXXFLAGS='-pipe -Os -DNDEBUG -I/opt/local/include -I/opt/local/include -stdlib=libc++ -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.s dk' DEBUG: SDKROOT='/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk'
-Joseph
comment:26 Changed 3 years ago by Chris Jones <jonesc@…>
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
I'm not sure this is related, but I'm also seeing odd failures for
py-pyarrow
: