python38 @3.8.0: build fails on Tiger: missing copyfile.h

[SerpentChris (Chris Calderon)]

I'm trying to build Python 3.8 on Mac OS X 10.4.11 PPC, and the build seems to fail here:

/opt/local/bin/gcc-apple-4.2 -fno-strict-aliasing -Wsign-compare -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -arch ppc   -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I./Include/internal  -I. -I./Include -I/opt/local/include   -DPy_BUILD_CORE_BUILTIN  -DPy_BUILD_CORE_BUILTIN -I./Include/internal -c ./Modules/posixmodule.c -o Modules/posixmodule.o
./Modules/posixmodule.c:113:22: error: copyfile.h: No such file or directory
./Modules/posixmodule.c: In function 'os__fcopyfile_impl':
./Modules/posixmodule.c:9175: warning: implicit declaration of function 'fcopyfile'
./Modules/posixmodule.c: In function 'all_ins':
./Modules/posixmodule.c:14226: error: 'COPYFILE_DATA' undeclared (first use in this function)
./Modules/posixmodule.c:14226: error: (Each undeclared identifier is reported only once
./Modules/posixmodule.c:14226: error: for each function it appears in.)

I've noticed there is also a build failure for Leopard, might these be related?

[reneeotten (Renee Otten)]

that file should be part of the CommandLineTools (at least on newer systems) - did you install those? Also, in the log on my local machine and Travis CI, I do see a line specifiying SDKROOT that is not present in your logfile. I guess that's a starting point, besides that I'm not likely to be of any help....

[jmroot (Joshua Root)]

It's Tiger, there is no copyfile. If /usr/include exists then no SDK will be used.

[kencu (Ken)]

You'll have to stick with python37 until this is sorted out. Some software is defaulting to python38 -- I think it was youtube-dl -- just change it to 37 until we fix this, if it is fixed.

[dgelessus (dgelessus)]

I've submitted a possible fix (​https://github.com/macports/macports-ports/pull/5987). It fixes the copyfile compile error, but I can't check whether it's fully working - on my Intel Tiger machine, Python 3.8 doesn't build because of #59772.

[ballapete (Peter "Pete" Dyballa)]

I a have an a bit different error report WRT copyfile:

/opt/local/bin/gcc-apple-4.2 -c -fno-strict-aliasing -Wsign-compare -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -arch ppc   -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I./Include/internal  -I. -I./Include -I/opt/local/include   -DPy_BUILD_CORE \
      -DGITVERSION="\"`LC_ALL=C `\"" \
      -DGITTAG="\"`LC_ALL=C `\"" \
      -DGITBRANCH="\"`LC_ALL=C `\"" \
      -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
/opt/local/bin/gcc-apple-4.2 -c -fno-strict-aliasing -Wsign-compare -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -arch ppc   -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I./Include/internal  -I. -I./Include -I/opt/local/include   -DPy_BUILD_CORE -o Python/frozen.o Python/frozen.c
rm -f libpython3.8.a
ar rcs libpython3.8.a Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/token.o  Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/hamt.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/peephole.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o    Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o  Modules/_collectionsmodule.o  Modules/_abc.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_stat.o  Modules/timemodule.o  Modules/_threadmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/faulthandler.o  Modules/_tracemalloc.o Modules/hashtable.o  Modules/symtablemodule.o  Modules/xxsubtype.o Python/frozen.o
/opt/local/bin/ranlib: file: libpython3.8.a(dynamic_annotations.o) has no symbols
/opt/local/bin/ranlib: file: libpython3.8.a(pymath.o) has no symbols
/usr/bin/install -c -d -m 755 Python.framework/Versions/3.8
/opt/local/bin/gcc-apple-4.2 -o Python.framework/Versions/3.8/Python -L/opt/local/lib -Wl,-headerpad_max_install_names -arch ppc    -dynamiclib \
-all_load libpython3.8.a -Wl,-single_module \
-install_name /opt/local/Library/Frameworks/Python.framework/Versions/3.8/Python \
-compatibility_version 3.8 \
-current_version 3.8 \
-framework CoreFoundation -lintl -ldl   -framework CoreFoundation;
Undefined symbols:
  "_fcopyfile", referenced from:
      _os__fcopyfile in libpython3.8.a(posixmodule.o)
     (maybe you meant: cstring=_fcopyfile)
  "_pthread_threadid_np", referenced from:
      _PyThread_get_thread_native_id in libpython3.8.a(thread.o)
ld: symbol(s) not found
collect2: ld returned 1 exit status
make: *** [Python.framework/Versions/3.8/Python] Error 1
make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_lil.fr.rsync.macports.org_release_tarballs_ports_lang_python38/python38/work/Python-3.8.1'
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_lil.fr.rsync.macports.org_release_tarballs_ports_lang_python38/python38/work/Python-3.8.1" && /usr/bin/make -w all 
Exit code: 2

PPC Tiger, Mac OS X 10.4.11.

[kencu (Ken)]

there is a not-quite-right-yet patch in 59772 to get you going.

[kencu (Ken)]

tiger intel too, fwiw

[ballapete (Peter "Pete" Dyballa)]

Replying to kencu:

there is a not-quite-right-yet patch in 59772 to get you going.

Since trac.macports.org is taking too long to respond I cannot report Python 3.8 just built. It is already installed, plus some extensions. gobject-introspection @1.58.3_3 is being built, meson @0.53.2_0 will soon build. It looks quite promising!

[ballapete (Peter "Pete" Dyballa)]

On PPC Leopard, Mac OS X 10.5.8, this error ahppens:

/usr/bin/gcc-4.2 -c -fno-strict-aliasing -Wsign-compare -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -arch ppc   -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I./Include/internal  -I. -I./Include -I/opt/local/include   -DPy_BUILD_CORE \
	      -DGITVERSION="\"`LC_ALL=C `\"" \
	      -DGITTAG="\"`LC_ALL=C `\"" \
	      -DGITBRANCH="\"`LC_ALL=C `\"" \
	      -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
/usr/bin/gcc-4.2 -c -fno-strict-aliasing -Wsign-compare -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -arch ppc   -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I./Include/internal  -I. -I./Include -I/opt/local/include   -DPy_BUILD_CORE -o Python/frozen.o Python/frozen.c
rm -f libpython3.8.a
ar rcs libpython3.8.a Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/token.o  Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/hamt.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/peephole.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o    Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o  Modules/_collectionsmodule.o  Modules/_abc.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_stat.o  Modules/timemodule.o  Modules/_threadmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/faulthandler.o  Modules/_tracemalloc.o Modules/hashtable.o  Modules/symtablemodule.o  Modules/xxsubtype.o Python/frozen.o
/opt/local/bin/ranlib: file: libpython3.8.a(dynamic_annotations.o) has no symbols
/opt/local/bin/ranlib: file: libpython3.8.a(pymath.o) has no symbols
/usr/bin/install -c -d -m 755 Python.framework/Versions/3.8
/usr/bin/gcc-4.2 -o Python.framework/Versions/3.8/Python -L/opt/local/lib -Wl,-headerpad_max_install_names -arch ppc    -dynamiclib \
		-all_load libpython3.8.a -Wl,-single_module \
		-install_name /opt/local/Library/Frameworks/Python.framework/Versions/3.8/Python \
		-compatibility_version 3.8 \
		-current_version 3.8 \
		-framework CoreFoundation -lintl -ldl   -framework CoreFoundation;
Undefined symbols:
  "_pthread_threadid_np", referenced from:
      _PyThread_get_thread_native_id in libpython3.8.a(thread.o)
ld: symbol(s) not found
collect2: ld returned 1 exit status
make: *** [Python.framework/Versions/3.8/Python] Error 1
make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_lang_python38/python38/work/Python-3.8.1'
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_lang_python38/python38/work/Python-3.8.1" && /usr/bin/make -w all 
Exit code: 2
Error: Failed to build python38: command execution failed

Ken, is your patch OK for this case too?

[kencu (Ken)]

yes it is.

[ballapete (Peter "Pete" Dyballa)]

Replying to kencu:

yes it is.

Python 3.8 built and installed on PPC Leopard with your patch.

[kencu (Ken)]

In 17a67bbbeb98bfbc27d006d41c699d5c84a489dc/macports-ports (master):

python38: fix build on older systems

there is no copyfile on Tiger
work around lack of pthread_threadid_np on < 10.6

closes: #59827
closes: #59772

[kencu (Ken)]

the patch i borrowed had a typo that I only noted on 14th reading. very sorry. will redo it once fixed up a rerun all the tests.

[ballapete (Peter "Pete" Dyballa)]

Version @3.8.2 shows this error:

/opt/local/bin/gcc-apple-4.2 -c -fno-strict-aliasing -Wsign-compare -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -arch ppc   -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I./Include/internal  -I. -I./Include -I/opt/local/include   -DPy_BUILD_CORE \
      -DGITVERSION="\"`LC_ALL=C `\"" \
      -DGITTAG="\"`LC_ALL=C `\"" \
      -DGITBRANCH="\"`LC_ALL=C `\"" \
      -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
/opt/local/bin/gcc-apple-4.2 -c -fno-strict-aliasing -Wsign-compare -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -arch ppc   -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I./Include/internal  -I. -I./Include -I/opt/local/include   -DPy_BUILD_CORE -o Python/frozen.o Python/frozen.c
rm -f libpython3.8.a
ar rcs libpython3.8.a Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/token.o  Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/graminit.o Python/hamt.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/peephole.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o    Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o  Modules/_collectionsmodule.o  Modules/_abc.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_stat.o  Modules/timemodule.o  Modules/_threadmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/faulthandler.o  Modules/_tracemalloc.o Modules/hashtable.o  Modules/symtablemodule.o  Modules/xxsubtype.o Python/frozen.o
/opt/local/bin/ranlib: file: libpython3.8.a(dynamic_annotations.o) has no symbols
/opt/local/bin/ranlib: file: libpython3.8.a(pymath.o) has no symbols
/usr/bin/install -c -d -m 755 Python.framework/Versions/3.8
/opt/local/bin/gcc-apple-4.2 -o Python.framework/Versions/3.8/Python -L/opt/local/lib -Wl,-headerpad_max_install_names -arch ppc    -dynamiclib \
-all_load libpython3.8.a -Wl,-single_module \
-install_name /opt/local/Library/Frameworks/Python.framework/Versions/3.8/Python \
-compatibility_version 3.8 \
-current_version 3.8 \
-framework CoreFoundation -lintl -ldl   -framework CoreFoundation;
Undefined symbols:
  "_fcopyfile", referenced from:
      _os__fcopyfile in libpython3.8.a(posixmodule.o)
     (maybe you meant: cstring=_fcopyfile)
  "_pthread_threadid_np", referenced from:
      _PyThread_get_thread_native_id in libpython3.8.a(thread.o)
ld: symbol(s) not found
collect2: ld returned 1 exit status
make: *** [Python.framework/Versions/3.8/Python] Error 1
make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_lil.fr.rsync.macports.org_release_tarballs_ports_lang_python38/python38/work/Python-3.8.2'
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_lil.fr.rsync.macports.org_release_tarballs_ports_lang_python38/python38/work/Python-3.8.2" && /usr/bin/make -w all 

[kencu (Ken)]

I fixed this once, but the fix I borrowed had a bad typo in it, so I reverted it. Use that same patch as I mentioned before <​https://trac.macports.org/ticket/59827#comment:7> until I get this fixed properly.

Just shell-shocked. It takes an evening of testing to be sure the fixes are working correctly, and then when a typo slips past me I get ticked off a bit.

[ballapete (Peter "Pete" Dyballa)]

Replying to kencu:

This patch lets python38 @3.8.2_0 build. What is needed now is an accelerator for installation. It feels like installing this version takes much longer than building it…

[kencu (Ken)]

four months. I still have a perfectly good fix sitting here. OK to push it?

Meanwhile, I am working on adding a copyfile implementation to legacysupport that this port, and all the others that need copyfile, can use. Dunno when it will be available, but it should help a lot of this trouble.

[reneeotten (Renee Otten)]

fine with me Ken. If you say it solves the issues on older systems and doesn't affect newer systems where Python 3.8 already builds now, I see no objection to commit it. Besides that, the port is "nomaintainer" (I've been trying to keep it up-to-date), so in principle there is no need to wait for anyone and you can just commit your changes.

[ken-cunningham-webuse]

In 7da358db764df2913c58385f3857e248191cc460/macports-ports (master):

python38: fix build on < 10.6

work around no copyfile on < 10.5
work around no pthread_threadid_np on < 10.6

closes: #60342
closes: #59772
closes: #59827

Thanks to @dgelessus as well for valuable input
There is a comprehensive patch upstream being considered
for a future python release.