python38: fails on Leopard: Undefined symbols: :info:build "_pthread_threadid_np"

[kencu (Ken)]

:info:build Undefined symbols:
:info:build   "_pthread_threadid_np", referenced from:
:info:build       _PyThread_get_thread_native_id in libpython3.8.a(thread.o)

that symbol exists on SnowLeopard and later. In llvm-9.0, I handled a similar issue like this:

diff --git llvm_master/lib/Support/Unix/Threading.inc macports_master/lib/Support/Unix/Threading.inc
index ed9a9656305..e8f9a13860f 100644
--- llvm_master/lib/Support/Unix/Threading.inc
+++ macports_master/lib/Support/Unix/Threading.inc
@@ -14,6 +14,7 @@
 #include "llvm/ADT/Twine.h"
 
 #if defined(__APPLE__)
+#include <Availability.h>
 #include <mach/mach_init.h>
 #include <mach/mach_port.h>
 #endif
@@ -153,8 +154,10 @@ void llvm::set_thread_name(const Twine &Name) {
   ::pthread_setname_np(::pthread_self(), "%s",
     const_cast<char *>(NameStr.data()));
 #elif defined(__APPLE__)
+#if __MAC_OS_X_VERSION_MAX_ALLOWED >= 1060
   ::pthread_setname_np(NameStr.data());
 #endif
+#endif
 }
 
 void llvm::get_thread_name(SmallVectorImpl<char> &Name) {
-- 

[jmroot (Joshua Root)]

Is there a reason they're not just using pthread_self() in the first place?

[kencu (Ken)]

I think that's the special insight you always bring to these questions, Josh.

[jmroot (Joshua Root)]

Probably because it returns a pointer to a struct and not a "real" thread id number. There may be concerns about that not being guaranteed to be unique over the lifetime of the process. I don't know if using an undocumented syscall will give better results though. Have you tested that it returns sane values? At minimum, I'd want to start a few threads and check that the ids it returns for them are all different, and maybe kill one and spawn a new one and check again.

[kencu (Ken)]

I copied exactly the exact same call block python uses now on linux, in the same file -- see the lines immediately above (in the first) and below (in the second).

[jmroot (Joshua Root)]

Yes but there's no guarantee that it does the same thing on Darwin as it does on Linux.

[kencu (Ken)]

(Heh -- I was typing as you were typing --- )

but -- if you show me how to do what you ask -- me or dgellieus (sp?) will do it.

I am out of commission for a week or two now, however, so if I don't answer, it's because of that.

[kencu (Ken)]

well well well <​http://elliotth.blogspot.com/2012/04/gettid-on-mac-os.html>

"If you poke around, you'll see that the Darwin/xgnu kernel actually has a gettid(2) system call. But before you get excited, you'll find that it's completely unrelated to the Linux gettid(2). It returns the "per-thread override identity", which is a uid and gid that a thread might be operating under (like a per-thread setuid(2) kind of facility). No use to us."

[SerpentChris (Chris Calderon)]

Can Ken's patch be added to this port so that it will build? Also does anyone know how to contact the Python developers to let them know about this? I believe the problems with building Python on older Macs stem from the recent removal of their Tiger build bot.

[kencu (Ken)]

My patch is not really right, Josh is correct. It does allow the build, but I need to fix it better. Just have one other thing to do first, then I'll work on it.

[kencu (Ken)]

I forgot about this ticket...sorry.

[kencu (Ken)]

Some resources I accumulated in hopes of coming up with some kind of implementation for pthread_threadid_np on < 10.6 that might not be total garbage:

<​https://stackoverflow.com/questions/1540603/mac-iphone-is-there-a-way-to-get-a-thread-identifier-without-using-objective-c>

<​https://stackoverflow.com/questions/21091000/how-to-get-thread-id-of-a-pthread-in-linux-c-program>

<​https://developer.apple.com/library/archive/documentation/General/Conceptual/ConcurrencyProgrammingGuide/ThreadMigration/ThreadMigration.html#//apple_ref/doc/uid/TP40008091-CH105-SW1>

<​https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/KernelProgramming/scheduler/scheduler.html>

<​https://robert.sesek.com/2018/7/debugging_early_program_startup_on_macos.html>

Alternatively, if none of these lead to something acceptable, it's not the end of the world to keep a local Tiger patch here hidden away in MacPorts; these threadid's are mostly used for debugging anyway, and goodness help anyone who is trying to do some serious debugging on Tiger these days...people just want python38 to build so they're not left high and dry...

[kencu (Ken)]

This snippet is getting very close, just need to flesh out the details of replicating the full functionality of pthread_threadid_np. This might need to go into legacysupport as it is a tad complicated, but perhaps the python source file will accept these headers being added ...

#include <unistd.h>

#include <errno.h>
#include <iostream>
#include <pthread.h>
#include <sys/syscall.h>

#include <mach/kern_return.h>
#include <mach/thread_info.h>
#include <mach/thread_act.h>

uint64_t my_pthread_threadid_np(pthread_t thread) {
    mach_port_name_t port=pthread_mach_thread_np(thread);

    thread_identifier_info_data_t info;
    mach_msg_type_number_t info_count=THREAD_IDENTIFIER_INFO_COUNT;
    kern_return_t kr=thread_info(port,
                                 THREAD_IDENTIFIER_INFO,
                                 (thread_info_t)&info,
                                 &info_count);
    if(kr!=KERN_SUCCESS) {
        /* you can get a description of the error by calling
         * mach_error_string(kr)
         */
        return 0;
    } else {
        return info.thread_id;
    }
}

int main() {
 std::cout << "getpid()=" << getpid() << std::endl;

 std::cout << "pthread_self()=" << pthread_self() << std::endl;
 uint64_t tid;
 pthread_threadid_np(NULL, &tid);
 std::cout << "pthread_threadid_np()=" << tid << std::endl;

 std::cout << "syscall(SYS_thread_selfid)=" << syscall(SYS_thread_selfid) << std::endl;
 
//for (int i=0; i < 100; i++) { 
 mach_port_t newtid = pthread_mach_thread_np(pthread_self());
 std::cout << "pthread_mach_thread_np()=" << (uint64_t)newtid << std::endl;
//}
 
 std::cout << "my_pthread_threadid_np()=" << my_pthread_threadid_np(pthread_self()) << std::endl;
 
 return 0;
}


// getting very close to replicating the function
// check the man page for pthread_threadid_np for details on implementation
// probably can find it in Apple Open Source as well

[kencu (Ken)]

this looks like it will do it:

#include <errno.h>
#include <pthread.h>
#include <mach/kern_return.h>
#include <mach/thread_info.h>
#include <mach/thread_act.h>

int pthread_threadid_np(pthread_t thread, __uint64_t *thread_id) {

    if (thread_id == NULL) {
        return EINVAL;
    }

    if (thread == NULL) {
        thread = pthread_self();
        
        if (thread == NULL ) {
          return ESRCH;
        }
    }

    mach_port_name_t port = pthread_mach_thread_np(thread);

    thread_identifier_info_data_t info;
    mach_msg_type_number_t info_count = THREAD_IDENTIFIER_INFO_COUNT;

    kern_return_t kr = thread_info(port,
                                   THREAD_IDENTIFIER_INFO,
                                   (thread_info_t)&info,
                                   &info_count);
    if(kr != KERN_SUCCESS) {
        return ESRCH;
    } else {
        *thread_id = info.thread_id;
        return 0;
    }
}

[kencu (Ken)]

but not on Tiger or Leopard -- grrr -- where these are actually needed.

I have surveyed the web, and prior to SnowLeopard, and even now in many cases, this is used for a thread_id:

mach_port_name_t port = pthread_mach_thread_np(thread);

I think for the purposes of a reasonable approximation, that is what we should use on Tiger and Leopard until (and if) such time as we figure out that there is actually kernel support in Tiger and Leopard for the full pthread_threadid_np that is used on SnowLeopard and later.

[kencu (Ken)]

In 17a67bbbeb98bfbc27d006d41c699d5c84a489dc/macports-ports (master):

python38: fix build on older systems

there is no copyfile on Tiger
work around lack of pthread_threadid_np on < 10.6

closes: #59827
closes: #59772

[kencu (Ken)]

the patch i borrowed had a typo that I only noted on 14th reading. very sorry. will redo it once fixed up and I rerun all the tests.

[kencu (Ken)]

OK -- four months now. I still have the fix sitting here. OK to just push it now?

[proxsi]

I have the same issue.

[ken-cunningham-webuse]

In 7da358db764df2913c58385f3857e248191cc460/macports-ports (master):

python38: fix build on < 10.6

work around no copyfile on < 10.5
work around no pthread_threadid_np on < 10.6

closes: #60342
closes: #59772
closes: #59827

Thanks to @dgelessus as well for valuable input
There is a comprehensive patch upstream being considered
for a future python release.

[barracuda156]

Replying to kencu:

This snippet is getting very close, just need to flesh out the details of replicating the full functionality of pthread_threadid_np. This might need to go into legacysupport as it is a tad complicated, but perhaps the python source file will accept these headers being added ...

Has it ever made into legacysupport?

[kencu (Ken)]

Nah, I ran out of steam for the proper fix -- but you can take over and get it done!

[barracuda156]

Replying to kencu:

Nah, I ran out of steam for the proper fix -- but you can take over and get it done!

What do we need to implement? pthread_setname_np, pthread_threadid_np, any friends?

Yeah, it gets annoying to have to deal with these every time. Makes better sense to fix once for all.