openjdk11 has broken files after multiple rev-upgrade

[fracai]

The openjdk11 port reports multiple broken files after repeated attempts to rebuild. This is on an M1 Mac, but I know I saw this on an Intel Mac running Big Sur and Catalina at least. This seems similar to #57677, though that ticket seems to indicate it's an issue for older OS releases.

[fracai]

port -d -y rev-upgrade

[breun (Nils Breunese)]

I don't see this issue on my Intel Mac running macOS Big Sur 11.3. The openjdk1 port is installing a prebuilt binaries from AdoptOpenJDK, so if there really is something wrong with these files, then that would need to be fixed in AdoptOpenJDK. However, I don't know if AdoptOpenJDK's builds are supposed to work on an M1. I don't have access to an M1 Mac, so I cannot verify this myself. I believe AdoptOpenJDK is planning to add ARM builds for macOS, but these haven't been released yet as far as I know.

I do know that Azul Zulu OpenJDK has ​arm64 builds for macOS. I could add those for the openjdk*-zulu ports, but I can't test them myself.

[breun (Nils Breunese)]

I'll try adding arm64 for openjdk*-zulu via ​https://github.com/macports/macports-ports/pull/10881

[fracai]

Are there any additional logs I could add or info to provide with more info?

[breun (Nils Breunese)]

The openjdk11 port basically just downloads ​https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.11%2B9/OpenJDK11U-jdk_x64_mac_hotspot_11.0.11_9.tar.gz and unpacks it under /Library/Java/JavaVirtualMachines.

If you do the following, does this work on an M1?

% curl -OL https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.11%2B9/OpenJDK11U-jdk_x64_mac_hotspot_11.0.11_9.tar.gz
% tar zxf OpenJDK11U-jdk_x64_mac_hotspot_11.0.11_9.tar.gz
% ./jdk-11.0.11+9/Contents/Home/bin/java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)

[fracai]

Yes:

% ./jdk-11.0.11+9/Contents/Home/bin/java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)

As does:

% /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/bin/java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)

What I'm reporting as "broken" is what: port rev-upgrade reports about the openjdk11 port.

Is this just a false positive from rev-upgrade?

[breun (Nils Breunese)]

I'm sorry, but I don't know what command/process runs during rev-upgrade that decides those files are 'broken'. I'll see if I can find out.

[cjones051073 (Chris Jones)]

The core issue is

Could not open /System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaNativeFoundation.framework/Versions/A/JavaNativeFoundation: Error opening or reading file (referenced from /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libawt.dylib)

So the (x86_64) libawt.dylib could not open a system framework. Quite why I don’t know and as I don’t have an arm machine I cannot investigate.

Can you try running otool -L on the above dylib and framework to see what that reports ?

[breun (Nils Breunese)]

For reference, on an x86_64 machine running macOS 11.3.1 I get this:

% otool -L /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libawt.dylib
/Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libawt.dylib:
	@rpath/libawt.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libjvm.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libjava.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
	@rpath/libmlib_image.dylib (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaNativeFoundation.framework/Versions/A/JavaNativeFoundation (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaRuntimeSupport.framework/Versions/A/JavaRuntimeSupport (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 50.1.0)
	/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 492.0.0)

[breun (Nils Breunese)]

@cjones051073 There is a second class of messages in the output that look worrying:

DEBUG: Marking /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libosx.dylib as broken
DEBUG: Marking /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libosxapp.dylib as broken
DEBUG: Marking /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libosxkrb5.dylib as broken
DEBUG: Marking /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libosxsecurity.dylib as broken
DEBUG: Marking /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libosxui.dylib as broken
DEBUG: Marking /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libsaproc.dylib as broken
DEBUG: Marking /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libsplashscreen.dylib as broken

Can these be ignored?

[cjones051073 (Chris Jones)]

Probably the same reason as the first one. Focus on that first.

[fracai]

On 11.3 arm64. Edited with the result after removing the old plugins.

$ otool -L /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libawt.dylib
/Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libawt.dylib:
	@rpath/libawt.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libjvm.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libjava.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
	@rpath/libmlib_image.dylib (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaNativeFoundation.framework/Versions/A/JavaNativeFoundation (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaRuntimeSupport.framework/Versions/A/JavaRuntimeSupport (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 50.1.0)
	/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 492.0.0)

[cjones051073 (Chris Jones)]

so that worked... I don't know why it doesn't work in the MP check. I think you should ping the devel list to see if anyone there has an idea.

The best solution of course is to use a proper native arm version....

[fracai]

Also, I don't have a /System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaNativeFoundation.framework/Versions/A/JavaNativeFoundation

But, I do have: /System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaNativeFoundation.framework/Versions/A/Resources/BridgeSupport/JavaNativeFoundation.bridgesupport

$ tree /System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaNativeFoundation.framework/Versions/A/
/System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaNativeFoundation.framework/Versions/A/
├── Resources
│   ├── BridgeSupport
│   │   └── JavaNativeFoundation.bridgesupport
│   ├── Info.plist
│   └── version.plist
└── _CodeSignature
    └── CodeResources

[neverpanic (Clemens Lang)]

One of the binaries references a file that doesn't exist. Usually that means it won't start when you attempt to run it. Is this actually the case on arm64? If it works, are the files that are marked as broken actually used? If they're just unused, you could delete them to fix the issue.

[breun (Nils Breunese)]

I also don't have /usr/lib/libSystem.B.dylib or any of those /System/Library/Frameworks/* files in the output of otool -L /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libawt.dylib on my Intel Mac, but I don't see anything complaining about that.

I don't know how I can be sure they are not used. I guess libawt.dylib might only be used by Java applications that use the Java AWT framework.

[breun (Nils Breunese)]

@fracai I have added arm64 native builds of Azul Zulu OpenJDK to MacPorts. Could you try installing the openjdk11-zulu port and see if that works?

[breun (Nils Breunese)]

For the x86_64 openjdk11-zulu I see the same non-existent files in the output of running otool -L on its libawt.dylib:

% otool -L /Library/Java/JavaVirtualMachines/openjdk11-zulu/Contents/Home/lib/libawt.dylib
/Library/Java/JavaVirtualMachines/openjdk11-zulu/Contents/Home/lib/libawt.dylib:
	@rpath/libawt.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libjvm.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libjava.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
	@rpath/libmlib_image.dylib (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 22.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaNativeFoundation.framework/Versions/A/JavaNativeFoundation (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/JavaVM.framework/Versions/A/Frameworks/JavaRuntimeSupport.framework/Versions/A/JavaRuntimeSupport (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 50.0.0)
	/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 492.0.0)

[fracai]

I installed openjdk11-zulu without any rev-upgrade complaints. And I see the same list of files from otool. Still not sure what this means for the regular openjdk11. And unfortunately it's a dependency of maven3, so I can't switch. Maybe a variant for maven3? Or a file dependency?

In any event, this maybe qualify as a workaround for now?

[breun (Nils Breunese)]

maven3 works with Java 7 or higher, but if you don't have that already installed it will try to install openjdk11 as a fallback: ​https://github.com/macports/macports-ports/blob/master/java/maven3/Portfile#L42-L43

I'm not sure how the detection logic is implemented in the java PortGroup, but does sudo port install maven3 still try to install openjdk11 if you first install another JDK? If so the java PortGroup might need to be updated to also recognise Azul Zulu OpenJDK.

[fracai]

I uninstalled maven3 and openjdk11, noted that I still have 11-zulu installed, tried to install maven3, and openjdk11 was listed as a dep. It's not a huge headache and I can open a separate ticket for maven3 to avoid bloating this one.

[fracai]

Or a ticket for the java portgroup that is.

[breun (Nils Breunese)]

maven3 could set a different java.fallback when the arch is arm64, but that would imply that any port that has a Java dependency would need such a construct.

Maybe it would indeed be best that the java portgroup adds -zulu to the java.fallback value when running on arm64 until openjdk11 properly supports arm64.

[cjones051073 (Chris Jones)]

Yes, this is definitely best handled by the port group.

[ryandesign (Ryan Carsten Schmidt)]

Replying to breun:

I also don't have /usr/lib/libSystem.B.dylib or any of those /System/Library/Frameworks/* files in the output of otool -L /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/lib/libawt.dylib on my Intel Mac, but I don't see anything complaining about that.

See ​https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes:

"New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)"