request for lldb

[RJVB (René Bertin)]

Would it be possible to provide a port for the debugger component of llvm, lldb?

[jeremyhu (Jeremy Huddleston Sequoia)]

Yeah, probably... if someone is interested in providing it and maintaining it.

[RJVB (René Bertin)]

I would if the Portfile code in the other llvm ports weren't so complicated, and lldb's own build system were a little less OS version specific ...

clang-3.4 clearly builds on 10.6.8 , is there any reason to believe lldb-3.4 wouldn't with the proper build scripts?

[larryv (Lawrence Velázquez)]

I’ll put this on my todo list.

[cooljeanius (Eric Gallager)]

personally, besides just lldb, I'd like every single llvm sub-project on the front page of llvm.org to have its own port, but then again I'm kind of OCD about checking off stuff on checklists, and lldb is probably the most important of the sub-projects that is still missing a port...

[cooljeanius (Eric Gallager)]

Cc Me!

[alexreg (Alexander Regueiro)]

Cc Me!

[eborisch (Eric A. Borisch)]

Cc Me!

[eborisch (Eric A. Borisch)]

Would also (hopefully) provide _lldb.so linked against MacPorts' python27 rather than the system one (and vim-lldb won't crash on startup.)

[eborisch (Eric A. Borisch)]

This is (likely) a big stumbling block for a MacPorts-provided lldb.

I suppose users who actually want it could create the certificate manually and then build lldb via MacPorts...

docs/code-signing.txt:

On MacOSX lldb needs to be code signed. The Debug, DebugClang and Release
builds  are set to code sign using a code signing certificate named
"lldb_codesign".

If you have re-installed a new OS, please delete all old lldb_codesign items
from your keychain. There will be a code signing certification and a public
and private key. Reboot after deleting them. You will also need to delete and
build folders that contained old signed items. The darwin kernel will cache
code signing using the executable's file system node, so you will need to
delete the file so the kernel clears its cache.

If you don't have one yet you will need to:
- Launch /Applications/Utilities/Keychain Access.app

- In Keychain Access select the "login" keychain in the "Keychains"
  list in the upper left hand corner of the window.

- Select the following menu item:

	Keychain Access->Certificate Assistant->Create a Certificate...

- Set the following settings

	Name = lldb_codesign
	Identity Type = Self Signed Root
	Certificate Type = Code Signing

- Click Create
- Click Continue
- Click Done
- Click on the "My Certificates"
- Double click on your new lldb_codesign certificate
- Turn down the "Trust" disclosure triangle, scroll to the "Code Signing" trust
  pulldown menu and select "Always Trust" and authenticate as needed using your
  username and password.
- Drag the new "lldb_codesign" code signing certificate (not the public or private
  keys of the same name) from the "login" keychain to the "System" keychain in the
  Keychains pane on the left hand side of the main Keychain Access window. This will
  move this certificate to the "System" keychain. You'll have to authorize a few
  more times, set it to be "Always trusted" when asked.
- Remove "~/Desktop/lldb_codesign.cer" file on your desktop if there is one.
- In the Keychain Access GUI, click and drag "lldb_codesign" in the "System" keychain
  onto the desktop. The drag will create a "~/Desktop/lldb_codesign.cer" file used in
  the next step.
- Switch to Terminal, and run the following:

sudo security add-trust -d -r trustRoot -p basic -p codeSign -k /Library/Keychains/System.keychain ~/Desktop/lldb_codesign.cer
rm -f ~/Desktop/lldb_codesign.cer

- Drag the "lldb_codesign" certificate from the "System" keychain back into the
  "login" keychain
- Quit Keychain Access
- Reboot
- Clean by removing all previously creating code signed binaries and rebuild
  lldb and you should be able to debug.

When you build your LLDB for the first time, the Xcode GUI will prompt you for permission
to use the "lldb_codesign" keychain. Be sure to click "Always Allow" on your first
build. From here on out, the "lldb_codesign" will be trusted and you can build from the
command line without having to authorize. Also the first time you debug using a LLDB that
was built with this code signing certificate, you will need to authenticate once.

Ugh.

[mmpestorich (Mike M Pestorich)]

Cc Me!

[RJVB (René Bertin)]

As mentioned on the dev ML too:

I have a first draft for an lldb-3.8 subport:

​https://github.com/RJVB/macstrop/commit/4608ad87a12075e63125e5ee2ffc4c3fc2bfb668#diff-6409887f54daa21d3d10e36ddda20f05

This appears to work, except for the whole signing business. Running the debugger via sudo allows it to attach but is of course no viable solution.

I wonder if there is no more elegant method to change an executables signing identity that doesn't involve rebooting as lldb's code-signing document suggests?

[RJVB (René Bertin)]

Setting the debugserver executable SETUID root works too, but is that less risky?

[RJVB (René Bertin)]

Either way, the existing codesigning instructions in the buildsystem do not appear to work through port build:

> port -nvok build lldb-3.8
...
cd /opt/local/var/macports/build/_opt_local_site-ports_lang_llvm-3.8/lldb-3.8/work/build/tools/lldb/tools/debugserver/source/MacOSX && /opt/local/bin/cmake -E cmake_link_script CMakeFiles/debugserver.dir/link.txt --verbose=ON
/opt/local/bin/clang++-mp-3.7   -O3 -march=native -std=c++11 -DNDEBUG -stdlib=libc++  -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -std=c++11 -Wno-deprecated-declarations -Wno-unknown-pragmas -Wno-strict-aliasing -Wno-deprecated-register -Wno-vla-extension  -fno-exceptions -fno-rtti -Wno-gnu-zero-variadic-macro-arguments -Wno-zero-length-array -Wno-extended-offsetof -arch x86_64 -mmacosx-version-min=10.9 -Wl,-search_paths_first -Wl,-headerpad_max_install_names  -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,@loader_path -Wl,-rpath,@loader_path/ -stdlib=libc++ -Wl,-sectcreate,__TEXT,__info_plist,/opt/local/var/macports/build/_opt_local_site-ports_lang_llvm-3.8/lldb-3.8/work/llvm-3.8.1.src/tools/lldb/tools/debugserver/source/../resources/lldb-debugserver-Info.plist  -Wl,-dead_strip CMakeFiles/debugserver.dir/HasAVX.s.o CMakeFiles/debugserver.dir/CFBundle.cpp.o CMakeFiles/debugserver.dir/CFData.cpp.o CMakeFiles/debugserver.dir/CFString.cpp.o CMakeFiles/debugserver.dir/Genealogy.cpp.o CMakeFiles/debugserver.dir/MachException.cpp.o CMakeFiles/debugserver.dir/MachProcess.mm.o CMakeFiles/debugserver.dir/MachTask.mm.o CMakeFiles/debugserver.dir/MachThread.cpp.o CMakeFiles/debugserver.dir/MachThreadList.cpp.o CMakeFiles/debugserver.dir/MachVMMemory.cpp.o CMakeFiles/debugserver.dir/MachVMRegion.cpp.o CMakeFiles/debugserver.dir/mach_excServer.c.o CMakeFiles/debugserver.dir/mach_excUser.c.o CMakeFiles/debugserver.dir/debugserver_vers.c.o  -o ../../../../../../bin/debugserver  ../../../../../../lib/liblldbDebugserverCommon.a ../../../../../../lib/liblldbUtility.a ../../../../../../lib/liblldbDebugserverMacOSX_I386.a ../../../../../../lib/liblldbDebugserverMacOSX_X86_64.a -framework Cocoa -Wl,-rpath,@executable_path/../lib 
cd /opt/local/var/macports/build/_opt_local_site-ports_lang_llvm-3.8/lldb-3.8/work/build/bin && /opt/local/bin/cmake -E env CODESIGN_ALLOCATE=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate codesign --force --sign lldb_codesign debugserver
lldb_codesign: no identity found
make[2]: *** [bin/debugserver] Error 1
make[2]: *** Deleting file `bin/debugserver'
make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_site-ports_lang_llvm-3.8/lldb-3.8/work/build'
make[1]: *** [tools/lldb/tools/debugserver/source/MacOSX/CMakeFiles/debugserver.dir/all] Error 2

The signing does work when running make by hand in the debugserver's build directory (port work lldb-3.8/build/tools/lldb/tools/debugserver) , but it doesn't yield a functional (sanctioned) debugserver executable. Of course I didn't reboot after each different signing attempt ...

[raimue (Rainer Müller)]

If the certificate named 'lldb_codesign' is only in you login keychain, the user macports will not have access to it during building.

See also #51504, where you started to implement codesigning as a port group already.

[RJVB (René Bertin)]

Well, my macports user setting is to use my own username. I know the risks, but I think I discovered why I rarely ever saw those realised; the same reason for which the codesign app didn't find my signing certificate. MacPorts "base" sets the HOME variable to something safe, even if you're using an existing username as the macports user.

I just updated the code signing PortGroup, I hope this discussion will draw some activity to that ticket.

[RJVB (René Bertin)]

Hah, all I had to do was uninstall lldb-3.8, restart taskgated (sudo killall -1 taskgated) and reinstall lldb-3.8 . That took care of kernel certificate caching.

I must say I am running taskgated with the -p option, so it's still possible that YMMV.

[RJVB (René Bertin)]

The port still needs the registry file for the port select feature.

[RJVB (René Bertin)]

Replying to eborisch@…:

Would also (hopefully) provide _lldb.so linked against MacPorts' python27 rather than the system one (and vim-lldb won't crash on startup.)

> otool -L /opt/local/libexec/llvm-3.8/lib/liblldb.dylib 
/opt/local/libexec/llvm-3.8/lib/liblldb.dylib:
        /opt/local/libexec/llvm-3.8/lib/liblldb.dylib (compatibility version 0.0.0, current version 3.8.1)
        /opt/local/lib/libedit.0.dylib (compatibility version 1.0.0, current version 1.51.0)
        /opt/local/lib/libncurses.6.dylib (compatibility version 6.0.0, current version 6.0.0)
        /opt/local/lib/libform.6.dylib (compatibility version 6.0.0, current version 6.0.0)
        /opt/local/lib/libpanel.6.dylib (compatibility version 6.0.0, current version 6.0.0)
        /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
        /opt/local/lib/libxml2.2.dylib (compatibility version 12.0.0, current version 12.2.0)
        /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 157.0.0)
        /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1056.17.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.17.0)
        /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 59.0.0)
        /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 55471.14.40)
        /System/Library/PrivateFrameworks/DebugSymbols.framework/Versions/A/DebugSymbols (compatibility version 1.0.0, current version 106.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
        /opt/local/lib/libffi.6.dylib (compatibility version 7.0.0, current version 7.4.0)
        /opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
        /opt/local/libexec/llvm-3.8/lib/libLLVM.dylib (compatibility version 1.0.0, current version 3.8.1)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
        /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

[RJVB (René Bertin)]

I've added an lldb subport to the new release version of port:llvm-3.9 . Initial testing indicates it is functional.

[jeremyhu (Jeremy Huddleston Sequoia)]

Some comments:

1. Don't skip codesigning. There's not really a reason that I can see to do patch-no-codesign-3.9.diff​. If the developer doesn't have a CODESIGN_IDENTITY set, just adhoc sign it by setting LLDB_CODESIGN_IDENTITY to - when executing cmake.
2. We should probably allow users to specifiy a codesigning identity in macports.conf and honor that.
3. Where is the code-sign-1.0.tcl that you reference in the Portfile changes?
4. We should not rebuild everything for lldb. The cmake build system should be capable of building lldb against the installed llvm and libclang. Can you try that out. There are some reasons why we can't yet do that for clang, but it's a goal to not have to rebuild what's already built.
5. We should not need to use install_name_tool to change the dylib identifiers.
6. Why patch-accept-build_types.diff? What's being passed that isn't accepted? Should you instead update that list and push the patch upstream?

[RJVB (René Bertin)]

Replying to jeremyhu@…:

1. Don't skip codesigning. There's not really a reason that I can see to do patch-no-codesign-3.9.diff​. If the developer doesn't have a CODESIGN_IDENTITY set, just adhoc sign it

by setting LLDB_CODESIGN_IDENTITY to - when executing cmake.

That's what I did at first, but as discussed elsewhere, the only location where we can really codesign from within MacPorts is in the (post) activate step. During the build step where lldb is normally signed, the user is the macports user, who typically won't have access to the *operator*'s keychain (and HOME will be set to point elsewhere anyway). We can of course sign twice, but that would mean using -f the 2nd time, and are we sure that will work, what with the kernel caching?

2. We should probably allow users to specifiy a codesigning identity in macports.conf and honor that.

There's a parallel discussion on the devel ML about this.

3. Where is the code-sign-1.0.tcl that you reference in the Portfile changes?

Submitted elsewhere, ticket #51504

4. We should not rebuild everything for lldb. The cmake build system should be capable of building lldb against the installed llvm and libclang. Can you try that out. There are some reasons why we can't yet do that for clang, but it's a goal to not have to rebuild what's already built.

When I can find a couple of hours I don't need my machine for other things ... Note that I build within the lldb subdirectory. CMake's Makefiles are clever enough to rebuild only what's absolutely necessary in that case.

5. We should not need to use install_name_tool to change the dylib identifiers.

Is there a reason why it's necessary for the other subports?

6. Why patch-accept-build_types.diff? What's being passed that isn't accepted? Should you instead update that list and push the patch upstream?

That's because of a change I proposed for the CMake portgroup which still hasn't been included (or rejected). Using another than the 4 predefined build types it becomes possible to control the compiler flags (upstreams of what the buildsystem might do with them). Debian has a similar tactic (-DCMAKE_BUILD_TYPE=Debian), I presume for the same reason.

[RJVB (René Bertin)]

In fact, do you have any guidelines or at least arguments to pass to CMake for building lldb in isolation? The online build instructions don't seem to consider that possibility, except by using one of the Xcode projects (but I reckon that's not a good idea here).

[jeremyhu (Jeremy Huddleston Sequoia)]

Replying to rjvbertin@…:

the only location where we can really codesign from within MacPorts is in the (post) activate step.

I'm not suggesting that you somehow get the macports user to access the real user's keychain. That's not needed to adhoc sign. Just set LLDB_CODESIGN_IDENTITY="-" to get it adhoc signed.

3. Where is the code-sign-1.0.tcl that you reference in the Portfile changes?

Submitted elsewhere, ticket #51504

4. We should not rebuild everything for lldb. The cmake build system should be capable of building lldb against the installed llvm and libclang. Can you try that out. There are some reasons why we can't yet do that for clang, but it's a goal to not have to rebuild what's already built.

When I can find a couple of hours I don't need my machine for other things ... Note that I build within the lldb subdirectory. CMake's Makefiles are clever enough to rebuild only what's absolutely necessary in that case.

Ok, that's good enough for now, but we should look into that a bit more.

5. We should not need to use install_name_tool to change the dylib identifiers.

Is there a reason why it's necessary for the other subports?

It's not. The other subports don't do it (as of a year or so ago).

6. Why patch-accept-build_types.diff? What's being passed that isn't accepted? Should you instead update that list and push the patch upstream?

That's because of a change I proposed for the CMake portgroup which still hasn't been included (or rejected). Using another than the 4 predefined build types it becomes possible to control the compiler flags (upstreams of what the buildsystem might do with them). Debian has a similar tactic (-DCMAKE_BUILD_TYPE=Debian), I presume for the same reason.

Ok. Perhaps send this patch upstream to llvm.org then.

[RJVB (René Bertin)]

Replying to jeremyhu@…:

Ok, that's good enough for now, but we should look into that a bit more.

I could raise the question on the lldb ML...

Is there a reason why it's necessary for the other subports?

It's not. The other subports don't do it (as of a year or so ago).

My bad. I introduced the identifier change when I was setting up a variant that builds a static libllvm and thus allows to build a redistributable libclang. (Potential) ports that provide a clang-based parser could benefit from that as they could simply depend on *a* port providing libclang, without the need for installing a full clang port or clang-version variants. KDevelop is one such port, slated for release in a near future along with the other KF5 ports. It's been a while, but it looks that the identify change is required to make all the libraries load when using KDevelop and NOT using a redistributable build.

The change basically locks the dylibs to their intended location; is there anything wrong with that?

That's because of a change I proposed for the CMake portgroup which still hasn't been included (or rejected). Using another than the 4 predefined build types it becomes possible to control the compiler flags (upstreams of what the buildsystem might do with them). Debian has a similar tactic (-DCMAKE_BUILD_TYPE=Debian), I presume for the same reason.

Ok. Perhaps send this patch upstream to llvm.org then.

OK, I'll see about that. I have a hunch that "they" will tell me just to use one of CMake's predefined types, and that if MacPorts wants better control over the compiler options it's their problem ;)

[RJVB (René Bertin)]

Re: install_name_tool: I thought I'd do a rapid test not requiring a full rebuild by building KDEvelop's clang parser against 3.6.2, and discovered that the clang/llvm dylibs use the full path instead of @rpath. Exactly the change I introduced in my Portfiles...

[RJVB (René Bertin)]

Re: install_name_tool (2)

I installed llvm+clang 3.8 from the servers, so with @rpath in libLLVM and libclang. This clearly no longer poses a problem for KDevelop's clang-based parser. I don't have any other cross-platform projects that link against libclang and/or libLLVM that I can test though, so don't really know what to do here. Should we assume that it's up to the dependent software to add the path to libclang to its rpath?

FWIW: I also timed my build (-O3 and the nortti variant) vs. the default build. A C++ project that takes about 1 minute to build with the former takes roughly 20 seconds more with the latter (i.e. about 30% slower).

[jeremyhu (Jeremy Huddleston Sequoia)]

a redistributable libclang

I'm ok with splitting up the port and would be happy do to so if we can avoid build time regressions and setup sane dependencies. I don't like the current situation in which clang rebuilds libLLVM.dylib at build time instead of using llvm-config to find the installed one to build and link against.

The change basically locks the dylibs to their intended location; is there anything wrong with that?

If that is needed, it should be addressed as a patch to the build system that can be integrated upstream. Using install_name_tool can introduce a dependency on cctools, which is a pain point as it can cause a dependncy cycle (eg: building clang-3.7 on Leopard requires us to build two older llvm versions in order to bootstrap the process: llvm-3.3 -> cctools -> llvm-3.4 -> ld64 -> clang-3.4 -> llvm-3.7 -> clang-3.7).

Should we assume that it's up to the dependent software to add the path to libclang to its rpath?

Let's track that in a separate bug (I filed #52196). I'm of the opinion that llvm-config --ldflags should "do the right thing" but it doesn't setup rpath.

[RJVB (René Bertin)]

[this inadvertently went out as an email first, sorry about that]

I'm ok with splitting up the port and would be happy do to so if we can avoid build time regressions and setup sane dependencies.

Building a redistributable, standalone libclang is a bit of a separate issue, no? That said, it's also supposed to speed up compile times when using clang, as there are less shared libraries to load at each invocation.

I don't like the current situation in which clang rebuilds libLLVM.dylib at build time instead of using llvm-config to find the installed one to build and link against.

No, I agree (all the more since build times for 3.9 appear to be almost 50% longer than the ones for 3.8).

That said, I tried my trick of calling make in the tools/clang. For once it doesn't work, probably because there are apparently 3 parallel directories that have to be built in the right order.

You probably saw my thread about standalone building of lldb. In short: it almost works. I copied the missing CheckAtomic.cmake file into the installed cmake/llvm directory, and tried to build lldb. It aborted because of trying to include a headerfile from the llvm source:

In file included from /opt/local/var/macports/build/_opt_local_site-ports_lang_llvm-3.9/lldb-3.9/work/llvm-3.9.0.src/tools/lldb/tools/lldb-mi/MICmdCmdData.cpp:45:
/opt/local/var/macports/build/_opt_local_site-ports_lang_llvm-3.9/lldb-3.9/work/llvm-3.9.0.src/tools/lldb/tools/lldb-mi/MIUtilParse.h:13:10: fatal error: 
      '../lib/Support/regex_impl.h' file not found
#include "../lib/Support/regex_impl.h"
         ^
1 error generated.

But, the extrapolated build time would be around 45 minutes (against IIRC almost 1h15 for my current approach) and the extrapolated build directory size is over 2x smaller.

If that is needed, it should be addressed as a patch to the build system that can be integrated upstream. Using install_name_tool can introduce a dependency on cctools, which is a pain point as it can cause a dependncy

Can you explain? Why would that the be case as long as we don't tell the port to use the copy from cctools?

Let's track that in a separate bug (I filed #52196). I'm of the opinion that llvm-config --ldflags should "do the right thing" but it doesn't setup rpath.

I've thought about this a bit more, and it may be a moot point. The official Linux builds I have on my Ubuntu rig also install libclang, libLLVM etc. somewhere that's not part of the standard ldconfig search path. IOW, cross-platform code already has to add the path to those libraries to its rpath, which should solve the issue on OS X too. No?

[jeremyhu (Jeremy Huddleston Sequoia)]

Why would that the be case as long as we don't tell the port to use the copy from cctools?

Because the cctools port might be newer than the Xcode.app-installed version, and the toolchain building the port might be using emitting load commands that the Xcode.app-provided version does not understand. This becomes more obvious when you look back at the Leopard case.

[jeremyhu (Jeremy Huddleston Sequoia)]

Ok, well the parts that are not controversial are now in llvm-devel as r152453. We can iterate on it.

[RJVB (René Bertin)]

OK, I on my end will maintain the 3.9 implementation in my macstrop github repo. I don't have the resources to do much more.

I did manage to figure out how to do a standalone build, though. That currently requires installing an additional .cmake file with port:llvm and unpacking the lldb sources in the llvm source tree, but the resulting build directory is over twice smaller. The extrapolated estimates in an earlier post were clearly too large. These findings have been fed back to the lldb ML so someone upstream will probably take care of the issue.

[RJVB (René Bertin)]

Replying to jeremyhu@…:

Because the cctools port might be newer than the Xcode.app-installed version, and the toolchain building the port might be using emitting load commands that the Xcode.app-provided version does not understand. This becomes more obvious when you look back at the Leopard case.

You do realise that port:cctools is already listed as a build dependency?

[RJVB (René Bertin)]

FWIW, I now have a port:lldb-3.9 that uses a standalone build and contains the necessary patch and tweaks to achieve this.

​https://github.com/RJVB/macstrop/tree/master/lang/llvm-3.9

[jeremyhu (Jeremy Huddleston Sequoia)]

You do realise that port:cctools is already listed as a build dependency?

Of clang, yes. We want to make sure it's not a dependency of llvm