arb @6.0.1_3: LD_LIBRARY_PATH is undefined at make_arbperl_makefile.pl line 21

[cruiz_perez@…]

--->  Computing dependencies for arb
--->  Fetching archive for arb
--->  Attempting to fetch arb-6.0.1_3.darwin_15.x86_64.tbz2 from http://packages.macports.org/arb
--->  Attempting to fetch arb-6.0.1_3.darwin_15.x86_64.tbz2 from http://lil.fr.packages.macports.org/arb
--->  Attempting to fetch arb-6.0.1_3.darwin_15.x86_64.tbz2 from http://nue.de.packages.macports.org/macports/packages/arb
--->  Fetching distfiles for arb
--->  Verifying checksums for arb
--->  Extracting arb
--->  Applying patches to arb
--->  Configuring arb
--->  Building arb
Error: org.macports.build for port arb returned: command execution failed

[cruiz_perez@…]

Log File

[ryandesign (Ryan Carsten Schmidt)]

It looks like the relevant error in the log is:

:info:build LD_LIBRARY_PATH is undefined at make_arbperl_makefile.pl line 21.
:info:build make[5]: *** [Makefile] Error 255
:info:build make[5]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_science_arb/arb/work/arbsrc_12565/PERL2ARB'
:info:build make[4]: *** [.depends] Error 2
:info:build make[4]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_science_arb/arb/work/arbsrc_12565/PERL2ARB'
:info:build make[3]: *** [all] Error 2
:info:build make[3]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_science_arb/arb/work/arbsrc_12565/PERL2ARB'
:info:build Command exited with non-zero status 2
:info:build 0.06user 0.02system 0:00.25elapsed 36%CPU (0avgtext+0avgdata 29736960maxresident)k
:info:build 0inputs+2outputs (3major+5798minor)pagefaults 0swaps
:info:build make[2]: *** [realperl] Error 1
:info:build make[2]: *** Waiting for unfinished jobs....

[mattcottrell]

You need to disable the new (El Capitan) OS X System Integrity Protection (SIP), also known as “rootless” mode.

I was able to re-enable SIP and run arb after the install was complete.

Follow these steps to disable SIP:

Restart your Mac.
Before OS X starts up, hold down Command-R and keep it held down until you see an Apple icon and a progress bar. Release. This boots you into Recovery.
From the Utilities menu, select Terminal.
At the prompt type exactly the following and then press Return: csrutil disable
Terminal should display a message that SIP was disabled.
From the  menu, select Restart.
You can re-enable SIP by following the above steps, but using csrutil enable instead.

I learned about it from MacWorld here: ​http://www.macworld.com/article/2986118/security/how-to-modify-system-integrity-protection-in-el-capitan.html

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #50028.

Replying to matt.cottrell@…:

You need to disable the new (El Capitan) OS X System Integrity Protection (SIP), also known as “rootless” mode.

SIP exists for a reason. We do not want to encourage users to turn it off, even for a moment. Why does arb's install process require SIP to be turned off? Are the arb developers aware of this problem? Are they working on a fix? Is there an upstream bug report URL you could give us?

[mattcottrell]

The LD_LIBRARY_PATH environment variable seems to be the culprit. I'll submit a bug report to the upstream arb developers.

[mattcottrell]

The upstream arb developers are aware of the problem with Apple's SIP and building the arb software.

Here are their notes (​http://bugs.arb-home.de/ticket/667):

Since El Capitan, apple has added something called "System Integrity Protection" aka "rootless". 
Meant to protect from malware messing with the system, one of the things that mechanism does is deleting 
LD_LIBRARY_PATH and all DYLD_* from the environment for "protected" processes.

"make" itself isn't protected, but it uses /bin/sh to execute the recipes, and that's protected, 
so LD_LIBRARY_PATH is removed from the environment.

Still investigating this. For now, copying /bin/sh to $ARBHOME/bin and setting SHELL 
to $ARBHOME/bin/sh seems to fix the issue.

Is there some way that we could implement this workaround in the arb MacPort?

There are several shells available for installation by MacPorts (browser:trunk/dports/shells?order=name). Could we simply make one of those an arb port dependency and force it's use to run make? Would it be installed under $prefix and therefore immune to SIP?

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #51172.

Please check whether using MacPorts trunk and using trace mode helps. Some changes were made to trunk in r141420 to copy system binaries into a temporary directory to work around this kind of problem, but if I understand correctly that only applies when using trace mode (e.g. sudo port -t install arb).

This is merely a workaround. The developers of arb should fix their build system so that it works even on El Capitan and later with SIP enabled and without workarounds like copying the shell.

[elmar@…]

Just to explain exactly what's going on here:

1. ARB passes the location of dynamic libraries to be used both during build and after installation using the environment variables LD_LIBRARY_PATH or DYLD_LIBRARY_PATH, depending on host OS. This approach has thus far been a fool-proof way to make sure than even with multiple versions of ARB installed and users moving ARB installations around, the dynamic linker always uses the correct set of libraries. It still works fine, even with SIP, except during build.
2. As per POSIX.1 standard, which the Apple documentation claims their implementation adheres to (see man /usr/share/man/man3/exec.3), the environment for a new process image "shall" be a copy of the environment of the parent process, or the array passed to execle.
3. Apple's "SIP" breaks this behavior by deleting all variables matching /LD_LIBRARY_PATH|DYLD_.*/ from the process images of binaries if they are loaded from a path flagged as protected (e.g. /bin, /sbin, /usr). This is not mentioned in any of the pertinent man files nor in the System Integrity Protection Guide on the Apple Developer website, nor anywhere else I checked.
4. Gnu Make includes a hard coded path to /bin/sh to make sure all recipes are executed sanely and predictably, without interference by user configured environments. It ignores both the PATH and SHELL environment variables for this purpose, unless SHELL is explicitly set on the command line or in a loaded script.
5. During the build, a Perl script is called from a Make recipe. The Gnu Make installed via MacPorts passes the recipe line calling Perl (also installed via MacPorts) to /bin/sh for expansion and execution. At this point, OS X's SIP interferes and deletes the library path variables from the environment. The Perl script then fails.

This is merely a workaround. The developers of arb should fix their build system so that it works even on El Capitan and later with SIP enabled and without workarounds like copying the shell.

A behavior that is both unexpected and in conflict with the documentation is commonly called a bug. "Fixing" the issue within the ARB build system is therefore what I would call a workaround for a broken target platform/distribution.

That said, ARB is already littered with workarounds for various target platforms anyway. In this case, however, the failing script is part of the code building the Perl bindings for the ARB database library. This is done using Perl's MakeMaker utility, which does frightening things like rebuilding its own Makefile and has been fragile in the past. In order to avoid regression on other platforms, I'd rather not rashly touch this part. If anyone wants to give it a try, please do. Do test on SLES, RHEL, Debian, Centos and Ubuntu as well, though.

I would favor a fix on distribution level. Environment variables "mysteriously disappearing" between Make and the tool called from a Makefile recipe is just wrong. The suggestion to copy /bin/sh into a path not flagged for protection and setting SHELL in Makewas meant as a hot fix and as a means for verifying the source of the problem. A more general solution would be to make it so that MacPort's Gnu Make always uses a fully working shell, rather than /bin/sh. AFAIK this would require patching Gnu Make and/or adding a shell to its dependencies.

As a more direct, short term fix, the ARB package could simply require Bash as a dependency and set PATH and SHELL appropriately to avoid usage of /bin/bash or /bin/sh (with SHELL set either on the Make command line or by adding a line to config.makefile.

The sad part is that after it originally took me almost a day to figure out what was going on (the idea that some part of the OS was stealing variables didn't occur to me until everything else had been excluded), it took all of 5 seconds to work around it. I would love to see a discussion piece from Apple on why they thought this was a useful approach to improve security, more so than say the time honored method of linking high-value targets statically. Also, I'd love to hear in which scenarios an attack that is possible by getting a binary on the target host and modifying a processes' environment can be done using DYLD_LIBRARY_PATH and a malicious library, but not by modifying PATH and adding a malicious binary... (let's hope PATH isn't next up on Apple's list for lock down).

[elmar@…]

Cc Me!

[kurthindenburg (Kurt Hindenburg)]

Cc Me!

[kurthindenburg (Kurt Hindenburg)]

should be fixed in r148322