caff cannot find public keys due to stderr redirection

[ewen-naos-nz (Ewen McNeill)]

caff is a tool to sign the keys after a keysigning party. As far as I can tell on OS X (10.9) with perl 5.16 from MacPorts, caff's decision to do:

open my $NULL, '+<', '/dev/null';
[...]
        # get key listing (and ensure there is no collision)
        my $handles = make_gpg_fds( stdin => $NULL, stdout => undef, stderr => $NULL );

results in GPG not being run correctly (at all?), which means caff never finds the key to sign:

ewen@ashram:~$ caff --keys-from-gnupg -R e4d3e863
[INFO] Key E4D3E863 imported from your normal GnuPGHOME.
[WARN] No public keys found with list-key E4D3E863 (note that caff uses its own keyring in /Users/ewen/.caff/gnupghome).
[NOTICE] No keys to sign found
ewen@ashram:~$ 

which makes it not very useful at all :-(

However if /opt/local/bin/caff is edited to avoid redirecting stderr to +</dev/null, then the same command works (ie, same keyid):

ewen@ashram:~$ caff --keys-from-gnupg -R e4d3e863
[INFO] Key E4D3E863 imported from your normal GnuPGHOME.
[INFO] Sign the following keys according to your policy, then exit gpg with 'save' after signing each key
gpg --homedir=/Users/ewen/.caff/gnupghome --secret-keyring /Users/ewen/.gnupg/secring.gpg --no-auto-check-trustdb --trust-model=always --edit E4D3E863 sign
gpg (GnuPG) 1.4.18; Copyright (C) 2014 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[....]

Since this took a while to debug, it'd be nice if it could be fixed at least in the MacPorts version. (AFAICT the same code is still in, eg, signing-party 1.12 in Debian Unstable, but I'm not sure if it accidentally works there or not.)

I tracked it down to getting part way into the fork child code in GnuPG::Interface before failing, but not precisely to which point it is failing. However in the failing case there is no output at all to stdout, and it appears not to reach the point where gpg gets exec()ed at all. Without that redirect it seems to work fine. The redirection of stderr to +</dev/null on listing keys is the only one in the code.

Ewen

[ewen-naos-nz (Ewen McNeill)]

caff: avoid redirecting stderr when listing keys

[ewen-naos-nz (Ewen McNeill)]

Based on yet more frustrating debugging, it appears extracting the signatures to send also does not work without extra effort, and hence emailing signatures does not work. By enabling all the tracing code in caff for tracing, I got the hint that stderr was being set to ${GPG_TTY}, which I had not set:

[trace2] stderr is now GPG_TTY must be set in shell (cannot determine automatically).

(part of that message comes from caff's readwrite_gpg(), and part of it apparently from gpg itself; the function says "stderr is now", and the rest seems to be output from gpg.)

It appears that may be the trigger for both problems, ie doing:

GPG_TTY=$(tty)
export GPG_TTY
caff --keys-from-gnupg -R e4d3e863

means even the unpatched code will run in a useful manner. So a better patch may be to ensure that GPG_TTY is actually set to a sensible value in caff if it is not already set it the environment. (AFAICT GPG_TTY is mostly needed when using gpg-agent -- but that's obviously something one would normally want to do with caff, so ensuring it is always set and/or complaining if it is not set, seem a useful idea.)

Ewen

[neverpanic (Clemens Lang)]

So what do you propose? Should I apply your patch? Are the caff maintainers aware of this?

[ewen-naos-nz (Ewen McNeill)]

I have not told the caff maintainers (so far), but that may be the most useful next step.

Having spent a bunch more time fighting with caff (to get email sending working), it seems to be built on some fairly fragile foundations and have quite a few (mostly undocumented) assumptions about the environment. It also appears to try to hide the underlying errors from the user to present a "nicer" experience (eg, not expose gpg reports of missing keys), but that results in making debugging much more difficult because things "just fail" sometimes with obscure high level symptoms and all clues for why the failed are hidden.

At present I think I'd suggest:

1. not applying my original patch, as it's at best a partial solution with an unfortunate side effect (but hopefully at least this ticket will be search bait for anyone else trying to debug the same problem)

2. considering/writing a patch which checks for GPG_AGENT_INFO being set, and if it is warns if GPG_TTY is not set (of the "gpg-agent is in use but GPG_TTY is not set, some operations may unexpectedly fail" variety)

3. talking with caff maintainers about applying it upstream, rather than just in MacPorts, as with the benefit of many hours more debugging I no longer think it is OS X specific (but rather "using gpg-agent" specific). And possibly suggesting adding some comments for why stderr is being hidden at certain points.

Ewen

[ewen-naos-nz (Ewen McNeill)]

Since upstream appears to be Debian (​http://pgp-tools.alioth.debian.org/), I have reported what appears to be the underlying issue (GPG_TTY not set causing silent failures) to Debian (​https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775702) with a link to this bug, and my suggestion of "warn if GPG_AGENT_INFO set and GPG_TTY is not set".

I've also made a blog post (​http://ewen.mcneill.gen.nz/blog/entry/2015-01-19-keysigning-with-caff/) so I do not forget how I made it work, because I really do not want to have to debug this all over again (it literally took 4+ hours of digging in the perl source/library source to figure out why it was not working for the various things it claimed to do).

Up to you if you want to consider this MacPorts bug "resolved" for now (by being reported upstream) or keep it open for possible discussion with Debian (or even make a local patch -- but at this point I no longer think it is OS X specific, which I did at the beginning).

Ewen

[ewen-naos-nz (Ewen McNeill)]

After further investigating prompted by the Debian maintainer, the problem does appear to be MacPorts (or OS X) specific, and to not involve gpg-agent directly. In particular with the MacPorts version of gnupg, this fails:

gpg --trust-model=always --no-auto-check-trustdb --fingerprint --with-colons --list-public-keys --no-tty --batch -- E4D3E863 </dev/null 2>/dev/null

with exit code 1 and no output. But on Debian unstable, using the same gnupg version, the command works and produces the expected output.

With the MacPorts version it is necessary to do:

GPG_TTY=$(tty) gpg --trust-model=always --no-auto-check-trustdb --fingerprint --with-colons --list-public-keys --no-tty --batch -- E4D3E863 </dev/null 2>/dev/null

(or otherwise have set GPG_TTY) in order to get the expected output if stderr is redirected to /dev/null (the command also works without GPG_TTY set providing stderr is not redirected).

It does not seem to matter whether GPG_AGENT_INFO is set or not. (The Debian bug has some full command output illustrating this, at ​https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775702#15)

At this stage I do not know what is different about the MacPorts/OS X build (and/or OS X) to trigger this.

For reference, these are the versions I have installed from MacPorts at present:

ewen@ashram:~$ port installed | grep gnupg
  gnupg @1.4.18_0
  gnupg @1.4.18_1 (active)
  p5.16-gnupg-interface @0.500.0_3 (active)
ewen@ashram:~$ port installed | grep gpg
  gpg-agent @2.0.26_4+pinentry_mac (active)
  libgpg-error @1.16_0
  libgpg-error @1.17_0 (active)
ewen@ashram:~$ port installed | grep signing-party
  signing-party @1.1.9_0 (active)
ewen@ashram:~$ 

Possibly this will want either (a) a MacPorts-specific work around patch like the one I originally posted (but that didn't seem to be sufficient by itself), (b) a MacPorts-specific warning patch ("you don't have GPG_TTY set, caff may fail") or (c) some investigation of how gnupg is built differently in MacPorts from Debian. But it might be worth waiting and seeing if there is any follow up on the Debian ticket first.

Ewen

[ewen-naos-nz (Ewen McNeill)]

The Debian maintainer ​applied my patch, so the immediate fix (not redirecting stderr when listing keys) should come out in the next release (whatever comes after 1.1.12 that ​is in Debian Unstable already I assume; 1.1.13 maybe?). It may be easiest to wait for that next release, and pull that later version in.

On OS X it doesn't seem to be a complete fix for full caff functionality (since there are other things caff does that seems to trigger the gpg "stderr not usable, GPG_TTY not set, just exit 1" functionality). But between that update, and these two bug reports as search bait hopefully anyone else finding problems on OS X will find a place to start.

Ewen

[neverpanic (Clemens Lang)]

OK, to make matters more complicated, my gpg actually does not have the problem:

gpg --trust-model=always --no-auto-check-trustdb --fingerprint --with-colons --list-public-keys --no-tty --batch -- 4C6F6B99 </dev/null 2>/dev/null
tru:t:1:1412878257:1439900531:3:1:5
pub:u:4096:1:E26CAD774C6F6B99:2011-02-07:2016-02-06::u:Clemens Lang <****>::scESC:
fpr:::::::::EB607CF66858E04BA440BAE0E26CAD774C6F6B99:
uid:u::::2014-10-09::6A5667688E367B0B3F5A470002E55F718E009E0A::Clemens Lang <****>:
uid:u::::2011-02-07::2EAAEA1A395225E40BF2C5F4ADFEB7F46B3E38EA::Clemens Lang (University of Erlangen-Nuremberg) <****>:
uid:u::::2011-02-07::AAF0301C6B0CED1FCF65CC8D41C7DFB04389D172::Clemens Lang <****>:
uid:u::::2012-07-10::143A87B4F935D4AB24DF0D918AE217FE8BC90AB8::Clemens Lang (University of Erlangen-Nuremberg) <****>:
uid:u::::2014-10-09::5B7DC41549741432A20570DA5E1F22CAF61A2137::Clemens Lang (BMW Car IT GmbH) <****>:
sub:u:4096:1:D2861130CC18EB31:2011-02-07:2016-02-06:::::e:

Are you actually using MacPorts' gpg? Which gpg is first on your $PATH? Does the command prompt you (e.g., for a password) when you run it with GPG_TTY set?

[ewen-naos-nz (Ewen McNeill)]

That's a very insightful question, it turns out:

ewen@ashram:~$ echo $PATH
/Users/ewen/.bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
ewen@ashram:~$ which gpg
/Users/ewen/.bin/gpg
ewen@ashram:~$ 

where it turns out that ~/.bin/gpg is an old shell script wrapper I clearly put in Some Time Ago (tm) and then forgot about over the years, which tries to auto-set GPG_TTY if it is not set... by looking at stdin and stderr. So if neither stdin nor stderr points at a tty it is unable to set GPG_TTY. (IIRC I ignored stdout because gpg may well be in a pipeline.)

In particular that solves the mystery for where the error message is coming from:

  if [ "$GPG_TTY" = "not a tty" ]; then
    echo "GPG_TTY must be set in shell (cannot determine automatically)." >&2
    exit 1
  fi

which is triggered when neither stdin or stderr is a tty (and GPG_TTY is not already set). (And in hindsight explicitly redirecting the error message to /dev/null when it's only triggerable if stderr is redirected to /dev/null is probably a.... less useful reflex!)

Looking at the date on that particular wrapper script (and the comments in it for why it exists) this appears to be a pretty old (ie, many years) work around for other issues that I was seeing.

So... the problem turns out to be at least partly self inflicted. But the trigger condition of redirecting stdin and stderr to /dev/null does mean it's rather hard to debug. So picking up the eventual upstream (Debian) caff patch seems useful for other issues.

However at this point, closing this ticket and eventually picking the up a later upstream caff with the fix, is the best resolution.

Thanks for pointing me in the right direction.

Ewen

[Ionic (Mihai Moldovan)]

Neither neverpanic nor I were able to reproduce that failure.

Good you figured it out eventually.

Invalidating this.

[neverpanic (Clemens Lang)]

We could still patch caff to make sure it actually uses MacPorts' copy of GnuPG. That would have prevented this issue completely.

[neverpanic (Clemens Lang)]

r132348, unless you set a specific gpg binary in ~/.caffrc it'll default to MacPorts' gpg now.