libgcrypt abort/crash when using krdc to connect to an OS X host

[RJVB (René Bertin)]

Using port:krdc to connect to an OS X host over VNC leads to an abort in libgcrypt:

Process 25698 launched: '/Applications/MacPorts/KDE4/krdc.app/Contents/MacOS/krdc' (x86_64)
tp-qt 0.9.5 DEBUG: Register StreamTubeClient with name  "krdc_rfb_handler"  
tp-qt 0.9.5 DEBUG: Client registered - busName: "org.freedesktop.Telepathy.Client.krdc_rfb_handler" objectPath: "/org/freedesktop/Telepathy/Client/krdc_rfb_handler" interfaces: ("org.freedesktop.Telepathy.Client.Handler")  
QDBusObjectPath: invalid path ""
krdc(25698)/kdeui (kdelibs): Attempt to use QAction "remote_desktop_dockwidget" with KXMLGUIFactory! 
QSystemTrayIcon::setVisible: No Icon set
2015-05-19 15:13:43.893 krdc[25698:d0b] modalSession has been exited prematurely - check for a reentrant call to endModalSession:
### WalletPrivate ->handleIdleTiming: created QTimer(0x10873a780, name = "idleTimer for process/wallet ""/kdewallet") for wallet "kdewallet" handle 160642784 timeout= 6 
Opened already unlocked wallet ' "kdewallet" ':  KWallet::Wallet(0x108b4d200) 
2015-05-19 15:13:50.040 krdc[25698:d0b] modalSession has been exited prematurely - check for a reentrant call to endModalSession:
Fatal: failed to create the RNG lock: Invalid argument
FATAL: failed to acquire the FSM lock in libgrypt: Invalid argument
Process 25698 stopped
* thread #7: tid = 0x584c64, 0x00007fff8dcee866 libsystem_kernel.dylib`__pthread_kill + 10, name = 'VncClientThread', stop reason = signal SIGABRT
    frame #0: 0x00007fff8dcee866 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill + 10:
-> 0x7fff8dcee866:  jae    0x7fff8dcee870            ; __pthread_kill + 20
   0x7fff8dcee868:  movq   %rax, %rdi
   0x7fff8dcee86b:  jmp    0x7fff8dceb175            ; cerror_nocancel
   0x7fff8dcee870:  retq   
(lldb) bt
* thread #7: tid = 0x584c64, 0x00007fff8dcee866 libsystem_kernel.dylib`__pthread_kill + 10, name = 'VncClientThread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff8dcee866 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff8e4dd35c libsystem_pthread.dylib`pthread_kill + 92
    frame #2: 0x00007fff91d85b1a libsystem_c.dylib`abort + 125
    frame #3: 0x000000010a61e1d8 libgcrypt.20.dylib`fips_new_state + 520
    frame #4: 0x000000010a61e501 libgcrypt.20.dylib`_gcry_fips_signal_error + 81
    frame #5: 0x000000010a6153a6 libgcrypt.20.dylib`_gcry_log_fatal + 246
    frame #6: 0x000000010a66e911 libgcrypt.20.dylib`_gcry_rngfips_initialize + 689
    frame #7: 0x000000010a66e9ec libgcrypt.20.dylib`_gcry_rngfips_randomize + 28
    frame #8: 0x000000010a66bf26 libgcrypt.20.dylib`_gcry_random_bytes + 38
    frame #9: 0x000000010a67a10f libgcrypt.20.dylib`_gcry_mpi_randomize + 95
    frame #10: 0x000000010890d439 libvncclient.0.dylib`InitialiseRFBConnection + 3401
    frame #11: 0x000000010891ffe9 libvncclient.0.dylib`rfbInitClient + 969
    frame #12: 0x00000001088e752f krdc_vncplugin.so`VncClientThread::clientCreate(this=0x0000000108b97170, reinitialising=false) + 319 at vncclientthread.cpp:555
    frame #13: 0x00000001088e6dba krdc_vncplugin.so`VncClientThread::run(this=0x0000000108b97170) + 282 at vncclientthread.cpp:463
    frame #14: 0x000000010106e272 QtCore`QThreadPrivate::start(arg=<unavailable>) + 386 at qthread_unix.cpp:352
    frame #15: 0x00007fff8e4dc899 libsystem_pthread.dylib`_pthread_body + 138
    frame #16: 0x00007fff8e4dc72a libsystem_pthread.dylib`_pthread_start + 137
    frame #17: 0x00007fff8e4e0fc9 libsystem_pthread.dylib`thread_start + 13

I've tried to install libgcrypt from source using configure.optflags=-g : while the -g flag did show up in the build log, the resulting binaries are apparently stripped post-build. The RNG lock error message is printed at line 201 of file random-fips.c

I tested this with "localhost" but also with a remote OS X VM; both systems run 10.9.5 . Connecting to Linux hosts running the x11vnc server works fine.

[RJVB (René Bertin)]

Managed to get a better backtrace:

# lldb /Applications/MacPorts/KDE4/krdc.app/Contents/MacOS/krdc
(lldb) target create "/Applications/MacPorts/KDE4/krdc.app/Contents/MacOS/krdc"
Current executable set to '/Applications/MacPorts/KDE4/krdc.app/Contents/MacOS/krdc' (x86_64).
(lldb) r
Process 68384 launched: '/Applications/MacPorts/KDE4/krdc.app/Contents/MacOS/krdc' (x86_64)
tp-qt 0.9.5 DEBUG: Register StreamTubeClient with name  "krdc_rfb_handler"  
tp-qt 0.9.5 DEBUG: Client registered - busName: "org.freedesktop.Telepathy.Client.krdc_rfb_handler" objectPath: "/org/freedesktop/Telepathy/Client/krdc_rfb_handler" interfaces: ("org.freedesktop.Telepathy.Client.Handler")  
QDBusObjectPath: invalid path ""
krdc(68384)/kdeui (kdelibs): Attempt to use QAction "remote_desktop_dockwidget" with KXMLGUIFactory! 
QSystemTrayIcon::setVisible: No Icon set
2015-05-19 17:13:40.958 krdc[68384:d0b] modalSession has been exited prematurely - check for a reentrant call to endModalSession:
### WalletPrivate ->handleIdleTiming: created QTimer(0x103bea860, name = "idleTimer for process/wallet ""/kdewallet") for wallet "kdewallet" handle 92408080 timeout= 6 
Opened already unlocked wallet ' "kdewallet" ':  KWallet::Wallet(0x109bc9690) 
2015-05-19 17:13:46.368 krdc[68384:d0b] modalSession has been exited prematurely - check for a reentrant call to endModalSession:
Fatal: failed to create the RNG lock: Invalid argument
FATAL: failed to acquire the FSM lock in libgrypt: Invalid argument
Process 68384 stopped
* thread #8: tid = 0x59d336, 0x00007fff8dcee866 libsystem_kernel.dylib`__pthread_kill + 10, name = 'VncClientThread', stop reason = signal SIGABRT
    frame #0: 0x00007fff8dcee866 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill + 10:
-> 0x7fff8dcee866:  jae    0x7fff8dcee870            ; __pthread_kill + 20
   0x7fff8dcee868:  movq   %rax, %rdi
   0x7fff8dcee86b:  jmp    0x7fff8dceb175            ; cerror_nocancel
   0x7fff8dcee870:  retq   
(lldb) bt
* thread #8: tid = 0x59d336, 0x00007fff8dcee866 libsystem_kernel.dylib`__pthread_kill + 10, name = 'VncClientThread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff8dcee866 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff8e4dd35c libsystem_pthread.dylib`pthread_kill + 92
    frame #2: 0x00007fff91d85b1a libsystem_c.dylib`abort + 125
    frame #3: 0x000000010a8561d8 libgcrypt.20.dylib`fips_new_state [inlined] lock_fsm + 520 at fips.c:235
    frame #4: 0x000000010a8561c8 libgcrypt.20.dylib`fips_new_state(new_state=STATE_FATALERROR) + 504 at fips.c:759
    frame #5: 0x000000010a856501 libgcrypt.20.dylib`_gcry_fips_signal_error(srcfile=0x000000010a8b59e8, srcline=<unavailable>, srcfunc=0x000000010a8b5a47, is_fatal=<unavailable>, description=0x000000010a8b5a52) + 81 at fips.c:732
    frame #6: 0x000000010a84d3a6 libgcrypt.20.dylib`_gcry_log_fatal [inlined] _gcry_logv(level=<unavailable>) + 108 at misc.c:140
    frame #7: 0x000000010a84d33a libgcrypt.20.dylib`_gcry_log_fatal(fmt=<unavailable>) + 138 at misc.c:230
    frame #8: 0x000000010a8a6911 libgcrypt.20.dylib`_gcry_rngfips_initialize [inlined] check_guards(rng_ctx=<unavailable>) + 689 at random-fips.c:255
    frame #9: 0x000000010a8a6906 libgcrypt.20.dylib`_gcry_rngfips_initialize(full=<unavailable>) + 678 at random-fips.c:777
    frame #10: 0x000000010a8a69ec libgcrypt.20.dylib`_gcry_rngfips_randomize(buffer=0x0000000109b04b00, length=16, level=GCRY_STRONG_RANDOM) + 28 at random-fips.c:835
    frame #11: 0x000000010a8a3f26 libgcrypt.20.dylib`_gcry_random_bytes(nbytes=16, level=<unavailable>) + 38 at random.c:324
    frame #12: 0x000000010a8b210f libgcrypt.20.dylib`_gcry_mpi_randomize(w=<unavailable>, nbits=<unavailable>, level=<unavailable>) + 95 at mpiutil.c:612
    frame #13: 0x0000000105db5439 libvncclient.0.dylib`InitialiseRFBConnection + 3401
    frame #14: 0x0000000105dc7fe9 libvncclient.0.dylib`rfbInitClient + 969
    frame #15: 0x0000000105d8f52f krdc_vncplugin.so`VncClientThread::clientCreate(this=0x0000000103bdf2f0, reinitialising=false) + 319 at vncclientthread.cpp:555
    frame #16: 0x0000000105d8edba krdc_vncplugin.so`VncClientThread::run(this=0x0000000103bdf2f0) + 282 at vncclientthread.cpp:463
    frame #17: 0x000000010106e272 QtCore`QThreadPrivate::start(arg=<unavailable>) + 386 at qthread_unix.cpp:352
    frame #18: 0x00007fff8e4dc899 libsystem_pthread.dylib`_pthread_body + 138
    frame #19: 0x00007fff8e4dc72a libsystem_pthread.dylib`_pthread_start + 137
    frame #20: 0x00007fff8e4e0fc9 libsystem_pthread.dylib`thread_start + 13

It looks like ath_init() is never called; this may in fact be a bug in LibVNCServer (to be confirmed)

[Schamschula (Marius Schamschula)]

See ​https://github.com/LibVNC/libvncserver/issues/45 and ​https://github.com/LibVNC/libvncserver/pull/51

[RJVB (René Bertin)]

Ah, thanks. I guess my Ubuntu rig had a similar patch, and this one worked perfectly. I'll be creating a ticket for a LibVNCServer update.

Now to get krdc to send the correct keysymbols ... for now it sends gibberish making it "point and click only" :-/

[mkae (Marko Käning)]

Cc Me!