librsync doesn't correctly export inlined functions when compiled with clang

[mzch@…]

rdiff-backup can successfully be built and installed, but doesn't work. Error messages as below:

Traceback (most recent call last):
  File "/opt/local/bin/rdiff-backup", line 20, in <module>
    import rdiff_backup.Main
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/rdiff_backup/Main.py", line 25, in <module>
    import Globals, Time, SetConnections, selection, robust, rpath, \
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/rdiff_backup/SetConnections.py", line 30, in <module>
    import Globals, connection, rpath
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/rdiff_backup/connection.py", line 539, in <module>
    import Globals, Time, Rdiff, Hardlink, FilenameMapping, C, Security, \
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/rdiff_backup/Rdiff.py", line 22, in <module>
    import os, librsync
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/rdiff_backup/librsync.py", line 28, in <module>
    import _librsync
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/rdiff_backup/_librsync.so, 2): Symbol not found: _rs_appendflush
  Referenced from: /opt/local/lib/librsync.1.dylib
  Expected in: flat namespace
 in /opt/local/lib/librsync.1.dylib

[mzch@…]

Hi,

Librsync 0.9.8 causes this issue. In delta.c all _rs_append* functions are defined as inline, so those symbols are lost in librsync.dynlib.

The following log is a symbol table of delta.o (include rs_append* functions). Somehow, rs_appendflush is listed.

0000000000000918 s EH_frame0
000000000000071a s L_.str
000000000000072f s L_.str1
00000000000008eb s L_.str10
0000000000000756 s L_.str2
0000000000000794 s L_.str3
00000000000007b8 s L_.str4
000000000000080b s L_.str5
0000000000000844 s L_.str6
0000000000000883 s L_.str7
00000000000008a6 s L_.str8
00000000000008c0 s L_.str9
0000000000000897 s L___FUNCTION__.rs_appendflush
0000000000000720 s L___FUNCTION__.rs_delta_begin
0000000000000872 s L___FUNCTION__.rs_delta_s_flush
0000000000000782 s L___FUNCTION__.rs_delta_s_header
0000000000000834 s L___FUNCTION__.rs_delta_s_scan
00000000000007fa s L___FUNCTION__.rs_delta_s_slack
00000000000008de s L___FUNCTION__.rs_findmatch
                 U _RollsumUpdate
                 U _abort
                 U _rs_appendflush
0000000000000040 T _rs_delta_begin
0000000000000958 S _rs_delta_begin.eh
00000000000004b0 t _rs_delta_s_end
0000000000000a00 s _rs_delta_s_end.eh
00000000000004c0 t _rs_delta_s_flush
0000000000000a28 s _rs_delta_s_flush.eh
00000000000000c0 t _rs_delta_s_header
0000000000000980 s _rs_delta_s_header.eh
0000000000000140 t _rs_delta_s_scan
00000000000009a8 s _rs_delta_s_scan.eh
0000000000000430 t _rs_delta_s_slack
00000000000009d8 s _rs_delta_s_slack.eh
                 U _rs_emit_delta_header
                 U _rs_emit_end_cmd
                 U _rs_emit_literal_cmd
0000000000000000 T _rs_getinput
0000000000000930 S _rs_getinput.eh
                 U _rs_job_check
                 U _rs_job_input_is_ending
                 U _rs_job_new
                 U _rs_log0
                 U _rs_outbuflen
0000000000000a58 S _rs_roll_paranoia
                 U _rs_scoop_input
                 U _rs_scoop_total_avail
                 U _rs_search_for_block
                 U _rs_tube_catchup
                 U _rs_tube_copy

I don't understand why inline function is listed in the symbol table, but I applied the following dirty-hack and made sure to fix this issue.

--- delta.c.orig	2011-10-28 07:04:32.000000000 +0900
+++ delta.c	2011-10-28 07:05:08.000000000 +0900
@@ -254,7 +254,7 @@
  * forwards beyond the block boundaries. Extending backwards would require
  * decrementing scoop_pos as appropriate.
  */
-inline int rs_findmatch(rs_job_t *job, rs_long_t *match_pos, size_t *match_len) {
+int rs_findmatch(rs_job_t *job, rs_long_t *match_pos, size_t *match_len) {
     /* calculate the weak_sum if we don't have one */
     if (job->weak_sum.count == 0) {
         /* set match_len to min(block_len, scan_avail) */
@@ -281,7 +281,7 @@
 /**
  * Append a match at match_pos of length match_len to the delta, extending
  * a previous match if possible, or flushing any previous miss/match. */
-inline rs_result rs_appendmatch(rs_job_t *job, rs_long_t match_pos, size_t match_len)
+rs_result rs_appendmatch(rs_job_t *job, rs_long_t match_pos, size_t match_len)
 {
     rs_result result=RS_DONE;
     
@@ -312,7 +312,7 @@
  * 
  * This also breaks misses up into block_len segments to avoid accumulating
  * too much in memory. */
-inline rs_result rs_appendmiss(rs_job_t *job, size_t miss_len)
+rs_result rs_appendmiss(rs_job_t *job, size_t miss_len)
 {
     rs_result result=RS_DONE;
     
@@ -329,7 +329,7 @@
 /**
  * Flush any accumulating hit or miss, appending it to the delta.
  */
-inline rs_result rs_appendflush(rs_job_t *job)
+rs_result rs_appendflush(rs_job_t *job)
 {
     /* if last is a match, emit it and reset last by resetting basis_len */
     if (job->basis_len) {
@@ -360,7 +360,7 @@
  * scoop_pos appropriately. In the future this could be used for something
  * like context compressing of miss data. Note that it also calls
  * rs_tube_catchup to output any pending output. */
-inline rs_result rs_processmatch(rs_job_t *job)
+rs_result rs_processmatch(rs_job_t *job)
 {
     job->scoop_avail-=job->scoop_pos;
     job->scoop_next+=job->scoop_pos;
@@ -382,7 +382,7 @@
  * 
  * In the future this could do compression of miss data before outputing
  * it. */
-inline rs_result rs_processmiss(rs_job_t *job)
+rs_result rs_processmiss(rs_job_t *job)
 {
     rs_tube_copy(job, job->scoop_pos);
     job->scoop_pos=0;

[thomas@…]

Replying to mzch@…:

Hi,

Librsync 0.9.8 causes this issue. In delta.c all _rs_append* functions are defined as inline, so those symbols are lost in librsync.dynlib.

The following log is a symbol table of delta.o (include rs_append* functions). Somehow, rs_appendflush is listed.

duplicity also seems to be affected by this bug:

Traceback (most recent call last):
  File "/opt/local/bin/duplicity", line 41, in <module>
    from duplicity import collections
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/duplicity/collections.py", line 29, in <module>
    from duplicity import path
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/duplicity/path.py", line 36, in <module>
    from duplicity import librsync
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/duplicity/librsync.py", line 29, in <module>
    import _librsync
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/duplicity/_librsync.so, 2): Symbol not found: _rs_appendflush
  Referenced from: /opt/local/lib/librsync.1.dylib
  Expected in: flat namespace
 in /opt/local/lib/librsync.1.dylib

[miha.valencic@…]

Cc Me!

[miha.valencic@…]

Is there a workaround? I'm having the same problem on OS X Lion, MacPorts 2.0.3. On Snow Leopard, MacPorts version 1.9.2 it works.

[miha.valencic@…]

If someone else want's a workaround, I can confirm that modifying delta.c (removing inline function modifier) on OS X Lion solves the problem.

I issued sudo port -d -k install librsync and then modified delta.c by hand and issued sudo make install && sudo make clean and rdiff is working again.

[elliotj@…]

Thank you! My backups work again!

Replying to miha.valencic@…:

If someone else want's a workaround, I can confirm that modifying delta.c (removing inline function modifier) on OS X Lion solves the problem. ...

[neverpanic (Clemens Lang)]

The problem is related to whether you use gcc or clang:

GCC by default compiles GNU89, which is C89 with some GNU extensions. C89 doesn't have inline, but GNU89 adds it. C99, which is what clang compiles by default, however, _has_ a specification for the inline keyword. GNU89's interpretation of the inline keyword differs from the one specified in C99, which is why this problem occurs.

For more information, read ​http://clang.llvm.org/compatibility.html#inline.

A working solution is to drop the inline keyword whenever librsync is compiled with clang and let the compiler figure out, whether it should inline the function or not. This is what I have done in r88255.

[neverpanic (Clemens Lang)]

Reported upstream at ​https://sourceforge.net/tracker/?func=detail&aid=3464437&group_id=56125&atid=479439.

[ranauei@…]

Cc Me!