cyrus-sasl2 @2.1.27_2+kerberos: _sasl_locate_entry() in Cyrus SASL's lib/dlopen.c assumes a leading underscore is needed for calls to dlsym()

[steven-michaud (Steven Michaud)]

I've installed cyrus-sasl2 @2.1.27_2+kerberos on macOS 11.2.3 and have been using it with postfix @3.5.9_0+dovecot_sasl+pcre+sasl+tls and dovecot @2.3.11.3_2. Sending mail with STARTTLS and the SASL PLAIN mechanism simply didn't work. I've spent the last several days debugging the problem, and now I've found the cause.

Here's a fragment of code from _sasl_locate_entry():

#if defined(DLSYM_NEEDS_UNDERSCORE) || (defined(__OpenBSD__) && !defined(__ELF__))
    snprintf(adj_entryname, sizeof adj_entryname, "_%s", entryname);
#endif

    *entry_point = NULL;
    *entry_point = dlsym(library, adj_entryname);
    if (*entry_point == NULL) {
#if 0 /* This message appears to confuse people */
      _sasl_log(NULL, SASL_LOG_DEBUG,
                "unable to get entry point %s: %s", adj_entryname,
                dlerror());
#endif
      return SASL_FAIL;
    }

By means of various tests, and by looking at libsasl2.3.dylib in a disassembler, I've found that (in your distro) this code is behaving as if DLSYM_NEEDS_UNDERSCORE is defined. Of course it shouldn't be. I don't know why. The sasl_cv_dlsym_adds_uscore test in configure.ac looks like it should work. Could your cyrus_sasl2 distro have been compiled on something other than a Mac? In other words, might it have been cross-compiled on some other kind of system (e.g. Linux)?

Every time I try to send mail via STARTTLS and the SASL PLAIN mechanism, I get the error message "No worthy mechs found". Postfix is correctly set up -- the same setup works fine on other machines which use Cyrus SASL from other providers (like Apple). The reason is that my connection's list of mechanisms (sasl_client_conn_t.mech_list) only includes the "EXTERNAL" mechanism. sasl_client_add_plugin() is only ever called on this mechanism. The reason it isn't called on all the others (including the PLAIN mechanism) is that _sasl_locate_entry() (called from _sasl_plugin_load()) always fails, because it always adds an underscore in front of entryname.

[steven-michaud (Steven Michaud)]

HookCase hook library I tested with

[steven-michaud (Steven Michaud)]

I just added the source code for a HookCase hook library I've been testing with, as a patch on ​https://github.com/steven-michaud/HookCase/blob/master/HookLibraryTemplate/hook.mm.

​https://github.com/steven-michaud/HookCase

[steven-michaud (Steven Michaud)]

'smtp' launcher that loads my hook library

[kencu (Ken)]

it checks here (configure.ac)

dnl Do we need leading underscores on our symbols?

AC_CHECK_PROGS(NM, nm)

AC_MSG_CHECKING(for underscore before symbols)
AC_CACHE_VAL(sasl_cv_uscore,[
    echo "int main(){int i=1; return 0;}
    void foo(){int i=6;}" > conftest.c
    ${CC} -o a.out conftest.c > /dev/null
    if (${NM} a.out | grep _foo) > /dev/null; then
      sasl_cv_uscore=yes
    else
      sasl_cv_uscore=no
    fi])
AC_MSG_RESULT($sasl_cv_uscore)
rm -f conftest.c a.out

and on my Mojave system, finds this:

configure:15311: checking for underscore before symbols
configure:15327: result: yes

[kencu (Ken)]

but in the next section, comes something interesting that could be the issue. What system are you on? On some systems, this implicit-function-declaration is raised to an error.

configure:15333: checking whether dlsym adds the underscore for us
configure:15355: /usr/bin/clang -o conftest -Wall -W -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -arch x86_64 -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -arch x86_64 conftest.c  -ldl >&5
conftest.c:37:18: warning: unused variable 'i' [-Wunused-variable]
void foo() { int i=0;}
                 ^
conftest.c:40:23: warning: implicitly declaring library function 'exit' with type 'void (int) __attribute__((noreturn))' [-Wimplicit-function-declaration]
    if(ptr1 && !ptr2) exit(0); } exit(1); }
                      ^
conftest.c:40:23: note: include the header <stdlib.h> or explicitly provide a declaration for 'exit'
2 warnings generated.
configure:15355: $? = 0
configure:15355: ./conftest
configure:15355: $? = 0
configure:15370: result: yes

[steven-michaud (Steven Michaud)]

I used 'smtp.sh' (above) to load my hook library. I replaced 'smtp' in Postfix's master.cf with 'smtp.sh'.

[steven-michaud (Steven Michaud)]

I'm on macOS 11.2.3. I used "port install" for everything (including cyrus-sasl2 @2.1.27_2+kerberos). I didn't do any explicit compiling.

[kencu (Ken)]

Yeah, it probably fails that dlsym adds the underscore for us test because of Wimplicit-function-declaration being raised to an error.

can you check in your config.log?

[kencu (Ken)]

cd `port work cyrus-sasl2`
cd cy*
bbedit config.log

and search for:

dlsym adds the underscore for us

[steven-michaud (Steven Michaud)]

can you check in your config.log?

Where is is? I couldn't find it anywhere under /opt/local.

[kencu (Ken)]

I figured that might happen, so I just outlined how to find it one box above :>

[steven-michaud (Steven Michaud)]

Oh well. In any case I still can't find it. port work cyrus-sasl2 resolves to my home directory, and there's nothing there that I didn't explicitly create myself.

[kencu (Ken)]

really? here it is then

$ port work cyrus-sasl2
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_security_cyrus-sasl2/cyrus-sasl2/work

[kencu (Ken)]

Oh -- you have to have the work directory for it to be found, of course. Do this:

sudo port -v configure cyrus-sasl2
cd `port work cyrus-sasl2`
cd cy*
bbedit config.log

or just wait if you like, and someone will do it in time. It's about 99.9999% likely that is going to be the issue.

We've seen hundreds of these since Apple raised Wimplicit-function-declaration to an error in Xcode 12.

[steven-michaud (Steven Michaud)]

"port work cyrus-sasl2" produces no output. And there are no "work" directories at all under /opt/local. I used "sudo find /opt/local -name work".

[kencu (Ken)]

see one box above.

[steven-michaud (Steven Michaud)]

Yup. Now I see exactly what you reported in comment 4.

[kencu (Ken)]

You probably don't. You probably see this:

configure:15333: checking whether dlsym adds the underscore for us
configure:15355: /usr/bin/clang -o conftest -Wall -W -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -arch x86_64 -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -arch x86_64 conftest.c  -ldl >&5
conftest.c:37:18: warning: unused variable 'i' [-Wunused-variable]
void foo() { int i=0;}
                 ^
conftest.c:40:23: error: implicitly declaring library function 'exit' with type 'void (int) __attribute__((noreturn))' [-Werror,-Wimplicit-function-declaration]
    if(ptr1 && !ptr2) exit(0); } exit(1); }
                      ^
conftest.c:40:23: note: include the header <stdlib.h> or explicitly provide a declaration for 'exit'
1 warning and 1 error generated.
configure:15355: $? = 1
configure: program exited with status 1

which is what I see on BigSur, and that exit status 1 is what gives the (incorrect) failure that leads to the issue, by defining this:

#define DLSYM_NEEDS_UNDERSCORE /**/

because it (falsely) thinks it is needed.

The fix is to do exactly what it says to do:

include the header <stdlib.h> or explicitly provide a declaration for 'exit'

[kencu (Ken)]

There are also a bunch of these:

error: implicit declaration of function 'db_create' is invalid in C99 [-Werror,-Wimplicit-function-declaration]

that may, or may not, be another issue that needs fixing.

Good for you to notice what was wrong and to steer it in the right direction. This is an upstream issue to fix, of course, as every system running Xcode 12 on Earth is broken right now (nothing to do with MacPorts). But we'll see if we can get it fixed for you.

[kencu (Ken)]

It looks like homebrew has the same version, and no patches, so theirs is likely just as broken as you found ours to be.

<​https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/cyrus-sasl.rb>

So we'll have to make up our own patches, it seems.

[steven-michaud (Steven Michaud)]

Oops, sorry. What you say in comment 18 is correct.

But I don't see the error from comment 19 at all. Edit: This is wrong. I wasn't searching from the top of the file :-(

Is there any way to just do my own patch and rebuild your distro? Please point me to documentation that tells me how. (Needless to say, I can deal with the source code. I just don't know how to get from there to something that will fit with the rest of your distros.)

[steven-michaud (Steven Michaud)]

implicit-function-declaration is raised to an error

For the time being, I'm strongly tempted to just turn this off in XCode 12. There must be a way.

[kencu (Ken)]

for a quick fix, use a macports compiler if you're on Intel. This should do it:

sudo port clean cyrus-sasl2
sudo port install clang-11
sudo port -v install cyrus-sasl2 configure.compiler=macports-clang-11

the macports-clang compilers don't have that Werror turned on (but this needs to be turned on for arm64 to work).

What we will do is patch configure to add this:

#include <stdlib.h> 

at the right spot to make the test pass, as it should.

[kencu (Ken)]

Here's how we generate the patch, by the way:

sudo port -v extract cyrus-sasl2
cd `port work cyrus-sasl2`
cd cy*
sudo cp configure configure.orig
bbedit configure

add this in the right place:

#include <stdlib.h> 

then

diff -u configure.orig configure > ~/patch-cyrus-fix-implicit-dlysm.diff

then

cd ~
sudo port -v install cyrus-sasl2

(builds with your patched configure, hopefully fixed up as it should be).

and then if you get that far and are motivated, I can show you how to make a PR and fix this for everyone.

If you are not that motivated, attach your patch here to this ticket instead.

If you really like this port, and you (obviously) know what you're doing, volunteer to be the worldwide port maintainer, and make sure it always works correctly for everyone!

[ryandesign (Ryan Carsten Schmidt)]

In ba36fd455188921e2bf22d73e23a2e91f36f332f/macports-ports (master):

cyrus-sasl2: Fix implicit declaration of function

Closes: #62468

[steven-michaud (Steven Michaud)]

So can I upgrade now to cyrus-sasl2 @2.1.27_3? I tried "sudo port upgrade outdated" and "sudo port -d sync". Neither of them did the trick.

[ryandesign (Ryan Carsten Schmidt)]

Wait at least 1 hour after the commit for it to propagate to the public rsync server.

[kencu (Ken)]

So Steven, I see someone did the patch for you, but in that "teach a man to fish" way I think you were coming along nicely.

MacPorts needs more maintainers, especially when people actually use the software and know when it's not working right.

So if this has given you any enthusiasm dive in a bit, and if you get there, please feel free volunteer to help us maintain this and other software you know about. You can ask any questions you may have on the developer mailing list.

[steven-michaud (Steven Michaud)]

I still can't upgrade cyrus-sasl2 @2.1.27_2+kerberos. Even the command "sudo port upgrade cyrus-sasl2" didn't work.

And in any case my service provider seems to have blacklisted my test server :-( Once I found out how to use my hook library to work around the bug, something like 25 test messages (to myself) got delivered from my mail queue almost simultaneously. That seems to have been what triggered the blacklisting. Now I get a different error ("Relay access denied (in reply to RCPT TO command))"), before the SASL client even gets initialized.

I'll try to persuade them to undo the blacklisting. Then, once I'm able to upgrade to cyrus-sasl2 @2.1.27_3+kerberos, I'll be able to test it.

[steven-michaud (Steven Michaud)]

Thanks for the offer to make me a port maintainer, but I'm going to decline it. I'm retired :-) I don't want to shoulder that kind of burden. However, I'm happy to report any more problems I find, and to fix them if possible.

My current interest in MacPorts is to work up a replacement for Apple's late and lamented macOS/OS X Server. It actually still exists, but only as a pale ghost of what it used to be. As of macOS 10.14 (Mojave) it no longer supports the services that most people want -- things like mail, web service, dns and dhcp. I've run the "original" OS X Server for many years, but now it's getting very long in the tooth, and doesn't run on current versions of macOS. I figured that if I based my replacement on MacPorts, it'd be much easier to keep it up to date.

This replacement is just for my own use. But if it goes well I might create a Github project to describe how I implemented it.

[steven-michaud (Steven Michaud)]

Turns out what I called blacklisting was actually my fault. I inadvertently changed the Postfix "smtp_sasl_auth_enable" setting from "yes" to "no". When I changed it back my hook library workaround started working correctly again.

But I'm still not able to upgrade to cyrus-sasl2 @2.1.27_3+kerberos. So I can't test it.

[ryandesign (Ryan Carsten Schmidt)]

You have run sudo port selfupdate to receive the fix?

Note that there is already some work in MacPorts to provide replacements for what Apple provided in OS X Server, such as the calendar-contacts-server, clamav-server, dns-server, macos-vpn-server, and mail-server ports.

[steven-michaud (Steven Michaud)]

"sudo port selfupdate" made no difference, as you'd expect since it only effects "base sources". But I tried "sudo port upgrade outdated" again, as it recommended, and this time it did upgrade cyrus-sasl2 to 2.1.27_3. I guess it takes about 24 hours for new updates to migrate out fully.

I'll test it once I've had my dinner :-)

[ryandesign (Ryan Carsten Schmidt)]

Replying to steven-michaud:

"sudo port selfupdate" made no difference, as you'd expect since it only effects "base sources".

It does not. selfupdate updates base, if needed, and always updates your ports tree, unless you use the --no-sync flag.

But I tried "sudo port upgrade outdated" again, as it recommended, and this time it did upgrade cyrus-sasl2 to 2.1.27_3. I guess it takes about 24 hours for new updates to migrate out fully.

It takes about 1 hour for commits to move from our repository to the public rsync server, after which time you can run sudo port selfupdate or sudo port sync to receive those changes, and then do whatever install or upgrade you were trying to do.

[steven-michaud (Steven Michaud)]

Here's the output I got from "sudo port selfupdate":

--->  Updating MacPorts base sources using rsync
MacPorts base version 2.6.4 installed,
MacPorts base version 2.6.4 downloaded.
--->  Updating the ports tree
--->  MacPorts base is already the latest version

The ports tree has been updated. To upgrade your installed ports, you should run
  port upgrade outdated

So it updates both the "base sources" and the "ports tree". But that isn't enough by itself.

It seems you need to do both "sudo port selfupdate" and "sudo port upgrade outdated". And if I'd done both about an hour after you landed your fix, my copy of cyrus-sasl2 would have been upgraded to 2.1.27_3. I'll remember that for future reference.

[ryandesign (Ryan Carsten Schmidt)]

Updating the ports tree by itself using sudo port sync is enough. sudo port selfupdate does nothing more than that, unless a new version of MacPorts base had been released, but it hasn't.

[steven-michaud (Steven Michaud)]

I just redid my test of sending email to myself via STARTTLS and SASL PLAIN authentication to my service provider. It worked fine.

Thank you for fixing this so quickly!

I'll take a look at the ports you mentioned in comment 33, particularly "mail-server". I didn't know about them.