aiccu - daemondo automatically restarts it, the reason why it exists is because something is broken.....

[jeroen@…]

"daemondo launches the daemon directly, rather than indirectly via a script, and therefore it automatically knows how to monitor a daemon process and restart it if it dies."

And thus, when aiccu exits because there is an unresolvable error case daemondo automatically restarts it, aiccu exists, daemondo restarts it, aiccu exits..... ad infinitum.

Otherwise said: daemondo is not the proper tool for running AICCU in this way as it clearly avoids to check the exit code of AICCU, which exits because there is an error condition, which means it can't do anything else and that user intervention is needed.

A couple of solutions that should be applied in this case and most likely in the cases of many other daemons which will abort on startup:

• don't use it specifically for AICCU
• rate limit the number of 'startups', if it fails X times, then don't start it any more.
• If it fails after the first startup, tell the user, and don't start it again.
• etc etc etc.

From the 'README' file in the AICCU distribution: 8<------------------------------- WARNING: never run AICCU from DaemonTools or a similar automated 'restart' tool/script. When AICCU does not start, it has a reason not to start which it gives on either the stdout or in the (sys)log file. The TIC server *will* automatically disable accounts which are detected to run in this mode. Use 'verbose true' to see more information which is especially handy when starting fails.

---

Oh and as clearly logging is not configured properly, nor the user ever cares to look at it they will never figure this problem out.

Most likely the same goes for a lot of other daemons.

In this case it is especially annoying as it will contact the TIC servers of SixXS, which will then have to rate limit the client and disable them, as per the above. Still, the client, which gets repeatedly restarted by daemondo will keep on trying to hammer those TIC servers. Thank you macport folks for another attempt at DDoSSing them. (attempt, as it fails due to the ratelimmitting and fortunately also the limited number of clients of which we had to tell and explain the users that they had a broken client....)

But lets solve this properly, as macports is where the problem happens. Please resolve this.

Additionally "startupitem.netchange" which is described as "Cause the daemon to be restarted when a change in network state is detected." is completely ridiculous in the case of AICCU as AICCU already KNOWS how to handle network changes, that is why there are "specialized" protocols called heartbeat and AYIYA which AICCU supports. Thus that option can also go to /dev/null for AICCU.

[nolith@…]

I completely agree with the reporter

[grimreaper (Eitan Adler)]

Cc Me!

[george@…]

Cc Me!

[jinn.koriech@…]

Please merge the following solution to resolve this bug.

Currently the plist definition configures daemondo to launch in mode 2, causing aiccu to be restarted indefinitely.

$ /opt/local/bin/daemondo --help
...
daemondo runs in one of two modes: (1) If no stop-cmd is given, daemondo
executes start-cmd asyncronously, and tracks the process id; that process id
is used to signal the daemon for later stop and/or restart. (2) If stop-cmd
is given, then both start-cmd and stop-cmd are issued syncronously, and are
assumed to do all the work of controlling the daemon. In such cases there is
no process id to track. In either mode, restart-cmd, if present, is used to
restart the daemon. If in mode 1, restart-cmd must not disrupt the process id.
If restart-cmd is not provided, the daemon is restarted via a stop/start
sequence.
...
In mode 1 only, daemondo will exit when it detects that the daemon being
monitored has exited.

Therefore by removing the --stop-cmd and --restart-cmd we can ensure daemondo will only start the aiccu daemon once, however if a netchange event is detected daemondo should correctly restart aiccu.

$ pwd
/opt/local/etc/LaunchDaemons/org.macports.aiccu

$ diff org.macports.aiccu.plist.orig org.macports.aiccu.plist 
15,22d14
< 	<string>--stop-cmd</string>
< 	<string>/opt/local/etc/LaunchDaemons/org.macports.aiccu/aiccu.wrapper</string>
< 	<string>stop</string>
< 	<string>;</string>
< 	<string>--restart-cmd</string>
< 	<string>/opt/local/etc/LaunchDaemons/org.macports.aiccu/aiccu.wrapper</string>
< 	<string>restart</string>
< 	<string>;</string>

Above diff is against aiccu @20070115.

$ sudo port info aiccu   
aiccu @20070115, Revision 2 (net, ipv6)
Variants:             gnutls

Description:          AICCU makes it very easy for SixXS users to get IPv6 connectivity everywhere they
                      want. It uses the TIC (Tunnel Information & Control) protocol to request the
                      information needed to setup a tunnel through which the connectivity is created. It
                      supports 6in4 static (RFC 2893), 6in4 heartbeat (RFC 2893 +
                      draft-massar-v6ops-heartbeat) and AYIYA (draft-massar-v6ops-ayiya) tunnel
                      protocols.
Homepage:             http://www.sixxs.net/tools/aiccu/

Library Dependencies: tuntaposx
Platforms:            darwin
License:              unknown
Maintainers:          cr@23bit.net

[jeroen@…]

Replying to jinn.koriech@…:

As also mentioned per email, but apparently one cannot just reply to the bug email that is sent out...

Therefore by removing the --stop-cmd and --restart-cmd we can ensure daemondo will only start the aiccu daemon once,

That is correct.

however if a netchange event is detected daemondo should correctly restart aiccu.

That is *NOT* correct. AICCU must *NEVER be restarted*

Network changes is something that AICCU, or actually the protocols that it implements will handle and it does not need any restarts for network changes.

[suominen (Kimmo Suominen)]

Cc Me!

[fgp (Florian G. Pflug)]

Replying to jeroen@…:

That is *NOT* correct. AICCU must *NEVER be restarted*

This doesn't seem to be true. I've just tested this. Here's what I did

1. port install aiccu

2. Created /opt/local/etc/aiccu.conf and configured it for my SIXXS AYIAY tunnel

2. I left the aiccu launchd service disabled, but did of course enable the tuntap service

3. aiccu start

4. Tunnel came up, works as expected

5. I disabled WiFi and instead connected my phone over USB. OS X started using my phone's internet connection as expected (Using this connect to write this replay this very second, so it certainly works!)

6. Tried pinging the other SIXXS tunnel endpoint, but no luck. aiccu logs the following once per minute

     [AYIYA-beat] : Error (-1) while sending 44 bytes sent to network: Can't assign requested address (49)
     [AYIYA-tundev->tun] : Error (-1) while sending 100 bytes to network: Can't assign requested address (49)
   

7. aiccu stop, aiccu start.

8. Pinging the tunnel endpoint works again.

So no, aiccu does not react to network changes correctly, at least not on OS X 10.9.5

Network changes is something that AICCU, or actually the protocols that it implements will handle and it does not need any restarts for network changes.

Nope, unfortunately not! From the errors logged to system.log, my guess is that aiccu - for whatever reason - sets a source address for its socket(s), and fails to change it if that address is removed from the system. I haven't looked at the source to verify this, but it doesn't really matter anyway - fact is, never restarting aiccu means the tunnel will break if the primary interface changes!

So we either need to patch AICCU to correct rebind its sockets (if that is indeed the problem here), or do restart it whenever the network configuration changes.

[fgp (Florian G. Pflug)]

Replying to jinn.koriech@…:

daemondo runs in one of two modes: (1) If no stop-cmd is given, daemondo executes start-cmd asyncronously, and tracks the process id; that process id is used to signal the daemon for later stop and/or restart. (2) If stop-cmd is given, then both start-cmd and stop-cmd are issued syncronously, and are assumed to do all the work of controlling the daemon. In such cases there is no process id to track. In either mode, restart-cmd, if present, is used to restart the daemon. If in mode 1, restart-cmd must not disrupt the process id. If restart-cmd is not provided, the daemon is restarted via a stop/start sequence. ... In mode 1 only, daemondo will exit when it detects that the daemon being monitored has exited. }}}

Therefore by removing the --stop-cmd and --restart-cmd we can ensure daemondo will only start the aiccu daemon once, however if a netchange event is detected daemondo should correctly restart aiccu.

You have that exactly backwards, I think. If there's only a --start-cmd, then daemondo will assume that the process it launches does not daemonize. It will then assume that the daemon has dies as soon as that exits -- which, for aiccu in the default configuration, happens immediately.

We do want daemondo to run in mode (2). We don't want it to track the aiccu.wrapper start process, we want it to trust that aiccu is running after it executed that command, and to never re-execute aiccu start unless

1. the network configuration changes, or

2. launchctl is used to stop or start aiccu.

So that part of the current org.macports.aiccu.plist is fine. The part that isn't is the part about pidfile handling. Currently, org.macports.aiccu.plist passes the --pidfile and --pid=fileclean options to daemondo. That causes daemondo to read the daemon's PID from that file, and to assume that the daemon has exited once the corresponding process terminates (maybe also if the PID file is deleted -- documentation on daemondo is unfortunately quite scarce). It seems that daemondo does not re-execute the --start-cmd in that case, but it does seems to exit, causing launchd to re-start daemondo and that new daemondo instance than (of course) executes the --start-cmd again.

Setting --pid=none should, I think, avoid that. In that case, daemondo really should have no way of knowing whether aiccu is actually running or not, and so shouldn't ever have a reason to re-execute aiccu.wrapper start without user intervention (or a network change being detected), which is the behaviour we want.

For added safety, we should also ensure that aiccu.wrapper never exits with a non-zero exit code, even if aiccu start does. aiccu start seems to daemonize only after requesting the tunnel configuration from the TIC server, so a wrong username or password, for example, will cause its exit code to be non-zero. If that is passed on to daemondo, daemondo might decide to re-run the command - it doesn't seem to do so in my tests, but the behaviour in that case isn't documented, so it might change.

I'll try to create a patch which implementes these changes - could take me a while, though, I really haven't much time to spare right now.

[jmehnle (Julian Mehnle)]

Cc Me!

[jmehnle (Julian Mehnle)]

Florian, any chance you can come up with a patch? It seems you're the one person who understands the scenario best, and it would be a shame for that knowledge to disappear into the ether.

[taylanbayirli@…]

For the time being I would simply rip out the launch service from the package.

It just caused my SixXS account to be suspended! This is unacceptably broken, so not having it in the package would be a big improvement.

[ryandesign (Ryan Carsten Schmidt)]

Since the maintainer is clearly not responding to this ticket, and since it is broken but I don't have time to understand how to fix it so it works properly, I'll disable the startupitem part as suggested. r135910

[danchr (Dan Villiom Podlaski Christiansen)]

Cc Me!

[danchr (Dan Villiom Podlaski Christiansen)]

AICCU patch that implements socket recreation

[danchr (Dan Villiom Podlaski Christiansen)]

Replying to fgp@…:

So we either need to patch AICCU to correct rebind its sockets (if that is indeed the problem here), or do restart it whenever the network configuration changes.

I've attached such a patch, and as far as I can tell it resolves the issue; AICCU now handles network changes just fine. Would any of you please test it and report whether it fixes the issue for you as well? Thanks!

I believe a next step might be to find a better way for making the routing changes; the regular UNIX way can be a bit fragile. The SystemConfiguration framework might be able to do this, but I haven't looked into it in detail.

[danchr (Dan Villiom Podlaski Christiansen)]

FWIW I also posted the patch to the ​SixXS forums, but I've yet to receive a response.

[danchr (Dan Villiom Podlaski Christiansen)]

In 2ece73c5c3fcffebe5a1389632c778d812c96df8/macports-ports:

aiccu: implement reconnection when interface address changes

The upstream author originally filed ticket 22872, complaining that
AICCU should never, ever run under a job manager and just stay alive
forever. Ryan Schmidt eventually addressed this by disabling said
startup item.

In the same bug, however, a user reported that AICCU did, in fact,
fail under some circumstances; when changing networks. A bit of
investigation found that the cause of this that on macOS, sockets
bound to a wildcard UDP port actually get an IP address assigned
internally. If the computer later switches networks and loses that
address, all subsequent reads fail with EADDRNOTAVAIL. I've added a
patch that addresses this by detecting this error and reestablishing
the connection.

Under normal circumstances, package maintainers shouldn't fix bugs in
upstream code, without at the very least having upstream accept the
fix. However, upstream has effectively abandoned the port; I posted a
patch to their forum a year ago, and received no reply. Not even from
the maintainer of the port, so I've taken the liberty of adopting it.

(Anyway, the port itself is somewhat esoteric: it's for an IPv6
tunneling service that no longer accepts new sign-ups.)

Closes: #22872

[danchr (Dan Villiom Podlaski Christiansen)]

Ironically, SixXS just announced the closure of the service in June, see <​https://www.sixxs.net/sunset/>. At some point, they also deleted the post in their forum where I posted the patch, which seems a bit odd…