Squid3 fails to restart correctly

[dazuelos (dan)]

This problem was tracked and closed under #11065 .

I can reproduce it under Lion 10.7.3 with squid3.1.16 .

/Library/LaunchDaemons/org.macports.Squid.plist is correctly loaded.

[dazuelos (dan)]

Cc Me!

[dazuelos (dan)]

I have pinpointed the problem to a problem in a way "daemondo" does a restart on any network change. When a network change occurs, daemondo does:

${prefix}/etc/LaunchDaemons/org.macports.Squid/Squid.wrapper restart

.

Unfortunately this "Squid.wrapper restart" does work as a switch:

# ./Squid.wrapper restart
# ps ax | grep '[(]squid'
#
### nothing: in fact Squid.wrapper did the Stop and not the Start

# ./Squid.wrapper restart
squid: No running copy
# ps ax | grep '[(]squid'
 4781   ??  S      0:00.02 (squid) -s
#
### this time Squid.wrapper failed the Stop and did the Start

Squid.wrapper is killing the actual squid daemon if it's running and doesn't start a new one. If no running daemon is there, the Stop fails and the Start works.

A network change does occurs during the boot of MacOS X… just after launchd launched the first squid daemon. Hence this first call to "Squid.wrapper restart" does a "Stop". This is the reason why squid never starts on its own during normal boot, when a "launchd stop org.macports.Squid" does start it.

[jmroot (Joshua Root)]

So the summary is wrong and stop doesn't actually start the daemon?

[dazuelos (dan)]

Replying to jmr@…:

So the summary is wrong and stop doesn't actually start the daemon?

The summary is correct. "launchctl stop" does start squid, because it kills daemondo and on restart (of daemondo) squid is started.

There are 3 levels of daemons to start involved here:

launchd daemondo squid

The one which I noticed the wrong behaviour was launchd. The one which is the real culprit is the shell script called by daemondo:

Squid.wrapper

I'm working on a fix and testing it.

[jmroot (Joshua Root)]

I just tested and launchctl stop does kill daemondo. However, telling launchctl to stop a job that is marked KeepAlive (as all daemondo jobs are) will only stop it as long as it takes for launchd to detect it's stopped and restart it. And of course starting a job that is already running does nothing.

So that just leaves restart, and I think that is problematic because squid takes 30 seconds to shut down cleanly, so it's not done yet when the Start command is run immediately afterwards. Daemondo actually has a --restart-wait option that would be suited to this, but unfortunately the portstartupitem code doesn't provide an interface to it, and also insists on passing an explicit --restart-cmd when using a wrapper script, which bypasses the built-in code that would do the waiting.

So the options would appear to be: 1. Kill squid outright instead of asking it to shut down cleanly; or 2. Insert a sleep into the restart command. The former could be problematic for obvious reasons, and the latter could also be problematic because the shutdown time is configurable (shutdown_lifetime in squid.conf).

[dazuelos (dan)]

Fix proposal:

*** Squid.wrapper       2012/03/02 14:24:40     1.1
--- Squid.wrapper       2012/03/02 17:42:21
***************
*** 8,13 ****
--- 8,14 ----
  #
  prefix=/opt
  
+ logger -i -t daemondo "args=$@"
  #
  # Start
  #
***************
*** 27,32 ****
--- 28,37 ----
  {
        cd /opt/var/squid
        su -fm root -c "exec /opt/sbin/squid -k shutdown"
+       su -fm root -c "exec /bin/sh -c \"while /opt/sbin/squid -k check ; do
+               sleep 5
+       done
+       exit 0\""
  }
  
  #

Line 14 is just debugging, but may be useful to keep. Gone through the monkey test.

Launchd restarts daemondo which restarts squid.

[jmroot (Joshua Root)]

The patch isn't usable because the wrapper is generated automatically, but thanks for the idea of using -k check.

[jmroot (Joshua Root)]

r90334

[dazuelos (dan)]

Wrong! Squid.wrapper restarts correctly squid when it is shuting down quickly.

Mar  2 19:46:49 milky-way org.macports.Squid[6093]: 2012-03-02 19:46:49 Squid: Child 6109 didn't die; Killing with SIGKILL.
Mar  2 19:47:01: --- last message repeated 42 times ---
Mar  2 19:47:01 milky-way squid[6101]: Shutting down...

daemondo is here running with "-verbosity=9" . My hypothesis is that it is killing the script "Squid.wrapper restart" before the "Start" part.

Where is the source code of "daemondo"?

[dazuelos (dan)]

Replying to dan+macports@…: [...] | Where is the source code of "daemondo"?

Found it. And the cause of my problems is:

const CFTimeInterval kChildDeathTimeout = 20.0;

Hence I can't use a shell script as "Squid.wrapper" to wait for a real squid shutdown.

I found another workaround, based on 2 modif. within

${prefix}/etc/LaunchDaemons/org.macports.Squid

daemondo will use the standard pidfile of squid, Squid.wrapper will be simpler.

*** org.macports.Squid.plist    2012/02/29 17:13:27     1.1
--- org.macports.Squid.plist    2012/03/03 17:47:52
***************
*** 21,30 ****
        <string>restart</string>
        <string>;</string>
        <string>--restart-netchange</string>
!       <string>--pid=none</string>
  </array>
  <key>Debug</key><false/>
! <key>Disabled</key><true/>
  <key>KeepAlive</key><true/>
  </dict>
  </plist>
--- 21,31 ----
        <string>restart</string>
        <string>;</string>
        <string>--restart-netchange</string>
!       <string>--pid=fileauto</string>
!       <string>--pidfile=/opt/var/run/squid/squid.pid</string>
  </array>
  <key>Debug</key><false/>
! <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
  </dict>
  </plist>

*** Squid.wrapper       2012/03/02 14:24:40     1.1
--- Squid.wrapper       2012/03/03 17:56:10
***************
*** 15,23 ****
  {
        cd /opt/var/squid
        if [ ! -d "/opt/var/squid/cache/00" ]; then
!               su -fm root -c "exec /opt/sbin/squid -s -z"
        fi
!       su -fm root -c "exec /opt/sbin/squid -s"
  }
  
  #
--- 15,23 ----
  {
        cd /opt/var/squid
        if [ ! -d "/opt/var/squid/cache/00" ]; then
!               /opt/sbin/squid -s -z
        fi
!       /opt/sbin/squid -s
  }
  
  #
***************
*** 26,32 ****
  Stop()
  {
        cd /opt/var/squid
!       su -fm root -c "exec /opt/sbin/squid -k shutdown"
  }
  
  #
--- 26,32 ----
  Stop()
  {
        cd /opt/var/squid
!       /opt/sbin/squid -k shutdown
  }
  
  #

Tests done so far:

Squid.wrapper restart
launchctl start org.macports.Squid.plist
launchctl stop org.macports.Squid.plist
reboot
switch Location

[jmroot (Joshua Root)]

r91290 (Again, the patches don't really help because the files are automatically generated.)