[16:17] <Tack> "start rsyslog" outputs "start: Job is already running: rsyslog" but it definitely isn't.  "stop rsyslog" hangs indefinitely, but after which (if I ctrl-c it), "start rsyslog" no longer says it's running, but rather it hangs too.  Once I ctrl-c that, "start rsyslog" again says it's running.
[16:17] <Tack> At no time does it get started though, and I'm quite confused as to what's happening.  Any ideas?
[16:18] <Tack> (BTW this is on Ubuntu 10.04)
[16:36] <ion> Have you modified /etc/init/rsyslog.conf at any time?
[16:39] <Tack> ion: nope, it's stock.
[16:40] <Tack> However ...
[16:41] <Tack> https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/692919 is almost certainly what started it.  This happened following a package update last month some time.  I havne't had a chance to look into it until now.
[16:42] <Tack> So it's entirely possible /etc/init/rsyslog.conf got replaced during the package upgrade.  It's dated 2010-12-10, so it's likely. 
[16:42] <Tack> I've not modified it personally, though, nor changed its mtime.
[16:43] <ion> What does ‘status rsyslog’ print?
[16:44] <Tack> rsyslog start/starting
[16:44] <ion> What does ‘pgrep -l rsyslogd’ print?
[16:45] <Tack> Nothing, and it returns exitcode 1.
[16:45] <ion> sorry
[16:45] <ion> Ah, ok. (Had it printed anything, i’d have been interested to see pgrep -lf rsyslogd.)
[16:45] <ion> ‘status rsyslog-kmsg’?
[16:46] <Tack> status: Unknown job: rsyslog-kmsg
[16:46] <Tack> Possibly not present in Ubuntu 10.04
[16:46] <ion> Ah, indeed.
[16:48] <Tack> If I "strace stop rsyslog" it seems to be doing some dbus chatter over a unix socket (connect(3, {sa_family=AF_FILE, path=@"/com/ubuntu/upstart"}) and it hangs on poll() for that fd.   Maybe I need to bounce dbus-daemon?
[16:48] <Tack> Nah, no effect. :)
[16:48] <ion> No, that’s not the problem and stracing start/stop/initctl doesn’t help in diagnosing problems like this.
[16:49] <Tack> Ah, I was afraid of that.
[16:50] <ion> What does ‘stop --no-wait rsyslog’ print? How about ‘status rsyslog’ after that?
[16:51] <Tack> # stop --no-wait rsyslog
[16:51] <Tack> rsyslog stop/starting
[16:51] <Tack> # status rsyslog
[16:51] <Tack> rsyslog stop/starting
[16:59] <ion> I guess you’ll need to consult Keybuk for more ideas about debugging the issue. If you don’t feel like waiting for him to appear on IRC, please file a bug in https://bugs.launchpad.net/ubuntu/+source/upstart (i’m not sure if it’s a bug in the upstart package or the rsyslog one) describing the symptoms (e.g. that ‘stop rsyslog’ blocks forever) and providing the output of ‘pgrep -lf rsyslogd’, ‘stop --no-wait rsyslog’, ‘status rsyslog’, ‘start
[16:59] <ion> rsyslog’ and ‘status rsyslog’ again. Might as well attach the file /etc/init/rsyslog.conf as well, so it can be verified that the job definition hasn’t become broken due to some bug.
[17:02] <Tack> Can do, thanks.  Unfortunately I need to reboot this box soonish for a kernel upgrade, and that will almost certainly fix the problem.  I'll try to hold off for a couple days.
[17:03] <ion> Alright, thanks
[17:03] <Tack> Thanks for the tech support. :)