[06:39] <pmjdebru1jn> is there an upstart config file too? so these parameters won't have to be passed on the kernel commandline?
[10:15] <intgr> Hi! So I have a custom init conf script. Whatever I do, upstart doesn't even attempt to start the service. It just says it's either "start/killed" or "stop/killed" with the same old PID, every time.
[10:20] <intgr> http://codepad.org/3oSxyR4p
[10:21] <intgr> The PID doesn't even change when I try to start/stop it
[10:21] <intgr> Nothing gets written to syslog or /var/log/upstart/daemonlogger.log
[10:21] <intgr> Nor is there a "daemonlogger" process
[10:24] <intgr> And the process 11920 hasn't existed for ages
[10:26] <intgr> Oh great, now I ran "reboot" and it hung at "Asking all remaining processes to terminate..."
[10:34] <intgr> Well the service used to include "expect fork" in it. When I re-add that back I can reproduce this problem, which seems like a bug.
[10:41] <jodh> intgr: take a look at http://upstart.ubuntu.com/cookbook/#precepts-for-creating-a-job-configuration-file, http://upstart.ubuntu.com/cookbook/#determining-the-value-of-expect.
[10:42] <jodh> intgr: as the cookbook mentions, never add "respawn" until you are convinced the job is behaving correctly.
[10:42] <jodh> Also, please see http://upstart.ubuntu.com/cookbook/#checking-how-a-service-might-react-when-run-as-a-job
[10:42] <intgr> jodh: Regardless, a bad configuration file shouldn't prevent the machine from rebooting, no?
[10:42] <intgr> ?c
[10:44] <jodh> intgr: it shouldn't.
[10:46] <jodh> intgr: at a guess, I'd say your daemonlogger is forking and then dying. You need to know how many times it forks - http://upstart.ubuntu.com/cookbook/#how-to-establish-fork-count
[10:47] <intgr> I'll just exec it without daemonization, hopefully then Upstart won't crash
[10:48] <jodh> intgr: I don't think Upstart is crashing here - you'll get a kernel panic if it is.
[10:48] <intgr> Well a hung reboot is a "crash" in my mind.
[10:51] <jodh> intgr: if you are running on Ubuntu, that shouldn't be possible.
[10:51] <intgr> Ubuntu 12.04, up to date.
[10:52] <jodh> intgr: please raise a bug with full step-by-step details on how to recreate then.
[10:53] <intgr> My reset button is wearing out already...
[10:53] <intgr> :)
[10:59] <intgr> jodh: Any ideas how to fix this without a cold reboot? I already started this job on our live server.
[11:23] <jodh> intgr: the links above explain how to build up a .conf file. I suggest you try to run daemonlogger outside of Upstart (http://upstart.ubuntu.com/cookbook/#checking-how-a-service-might-react-when-run-as-a-job) and once you have that working and know how many times it forks, start testing your .conf job without 'respawn' initially.
[11:23] <jodh> intgr: or are you referring to how to reboot your system?
[11:40] <intgr> jodh: I mean how do I fix upstart so the next shutdown won't hang forever.
[11:51] <jodh> intgr: well, since I don't know what is causing the hang, difficult to say aside from disabling the job you are developing.
[11:56] <intgr> Ok...
[12:16] <intgr> jodh: Do you want to try to reproduce? http://codepad.org/lgj4aS2j
[12:32] <jodh> intgr: I cannot recreate your hang - even mis-specifying the .conf file does not cause a problem on shutdown.
[12:32] <jodh> intgr: looks like daemonlogger is only forking once btw so your job should specify "expect fork".
[12:37] <intgr> jodh: Same thing happens when I use "expect fork"
[12:38] <jodh> intgr: try creating a fresh 12.04 install as I suspect something else has changed on your system. If that still fails, please raise a bug with full details.
[12:39] <intgr> This is pretty fresh -- I just cloned it from our VM template (which is a clean 12.04 install with a few additional services)
[12:40] <jodh> intgr: what services? does a clean cloned system with no daemonlogger.conf reliably shut down?
[12:40] <intgr> And the machine I initially experienced this on was installed yesterday, with a different configuration.
[12:40] <intgr> If I remove the daemonlogger.conf file, it shuts down nicely again.
[12:40] <intgr> Or if I simply don't start the service
[12:41] <intgr> Oh, and the "stop daemonlogger" command is necessary too -- it doesn't hang if I don't run that.
[12:41] <intgr> Did you run that?
[12:43] <intgr> The VM template has acpid, atop, ntpd and is using linux-virtual kernel
[12:49] <jodh> intgr: ah - because you're running "stop", but that can never complete successfully (as you've mis-specified 'expect'), yes Upstart will wait until it gets confirmation your process has ended.
[12:49] <intgr> And this prevents shutdown, too?
[12:53] <jodh> In the scenario where a user has manually run 'stop', not interrupted it, mis-specified 'expect' and not specified 'stop on'... yes.
[12:53] <jodh> intgr: please raise a bug so we can try to find a way to identify such jobs and handle them.
[12:54] <intgr> Sure
[12:54] <intgr> Thanks for being heplful :)
[12:58] <jodh> intgr: np. Your job prolly wants to look something like this: http://paste.ubuntu.com/1074736/
[12:58] <jodh> intgr: ... although, putting logs in /tmp prolly isn't such a good idea ;)
[12:58] <intgr> :)
[12:59] <jodh> intgr: in fact that "pre-start script" can be just "pre-start exec mkdir $dir". It'll be handled the same internally, but it's results in a cleaner conf file.
[12:59] <intgr> jodh: Does "expect" work with "script/end script"?
[13:00] <jodh> intgr: "yes", but be careful since Upstart starts counting forks immediately so if your script starts by doing something like "var=$(cat /etc/somefile)", Upstart will follow the fork of /bin/cat - not what you want.
[13:00] <intgr> I see
[13:01] <jodh> intgr: but crucially 'expect' only works for the "main" exec or script section. Hence, makes sense to do setup in pre-start for example.
[13:04] <intgr> jodh: Well it works with exec, but still not with script: http://paste.ubuntu.com/1074741/
[13:04] <intgr> Oh, it doesn't hang anymore, just doesn't get the right PID
[13:04] <jodh> intgr: you can overcome that issue btw by having 2 conf files - one which does the setup and reads files, etc and then runs "start job2 var1=a var2=b varc='hello world'" or similar. All those variables will be available to job2 which can just do "exec mydaemon -l $var1 -i $var2" and specify "respawn" too.
[13:05] <jodh> intgr: if the pid is incorrect, you've mis-specified expect somehow.
[13:05] <intgr> I'm using "expect fork" just like you did
[13:05] <jodh> intgr: you need "exec daemonlogger" *within* that script section.
[13:06] <intgr> Ah ok
[13:17] <intgr> jodh: https://bugs.launchpad.net/upstart/+bug/1020951
[13:18] <jodh> intgr: thanks.
[14:28] <benbro> how can I run a service only after postgresql is running?
[14:28] <benbro> I currently have this in my upstart script:
[14:28] <benbro> start on (local-filesystems and net-device-up IFACE!=lo)
[14:31] <jodh> benbro: nominally, "start on stopped rc" (since postgresql is a SysV service in Ubuntu atm I think).
[14:32] <jodh> benbro: however, that assumes that when the postgresql SysV service script finishes that the postgresql server is ready to accept connections.
[14:41] <benbro> jodh: and it is not always the case?
[14:56] <intgr> benbro: I believe the Ubuntu Postgres init script returns immediately after launching postmaster.
[14:56] <intgr> If your machine crashes/loses power, Postgres has to perform recovery at startup, which can take a long time
[14:56] <intgr> (depending on your checkpoint_* settings)
[14:58] <benbro> intgr: so there is no way to start my service after postgres?
[14:58] <benbro> intgr: maybe my service should just retry creating the connection
[14:59] <intgr> Yeah, that's what I do
[14:59] <benbro> thanks
[14:59] <benbro> in that case I should still use "start on stopped rc" or something else?
[14:59] <intgr> benbro: Also beware that PostgreSQL may be restarted without warning if you have unattended upgrades enabled.
[15:00] <benbro> I don't think I allow automatic upgrades
[15:00] <benbro> should I use "start on (local-filesystems and net-device-up IFACE!=lo)" if I'll just retry creating the connection to postgres?
[15:05] <jodh> benbro: use "start on stopped rc" but you may have to poll as intgr suggests as you cannot guarantee that the db is servicing requests just because some process is running.
[15:05] <benbro> jodh: thanks
[23:43] <axisys> what is update-rc.d -f name remove equivalent for upstart process?