[06:39] is there an upstart config file too? so these parameters won't have to be passed on the kernel commandline? === Guest59153 is now known as kklimonda [10:15] Hi! So I have a custom init conf script. Whatever I do, upstart doesn't even attempt to start the service. It just says it's either "start/killed" or "stop/killed" with the same old PID, every time. [10:20] http://codepad.org/3oSxyR4p [10:21] The PID doesn't even change when I try to start/stop it [10:21] Nothing gets written to syslog or /var/log/upstart/daemonlogger.log [10:21] Nor is there a "daemonlogger" process [10:24] And the process 11920 hasn't existed for ages [10:26] Oh great, now I ran "reboot" and it hung at "Asking all remaining processes to terminate..." [10:34] Well the service used to include "expect fork" in it. When I re-add that back I can reproduce this problem, which seems like a bug. [10:41] intgr: take a look at http://upstart.ubuntu.com/cookbook/#precepts-for-creating-a-job-configuration-file, http://upstart.ubuntu.com/cookbook/#determining-the-value-of-expect. [10:42] intgr: as the cookbook mentions, never add "respawn" until you are convinced the job is behaving correctly. [10:42] Also, please see http://upstart.ubuntu.com/cookbook/#checking-how-a-service-might-react-when-run-as-a-job [10:42] jodh: Regardless, a bad configuration file shouldn't prevent the machine from rebooting, no? [10:42] ?c [10:44] intgr: it shouldn't. [10:46] intgr: at a guess, I'd say your daemonlogger is forking and then dying. You need to know how many times it forks - http://upstart.ubuntu.com/cookbook/#how-to-establish-fork-count [10:47] I'll just exec it without daemonization, hopefully then Upstart won't crash [10:48] intgr: I don't think Upstart is crashing here - you'll get a kernel panic if it is. [10:48] Well a hung reboot is a "crash" in my mind. [10:51] intgr: if you are running on Ubuntu, that shouldn't be possible. [10:51] Ubuntu 12.04, up to date. [10:52] intgr: please raise a bug with full step-by-step details on how to recreate then. [10:53] My reset button is wearing out already... [10:53] :) [10:59] jodh: Any ideas how to fix this without a cold reboot? I already started this job on our live server. [11:23] intgr: the links above explain how to build up a .conf file. I suggest you try to run daemonlogger outside of Upstart (http://upstart.ubuntu.com/cookbook/#checking-how-a-service-might-react-when-run-as-a-job) and once you have that working and know how many times it forks, start testing your .conf job without 'respawn' initially. [11:23] intgr: or are you referring to how to reboot your system? [11:40] jodh: I mean how do I fix upstart so the next shutdown won't hang forever. [11:51] intgr: well, since I don't know what is causing the hang, difficult to say aside from disabling the job you are developing. [11:56] Ok... [12:16] jodh: Do you want to try to reproduce? http://codepad.org/lgj4aS2j [12:32] intgr: I cannot recreate your hang - even mis-specifying the .conf file does not cause a problem on shutdown. [12:32] intgr: looks like daemonlogger is only forking once btw so your job should specify "expect fork". [12:37] jodh: Same thing happens when I use "expect fork" [12:38] intgr: try creating a fresh 12.04 install as I suspect something else has changed on your system. If that still fails, please raise a bug with full details. [12:39] This is pretty fresh -- I just cloned it from our VM template (which is a clean 12.04 install with a few additional services) [12:40] intgr: what services? does a clean cloned system with no daemonlogger.conf reliably shut down? [12:40] And the machine I initially experienced this on was installed yesterday, with a different configuration. [12:40] If I remove the daemonlogger.conf file, it shuts down nicely again. [12:40] Or if I simply don't start the service [12:41] Oh, and the "stop daemonlogger" command is necessary too -- it doesn't hang if I don't run that. [12:41] Did you run that? [12:43] The VM template has acpid, atop, ntpd and is using linux-virtual kernel [12:49] intgr: ah - because you're running "stop", but that can never complete successfully (as you've mis-specified 'expect'), yes Upstart will wait until it gets confirmation your process has ended. [12:49] And this prevents shutdown, too? [12:53] In the scenario where a user has manually run 'stop', not interrupted it, mis-specified 'expect' and not specified 'stop on'... yes. [12:53] intgr: please raise a bug so we can try to find a way to identify such jobs and handle them. [12:54] Sure [12:54] Thanks for being heplful :) [12:58] intgr: np. Your job prolly wants to look something like this: http://paste.ubuntu.com/1074736/ [12:58] intgr: ... although, putting logs in /tmp prolly isn't such a good idea ;) [12:58] :) [12:59] intgr: in fact that "pre-start script" can be just "pre-start exec mkdir $dir". It'll be handled the same internally, but it's results in a cleaner conf file. [12:59] jodh: Does "expect" work with "script/end script"? [13:00] intgr: "yes", but be careful since Upstart starts counting forks immediately so if your script starts by doing something like "var=$(cat /etc/somefile)", Upstart will follow the fork of /bin/cat - not what you want. [13:00] I see [13:01] intgr: but crucially 'expect' only works for the "main" exec or script section. Hence, makes sense to do setup in pre-start for example. [13:04] jodh: Well it works with exec, but still not with script: http://paste.ubuntu.com/1074741/ [13:04] Oh, it doesn't hang anymore, just doesn't get the right PID [13:04] intgr: you can overcome that issue btw by having 2 conf files - one which does the setup and reads files, etc and then runs "start job2 var1=a var2=b varc='hello world'" or similar. All those variables will be available to job2 which can just do "exec mydaemon -l $var1 -i $var2" and specify "respawn" too. [13:05] intgr: if the pid is incorrect, you've mis-specified expect somehow. [13:05] I'm using "expect fork" just like you did [13:05] intgr: you need "exec daemonlogger" *within* that script section. [13:06] Ah ok [13:17] jodh: https://bugs.launchpad.net/upstart/+bug/1020951 [13:18] intgr: thanks. [14:28] how can I run a service only after postgresql is running? [14:28] I currently have this in my upstart script: [14:28] start on (local-filesystems and net-device-up IFACE!=lo) [14:31] benbro: nominally, "start on stopped rc" (since postgresql is a SysV service in Ubuntu atm I think). [14:32] benbro: however, that assumes that when the postgresql SysV service script finishes that the postgresql server is ready to accept connections. [14:41] jodh: and it is not always the case? [14:56] benbro: I believe the Ubuntu Postgres init script returns immediately after launching postmaster. [14:56] If your machine crashes/loses power, Postgres has to perform recovery at startup, which can take a long time [14:56] (depending on your checkpoint_* settings) [14:58] intgr: so there is no way to start my service after postgres? [14:58] intgr: maybe my service should just retry creating the connection [14:59] Yeah, that's what I do [14:59] thanks [14:59] in that case I should still use "start on stopped rc" or something else? [14:59] benbro: Also beware that PostgreSQL may be restarted without warning if you have unattended upgrades enabled. [15:00] I don't think I allow automatic upgrades [15:00] should I use "start on (local-filesystems and net-device-up IFACE!=lo)" if I'll just retry creating the connection to postgres? [15:05] benbro: use "start on stopped rc" but you may have to poll as intgr suggests as you cannot guarantee that the db is servicing requests just because some process is running. [15:05] jodh: thanks [23:43] what is update-rc.d -f name remove equivalent for upstart process?