| pmjdebru1jn | is there an upstart config file too? so these parameters won't have to be passed on the kernel commandline? | 06:39 |
|---|---|---|
| === Guest59153 is now known as kklimonda | ||
| intgr | Hi! So I have a custom init conf script. Whatever I do, upstart doesn't even attempt to start the service. It just says it's either "start/killed" or "stop/killed" with the same old PID, every time. | 10:15 |
| intgr | http://codepad.org/3oSxyR4p | 10:20 |
| intgr | The PID doesn't even change when I try to start/stop it | 10:21 |
| intgr | Nothing gets written to syslog or /var/log/upstart/daemonlogger.log | 10:21 |
| intgr | Nor is there a "daemonlogger" process | 10:21 |
| intgr | And the process 11920 hasn't existed for ages | 10:24 |
| intgr | Oh great, now I ran "reboot" and it hung at "Asking all remaining processes to terminate..." | 10:26 |
| intgr | Well the service used to include "expect fork" in it. When I re-add that back I can reproduce this problem, which seems like a bug. | 10:34 |
| jodh | intgr: take a look at http://upstart.ubuntu.com/cookbook/#precepts-for-creating-a-job-configuration-file, http://upstart.ubuntu.com/cookbook/#determining-the-value-of-expect. | 10:41 |
| jodh | intgr: as the cookbook mentions, never add "respawn" until you are convinced the job is behaving correctly. | 10:42 |
| jodh | Also, please see http://upstart.ubuntu.com/cookbook/#checking-how-a-service-might-react-when-run-as-a-job | 10:42 |
| intgr | jodh: Regardless, a bad configuration file shouldn't prevent the machine from rebooting, no? | 10:42 |
| intgr | ?c | 10:42 |
| jodh | intgr: it shouldn't. | 10:44 |
| jodh | intgr: at a guess, I'd say your daemonlogger is forking and then dying. You need to know how many times it forks - http://upstart.ubuntu.com/cookbook/#how-to-establish-fork-count | 10:46 |
| intgr | I'll just exec it without daemonization, hopefully then Upstart won't crash | 10:47 |
| jodh | intgr: I don't think Upstart is crashing here - you'll get a kernel panic if it is. | 10:48 |
| intgr | Well a hung reboot is a "crash" in my mind. | 10:48 |
| jodh | intgr: if you are running on Ubuntu, that shouldn't be possible. | 10:51 |
| intgr | Ubuntu 12.04, up to date. | 10:51 |
| jodh | intgr: please raise a bug with full step-by-step details on how to recreate then. | 10:52 |
| intgr | My reset button is wearing out already... | 10:53 |
| intgr | :) | 10:53 |
| intgr | jodh: Any ideas how to fix this without a cold reboot? I already started this job on our live server. | 10:59 |
| jodh | intgr: the links above explain how to build up a .conf file. I suggest you try to run daemonlogger outside of Upstart (http://upstart.ubuntu.com/cookbook/#checking-how-a-service-might-react-when-run-as-a-job) and once you have that working and know how many times it forks, start testing your .conf job without 'respawn' initially. | 11:23 |
| jodh | intgr: or are you referring to how to reboot your system? | 11:23 |
| intgr | jodh: I mean how do I fix upstart so the next shutdown won't hang forever. | 11:40 |
| jodh | intgr: well, since I don't know what is causing the hang, difficult to say aside from disabling the job you are developing. | 11:51 |
| intgr | Ok... | 11:56 |
| intgr | jodh: Do you want to try to reproduce? http://codepad.org/lgj4aS2j | 12:16 |
| jodh | intgr: I cannot recreate your hang - even mis-specifying the .conf file does not cause a problem on shutdown. | 12:32 |
| jodh | intgr: looks like daemonlogger is only forking once btw so your job should specify "expect fork". | 12:32 |
| intgr | jodh: Same thing happens when I use "expect fork" | 12:37 |
| jodh | intgr: try creating a fresh 12.04 install as I suspect something else has changed on your system. If that still fails, please raise a bug with full details. | 12:38 |
| intgr | This is pretty fresh -- I just cloned it from our VM template (which is a clean 12.04 install with a few additional services) | 12:39 |
| jodh | intgr: what services? does a clean cloned system with no daemonlogger.conf reliably shut down? | 12:40 |
| intgr | And the machine I initially experienced this on was installed yesterday, with a different configuration. | 12:40 |
| intgr | If I remove the daemonlogger.conf file, it shuts down nicely again. | 12:40 |
| intgr | Or if I simply don't start the service | 12:40 |
| intgr | Oh, and the "stop daemonlogger" command is necessary too -- it doesn't hang if I don't run that. | 12:41 |
| intgr | Did you run that? | 12:41 |
| intgr | The VM template has acpid, atop, ntpd and is using linux-virtual kernel | 12:43 |
| jodh | intgr: ah - because you're running "stop", but that can never complete successfully (as you've mis-specified 'expect'), yes Upstart will wait until it gets confirmation your process has ended. | 12:49 |
| intgr | And this prevents shutdown, too? | 12:49 |
| jodh | In the scenario where a user has manually run 'stop', not interrupted it, mis-specified 'expect' and not specified 'stop on'... yes. | 12:53 |
| jodh | intgr: please raise a bug so we can try to find a way to identify such jobs and handle them. | 12:53 |
| intgr | Sure | 12:54 |
| intgr | Thanks for being heplful :) | 12:54 |
| jodh | intgr: np. Your job prolly wants to look something like this: http://paste.ubuntu.com/1074736/ | 12:58 |
| jodh | intgr: ... although, putting logs in /tmp prolly isn't such a good idea ;) | 12:58 |
| intgr | :) | 12:58 |
| jodh | intgr: in fact that "pre-start script" can be just "pre-start exec mkdir $dir". It'll be handled the same internally, but it's results in a cleaner conf file. | 12:59 |
| intgr | jodh: Does "expect" work with "script/end script"? | 12:59 |
| jodh | intgr: "yes", but be careful since Upstart starts counting forks immediately so if your script starts by doing something like "var=$(cat /etc/somefile)", Upstart will follow the fork of /bin/cat - not what you want. | 13:00 |
| intgr | I see | 13:00 |
| jodh | intgr: but crucially 'expect' only works for the "main" exec or script section. Hence, makes sense to do setup in pre-start for example. | 13:01 |
| intgr | jodh: Well it works with exec, but still not with script: http://paste.ubuntu.com/1074741/ | 13:04 |
| intgr | Oh, it doesn't hang anymore, just doesn't get the right PID | 13:04 |
| jodh | intgr: you can overcome that issue btw by having 2 conf files - one which does the setup and reads files, etc and then runs "start job2 var1=a var2=b varc='hello world'" or similar. All those variables will be available to job2 which can just do "exec mydaemon -l $var1 -i $var2" and specify "respawn" too. | 13:04 |
| jodh | intgr: if the pid is incorrect, you've mis-specified expect somehow. | 13:05 |
| intgr | I'm using "expect fork" just like you did | 13:05 |
| jodh | intgr: you need "exec daemonlogger" *within* that script section. | 13:05 |
| intgr | Ah ok | 13:06 |
| intgr | jodh: https://bugs.launchpad.net/upstart/+bug/1020951 | 13:17 |
| jodh | intgr: thanks. | 13:18 |
| benbro | how can I run a service only after postgresql is running? | 14:28 |
| benbro | I currently have this in my upstart script: | 14:28 |
| benbro | start on (local-filesystems and net-device-up IFACE!=lo) | 14:28 |
| jodh | benbro: nominally, "start on stopped rc" (since postgresql is a SysV service in Ubuntu atm I think). | 14:31 |
| jodh | benbro: however, that assumes that when the postgresql SysV service script finishes that the postgresql server is ready to accept connections. | 14:32 |
| benbro | jodh: and it is not always the case? | 14:41 |
| intgr | benbro: I believe the Ubuntu Postgres init script returns immediately after launching postmaster. | 14:56 |
| intgr | If your machine crashes/loses power, Postgres has to perform recovery at startup, which can take a long time | 14:56 |
| intgr | (depending on your checkpoint_* settings) | 14:56 |
| benbro | intgr: so there is no way to start my service after postgres? | 14:58 |
| benbro | intgr: maybe my service should just retry creating the connection | 14:58 |
| intgr | Yeah, that's what I do | 14:59 |
| benbro | thanks | 14:59 |
| benbro | in that case I should still use "start on stopped rc" or something else? | 14:59 |
| intgr | benbro: Also beware that PostgreSQL may be restarted without warning if you have unattended upgrades enabled. | 14:59 |
| benbro | I don't think I allow automatic upgrades | 15:00 |
| benbro | should I use "start on (local-filesystems and net-device-up IFACE!=lo)" if I'll just retry creating the connection to postgres? | 15:00 |
| jodh | benbro: use "start on stopped rc" but you may have to poll as intgr suggests as you cannot guarantee that the db is servicing requests just because some process is running. | 15:05 |
| benbro | jodh: thanks | 15:05 |
| axisys | what is update-rc.d -f name remove equivalent for upstart process? | 23:43 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!