[17:10] Hi all - is there any way to provide a custom exec or script for the actual stop procedure? [17:11] use case is a server that stops gracefully when it receives a shutdown command over tcp, but loses data when it receives TERM [17:12] I tried doing it with "pre-stop exec server-cli shutdown" [17:12] which shuts it down properly, but it then respawns [17:12] what I really want isn't a pre-stop, but just a stop exec [17:43] jdsanders: I believe there is a 'kill signal' keyword [17:43] kill signal SIGNAL [17:43] SpamapS: thanks, but I actually want *no* signal [17:43] jdsanders: you can also do a 'normal exit SIGWHATEVER' and the signal will not be considered something to respawn on [17:43] or more importantly, I want a process killed in a pre-stop clause to not be respawned [17:43] oh interesting [17:43] i'll read about that command [17:44] jdsanders: so if it is graceful on SIGUSR1, then 'normal exit 0 USR1' means don't respawn if it is killed by USR1 [17:44] jdsanders: the '0' in there also means if the exit code was 0, don't respawn. Thats the default btw, so its surprising that your graceful shutdown doesn't exit(0) [17:45] jdsanders: though I believe even if you exit(0) from the USR1 handler, upstart will consider that a kill by SIGUSR1 [17:46] huh [17:46] thanks, sounds like i have a lot more respawn research to do [17:46] jdsanders: the default logging level *should* tell you what upstart thinks killed the process [17:47] jdsanders: would you mind pasting what you see? youre pre-stop, btw, is the right way to go [17:50] SpamapS: one moment, going back up my terminal.... [17:58] shoot can't reproduce! [17:59] ah, here [17:59] maybe a race condition actually... [17:59] deploy@NCVMLITSDEVQUE01:/var/www/apps/smartgrid/current/apollo$ sudo start redis-server [17:59] redis-server start/running, process 9535 [17:59] deploy@NCVMLITSDEVQUE01:/var/www/apps/smartgrid/current/apollo$ sudo status redis-server [17:59] redis-server start/running, process 9535 [17:59] deploy@NCVMLITSDEVQUE01:/var/www/apps/smartgrid/current/apollo$ sudo stop redis-server [17:59] redis-server start/running, process 9625 [17:59] deploy@NCVMLITSDEVQUE01:/var/www/apps/smartgrid/current/apollo$ sudo status redis-server [17:59] redis-server start/running, process 9625 [18:00] SpamapS: ^^ [18:01] SpamapS: here's the config script - https://gist.github.com/bf5b600645ff1ddc67cf [18:10] jdsanders: looking [19:13] SpamapS: just got back from a meeting, any luck understanding that redis-server issue? [19:14] jdsanders: oops no I got sdistracted. Looking again [19:15] jdsanders: I'd need to see your syslog entries to understand what happened [19:17] k, do you know what label to look for in syslog? [19:17] don't see anything like "upstart" [19:18] looks like "init" maybe [19:19] Jul 11 15:18:08 NCVMLITSDEVQUE01 init: redis-server pre-stop process (9765) terminated with status 1 [19:19] seems problematic maybe [19:29] hmmm, yeah it seems like my shutdown command is *expected* to have exit code 1 [19:29] (oddly enough) [19:30] so i think that's causing the problem? [20:19] by default it will consider exit != 0 as an error [20:20] so just override that if you want to ignore that [20:20] JanC: thanks, I'm now fairly certain my server is exiting with 0 when it shuts down, and I've modified the pre-stop to make sure it exits 0 [20:20] all I see in syslog is now basically "process died, respawning" [20:20] is there some way I can see more info about why it decided to respawn? [20:22] maybe use a script instead of the main binary [20:23] and make it log whatever seems relevant [20:28] ok thanks [22:12] jdsanders: you may also want to try 'initctl log-priority info' to get more about why it died [22:12] jdsanders: I suspect just setting 'normal exit' properly is all you need [22:32] SpamapS: i'll try the log level thing, but the reason I think the exit code is 0 is, I ran the same command that I'm executing from upstart in the foreground [22:32] then shut it down from another process [22:32] then checked its exit status [22:32] and it was 0 [22:33] jdsanders: right, so I am wondering what upstart thinks the exit code is [22:33] jdsanders: you might also just try 'normal exit 0' [22:33] I thought that was the default [22:33] but man 5 init isn't entirely crystal clear on it IIRC [22:34] "Tasks may exit with a zero exit status to prevent being respawned." [22:34] I believe that means jobs may not by default consider 0 a normal exit [22:34] " All [22:34] reasons for a service stopping, except the stop(8) command itself, are considered abnormal." [22:34] jdsanders: yeah, try 'normal exit 0' [22:37] huh [22:37] i'll try that! [22:39] jdsanders: what that means is that upstart won't be able to keep redis running in the face of accidental shutdowns. But thats probably fine. [22:40] well [22:40] I haven't tested this [22:40] but my theory is that any non-graceful shutdown will exit with a different exit code than 0 [22:40] it should [22:41] so I think if somebody *wants* to shut it down with the command that is meant for precisely that, it is reasonable to expect that it doesn't respawn [22:41] jdsanders: you're still in better shape than with a bare daemon. [22:54] alright, I think I've got this now, thanks for all the help! [22:54] SpamapS,JanC: ^^ [22:54] jdsanders: woot! [22:55] woot! [22:56] alright...I hate to do this, but here's another one [22:56] I'm on ubuntu 10.04 [22:56] which has upstart 0.6.5 [22:56] which doesn't support setuid [22:56] but I need to run my server as another user [22:57] so I'm doing [22:57] sudo -u $USER sh -c "/usr/local/bin/redis-server /etc/redis-server.conf 2>&1 >> /var/log/redis/server.log" [22:57] problem is that it makes the fork count jankier [22:57] when I start the server it actually spawns three processes - one for sudo, one for sh, and one for redis-server [22:58] the pid i'd really like to track is the redis-server one [22:58] (which isn't doing any forking) [22:58] any better way to change users? [23:00] jdsanders: sudo is one way [23:00] jdsanders: better is start-stop-daemon [23:00] ah [23:00] ok [23:00] I'll look into that [23:00] http://upstart.ubuntu.com/cookbook/#changing-user [23:01] jdsanders: ^^ [23:01] awesome, thank you [23:01] sorta related: how do i get out of the stop/killed state [23:08] I currently have: redis-server start/killed, process 20674 [23:08] but: sudo kill -0 20674; echo $? [23:08] kill: No such process [23:08] 1 [23:08] sudo stop redis-server seems to hang [23:09] jdsanders: ugh did you try 'expect fork' at one point? [23:09] i did [23:09] i was playing with it [23:09] Ok, thats a bug [23:09] and seem to have foobared it right up [23:09] http://paste.ubuntu.com/1087042/ [23:10] this script loops over the pid space until it reaches the pid given [23:10] jdsanders: its python 3.. might have to port it to python2 ;) [23:10] so the theory here is that it's running somewhere, but i don't know where? [23:10] no [23:10] its not running [23:10] upstart thinks it should be [23:10] and will wait forever for it to die [23:11] so how does that script help? [23:11] it creates processes until the given process exists.. then lets upstart kill it [23:11] haha oooooooooooh, i get it [23:11] clever [23:11] not really, its a hack on hack on hack [23:11] is there any way to re-initialize upstart? [23:11] I could always just reboot [23:11] jdsanders: yes but then it forgets all job state [23:11] well yeah, hacks are clever [23:11] yeah reboot works if you're just noodling around [23:12] jdsanders: the original fix was in ruby and about 10 lines of it at that.. this one is just more friendly [23:12] what do you mean by "forgets all job state" [23:12] jdsanders: you can send upstart a command to re-execute itself, but then it forgets what is running and what is not [23:12] which means your shutdown does not go so smoothly [23:15] hmmm [23:15] ok i'll figure it out