/srv/irclogs.ubuntu.com/2012/07/11/#upstart.txt

jdsandersHi all - is there any way to provide a custom exec or script for the actual stop procedure?17:10
jdsandersuse case is a server that stops gracefully when it receives a shutdown command over tcp, but loses data when it receives TERM17:11
jdsandersI tried doing it with "pre-stop exec server-cli shutdown"17:12
jdsanderswhich shuts it down properly, but it then respawns17:12
jdsanderswhat I really want isn't a pre-stop, but just a stop exec17:12
SpamapSjdsanders: I believe there is a 'kill signal' keyword17:43
SpamapS       kill signal SIGNAL17:43
jdsandersSpamapS: thanks, but I actually want *no* signal17:43
SpamapSjdsanders: you can also do a 'normal exit SIGWHATEVER' and the signal will not be considered something to respawn on17:43
jdsandersor more importantly, I want a process killed in a pre-stop clause to not be respawned17:43
jdsandersoh interesting17:43
jdsandersi'll read about that command17:43
SpamapSjdsanders: so if it is graceful on SIGUSR1, then 'normal exit 0 USR1' means don't respawn if it is killed by USR117:44
SpamapSjdsanders: the '0' in there also means if the exit code was 0, don't respawn. Thats the default btw, so its surprising that your graceful shutdown doesn't exit(0)17:44
SpamapSjdsanders: though I believe even if you exit(0) from the USR1 handler, upstart will consider that a kill by SIGUSR117:45
jdsandershuh17:46
jdsandersthanks, sounds like i have a lot more respawn research to do17:46
SpamapSjdsanders: the default logging level *should* tell you what upstart thinks killed the process17:46
SpamapSjdsanders: would you mind pasting what you see? youre pre-stop, btw, is the right way to go17:47
jdsandersSpamapS: one moment, going back up my terminal....17:50
jdsandersshoot can't reproduce!17:58
jdsandersah, here17:59
jdsandersmaybe a race condition actually...17:59
jdsandersdeploy@NCVMLITSDEVQUE01:/var/www/apps/smartgrid/current/apollo$ sudo start redis-server17:59
jdsandersredis-server start/running, process 953517:59
jdsandersdeploy@NCVMLITSDEVQUE01:/var/www/apps/smartgrid/current/apollo$ sudo status redis-server17:59
jdsandersredis-server start/running, process 953517:59
jdsandersdeploy@NCVMLITSDEVQUE01:/var/www/apps/smartgrid/current/apollo$ sudo stop redis-server17:59
jdsandersredis-server start/running, process 962517:59
jdsandersdeploy@NCVMLITSDEVQUE01:/var/www/apps/smartgrid/current/apollo$ sudo status redis-server17:59
jdsandersredis-server start/running, process 962517:59
jdsandersSpamapS: ^^18:00
jdsandersSpamapS: here's the config script - https://gist.github.com/bf5b600645ff1ddc67cf18:01
SpamapSjdsanders: looking18:10
jdsandersSpamapS: just got back from a meeting, any luck understanding that redis-server issue?19:13
SpamapSjdsanders: oops no I got sdistracted. Looking again19:14
SpamapSjdsanders: I'd need to see your syslog entries to understand what happened19:15
jdsandersk, do you know what label to look for in syslog?19:17
jdsandersdon't see anything like "upstart"19:17
jdsanderslooks like "init" maybe19:18
jdsandersJul 11 15:18:08 NCVMLITSDEVQUE01 init: redis-server pre-stop process (9765) terminated with status 119:19
jdsandersseems problematic maybe19:19
jdsandershmmm, yeah it seems like my shutdown command is *expected* to have exit code 119:29
jdsanders(oddly enough)19:29
jdsandersso i think that's causing the problem?19:30
JanCby default it will consider exit != 0 as an error20:19
JanCso just override that if you want to ignore that20:20
jdsandersJanC: thanks, I'm now fairly certain my server is exiting with 0 when it shuts down, and I've modified the pre-stop to make sure it exits 020:20
jdsandersall I see in syslog is now basically "process died, respawning"20:20
jdsandersis there some way I can see more info about why it decided to respawn?20:20
JanCmaybe use a script instead of the main binary20:22
JanCand make it log whatever seems relevant20:23
jdsandersok thanks20:28
SpamapSjdsanders: you may also want to try 'initctl log-priority info' to get more about why it died22:12
SpamapSjdsanders: I suspect just setting 'normal exit' properly is all you need22:12
jdsandersSpamapS: i'll try the log level thing, but the reason I think the exit code is 0 is, I ran the same command that I'm executing from upstart in the foreground22:32
jdsandersthen shut it down from another process22:32
jdsandersthen checked its exit status22:32
jdsandersand it was 022:32
SpamapSjdsanders: right, so I am wondering what upstart thinks the exit code is22:33
SpamapSjdsanders: you might also just try 'normal exit 0'22:33
SpamapSI thought that was the default22:33
SpamapSbut man 5 init isn't entirely crystal clear on it IIRC22:33
SpamapS"Tasks may exit with a zero exit status to prevent being respawned."22:34
SpamapSI believe that means jobs may not by default consider 0 a normal exit22:34
SpamapS" All22:34
SpamapS              reasons  for  a  service stopping, except the stop(8) command itself, are considered abnormal."22:34
SpamapSjdsanders: yeah, try 'normal exit 0'22:34
jdsandershuh22:37
jdsandersi'll try that!22:37
SpamapSjdsanders: what that means is that upstart won't be able to keep redis running in the face of accidental shutdowns. But thats probably fine.22:39
jdsanderswell22:40
jdsandersI haven't tested this22:40
jdsandersbut my theory is that any non-graceful shutdown will exit with a different exit code than 022:40
SpamapSit should22:40
jdsandersso I think if somebody *wants* to shut it down with the command that is meant for precisely that, it is reasonable to expect that it doesn't respawn22:41
SpamapSjdsanders: you're still in better shape than with a bare daemon.22:41
jdsandersalright, I think I've got this now, thanks for all the help!22:54
jdsandersSpamapS,JanC: ^^22:54
SpamapSjdsanders: woot!22:54
jdsanderswoot!22:55
jdsandersalright...I hate to do this, but here's another one22:56
jdsandersI'm on ubuntu 10.0422:56
jdsanderswhich has upstart 0.6.522:56
jdsanderswhich doesn't support setuid22:56
jdsandersbut I need to run my server as another user22:56
jdsandersso I'm doing22:57
jdsanderssudo -u $USER sh -c "/usr/local/bin/redis-server /etc/redis-server.conf 2>&1 >> /var/log/redis/server.log"22:57
jdsandersproblem is that it makes the fork count jankier22:57
jdsanderswhen I start the server it actually spawns three processes - one for sudo, one for sh, and one for redis-server22:57
jdsandersthe pid i'd really like to track is the redis-server one22:58
jdsanders(which isn't doing any forking)22:58
jdsandersany better way to change users?22:58
SpamapSjdsanders: sudo is one way23:00
SpamapSjdsanders: better is start-stop-daemon23:00
jdsandersah23:00
jdsandersok23:00
jdsandersI'll look into that23:00
SpamapShttp://upstart.ubuntu.com/cookbook/#changing-user23:00
SpamapSjdsanders: ^^23:01
jdsandersawesome, thank you23:01
jdsanderssorta related: how do i get out of the stop/killed state23:01
jdsandersI currently have: redis-server start/killed, process 2067423:08
jdsandersbut: sudo kill -0 20674; echo $?23:08
jdsanderskill: No such process23:08
jdsanders123:08
jdsanderssudo stop redis-server seems to hang23:08
SpamapSjdsanders: ugh did you try 'expect fork' at one point?23:09
jdsandersi did23:09
jdsandersi was playing with it23:09
SpamapSOk, thats a bug23:09
jdsandersand  seem to have foobared it right up23:09
SpamapShttp://paste.ubuntu.com/1087042/23:09
SpamapSthis script loops over the pid space until it reaches the pid given23:10
SpamapSjdsanders: its python 3.. might have to port it to python2 ;)23:10
jdsandersso the theory here is that it's running somewhere, but i don't know where?23:10
SpamapSno23:10
SpamapSits not running23:10
SpamapSupstart thinks it should be23:10
SpamapSand will wait forever for it to die23:10
jdsandersso how does that script help?23:11
SpamapSit creates processes until the given process exists.. then lets upstart kill it23:11
jdsandershaha oooooooooooh, i get it23:11
jdsandersclever23:11
SpamapSnot really, its a hack on hack on hack23:11
jdsandersis there any way to re-initialize upstart?23:11
jdsandersI could always just reboot23:11
SpamapSjdsanders: yes but then it forgets all job state23:11
jdsanderswell yeah, hacks are clever23:11
SpamapSyeah reboot works if you're just noodling around23:11
SpamapSjdsanders: the original fix was in ruby and about 10 lines of it at that.. this one is just more friendly23:12
jdsanderswhat do you mean by "forgets all job state"23:12
SpamapSjdsanders: you can send upstart a command to re-execute itself, but then it forgets what is running and what is not23:12
SpamapSwhich means your shutdown does not go so smoothly23:12
jdsandershmmm23:15
jdsandersok i'll figure it out23:15

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!