| glenn___ | spamaps: ive been trying out all the options, expect fork, expect daemon, expect stop... even tried no expect | 09:38 |
|---|---|---|
| jodh | glenn___: you need to know how many times the app forks initially before it is ready to start its real work (in other words its initialisation phase). | 09:39 |
| glenn___ | jodh: i tried the count on fork/clones... it is about 22 | 09:41 |
| glenn___ | but the strace is showing clones, not forks | 09:41 |
| jodh | glenn___: the maximum any process needs to fork is twice before it is fully initialized. | 09:42 |
| glenn___ | jodh: so the guys cant code at puppetlabs? :) | 09:42 |
| jodh | glenn___: any further forks are for setting up "helpers". You don't care about those though - you need the pid of the master parent process before it does its fork bomb. | 09:42 |
| glenn___ | jodh: the master process is about 22 pids later | 09:43 |
| jodh | glenn___: it can't be. | 09:43 |
| glenn___ | ill show you | 09:43 |
| glenn___ | root@puppetclient:/etc/init# start puppet | 09:46 |
| glenn___ | puppet start/running, process 1142 | 09:46 |
| glenn___ | root@puppetclient:/etc/init# ps auxf |grep puppet | 09:46 |
| glenn___ | root 1167 0.0 0.0 9744 876 pts/0 S+ 10:46 0:00 \_ grep --color=auto puppet | 09:46 |
| glenn___ | root 1163 7.0 3.9 122540 40488 ? Ssl 10:46 0:00 /usr/bin/ruby1.8 /usr/bin/puppet agent | 09:46 |
| glenn___ | root@puppetclient:/etc/init# | 09:46 |
| glenn___ | i can use the init script to shut it down, not with upstart | 09:47 |
| glenn___ | init script is using the pid file | 09:48 |
| glenn___ | root@puppetclient:/etc/init# grep clone /tmp/strace.log | wc -l | 09:49 |
| glenn___ | 24 | 09:49 |
| glenn___ | 1217 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f6ee51809f0) = 1218 | 09:50 |
| glenn___ | etc etc | 09:51 |
| glenn___ | 1217 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f6ee51809f0) = 1238 | 09:51 |
| glenn___ | it behave the same for version 0.25 2.6 and 2.7 | 09:51 |
| glenn___ | the expect options go to a max of 2, i think with daemon | 09:52 |
| glenn___ | so either upstart isnt the right tool, their daemon is weird, or i just dont get it | 09:53 |
| glenn___ | ive been trying for at least an hour :) | 09:53 |
| glenn___ | fyi, the daemon is just a ruby script, starting puppet from ruby | 09:59 |
| glenn___ | i suppose thats causing the problem | 10:00 |
| jodh | glenn___: as stated, there is no reason to fork more than twice (that's why upstart only tracks up to 2 forks). I think you should talk to the puppet guys to understand what is going on there. For now, you might need to consider using start-stop-daemon with upstart: http://upstart.ubuntu.com/cookbook/#alternative-method | 10:08 |
| glenn___ | jodh: i think i picked the wrong software to test upstart :) | 10:10 |
| jodh | glenn___: this isn't a problem with upstart - it's either ruby and/or puppet. | 10:11 |
| glenn___ | jodh: hence my point :) i picked the wrong software <- puppet | 10:11 |
| jodh | glenn___: are you using "respawn" by any chance? | 10:14 |
| glenn___ | jodh: if i use any expect it all start will hang | 10:15 |
| glenn___ | s/it/at | 10:15 |
| jodh | glenn___: but are you using "respawn"? | 10:15 |
| glenn___ | jodh: nope | 10:15 |
| glenn___ | that started the process several times | 10:16 |
| glenn___ | :) | 10:16 |
| glenn___ | jodh: sorry i misread i thought u said expect instead of respawn | 10:16 |
| jodh | glenn___: ok. please keep us posted on this as we need to understand exactly what is going on here. | 10:17 |
| glenn___ | jodh: im trying to get it to work with start-stop-daemon first, then ill try the puppetlabs guys | 10:17 |
| glenn___ | which doesnt work at all :( | 10:19 |
| glenn___ | exec start-stop-daemon --start --exec /usr/bin/puppet agent | 10:19 |
| glenn___ | i did a reboot though, now its starting | 10:23 |
| glenn___ | still the same issue though, seems to make no difference | 10:23 |
| jodh | glenn___: what problem? why are you rebooting? I'd get puppet working with start-stop-daemon on the command-line first, then put that into an upstart .conf file. | 10:25 |
| glenn___ | jodh: it is working on the commandline | 10:25 |
| glenn___ | jodh: i tried start puppet, it hangs, i reboot, i start it again and it works | 10:25 |
| glenn___ | jodh: i tried several reboots :) the upstart wass messed up | 10:26 |
| glenn___ | sometimes it even keeps track of the pid while that id is not even there anymore | 10:26 |
| jodh | glenn___: that's telling you your .conf file is incorrect - upstart just does what you tell it to :) | 10:27 |
| glenn___ | there is just a few lines | 10:27 |
| glenn___ | to be exact 4 | 10:27 |
| glenn___ | description "puppet client" | 10:28 |
| glenn___ | start on filesystem or runlevel [2345] | 10:28 |
| glenn___ | stop on runlevel [!2345] | 10:28 |
| glenn___ | exec start-stop-daemon --start --exec /usr/bin/puppet agent | 10:28 |
| glenn___ | also when i try stop, it says unknown instance, which would mean the job is not running, while it is... | 10:31 |
| glenn___ | need more spresso | 10:31 |
| jodh | glenn___: your 'start on' is incorrect - that will try to start *two* instances of the job (which will fail). | 10:32 |
| jodh | glenn___: make it 'start on runlevel [2345]' | 10:32 |
| glenn___ | same behaviour | 10:35 |
| glenn___ | i dont think this will work with upstart :( | 10:37 |
| glenn___ | for now | 10:37 |
| glenn___ | i have an idea perhaps it will help | 10:43 |
| === glenn___ is now known as glenn | ||
| jodh | glenn___: glenn: 'exec start-stop-daemon --start --exec /usr/bin/ruby /tmp/fork.rb' works as expected for me. | 11:05 |
| glenn | jodh: thanks for checking | 11:07 |
| glenn | jodh: it just does 1 fork i suppose? | 11:07 |
| glenn | jodh: could show me the contents of that ruby file | 11:19 |
| jodh | glenn: http://paste.ubuntu.com/816328/ | 11:20 |
| glenn | jodh: yup seems to work correct | 11:23 |
| glenn | jodh: when i include their code in the fork.rb script it forks 21 times | 11:50 |
| jodh | glenn: maybe you should raise a bug on that - sounds broken. | 11:51 |
| glenn | this is kind of nice | 12:17 |
| glenn | there is a flag called --no-daemonize, with that it works as expected | 12:18 |
| jodh | glenn: great. please raise the issue with the puppet folk though as it sounds like there is a bug in there. | 12:20 |
| glenn | im staring at their ruby code for half an hour now :) maybe ur right | 12:20 |
| glenn | interesting material though | 12:27 |
| glenn | they are using Process.detach | 12:28 |
| glenn | lol :) | 12:30 |
| glenn | jodh: jodh: im pretty sure its something weird in their code | 13:20 |
| glenn | http://paste.ubuntu.com/816410/ http://paste.ubuntu.com/816411/ | 13:22 |
| jodh | 'music | 13:23 |
| jodh | oops :) | 13:23 |
| glenn | music? :) | 13:23 |
| glenn | jodh: i appreciate ur help btw | 13:23 |
| jodh | ignore - tmux window searching :) | 13:23 |
| glenn | learning a lot regarding upstart | 13:23 |
| glenn | should just use spottify :) | 13:23 |
| jodh | glenn: glad your'e making progress. afk for a while ... | 13:24 |
| glenn | jodh: im filing a bug at puppetlabs right now after discussing it with someone in their irc-channel | 15:03 |
| jodh | glenn: great news! | 15:09 |
| glenn | jodh: Yup :) | 15:10 |
| glenn | jodh: im not exactly sure what to file but hey :P | 15:10 |
| glenn | its something with redmine facter within puppet | 15:10 |
| glenn | its doing system calls to which, ifconfig, hostname things like that | 15:10 |
| glenn | that seems to spawn the forks | 15:11 |
| glenn | which seems correct because in the strace there is exec's to that | 15:11 |
| glenn | jodh: and i probably have a workaround for now to upstart | 15:12 |
| glenn | woot :) | 15:14 |
| glenn | root@puppetclient:~# start puppet | 15:14 |
| glenn | puppet start/running, process 1806 | 15:14 |
| glenn | root@puppetclient:~# stop puppet | 15:14 |
| glenn | puppet stop/waiting | 15:14 |
| glenn | nice | 15:14 |
| glenn | :)_ | 15:14 |
| glenn | jodh: for my understanding, if software is daemonized, this should take no more then 2 forks? | 15:47 |
| jodh | glenn: there is no additional benefit to forking more than twice. | 15:47 |
| glenn | jodh: could you explain that a bit more | 15:47 |
| jodh | glenn: not right now (in a meeting). it's standard unix/linux semantics though. | 15:48 |
| SpamapS_ | glenn: I'm curious what your workaround was | 16:35 |
| glenn | spamaps: ah :) ill paste my script | 16:36 |
| glenn | spamaps: im kind of confused still, regarding the max of 2 forks in upstart | 16:36 |
| === SpamapS_ is now known as SpamapS | ||
| glenn | spamaps: im using the --no-daemonize function for puppet | 16:36 |
| SpamapS | glenn: puppet may be an example of something that reasonably *does* fork twice before daemonization for legitimate reasons.. I think this may be worth a bug report in upstart | 16:37 |
| SpamapS | Why shouldn't a program be able to exec stuff before it daemonizes itself? | 16:37 |
| glenn | spamaps; thats the same disucssion i had with our developers | 16:37 |
| SpamapS | jodh: ^^ Perhaps 'expect exit' will fix this? | 16:38 |
| glenn | spamaps: maybe it should track only forks of itself, instead of all other calls | 16:38 |
| glenn | spamaps: upstart seems to track all pid's instead of just the forking of the daemon itself | 16:38 |
| SpamapS | glenn: well its trying to figure out what the "main" pid is | 16:39 |
| glenn | spamaps: i understand | 16:39 |
| SpamapS | The method it uses works for most regular daemons. | 16:39 |
| glenn | spamaps: but in the puppet case, it does about 20 system calls, and then it has all the information for starting the daemon (dont ask me why) | 16:40 |
| SpamapS | glenn: I think I understand why.. and this is an example of where expect fork is just incapable of doing it right | 16:40 |
| SpamapS | So --no-daemonize is the only other option.. though then you have the trouble of needing to write a post-start which determines when the daemon is actually ready | 16:41 |
| glenn | its working properly already | 16:42 |
| glenn | when puppet is not daemonized it will run in the foreground, and thus not doing forks to system calls | 16:42 |
| glenn | or however i should pronounche this :) | 16:42 |
| SpamapS | glenn: perhaps puppet should add a --upstart mode where it sends itself SIGSTOP when it is ready (upstart will SIGCONT it) | 16:42 |
| SpamapS | glenn: then you could use 'expect stop' | 16:42 |
| glenn | spamaps: yeah that would be a solution | 16:42 |
| SpamapS | glenn: you're saying system calls, but what you mean is it is forking and execing system programs, I assume | 16:43 |
| glenn | spamaps: i think so yes :) | 16:43 |
| glenn | spamaps: correct me if im wrong, this is kind of new genre too me | 16:43 |
| glenn | the paste right | 16:43 |
| SpamapS | glenn: that should be valid and expected behavior for lots of daemons, not just puppet. | 16:44 |
| glenn | spamaps: but its not really nice of upstart to say: build in an upstart function :) | 16:45 |
| glenn | else we dont support you | 16:45 |
| glenn | the problem exists because upstart wants to track the pid itself | 16:46 |
| glenn | my mind is boggling | 16:46 |
| glenn | http://paste.ubuntu.com/816657/ | 16:50 |
| glenn | there this is one works | 16:50 |
| SpamapS | glenn: well its more of upstart saying "help us determine your pid" | 16:50 |
| glenn | spamaps: puppet creates a pid file already | 16:50 |
| glenn | spamaps: why would upstart need to find it again? :) | 16:50 |
| SpamapS | glenn: IMO, upstart should have the ability to use pidfiles, but Keybuk is (was?) convinced that pidfiles are too unreliable. | 16:50 |
| glenn | spamaps: upstart is more unreliable | 16:51 |
| glenn | :) | 16:51 |
| SpamapS | Not sure I agree with that | 16:51 |
| glenn | spamaps: concerning starting software | 16:51 |
| SpamapS | but I can see how it might feel that way given the difficulty in writing jobs | 16:51 |
| glenn | i was reading the cookbook | 16:51 |
| glenn | it seems really awesome | 16:51 |
| glenn | a new init | 16:51 |
| SpamapS | glenn: upstart is actually quite reliable.. its just that it reliably does weird things. :) | 16:51 |
| glenn | its fast, parallel et | 16:51 |
| glenn | but the pid tracking, is really weird | 16:52 |
| glenn | i did some reboots, and even 2 reinstalls (through auto deployment) | 16:52 |
| glenn | it was tracking pid's that didnt exist | 16:52 |
| SpamapS | using ptrace the way it does is the only thing I don't like | 16:52 |
| glenn | even restarting upstart didnt help | 16:52 |
| SpamapS | glenn: the tracking pids that don't exist thing is a known bug | 16:52 |
| SpamapS | glenn: happens whenever you use expect fork on something that doesn't fork | 16:53 |
| SpamapS | glenn: or.. something like that, I forget. Anyway, there's a workaround that doesn't require a reboot.. but it does require you to exhaust pid space and roll-over | 16:53 |
| SpamapS | glenn: anyway, the --no-daemonize option is still the simplest. But again, if you have other things that might depend on it running.. you need to have a post-start that verifies it is ready to serve their requests. | 16:54 |
| SpamapS | also the --no-daemonize thing fails on programs that fork/exec/exit on SIGHUP .. which I just figured out python-paste does. | 16:54 |
| glenn | spamaps: ah ok, i tried the expect option, that probably created the issue | 16:56 |
| glenn | spamaps: spamaps: --no-daemonize is a flag specific for puppet | 17:01 |
| SpamapS | glenn: right, its pretty common for daemons to have a "stay in the foreground" flag tho | 17:03 |
| glenn | spamaps: still, the pid tracking seems a bit strange | 17:10 |
| glenn | spamaps: i can imagine a lot of software does exec system programs before it starts | 17:11 |
| glenn | + as daemon | 17:11 |
| SpamapS | glenn: when daemons behave the way upstart expects them to, its quite nice. But I estimate thats only about 50% of daemons that I've tried to write jobs for | 17:11 |
| glenn | spamaps: well if upstart wants to be ground breaking, either a pid file support should be available, or software developers need to code in sigstop | 17:12 |
| glenn | spamaps: i dont think there is a common practice for this when developing software | 17:12 |
| glenn | spamaps: the only thing i could think of is that there should be a pid file | 17:13 |
| glenn | spamaps: which there is, but upstart wont support it | 17:13 |
| glenn | spamaps: also, software like puppet is supported on redhat and not on ubuntu | 17:14 |
| SpamapS | glenn: thats just crazy talk. puppet is very well supported on Ubuntu | 17:14 |
| SpamapS | glenn: upstart's pid tracking is not necessary to use services. | 17:14 |
| SpamapS | glenn: you can have an upstart job with no pid, that just tracks whether or not the boot/shutdown thinks it should be started/stopped | 17:15 |
| glenn | spamaps: the enterprise version is indeed supporting ubuntu now, sorry for that | 17:15 |
| SpamapS | and puppet is in main | 17:16 |
| SpamapS | so it is supported by Canonical | 17:16 |
| glenn | spamaps: i had the newer puppet backported to lucid | 17:16 |
| glenn | from onerice | 17:16 |
| glenn | -e | 17:16 |
| SpamapS | I think we (Ubuntu) should backport more stuff to LTS's | 17:16 |
| glenn | spamaps: i had some great help from people at ubuntu getting this to work | 17:17 |
| glenn | took about 1 month i guess | 17:17 |
| glenn | now 2.7.1 is in lucid backports | 17:17 |
| SpamapS | glenn: thank you for working on backports then!! | 17:17 |
| glenn | lol np | 17:17 |
| SpamapS | I hope to have some official Canonical resources working on backports going forward | 17:17 |
| SpamapS | at least, for server | 17:17 |
| glenn | that would be nice | 17:19 |
| glenn | i did some cool stuff to debian preseed too :) | 17:19 |
| glenn | but that was just on my side, not on the preseed package | 17:19 |
| trondm | can an upstart job tell the difference between a restart and stop? i.e. can I make my job do something else than a simple stop/start during a restart? | 17:39 |
| SpamapS | trondm: no | 17:41 |
| SpamapS | trondm: restart simply changes the goal to stop, then back to start | 17:42 |
| SpamapS | trondm: its a dangerously misunderstood command.. it does not re-load the upstart job. | 17:42 |
| trondm | Ah. Too bad. | 17:42 |
| trondm | In my case, I have a parent job than spawns several instances. I'd like each instance to be restarted separately when the parent is restarted, instead of having them all be stopped, then started | 17:43 |
| glenn | trondm: it sounds the same, restart them all, or stop them all and start them all | 17:44 |
| trondm | It's not a huge problem, though (and this is the way it worked before I started using upstart) | 17:45 |
| trondm | (I was just hoping it would be possible to improve on the current state) | 17:47 |
| glenn | trondm: is there is difference when you restart your instance or stop/start your instance? | 17:48 |
| trondm | glenn: there's a difference between stop/start 0; stop/start 1; stop/start 2; and stop 0 1 2; start 0 1 2. Basically I'm offline for a shorter period if everything stops, then starts. | 17:49 |
| glenn | as far as i knew restart == stop -> start | 17:49 |
| glenn | oh ic | 17:49 |
| glenn | maybe you can create several jobs? | 17:49 |
| glenn | or would that do the same | 17:50 |
| trondm | I think that would do the same. Part of the problem is that the parent is an instance itself, and it's not possible to use variables in the "stop on" stanza. So I have a "post-stop" script in the parent that stops all child-instances | 17:54 |
| glenn | hey spamaps you know what would be nice | 17:58 |
| glenn | the documentation regarding expect doesnt mention what to do if there is more then 2 pid forks | 17:59 |
| glenn | i.e. you should change the software, or use sigstop | 18:01 |
| glenn | jodh: spamaps: http://projects.puppetlabs.com/issues/12146 | 18:09 |
| glenn | i just made this issue | 18:09 |
| glenn | hopefully it makes sense | 18:09 |
| glenn | thanks a bunch again for your support/help really appreciate it | 18:20 |
| glenn | im off, gonna play starcraft 2 | 18:20 |
| glenn | yo | 18:20 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!