| === Md_ is now known as Md | ||
| rvgate | im creating a custom upstart script... when i say "service pentaho start" it tells me, job is already running... "service pentaho stop" hangs, and "initctl status pentaho" tells me pentaho is start/killed... what do i do now? :/ | 13:06 |
|---|---|---|
| rvgate | i want to be able to do a clean start/stop again | 13:07 |
| SpamapS | rvgate: does 'status pentaho' show a pid that does not exist? | 13:13 |
| rvgate | SpamapS, yes | 13:14 |
| SpamapS | rvgate: ok, you have 'expect fork' set wrong | 13:16 |
| SpamapS | rvgate: its a known bug, if you set it wrong then the pid will go missing | 13:16 |
| SpamapS | rvgate: you have to exhaust pid space to release the job | 13:17 |
| SpamapS | rvgate: (or reboot) | 13:17 |
| rvgate | reboot it is | 13:17 |
| rvgate | dev machine anyway :P | 13:17 |
| SpamapS | https://bugs.launchpad.net/upstart/+bug/406397 | 13:17 |
| SpamapS | http://heh.fi/tmp/workaround-upstart-snafu | 13:18 |
| SpamapS | has a script which does the pid space trick | 13:18 |
| SpamapS | rvgate: make sure before you reboot that you disable the start on for the job or it won't help right? :) | 13:19 |
| SpamapS | jodh: any progress on that bug btw? | 13:19 |
| SpamapS | slangasek: ^^ | 13:19 |
| rvgate | i didnt even bother defining a start on yet, since i need to make sure it works first :P | 13:19 |
| SpamapS | IIRC there was some thought that it would go away if we moved away from ptrace | 13:19 |
| rvgate | SpamapS, while the machine reboots... could you explain the "exhaust pid space"... it went way over my head | 13:21 |
| jodh | SpamapS: a little - watch this space... | 13:21 |
| SpamapS | jodh: :) | 13:23 |
| SpamapS | rvgate: so when you fork(), you get highestpid+1 | 13:23 |
| SpamapS | well.. actually its just a counter that goes up until you reach a free slot | 13:24 |
| SpamapS | rvgate: anyway, that counter wraps at 65535 .. so.. you have to keep fork()'ing until the missing pid is re-tried.. then upstart will be able to see that pid, reattach to it, and then watch it die | 13:24 |
| rvgate | haha | 13:25 |
| rvgate | so basicly the pid used gets lost | 13:25 |
| rvgate | unable to find it... unable to stop it.. and it hangs | 13:25 |
| SpamapS | There are a bunch of ways I'd love to fix it, but jodh and slangasek have told me "don't bother" for a while.. | 13:25 |
| rvgate | got the script working now again, thanks for the help SpamapS | 13:32 |
| slangasek | SpamapS: I don't remember the "don't bother"... I remember saying we should fix it right instead of extending initctl with an "oops" interface | 13:55 |
| SpamapS | slangasek: thats "don't bother" in my head anyway | 14:00 |
| slangasek | heh, well, I wouldn't mind help fixing it right ;) | 14:01 |
| SpamapS | jodh: If you're around.. was poking at the logging bits of upstart and wondering about what might happen if a system had way too many upstart processes actively logging.... | 19:52 |
| SpamapS | jodh: like, say, 50 active loggers producing > 8K/s | 19:53 |
| SpamapS | slangasek: ^^ ? | 19:53 |
| jodh | SpamapS: if upstart is unable to create a new pty to handle the logging, it emits a error, then degrades the job to "console none" automatically. | 19:54 |
| SpamapS | jodh: pty's aren't the problem. Processes blocked on write() to stdout/stderr would be though. | 19:55 |
| jodh | SpamapS: but if you're talking about lots of jobs producing a ton of output, then you might have issues, yeah. | 19:55 |
| SpamapS | jodh: seems to me that the buffer size might need to be configurable. | 19:55 |
| jodh | SpamapS: are you talking about specifying a buffer size limit? | 20:01 |
| SpamapS | jodh: I'm talking about the buffer on the pty. Its just a default buffer size AFAICT.. and my brain says that is only 8K | 20:03 |
| jodh | SpamapS: kernel code suggests it is 64k, but you might want to check with apw on that. init caches as required to minimise the possibility of jobs blocking on write so I don't think there is much else we can do. About to log off but if you can provoke bad behaviour, please let us know. | 20:36 |
| SpamapS | 64k is much better than 8k | 20:37 |
| SpamapS | well, 8x better anyway * | 20:39 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!