=== Md_ is now known as Md
[13:06] <rvgate> im creating a custom upstart script... when i say "service pentaho start" it tells me, job is already running... "service pentaho stop" hangs, and "initctl status pentaho" tells me pentaho is start/killed... what do i do now? :/
[13:07] <rvgate> i want to be able to do a clean start/stop again
[13:13] <SpamapS> rvgate: does 'status pentaho' show a pid that does not exist?
[13:14] <rvgate> SpamapS, yes
[13:16] <SpamapS> rvgate: ok, you have 'expect fork' set wrong
[13:16] <SpamapS> rvgate: its a known bug, if you set it wrong then the pid will go missing
[13:17] <SpamapS> rvgate: you have to exhaust pid space to release the job
[13:17] <SpamapS> rvgate: (or reboot)
[13:17] <rvgate> reboot it is
[13:17] <rvgate> dev machine anyway :P
[13:17] <SpamapS> https://bugs.launchpad.net/upstart/+bug/406397
[13:18] <SpamapS> http://heh.fi/tmp/workaround-upstart-snafu
[13:18] <SpamapS> has a script which does the pid space trick
[13:19] <SpamapS> rvgate: make sure before you reboot that you disable the start on for the job or it won't help right? :)
[13:19] <SpamapS> jodh: any progress on that bug btw?
[13:19] <SpamapS> slangasek: ^^
[13:19] <rvgate> i didnt even bother defining a start on yet, since i need to make sure it works first :P
[13:19] <SpamapS> IIRC there was some thought that it would go away if we moved away from ptrace
[13:21] <rvgate> SpamapS, while the machine reboots... could you explain the "exhaust pid space"... it went way over my head
[13:21] <jodh> SpamapS: a little - watch this space...
[13:23] <SpamapS> jodh: :)
[13:23] <SpamapS> rvgate: so when you fork(), you get highestpid+1
[13:24] <SpamapS> well.. actually its just a counter that goes up until you reach a free slot
[13:24] <SpamapS> rvgate: anyway, that counter wraps at 65535 .. so.. you have to keep fork()'ing until the missing pid is re-tried.. then upstart will be able to see that pid, reattach to it, and then watch it die
[13:25] <rvgate> haha
[13:25] <rvgate> so basicly the pid used gets lost
[13:25] <rvgate> unable to find it... unable to stop it.. and it hangs
[13:25] <SpamapS> There are a bunch of ways I'd love to fix it, but jodh and slangasek have told me "don't bother" for a while.. 
[13:32] <rvgate> got the script working now again, thanks for the help SpamapS 
[13:55] <slangasek> SpamapS: I don't remember the "don't bother"... I remember saying we should fix it right instead of extending initctl with an "oops" interface
[14:00] <SpamapS> slangasek: thats "don't bother" in my head anyway
[14:01] <slangasek> heh, well, I wouldn't mind help fixing it right ;)
[19:52] <SpamapS> jodh: If you're around.. was poking at the logging bits of upstart and wondering about what might happen if a system had way too many upstart processes actively logging....
[19:53] <SpamapS> jodh: like, say, 50 active loggers producing > 8K/s
[19:53] <SpamapS> slangasek: ^^ ?
[19:54] <jodh> SpamapS: if upstart is unable to create a new pty to handle the logging, it emits a error, then degrades the job to "console none" automatically.
[19:55] <SpamapS> jodh: pty's aren't the problem. Processes blocked on write() to stdout/stderr would be though.
[19:55] <jodh> SpamapS: but if you're talking about lots of jobs producing a ton of output, then you might have issues, yeah.
[19:55] <SpamapS> jodh: seems to me that the buffer size might need to be configurable.
[20:01] <jodh> SpamapS: are you talking about specifying a buffer size limit?
[20:03] <SpamapS> jodh: I'm talking about the buffer on the pty. Its just a default buffer size AFAICT.. and my brain says that is only 8K
[20:36] <jodh> SpamapS: kernel code suggests it is 64k, but you might want to check with apw on that. init caches as required to minimise the possibility of jobs blocking on write so I don't think there is much else we can do. About to log off but if you can provoke bad behaviour, please let us know.
[20:37] <SpamapS> 64k is much better than 8k
[20:39] <SpamapS> well, 8x better anyway *