[18:09] <SpamapS> Hrm.. jenkins forks and exits when you send it HUP
[18:09] <SpamapS> meaning the job appears to die, and upstart loses track of the pid
[18:10] <SpamapS> May have to drop pid tracking for it then. Bummer.
[22:56] <marrusl> what's a good late-in-boot-as-possible event normally?  runlevel 2?
[22:56] <marrusl> SpamapS, ^
[22:56] <SpamapS> Heh, we just had a big discussion about this yesterday
[22:57] <SpamapS> there isn't a good one
[22:57] <marrusl> :)
[22:57] <SpamapS> the best you have is 'stopped rc RUNLEVEL=[2345]'
[22:57] <SpamapS> Which is when tty1 pops up
[22:58] <marrusl> SpamapS, brings to mind the famous question...  what's your sleep(1) number?
[22:58] <SpamapS> marrusl: we need an intermediate job for mid-boot things that you want to track the state of. this goes back to the 'network-services' job.
[22:58] <SpamapS> lol
[22:58] <SpamapS> marrusl: what class of things do you want to be after?
[22:59] <SpamapS> If its "everything" .. we're probably going to transition nearly everything that is 'start on runlevel [2345]' to be 'start on starting network-services'
[22:59] <SpamapS> and then if you want to be "after that stuff" you will be 'start on started network-services'
[22:59] <marrusl> SpamapS, in this case networking indeed.
[22:59] <marrusl> and it's on lucid.  :-/
[22:59] <SpamapS> marrusl: for networking, runlevel 2 is good 11.10 and later
[23:00] <SpamapS> marrusl: for lucid, your best bet is to delay runlevel 2 for all of the network devices
[23:00] <SpamapS> I wonder..
[23:00] <SpamapS> perhaps we should backport the static-network-up event
[23:00] <SpamapS> not the change to /etc/init/rc.conf
[23:01] <marrusl> SpamapS, basically so far your idea to use "start on started udev" seems ok, but since the problem condition is so rare (~3-5% of boots) we have to script checking for the problem conditions...
[23:01] <marrusl> and if none, reboot.
[23:02] <marrusl> but when I had an invalid upstart job starting infiniband (well failing to start)
[23:03] <marrusl> the script didn't catch the error condition and kept rebooting. 
[23:04] <SpamapS> marrusl: burn-in test :)
[23:04] <marrusl> SpamapS, haha.  basically yup.