[15:15] <Md> I am trying to debug an ubuntu 10.04 system which hangs at boot. If look at what upstart is doing in a different console I see that it started a few processes but I am left with only plymouth running
[15:15] <Md> what can I do to find out what's wrong?
[15:25] <jodh> Md: take a look at http://upstart.ubuntu.com/cookbook/#debugging.
[15:25] <jodh> Md: first, I'd try booting with --debug on the kernel command-line and capturing the log output to see if anything looks suspicious.
[15:25] <jodh> Md: have you installed any new jobs recently or changed any default jobs maybe?
[15:25] <Md> jodh: this is a customer server and I have no idea if and how they broke it
[15:26] <Md> the problem does not appear to be a job hanging, unless it's plymouth itself
[15:26] <Md> I already checked the log with --verbose
[15:26] <jodh> Md: do you know if this server has ever booted successfully?
[15:27] <Md> jodh: yes, it was working until it was rebooted (or crashed, who knows)
[15:28] <Md> I can see that the only jobs started are plymouth, hwclock, ureadahead and hostname
[15:28] <Md> the only possibly suspicious message is two "ignored event 40 (1) for process ..." messages
[15:29] <Md> where the process is plymouth
[15:35] <jodh> Md: this will list missing standard jobs: http://paste.ubuntu.com/5798845/
[15:35] <jodh> Md: this will list modified jobs: http://paste.ubuntu.com/5798853/
[15:36] <jodh> Md: look at the newest files in /etc/init/ and try to identify any new jobs which might be causing problems.
[15:37] <whg> Is there any sort of "best practice" around managing Python processes inside a virtualenv?
[15:44] <Md> there are no new files
[15:45] <whg> Can I source scripts in shell script I'm running under upstart?
[15:45] <Md> no missing files, no modified files
[15:46] <jodh> whg: I suggest you read the Cookbook: http://upstart.ubuntu.com/cookbook/#sourcing-files
[15:46] <whg> Dammit
[15:46] <whg> I swear I searched the cookbook for that
[15:46] <whg> Thanks for the link.
[15:46] <jodh> whg: np.
[15:47] <jodh> Md: any new jobs? If not, has /etc/fstab been modified? are all expected disks mounted?
[15:47] <Md> no new jobs, only one disk
[15:49] <jodh> Md: I think we'll need to see the output from --verbose/--debug then.
[15:57] <Md> jodh: http://www.bofh.it/~md/u/
[16:13] <jodh> Md: looks like mountall might be the problem. Look for any odd lines/options in /etc/fstab.
[16:15] <Md> there is nothing unusual, the file is trivial
[16:15] <Md> if I run "initctl start upstart" it mounts / rw
[16:16] <Md> I mean "initctl start udev"
[16:16] <jodh> Md: well, from the files you've provided, it doesn't appear that mountall gets as far as spawning.
[16:16] <jodh> Md: mountall is supposed to do that
[16:22] <Md> yes, so what should start mountall?
[16:22] <Md> if I try to start it manually I get a "Job failed to start" error
[16:24] <Md> got it
[16:24] <Md> the problema was a syntax error in /etc/default/locale, which was over one year old
[16:36] <Md> can you make sure that a || true is added to mountall in the next release? :-)
[16:45] <JanC> Md: you mean when sourcing /etc/default/locale ?
[16:46] <Md> JanC: yes
[16:46] <Md> there are still some issues BTW, only some services have been started
[16:47] <Md> I can start them manually, but I have no clue why upstart is not starting them
[16:48] <JanC> considering that that file might get changed on many systems, and admins are humans, so make errors, maybe that wouldn't be a bad idea  :)
[16:48] <Md> nevermind, this one looks like a fuckup by a coworker
[16:58] <slangasek> Md: ah, that bug was just recently reported, https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1192514
[16:58] <slangasek> wonder why everyone has invalid /etc/default/locale all of a sudden :P
[16:59] <Md> there was no error message either at boot time or when starting mountall manuall BTW
[17:02] <slangasek> yes; job output handling only became sensible much later (13.04, I think?)
[18:03] <JanC> maybe upstart needs a library of functions that provides "best practice" methods for common tasks in upstart jobs...
[18:05] <JanC> (that also means they can be fixed centrally if they aren't really best practice ;) )
[18:05] <xnox> JanC: nah, ideally daemons/software is capable of running with just a single exec line.
[18:05] <xnox> JanC: without need of any scripts at all.
[18:06] <xnox> JanC: plus we have http://upstart.ubuntu.com/cookbook/ already.
[18:07] <xnox> JanC: problem with "helper" libraries is that they quickly gain feature creep trying to accommodate every use case.
[18:07] <JanC> xnox: ideally, all programs would be bugless  :p
[18:07] <xnox> JanC: please no, otherwise I'd be out of work :P
[18:08] <JanC> you can always write new bug-free software  :)
[18:13] <JanC> and to avoid too much feature creep, you just need a good gatekeeper  ;)