=== erkules_ is now known as erkules === soren_ is now known as soren [15:15] I am trying to debug an ubuntu 10.04 system which hangs at boot. If look at what upstart is doing in a different console I see that it started a few processes but I am left with only plymouth running [15:15] what can I do to find out what's wrong? [15:25] Md: take a look at http://upstart.ubuntu.com/cookbook/#debugging. [15:25] Md: first, I'd try booting with --debug on the kernel command-line and capturing the log output to see if anything looks suspicious. [15:25] Md: have you installed any new jobs recently or changed any default jobs maybe? [15:25] jodh: this is a customer server and I have no idea if and how they broke it [15:26] the problem does not appear to be a job hanging, unless it's plymouth itself [15:26] I already checked the log with --verbose [15:26] Md: do you know if this server has ever booted successfully? [15:27] jodh: yes, it was working until it was rebooted (or crashed, who knows) [15:28] I can see that the only jobs started are plymouth, hwclock, ureadahead and hostname [15:28] the only possibly suspicious message is two "ignored event 40 (1) for process ..." messages [15:29] where the process is plymouth [15:35] Md: this will list missing standard jobs: http://paste.ubuntu.com/5798845/ [15:35] Md: this will list modified jobs: http://paste.ubuntu.com/5798853/ [15:36] Md: look at the newest files in /etc/init/ and try to identify any new jobs which might be causing problems. [15:37] Is there any sort of "best practice" around managing Python processes inside a virtualenv? [15:44] there are no new files [15:45] Can I source scripts in shell script I'm running under upstart? [15:45] no missing files, no modified files [15:46] whg: I suggest you read the Cookbook: http://upstart.ubuntu.com/cookbook/#sourcing-files [15:46] Dammit [15:46] I swear I searched the cookbook for that [15:46] Thanks for the link. [15:46] whg: np. [15:47] Md: any new jobs? If not, has /etc/fstab been modified? are all expected disks mounted? [15:47] no new jobs, only one disk [15:49] Md: I think we'll need to see the output from --verbose/--debug then. [15:57] jodh: http://www.bofh.it/~md/u/ [16:13] Md: looks like mountall might be the problem. Look for any odd lines/options in /etc/fstab. [16:15] there is nothing unusual, the file is trivial [16:15] if I run "initctl start upstart" it mounts / rw [16:16] I mean "initctl start udev" [16:16] Md: well, from the files you've provided, it doesn't appear that mountall gets as far as spawning. [16:16] Md: mountall is supposed to do that === ion_ is now known as ion [16:22] yes, so what should start mountall? [16:22] if I try to start it manually I get a "Job failed to start" error [16:24] got it [16:24] the problema was a syntax error in /etc/default/locale, which was over one year old [16:36] can you make sure that a || true is added to mountall in the next release? :-) [16:45] Md: you mean when sourcing /etc/default/locale ? [16:46] JanC: yes [16:46] there are still some issues BTW, only some services have been started [16:47] I can start them manually, but I have no clue why upstart is not starting them [16:48] considering that that file might get changed on many systems, and admins are humans, so make errors, maybe that wouldn't be a bad idea :) [16:48] nevermind, this one looks like a fuckup by a coworker [16:58] Md: ah, that bug was just recently reported, https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1192514 [16:58] wonder why everyone has invalid /etc/default/locale all of a sudden :P [16:59] there was no error message either at boot time or when starting mountall manuall BTW [17:02] yes; job output handling only became sensible much later (13.04, I think?) [18:03] maybe upstart needs a library of functions that provides "best practice" methods for common tasks in upstart jobs... [18:05] (that also means they can be fixed centrally if they aren't really best practice ;) ) [18:05] JanC: nah, ideally daemons/software is capable of running with just a single exec line. [18:05] JanC: without need of any scripts at all. [18:06] JanC: plus we have http://upstart.ubuntu.com/cookbook/ already. [18:07] JanC: problem with "helper" libraries is that they quickly gain feature creep trying to accommodate every use case. [18:07] xnox: ideally, all programs would be bugless :p [18:07] JanC: please no, otherwise I'd be out of work :P [18:08] you can always write new bug-free software :) [18:13] and to avoid too much feature creep, you just need a good gatekeeper ;)