[09:27] <AlexExtreme> hey Keybuk, i'm having a few problems with my set of jobs I'm writing :/
[09:27] <Keybuk> oh, what's up?
[09:28] <AlexExtreme> i've got the mount-kernel-filesystems job which runs on startup, that runs ok
[09:28] <AlexExtreme> then there are jobs with 'start on stopped mount-kernel-filesystems ok'
[09:28] <AlexExtreme> none of those are run
[09:28] <Keybuk> sounds like mount-kernel-filesystems is failing
[09:28] <Keybuk> did you try --debug?
[09:28] <AlexExtreme> doh, there's a debug option
[09:29] <AlexExtreme> it also doesn't work if I omit the ok part
[09:29] <Keybuk> definitely sounds like it's failing
[09:30] <AlexExtreme> hmm
[09:30] <Keybuk> (need to get that information into "initctl jobs" somehow)
[09:31] <Keybuk> it shows up in initctl events
[09:31] <AlexExtreme> it seems to be hanging.... the last command in the mount-kernel-filesystems script is an echo which I put there for debugging, upstart gives no debug output after that
[09:31] <Keybuk> doubt it's hanging
[09:31] <AlexExtreme> so it handles the starting event, and it's now in the started state
[09:31] <AlexExtreme> nothing happens after that
[09:32] <Keybuk> I usually have jobs like "exec openvt -c 2 initctl jobs" and "exec openvt -c 3 initctl events"
[09:32] <Keybuk> run on startup
[09:32] <Keybuk> they're handy for debugging as well
[09:33] <AlexExtreme> http://frugalware.org/paste/886 << that's the job, can you see any problems with it?
[09:34] <Keybuk> not offhand
[09:35] <Keybuk> oh
[09:35] <Keybuk> sorry
[09:35] <Keybuk> it *doesn't* work if you omit "ok"
[09:35] <Keybuk> that's interesting
[09:35] <AlexExtreme> yeah
[09:36] <Keybuk> grep for mount-kernel-filesystems in other jobs
[09:36] <AlexExtreme> http://frugalware.org/paste/887
[09:37] <Keybuk> hmm
[09:38] <Keybuk> nothing blocking there
[09:38] <Keybuk> what does init do with --debug?
[09:38] <AlexExtreme> hold on, i'll take a screenshot
[09:39] <AlexExtreme> http://www.alex-smith.me.uk/files/upstartdebug.png
[09:40] <Keybuk> interesting
[09:40] <AlexExtreme> the "End of kernel fses" is an echo at the end of the job
[09:40] <AlexExtreme> last command in the job, should exit ok
[09:41] <Keybuk> it doesn't look like it's reaped the child
[09:41] <Keybuk> this is in vmware?
[09:41] <Keybuk> oh, qemu
[09:41] <Keybuk> ok
[09:41] <Keybuk> add a shell
[09:42] <Keybuk> start on startup
[09:42] <Keybuk> exec openvt -c 2 bash
[09:42] <Keybuk> -- 
[09:42] <AlexExtreme> k, hold on
[09:42] <Keybuk> then change m-k-f to be started when that's started (start on started shell)
[09:42] <Keybuk> you should then be able to Alt+F2 and see what's up
[09:43] <Keybuk> also would be useful:  init=/usr/bin/strace init
[09:43] <AlexExtreme> that's weird...
[09:44] <AlexExtreme> it won't start the shell
[09:44] <AlexExtreme> i think it's because i'm booting qemu with -kernel, -append, etc without using a real bootloader inside it
[09:46] <AlexExtreme> i'll try a real install, give me a few mins
[09:47] <Keybuk> that shouldn't matter?
[09:47] <AlexExtreme> no, it messes with the consoles i think, because openvt fails
[09:50] <Keybuk> odd
[10:07] <Keybuk> (at least you're not trying to run it in openvz :p)
[10:08] <AlexExtreme> :)
[10:09] <Keybuk> still not sure what *that* bug is
[10:09] <Keybuk> I strongly suspect openvz doesn't give init pid #1 :p
[10:09] <AlexExtreme> hmm
[10:10] <AlexExtreme> does openvt require the device node for the vt you're trying to open, by any chance?
[10:11] <Keybuk> oh, could do
[10:11] <Keybuk> heh
[10:11] <AlexExtreme> udev hasn't run yet ;)
[10:11] <Keybuk> I always debug this stuff with an initramfs that takes care of that kind of thing
[10:12] <AlexExtreme> ah, well
[10:12] <AlexExtreme> frugalware has no initramfs (yet)
[10:12] <AlexExtreme> we're gonna add support for initramfs soon though
[10:13] <Keybuk> can you try the strace thing?
[10:14] <AlexExtreme> i'm about to
[10:17] <AlexExtreme> it doesn't like that because init isn't pid 1
[10:18] <Keybuk> oh, meh
[10:18] <Keybuk> I so need to add something for that ;p
[10:19] <AlexExtreme> :)
[10:19] <AlexExtreme> brb, need to get some food
[10:29] <AlexExtreme> back
[10:29] <Keybuk> ;)
[10:29] <AlexExtreme> now
[10:29] <AlexExtreme> to me it seems that it's a bug in upstart
[10:30] <Keybuk> something's not right, certainly
[10:30] <Keybuk> what kernel?
[10:30] <AlexExtreme> 2.6.20.1
[10:31] <Keybuk> it looks like it's not getting SIGCHLD
[10:31] <Keybuk> or getting stuck there
[10:31] <Keybuk> run bash with console owner on startup
[10:35] <AlexExtreme> k
[10:35] <AlexExtreme> got a shell
[10:35] <AlexExtreme> weirsd
[10:35] <AlexExtreme> *weird
[10:35] <AlexExtreme> initctl list says mount-kernel-filesystems is waiting and in the stop stare
[10:35] <AlexExtreme> *state
[10:35] <AlexExtreme> damn keyboard
[10:36] <Keybuk> odd
[10:36] <Keybuk> --debug shows nothing?
[10:36] <AlexExtreme> debug doesn't say anything about stopping
[10:37] <Keybuk> hmm
[10:38] <Keybuk> you're always going to mount a fresh /dev over the top there, aren't you
[10:38] <Keybuk> is there a /dev/console FILE by any chance? :p
[10:39] <Keybuk> if so ... cat it
[10:41] <AlexExtreme> no console
[10:41] <Keybuk> that's your problem, anyhoo
[10:41] <AlexExtreme> it's there on the actual filesystem but not on the ramdisk
[10:41] <Keybuk> no /dev/console, no /dev/null
[10:41] <Keybuk> can't start jobs in that state
[10:41] <AlexExtreme> ack
[10:42] <AlexExtreme> i'll have mount-kernel-filesystems create it then
[10:42] <Keybuk> if you look at my example mount-kernel-filesystems job, I make sure to copy those devices into the ramdisk from /lib/udev/devices
[10:42] <AlexExtreme> yeah
[10:42] <AlexExtreme> but there's no null or console in that dir for some reason ;)
[10:48] <Keybuk> you don't copy them in at all
[10:59] <Keybuk> http://upstart.ubuntu.com/wiki/ComplexEventConfig/UseCases
[11:03] <AlexExtreme> hmm
[11:07] <AlexExtreme> quick question: the scripts are run with -e... how would i stop it from terminating the script if a command gives an error?
[11:13] <Keybuk> || true
[11:14] <Keybuk> on each command that you want to ignore the error for
[11:16] <AlexExtreme> k
[11:16] <Keybuk> (or ||: )
[11:17] <Keybuk> I tend to dislike that because it looks like Perl
[11:17] <AlexExtreme> yeah
[11:18] <Keybuk> :; is just as bad
[11:27] <AlexExtreme> yay
[11:27] <AlexExtreme> i've converted the whole of frugalware's rcS.d to upstart jobs
[11:28] <AlexExtreme> pretty simple ones though, i haven't done anything about responding to udev events yet
[11:29] <Keybuk> how are you handling mounting the filesystem?
[11:31] <AlexExtreme> it's just a straight conversion of the SysV scripts atm
[11:32] <Keybuk> ah right
[11:37] <_ion> Throwing random ideas around, episode 59:
[11:37] <_ion> start on filesystem-checked
[11:37] <_ion> emits filesystem-mounted
[11:37] <_ion> block script
[11:37] <_ion>     prereq_mountpoints=...
[11:37] <_ion>     for m in $prereq_mountpoints; do
[11:37] <_ion>         echo "wait filesystem-mounted MOUNTPOINT=${m}";
[11:37] <_ion>     done
[11:37] <_ion> end script
[11:37] <_ion> The block script is run whenever the job is being started. If it prints any 'wait' lines, the job only goes to 'starting' when all corresponding events have been received.
[11:37] <_ion> Race conditions are avoided (or are they?) by not processing new events until the currently running block script has finished (thus the writer of job must be careful with what she does inside a block script)
[11:39] <Keybuk> that's quite similar to something iwj and I discussed a while back
[11:39] <Keybuk> let me see if I can dig it out
[11:40] <Keybuk> its possible its on my old laptop
[11:41] <Keybuk> http://www.chiark.greenend.org.uk/~ijackson/d/upstart.text
[11:46] <Keybuk> (it's somewhat insance, since ian wrote it, but the core is the idea that upstart implements a turing complete programming language which jobs can use to respond to events in a race-free manner -- since the program is run before the next event, rather than alongside)
[11:48] <_ion> Interesting.
[11:52] <_ion> But determining the prerequisite mountpoints based on the current MOUNTPOINT env var surely needs shell scripting or running external processes some other way. Using that syntax, how would one achieve that?
[11:53] <Keybuk> that's what I thought
[11:53] <Keybuk> it was a lot of complexity for no visible gain
[12:03] <AlexExtreme> nice
[12:03] <AlexExtreme> http://frugalware.org/~alex/bootchart-upstart.png
[12:04] <AlexExtreme> compared to this from sysvinit: http://frugalware.org/~alex/bootchart-sysvinit.png
[12:07] <_ion> Looks like the connection to frugalware.org is going to timeout.
[12:07] <AlexExtreme> strange
[12:07] <AlexExtreme> works here
[12:09] <_ion> traceroute reaches taurus.int.elte.hu (157.181.141.18). After it it's all * * *s.
[12:10] <AlexExtreme> hold on, let me drop it on my server
[12:11] <AlexExtreme> http://www.alex-smith.me.uk/files/bootcharts/
[12:11] <_ion> Now the pictures loaded.
[12:11] <pkt> AlexExtreme: yes but it seems that with sysinit you were running more services, correct?
[12:11] <_ion> But if i reload, it's "connecting..." again for ages.
[12:11] <AlexExtreme> yes, i know, it's not completely accurate
[12:12] <Keybuk> 18s is nice
[12:12] <pkt> also, are you running SMP?
[12:12] <AlexExtreme> pkt, no, this was in a vmware VM, it was only using 1 core
[12:12] <pkt> That is nice :-)
[12:13] <pkt> I guess the worst thing in startup time is the shell scripts (especially when they do extensive fs ops)
[12:14] <AlexExtreme> yes
[12:14] <Keybuk> that's what einit tries to address
[12:14] <pkt> with very carefully tuned sysvinit + scripts I could do ~25s in a 386sx/25 (2.6.8 stripped down)
[12:15] <pkt> Of course some current subsystems take more time to initialize :-(
[12:15] <AlexExtreme> now i need to try and work out what the hell is wrong with my fsck script, it seems to interfere with the sysv compat stuff. if fsck gets run, it tries to run runlevel 4 and runlevel 0 at the same time
[12:15] <_ion> Would converting dash to a library and linking to it instead of execing shells help at all?
[12:15] <_ion> (with upstart, that is)
[12:15] <AlexExtreme> probably
[12:15] <AlexExtreme> but that wouldn't work for distros without dash
[12:16] <pkt> _ion I think redhat does something close to that with their shell
[12:16] <pkt> they have something that includes several typically external programs as libraries 
[12:16] <pkt> but is not usable as a shell (i.e., not interactive)
[12:17] <pkt> Of course, this caused both positive and negative feelings to many :-)
[12:25] <Keybuk> isn't that just busybox?
[12:26] <pkt> Keybuk: no, it is (was?) a redhat-specific shell-like thing that included e.g., functionality for setting up LVM/RAID etc
[12:26] <pkt> but I 'm forgetting its name ...
[12:26] <Keybuk> ok
[12:26] <Keybuk> what einit does it has C modules that handle the common things like setting hostname, checking and mounting filesystems, activating swap, etc.
[12:26] <Keybuk> in practice, these things aren't that configurable (it's not like a shell script, which you can just edit)
[12:27] <Keybuk> and everything is configured from one large einit.xml
[12:27] <_ion> The initialization of dash doesn't take long, so the answer to my question is probably "no".
[12:27] <_ion> dash -c ''  0.00s user 0.00s system 0% cpu 0.009 total
[12:27] <pkt> yep the xml part sucks
[12:27] <pkt> _ion, I don 't think that dash per se is the problem
[12:28] <pkt> usually the problem is like an "abstraction penalty" for using unix utilities for stuff that would only take a few lines of C code
[12:28] <_ion> (Side note: "perse" in Finnish means an arse.)
[12:29] <pkt> oops :-)
[12:29] <pkt> but I did leave a blank
[12:29] <Keybuk> ie. forking then execing grep, which loads localisation libraries and parses a file linewise matching against regular expression objects ... instead of using strstr()
[12:30] <pkt> exactly (and I could think of even worse stuff when amateur admins are involved :-)
[12:31] <Keybuk> e.g. /etc/event.d/hostname
[12:31] <Keybuk> the shell version looks something like:
[12:31] <Keybuk>  - stat /etc/hostname
[12:31] <Keybuk>  - fork/exec cat: fopen/stat /etc/hostname, read, write to stdout, close, capture stdout into variable
[12:31] <Keybuk>  - compare variable against empty string
[12:31] <Keybuk>  - fork/exec hostname: capture stdout
[12:32] <Keybuk>  - compare variable against empty string
[12:32] <Keybuk>  - set variable to localhost
[12:32] <Keybuk>  - fork/exec hostname, set hostname
[12:32] <Keybuk> in C, that might be:
[12:32] <Keybuk>  - gethostname()
[12:32] <Keybuk>  - fopen(/etc/hostname)
[12:32] <Keybuk>  - if opened: fgets
[12:33] <Keybuk>  - else if still unset: strcpy
[12:33] <Keybuk>  - sethostname()
[12:33] <Keybuk> 4 syscalls instead of ~100
[12:33] <_ion> Yeah
[12:34] <pkt> Keybuk: the only problem is that you can only provide "embedded" functionality in the "shell" for very simple stuff (those that cannot be configured in a gazillion different ways)
[12:35] <Keybuk> pkt: I don't follow?
[12:35] <pkt> E.g., redhat that provides the LVM functionality in their "shell" actually provides a "stripped-down" version
[12:35] <Keybuk> ah
[12:35] <Keybuk> I see
[12:36] <pkt> Otherwise you get a *huge* "shell/interpreter" which is again a loss
[12:36] <pkt> It is, as always, a tradeoff :-)
[12:36] <Keybuk> flexibility vs. speed
[12:37] <pkt> Actually "speed for a special case" or "robust speed"
[12:37] <Keybuk> a while ago, a friend and I wondered whether it'd be possible to write a scripting language based around syscalls
[12:38] <pkt> that is roughly what I thought as well, that sort of a "DSL" for booting would provide the largest benefit/cost ratio
[12:38] <Keybuk> DSL?
[12:39] <_ion> Domain-specific language IIRC
[12:39] <pkt> Domain-Specific Language, yep
[12:39] <Keybuk> right
[12:39] <pkt> but I view upstart as being an important step towards this direction
[12:40] <Keybuk> right now we have <action> exec and <action> script, we can always add <action> foo later :p
[12:41] <Keybuk> pre-start perl ...
[12:41] <Keybuk> end perl
[12:41] <Keybuk> :p
[12:41] <AlexExtreme> ugh :p
[12:41] <_ion> Haha
[12:41] <pkt> :)
[12:42] <pkt> I think the important think in upstart is to think really hard about the interactions of the base system components (especially for desktops)
[12:42] <pkt> This is no trivial problem (which is why I have seen no perfect "job" set or job language yet)
[12:43] <pkt> Once this is sorted out more or less it will be much easier to take the "extra step"
[12:43] <Keybuk> robust first, speed later
[12:44] <pkt> Plus it may be proven that the extra speed doesn 't really matter after all
[12:45] <pkt> But the really hard stuff (at least for me) is to figure out how to support as many common use-cases as possible with as few "artificial waits" as possible
[12:45] <pkt> And do that in a world where stuff (from all levels of the stack) come and go all the time
[12:46] <pkt> (think e.g., ATA over Ethernet over WLAN :-)
[12:46] <Keybuk> indeed
[12:48] <pkt> plus the fact that e.g., a developer will want to also start/stop stuff on demand
[12:51] <pkt> and the fact that some unix beasts (like shells) are a bit "special" on how you need to handle them 
[12:51] <pkt> Or they loose their settings :-)
[12:51] <pkt> s/settings/history
[01:00] <Keybuk> heh
[01:02] <pkt> yep, *every single pitfall* of sysvinit, I 've hit it ;-)
[01:02] <AlexExtreme> :)
[01:04] <AlexExtreme> random thought: sysvinit scripts provide a start and a stop. with upstart, to do the stop part you'd have to create a seperate job, no? (at least, for things that aren't daemons, like saving mixer settings. daemons would get SIGTERM)
[01:05] <Keybuk> you can do it either way
[01:05] <Keybuk> could you have a mixer-settings job that restored them in pre-start and saved them in post-stop
[01:05] <Keybuk> then you could "start mixer-settings" and "stop mixer-settings" :p
[01:05] <Keybuk> or you could have two separate jobs that do it in script
[01:06] <Keybuk> it depends what makes more sense really
[01:06] <Keybuk> you could do that for everything
[01:06] <Keybuk> ie. have a hostname job that sets the hostname in pre-start but does nothing in post-stop
[01:07] <Keybuk> (or even unsets it)
[01:07] <Keybuk> then the "hostname" job would be running all the time the hostname was set
[01:07] <Keybuk> and you could stop hostname to unset it
[01:07] <Keybuk> and restart the job to reset it
[01:07] <AlexExtreme> but
[01:07] <Keybuk> but that makes somewhat less sense than just having a set-hostname task
[01:07] <AlexExtreme> if you did it with pre-start and post-stop
[01:08] <AlexExtreme> wouldn't it not work because you need an exec/script/respawn/whatever ?
[01:08] <Keybuk> no
[01:08] <Keybuk> those are optional
[01:08] <AlexExtreme> hmm
[01:08] <Keybuk> you can have process-less jobs now
[01:08] <Keybuk> it was supposed to work in 0.3.5, but I broke it just before release
[01:08] <Keybuk> it works in trunk
[01:09] <AlexExtreme> but if they were missing and you did start mixer-settings, wouldn't it run the pre-start script, then jump straight to stopping and run post-stop?
[01:11] <Keybuk> no, sticks in running
[01:11] <AlexExtreme> ah
[01:11] <AlexExtreme> ok
[01:12] <Keybuk> http://upstart.ubuntu.com/wiki/JobsAsStates
[01:14] <AlexExtreme> ah
[01:14] <AlexExtreme> I see
[01:15] <Keybuk> I tend to think in terms of what a sysadmin would rather do
[01:15] <Keybuk> what makes more sense
[01:15] <AlexExtreme> :)
[01:15] <Keybuk> stop the mixer job to save them and mute the card
[01:15] <AlexExtreme> sounds logical
[01:15] <Keybuk> (and thus restart the mixer to save the settings)
[01:16] <Keybuk> or run the save-mixer-settings job to save them
[01:22] <Keybuk> I can't decide whether we need lots of small options for job control "start on", "stop on", "when", "from", "foo", etc.
[01:22] <Keybuk> or just one big option
[01:43] <AlexExtreme> IMO one big option would be too complex, i personally like it with small options
[01:43] <AlexExtreme> btw, what is the problem with logd?
[01:44] <Keybuk> kill it, any job with it as stdout ends up dying mysteriously and silently
[01:44] <AlexExtreme> ah :/
[02:53] <Keybuk> so interestingly:
[02:53] <Keybuk> some jobs can be defined by what starts or stops them
[02:53] <Keybuk> but others it's easier to define by when they should be running
[03:45] <AlexExtreme> hmm
[03:45] <AlexExtreme> ACTION=="add", SUBSYSTEM=="block", KERNEL=="*[!0-9] ", RUN="/sbin/initctl emit block-device-added %k"
[03:46] <AlexExtreme> is that rule ok for a block device added?
[03:46] <AlexExtreme> or should I add -eDEVNAME=%k too?
[04:42] <elpaso> hello
[04:43] <elpaso> is there anybody out there?