[00:40] <edgecase> last hope, oh my!
[00:43] <Keybuk> heh
[00:43] <edgecase> ok semantic question, on ubuntu mountall.conf takes care of mounting root fs rw, other fs, etc,
[00:44] <Keybuk> yes
[00:44] <edgecase> but, could it be assumed, that by definition, starting /sbin/init, means the rootfs is already mounted, possibly read-only?
[00:45] <Keybuk> yes
[00:45] <edgecase> i mean either a monolithic kernel, or an initrd's pivot_root, or say lxc container starting...
[00:45] <Keybuk>         /* Sanity check, the root filesystem should be already mounted */
[00:45] <Keybuk>         root = find_mount ("/");
[00:45] <Keybuk>         if (! root->mounted) {
[00:45] <Keybuk>                 nih_fatal ("%s", _("root filesystem isn't mounted"));
[00:45] <Keybuk>                 exit (EXIT_ERROR);
[00:45] <Keybuk>         }
[00:45] <Keybuk> even mountall assumes that :p
[00:45] <edgecase> so "start on startup" implies certain parts of FHS are available, yes?
[00:45] <Keybuk> edgecase: it implies that the root filesystem is read-only
[00:46] <Keybuk> we tend to assume that means you have /bin, /sbin, /lib and /etc
[00:46] <edgecase> but not /usr or /var
[00:46] <Keybuk> right
[00:46] <Keybuk> definitely not /usr, /var, /boot, /home, /proc, /sys, /dev, etc.
[00:46] <Keybuk> although with Upstart, you can assume /proc and /sys are mounted
[00:47] <edgecase> i'm just pondering the sysctl.conf race problem
[00:47] <Keybuk> and with recent kernels, /dev should be a devtmpfs
[00:47] <Keybuk> sysctl.conf race problem?
[00:47] <edgecase> race of setting things vs network interfaces coming up.  in Karmic for example
[00:48] <edgecase> there's a couple launchpad bug IDs
[00:48] <edgecase> i drew out an elaborate map of the issues, but let me try and focus on one aspect
[00:50] <edgecase> ok, with sysctl parameters, there are some that affect parts of the kernel which is only compiled in (not a module)
[00:50] <edgecase> but with many, it could be a module or compiled in, so there's no *right* place for sysctl.conf to be loaded
[00:52] <Keybuk> right
[00:53] <Keybuk> this is why I keep trying to eradicate that silly directory
[00:53] <Keybuk> but people keep insisting I put it back :p
[00:53] <edgecase> i wonder if the whole sysctl paradigm doesn't work so well 
[00:54] <Keybuk> I tend to think that, since we have the kernel source, changing a sysctl default is best done by changing the default in the kernel
[00:54] <Keybuk> otherwise you have races like "the sysctl doesn't get set until after ssh is running and there's a security hole for a while"
[00:54] <edgecase> i'm thinking of network configs here, default isn't enough
[00:54] <edgecase> the system needs to be configurable
[00:55] <Keybuk> sure
[00:55] <Keybuk> but you can do one off-things like that with udev
[00:55] <edgecase> there's a couple real problems with sysctl tho,
[00:56] <Keybuk> SUBSYSTEM=="module", KERNEL=="i915", RUN+="echo 4 > /proc/sys/..."
[00:56] <edgecase> a NIC module could be compiled in, loaded by udev, or by the initrd... hard to put it's settings in one place had have it affect all 3
[00:57] <Keybuk> you should still get module events for them
[00:57] <edgecase> will that read sysctl.conf ?
[00:57] <Keybuk> you only don't get module events for built-ins with no parameters
[00:57] <Keybuk> no
[00:57] <edgecase> hence the sticky issue
[00:57] <Keybuk> and reading sysctl.conf 800 times during boot is out of the question, before you ask ;)
[00:58] <edgecase> workaround for the parameters i need is to set in /etc/network/interfaces and avoid the issue
[00:58] <edgecase> but i'm also interested in the bigger picture of upstart
[00:59] <edgecase> maybe just agree that sysctl isn't good for things not compiled-in
[01:00] <Keybuk> I don't think there's a picture of sysctl in the upstart painting yet
[01:00] <Keybuk> it's probably in the big white space off to the right with a doodle of someone's dog in it
[01:01] <Keybuk> sorry, have to run now
[01:02] <Keybuk> it's 1am here
[01:02] <edgecase> yeah, that's ok, i can workaround for the network stuff  i need
[01:03] <Keybuk> feel free to mail suggestions for how this could work to the ML
[01:03] <Keybuk> upstart is very much still under development
[01:03] <Keybuk> so lots of things can be improved
[01:03] <edgecase> d'oh!
[01:03] <edgecase> be around tomorrow?
[01:04] <Keybuk> yes, though earlier!
[01:04] <Keybuk> nite
[01:04] <edgecase> ok
[01:04] <edgecase> thx
[01:15] <edgecase> hmm hostname.conf, nothing depends on it, 
[07:38] <NoReflex>  hello! I'm having some problems with upstart in karmic server 64bit. It won't start postgresql. I tried http://superuser.com/questions/98702/how-to-make-postgresql-start-at-boot-time-in-ubuntu but it didn't work. can I remove upstart? I'm afraid to test it because the machine is at a remote location and if I mess it up I won't be able to connect to it anymore
[07:38] <NoReflex> I can start postgresql using /etc/init.d/postgresql-8.4 manually after login. There are links in /etc/rc*.d to postgresql init script but for some reason it isn't started automatically after reboot
[08:21] <NoReflex> nothing?
[08:53] <elpargo> hi. I got a program that I want to "always be running" as it opens a http listener and waits for input "forever" 
[08:53] <elpargo> I'm not entirely sure if I should be doing an upstart script for it.
[09:54] <ct529> hi everybody .... I am having serious problems with upstart on ubuntu 910 64 bit .... it would not start CUPS at all ....
[09:54] <ct529> CUPS is properly configrued in /etc/rcX.d and in /etc/init.d ....
[10:07] <NoReflex> ct529: I have the same problem with upstart on Karmic 64bit. On my machine it won't start postgresql server. I tried http://superuser.com/questions/98702/how-to-make-postgresql-start-at-boot-time-in-ubuntu without luck
[10:09] <NoReflex> ct529: as a workaround you can use rc.local ... not perfect but at least it works...
[10:11] <ct529> NoReflex: how?
[10:11] <ct529> NoReflex: yes, I have the same problem with mysql and postgresql
[10:12] <ct529> NoReflex: what do you see when you run the command runlevel?
[10:13] <NoReflex> ct529: what I did was to remove the links from /etc/rc*.d for postgres (sudo update-rc.d -f  postgresql-8.4 remove) and add the start command to rc.local
[10:13] <ct529> NoReflex: did you add it manually?
[10:13] <ct529> NoReflex: could you pleas elet me know what do you see when  you run the command "runlevel"?
[10:14] <NoReflex> ct529: runlevel gives N 2
[10:14] <ct529> NoReflex: so it works on your machine
[14:05] <edgecase> elpargo, are you asking if you should write upstart script instead of sysVinit script?
[14:17] <kimju> hello.. Is there some documentation on howto debug the bootup sequence? I'm getting bitten by bind9 trying to start (and failing) before the interfaces with ip-addresses that it is configured to listen-on / listen-on-v6 are brought up.
[14:19] <kimju> (this is happening on ubuntu karmic. someone else has reported it as bug: https://bugs.launchpad.net/ubuntu/+source/bind9/+bug/510587 )
[14:21] <Keybuk> kimju: you should ask on #ubuntu about that
[14:21] <Keybuk> as far as I'm aware, bind9 in ubuntu is still using a sysvinit script and hasn't been converted to an upstart config file
[14:22] <kimju> Keybuk, ok, sorry. I just asked before you joined the channel if there is some documentation on how to debug the bootup sequence, as I'm getting bitten by that bug.
[14:23] <Keybuk> boot with --verbose on the kernel command-line
[14:34] <edgecase> Keybuk, top o' the morning to ya!
[14:35] <edgecase>  hmm hostname.conf, nothing depends on it, 
[14:35] <edgecase> so a race is possible with anything that uses hostname/utsname
[14:37] <Keybuk> edgecase: hey
[14:37] <Keybuk> ah yes, it's still St Patrick's Month
[14:37] <edgecase> i will now continue the barrage!
[14:37] <Keybuk> edgecase: can you think of anything that cares about the hostname? :p
[14:37] <edgecase> syslog?
[14:38] <Keybuk> it doesn't really depend on it
[14:38] <Keybuk> it queries it for every log
[14:38] <edgecase> ah ok
[14:38] <Keybuk> so if the hostname changes, it works just fine
[14:38] <edgecase> aside, is there anything to graph the dependencies visually?
[14:39] <edgecase> would make investigation easier
[14:39] <edgecase> for rsyslog, if you log to remote host, might get unknown hostname for a bit?
[14:40] <Keybuk> no, but there's no reason it shouldn't be easy to write something that makes dotty output
[14:40] <edgecase> ok will look at dotty
[14:40] <Keybuk> for rsyslog, if you log to remote host, you have to wait for the network to come up first :p
[14:40] <Keybuk> and on the typical laptop system, that won't happen until a user logs in and unlocks the WPA key :p
[14:40] <edgecase> you noticed my nick is "edgecase" right?
[14:41] <edgecase> i tend to ignore "typical" cases :P
[14:41] <Keybuk> :)
[14:41] <edgecase> anyhow i'll admit hostname is a trivial case
[14:42] <edgecase> i notice mountall.conf emits many variations of filesystems
[14:42] <edgecase> is there a formal definition of those, that would allow splitting them up?
[14:42] <edgecase> "emits filesystem", does that mean root fs is read-write?
[14:43] <edgecase> appologies if that info is published already
[14:50] <edgecase> hmm, having /var not a mandatory part of root fs, and yet having /var/run mounted on top, seems problematic
[14:51] <edgecase> I question the original unix paradigm of "map everything to a file" for daemon .pid files, esp. if they are going to just map back to ram(disk)
[14:52] <edgecase> just use unix socket or even abstract namespace one
[14:54] <Keybuk> edgecase: did you do "man 7 filesystem" ?
[14:54] <Keybuk> problematic> not really
[14:54] <Keybuk> it requires some juggling
[14:54] <Keybuk> but it works
[14:54] <Keybuk> we have /var/run before and after /var is mounted
[14:54] <Keybuk> in fact, even as /var is mounting, /var/run is still present
[14:54] <edgecase> ok, juggling is acceptable if you don't drop the ball
[14:55] <Keybuk> *shrug*
[14:55] <Keybuk> if the boot sequence drops any ball, you can't boot
[14:55] <edgecase> badness++
[14:56] <Keybuk> why?
[14:56] <Keybuk> just write things to be reliable
[14:57] <edgecase> i'll just assume "even as /var is mounting, /var/run is still present" is relable, moving on...
[14:57] <Keybuk> yes
[14:57] <Keybuk> I could teach a course on Theory of Atomic Operations in Operating Systems 101 :p
[14:57] <Keybuk> I think about them a lot
[14:57] <Keybuk> the details:
[14:57] <edgecase> oh thank god
[14:58] <Keybuk>   we always have a /var/run directory on the root filesystem, under the /var directory
[14:58] <Keybuk>   we mount /var/run there on the first pass
[14:58] <Keybuk>   then when it's time to mount /var, we actually mount it somewhere else (/dev/.var I think :P)
[14:58] <edgecase> is there a mount --under option or something?
[14:58] <Keybuk>   we bind mount the current /var/run to /dev/.var/run
[14:58] <Keybuk> so now it's available at both /var/run and /dev/.var/run
[14:59] <Keybuk>   we then move the /dev/.var mount to /var
[14:59] <edgecase> all atomic?
[14:59] <Keybuk> each of those is atomic yes
[14:59] <Keybuk> since a mount() operation holds the VFS lock
[14:59] <edgecase> ooh, snap!
[14:59] <Keybuk> mountinfo will show /var/run mounted twice
[14:59] <Keybuk> with different parent mounts
[15:00] <Keybuk> if you unmount /var, for whatever reason, /var/run is still mounted under it
[15:00] <edgecase> i find no mountinfo or man 7 filesystem
[15:00] <Keybuk> edgecase: are you on lucid?
[15:00] <edgecase> karmic
[15:00] <edgecase> i guess i need to be bleeding edge to get bleeding edge
[15:00] <Keybuk> ah, I think I added that manpage in lucid
[15:00] <edgecase> i'll get lucid on a box shortly
[15:01] <edgecase> so i can discuss current state of things
[15:01] <Keybuk> mountinfo => /proc/self/mountinfo
[15:01] <edgecase> ah /proc/mounts on steroids
[15:02] <edgecase> ok so, what provides /proc and /sys, upstart, before running any scripts?
[15:02] <Keybuk> I don't quite understand your question
[15:02] <Keybuk> you asked me what kind of trees apples, oranges and buses grow on :p
[15:03] <edgecase> earlier you said assume /proc and /sys when running upstart
[15:03] <Keybuk> yes
[15:03] <edgecase> so what mounts them?
[15:03] <Keybuk> init
[15:03] <edgecase> before running any "start on startup" jobs?
[15:03] <Keybuk> yup
[15:03] <edgecase> ok just getting my bearings
[15:03] <Keybuk> Upstart uses /proc/self/fd to run shell scripts
[15:04] <Keybuk> so it needs /proc mounted
[15:04] <Keybuk> and since it's probably going to need /sys at some point soon, I figured I'd mount that too
[15:04] <Keybuk> /dev gets mounted by the kernel these days
[15:04] <edgecase> is there a formal definition of state when "startup" event fires?
[15:04] <edgecase> ie "you get root /proc and /sys"
[15:05] <edgecase> and "but don't assume /var and /usr etc"
[15:05] <Keybuk> no written down one I guess
[15:05] <edgecase> use the source luke
[15:05] <Keybuk> the root filesystem is read-only
[15:05] <Keybuk> you have /proc and /sys
[15:05] <Keybuk> you probably have /dev
[15:05] <edgecase> reading source takes time tho
[15:05] <Keybuk> you have /bin, /sbin, /lib and /etc
[15:05] <Keybuk> but nothing else filesystem wise
[15:06] <edgecase> aside, i'm thinking mountall should be broken up, it's a syncronisation point, which defeats some parallelism
[15:06] <Keybuk> it isn't a synchronisation point
[15:06] <Keybuk> that's kinda the idea
[15:06] <edgecase> make a dummy job with the same syncronisation properties for dumb legacy jobs,
[15:06] <Keybuk> it replaces the shell script nightmares we had before that were
[15:07] <edgecase> but let smart jobs pick off say just root fs being rw
[15:07] <Keybuk> they can
[15:07] <Keybuk>   on mounted MOUNTPOINT=/
[15:07] <edgecase> mountall emits events?
[15:07] <edgecase> oh vfs emits mount events i'm guessing
[15:08] <Keybuk> no, mountall emits them
[15:08] <edgecase> back to upstart?
[15:08] <Keybuk> yes
[15:08] <Keybuk> like filesystem for a start ;)
[15:08] <edgecase> ok, then back to all the emits in mountall.conf...
[15:09] <edgecase> any definition what they mean?
[15:09] <Keybuk> emits virtual-filesystems
[15:09] <Keybuk> emits local-filesystems
[15:09] <Keybuk> emits remote-filesystems
[15:09] <Keybuk> emits all-swaps
[15:09] <Keybuk> emits filesystem
[15:09] <Keybuk> emits mounting
[15:09] <Keybuk> emits mounted
[15:09] <Keybuk> (from lucid)
[15:09] <Keybuk> for any of those, "man 7 EVENT-NAME"
[15:09] <Keybuk> (in lucid)
[15:09] <edgecase> ok will look there
[15:09] <Keybuk> I've tried to be reasonable well behaved about writing man pages for events
[15:09] <edgecase> that's why i say sync point, they all get emitted in parallel
[15:09] <Keybuk> they don't
[15:09] <edgecase> but only after *all* of mountall is done
[15:09] <Keybuk> no they don't
[15:10] <Keybuk> they get emitted at completely different times
[15:10] <edgecase> ok so it's a declaration, "may emit..."
[15:10] <Keybuk> yes
[15:10] <edgecase> right-o
[15:10] <Keybuk> did you read "man 5 init" ? :)
[15:10] <Keybuk>        emits EVENT...
[15:10] <Keybuk>               All processes on the system are free to emit their own events by
[15:10] <Keybuk>               using the initctl(8) tool, or by communicating directly with the
[15:10] <Keybuk>               init(8) daemon.
[15:10] <Keybuk>               This  stanza  allows  a job to document in its job configuration
[15:10] <Keybuk>               what events it emits itself, and may be useful for graphing pos‐
[15:10] <edgecase> i see now that i'm way to pessimistic coming into this
[15:10] <Keybuk>               sible transitions.
[15:11] <Keybuk> emits is basically there to make writing that graphing tool you were asking about possible ;)
[15:11] <Keybuk> documentation only
[15:11] <edgecase> ok things look in order then
[15:12] <edgecase> i'm thinking then that just the .conf files can be optimized, to depend only on what they *really* need,
[15:12] <edgecase> to enable some more unusual use-cases than "user logs in then unlocks WLAN key..."
[15:12] <Keybuk> yes, certainly
[15:12] <edgecase> like, network booting etc
[15:12] <Keybuk> the difficulty comes in adapting to users
[15:12] <Keybuk> like we could write the job to actually depend on /usr rather than filesystem
[15:13] <edgecase> exactly
[15:13] <Keybuk> but that wouldn't work if a user decided to put a different filesystem for /usr/bin and for /usr/lib
[15:13] <edgecase> FHS doesn't allow that does it?
[15:13] <Keybuk> yes it does
[15:13] <Keybuk> FHS only says directories you shouldn't put on different filesystems
[15:13] <edgecase> ok well on that topic,
[15:13] <Keybuk> things like /usr and /var are free-for-alls
[15:13] <edgecase> windows has "reparse points"...
[15:13] <Keybuk> it's pretty common for example to have /var/lib or /var/spool on different drives
[15:14] <edgecase> true
[15:14] <edgecase> with unix, a (empty) directory *could* be a mount point, no way to know
[15:14] <edgecase> you can read fstab, and hope it's in sync with reality
[15:15] <Keybuk> reparse points => those are basically crazy symlinks iirc?
[15:15] <edgecase> takes place of fstab entries
[15:15] <edgecase> you get file dir symlink, and now reparse point fs object
[15:15] <edgecase> so, it's like an empty dir, because it has a name, 
[15:15] <Keybuk> actually it might not even be in fstab
[15:15] <Keybuk> e.g. if you use LABEL to mean mountpoint
[15:15] <Keybuk> or if it's a removable device under /media
[15:15] <edgecase> but it's a type explicityly marking it as a mount point
[15:16] <edgecase> ok then that's an arguement for having reparse points in linux
[15:16] <edgecase> although 75% of the unix world will issue a death warrant to you
[15:16] <edgecase> for saying it
[15:16] <edgecase> the cases that are in fstab could be handled
[15:17] <edgecase> maybe that's a starting point
[15:19] <edgecase> but it's not /usr i'm interested in... but rather starting things that need root fs read-only and /var/run available, starting those asap
[15:19] <edgecase> anyhow i see that upstart is thankfully ahead of my expectations, i need to do some more reading now
[15:19] <edgecase> thanks for the good work!
[15:30] <Keybuk> local-filesystems should be ok for those on Ubuntu
[15:30] <Keybuk> ah
[15:30] <Keybuk> sorry
[15:30] <Keybuk> virtual-filesystems
[15:30] <Keybuk> virtual-filesystems means you have root read-only, and all the things like /var/run and stuff
[15:30] <Keybuk> we use it to start things like udev, dbus, etc.
[15:46] <edgecase> k thx
[17:33] <wasabi> oh hi/bye edgecase
[20:56] <halfline> sadmac: did you ever look at the display manager selection thing more?
[20:56] <halfline> i mean http://rstrode.fedorapeople.org/quest_to_find_a_dm.txt