[00:28] <smoser> gondoi, i do have a pretty good handle on upstart events and dependencies yes.
[00:57] <harlowja> gondoi can u provide the upstart config that starts nova-agent, that might help
[00:58] <harlowja> *since its likely a rackspace upstart config
[01:36] <smoser> gondoi, is this something i could jsut start on rackspace ?
[01:36] <smoser> ie , just public instances ?
[01:36] <smoser> i haven't started instances there in quite some time
[01:46] <harlowja> i can try also smoser, i deleted mine though since i got a damn bill for it
[01:46] <harlowja> although i wasn't aware that rackspace images had cloudinit
[01:46] <harlowja> *since they have that agent thingy
[01:47] <harlowja> https://github.com/rackerlabs/openstack-guest-agents-unix
[14:28] <smoser> gondoi, looking at an instance of saucy i started on rackspace
[14:28] <smoser> in general sysvinit will *start* only after cloud-init has finished.
[14:29] <smoser> so first glance, the reason for that (by design) is
[14:30] <smoser>  cloud-inti starts on
[14:30] <smoser> (cloud-init.conf)
[14:30] <smoser> starts on
[14:30] <smoser>  start on mounted MOUNTPOINT=/ and stopped cloud-init-nonet
[14:30] <smoser> cloud-init-nonet is a a job who's only purpose is to block running of cloud-init until static-networking is up
[14:31] <smoser> 'mounted MOUNTPOINT=/' is blocking event
[14:31] <smoser> meaning /etc/init/rc-sysinit.conf (start on (filesystem and static-network-up) or failsafe-boot)
[14:31] <smoser> will not run until that has finished.
[14:31] <smoser> so nova-agent, which is set to run via sysvinit will not run until after cloud-init has run
[14:32] <smoser> this is by design from cloud-init's perspective
[14:33] <smoser> hm..
[14:33]  * smoser wonders if gondoi was running a 'hvm' instance, because i just ran one, and it doesn't seem to get networking up.
[15:47] <gondoi> smoser, sorry i had to leave yesterday
[15:47] <gondoi> just catching up on your messages
[15:47] <gondoi> what is an hvm instance?
[15:48] <gondoi> i was using the performance class if that's what you mean
[15:48] <smoser> in my rackspace control panel, there was an option to run 13.10 (ubuntu) in something called "pvhvm"
[15:49] <smoser> which i suspect is xen HVM rather than PVM.
[15:49] <gondoi> ohhh no i was not using that one
[15:49] <smoser> but it never came up.
[15:49] <smoser> that was an image type.
[15:49] <smoser> (oddly an image type, not a instance type)
[15:50] <gondoi> yeah i'm not sure about that one :D
[15:50] <gondoi> let me see which one I was usning
[15:50] <gondoi> using
[15:53] <gondoi> i'm using Ubuntu 13.10 (Saucy Salamander) in ORD
[15:53] <gondoi> yeah the start on and stop on for rc is confusing to me since it's just looking for a runlevel change
[15:54] <smoser> gondoi, well its the 'rc-sysinit.conf' that emits that
[15:59] <gondoi> smoser, i'm sorry i don't see the connection.. I believe that's how it works, i'm just trying to understand it... 'start on mounted ...' that blocks cloud-init right?
[16:00] <smoser> cloud-init is 'start on mounted'
[16:00] <smoser> which is "really early"
[16:00] <smoser> and combined with the 'static-networking"
[16:00] <smoser> forces a "networking up and root mounted rw" bottleneck
[16:00] <smoser> so cloud-init blocks boot on itself
[16:01] <smoser> after cloud-init is done, rc-sysinit.conf can run
[16:01] <smoser> which will emit the 'runlevel'
[16:02] <smoser> which will trigger rc.conf to run the sysvinit scripts for the given runelvel (2)
[16:03] <gondoi> okay... i guess my confusion is on the filesystem state and fired events in upstart.. sorry
[16:03] <gondoi> upstart is async right?
[16:04] <smoser> yes.
[16:04] <smoser> but rc is not.
[16:05] <gondoi> or does it fire an event mounted MOUNTPOINT=/ and wait for the cloud-init task to finish before finishing all mounts and firing filesystem?
[16:05] <smoser> and by design cloud-init forces a synchronous bottleneck
[16:05] <smoser> mounted MOUNTPOINT= events are blocking
[16:05] <smoser> and nothing happens until after 'mounted MOUNTPOINT=/'
[16:05] <smoser> so that specific event is very globally blocking
[16:06] <smoser> this is all by design
[16:06] <smoser> so that cloud-init can do boothooks or config changes
[16:06] <smoser> and those things not be race condition enducing
[16:06] <smoser> inducing
[16:07] <gondoi> yeah
[16:07] <gondoi> thanks for the clarification... upstart is much more confusing than I ever expected it to be :/
[16:08] <gondoi> the docs are great on upstart.ubuntu.com, but i failed to find this one nugget of information in trying to troubleshoot this
[16:08] <smoser> gondoi, well, part of this is "upstart"
[16:08] <smoser> and part is "ubuntu"
[16:09] <smoser> ie, part of this is ubuntu-usage of upstart. not specifically upstart.
[16:09] <gondoi> oh ....
[16:09] <smoser> so thats why you didn't/don't find documentation on some of it on upstream upstart.
[16:10] <gondoi> oh man, it just clicked.. I think that's why this all works with the rackspace debian image.........
[16:10] <gondoi> is it blocking in debian too?
[16:11] <smoser> probably blocking in debian, yes.
[16:11] <gondoi> oh .. hmm
[16:11] <smoser> i'd have to actually look at debian
[16:11] <smoser> ie, do they use upstart there?
[16:11] <smoser> on rackpace debian ?
[16:11] <gondoi> oh... well.. i actually don't know the answer to that
[16:12] <gondoi> i'd have to boot one and check it out
[16:12] <smoser> if they're just using the sysvinit scripts, then ideally cloud-init forces a similar boot order
[16:12] <gondoi> yeah
[16:12] <smoser> (ie, it shoudl block just about everything other than '/' is mounted and network is up)