[00:02] <rharper> smoser: around ?
[00:02] <smoser> here
[00:02] <smoser> looking at why i broke your test suite
[00:03] <smoser> theres two issues i'm seeing
[00:03] <smoser> a.) a WARN in the log says the fallback datasource got used, but idont know how (and other parts say NoCloud got used)
[00:04] <smoser> b.) it seems the nics aren't being renamed appoprirately.
[00:04] <smoser> do you have an  easy way to get in to the test vm ?
[00:09] <smoser> ok. well, 'b' seems to be because we have a /etc/network/interfaces inside that doesnt' srouce /etc/network/interfaces.d
[00:13] <rharper> smoser: it's somewhat tricky to get it  but:  I usually drop the poweroff script in generate_payload_scripts();  this will leave the system up
[00:15] <rharper> the if you ps, you can get the /tmp/launch.XXXX/boot.img ; copy that out before you kill the vm and the qemu command line;  then I replace -serial file.* with -serial telnet:localhost:2446,server,nowait , quote the --append, then add -monitor stdio -S;  when you run it; you'll get monitor prompt on stdio, guest will be paused;  you can setup your telnet to localhost on the port; then type 'cont' on the monitor;
[00:18] <rharper> smoser: I've got the configdrive openstack helper updated to search for the network_data.json, so we can fetch this,  we use the same convert_to_json() logic and stuff that in configdrive.networkdata_raw;  I see in stages where userdata_raw and vendordata_raw are processed (and written out);  ideally I'd do that with networkdata and write out a networkdata.json ;  I'm trying to find where best to convert the json networ
[00:18] <rharper> kdata into network config;  it appears that I need to write out network_config.yaml where we write out userdata conf as well, and then the distro apply_net will do it's thing
[00:19] <smoser> oh. well the ps is easily fixable
[00:19] <smoser> rharper, CURTIN_VMTEST_KEEP_DATA_PASS=logs,collect
[00:19] <smoser> run with that, add 'disks' to it and you'll keep the disks too
[00:19] <rharper> I usually want the interactive nature
[00:20] <smoser> right. but i was just saying to keep the disk around
[00:20] <rharper> oh
[00:20] <smoser> so you dont have to race it.
[00:20] <rharper> yeah
[00:20] <rharper> ah, gotcha
[00:20] <rharper> or modify the power stage
[00:20] <rharper> yep
[00:20] <smoser> ok. so i understand most of the failure now
[00:20] <smoser> and have some changes.
[00:20] <smoser> but test_etc_network_interfaces
[00:20] <smoser> will have to be updated
[00:21] <smoser> as now i'm (by design) not writing to /etc/network/interfaces
[00:23] <rharper> http://paste.ubuntu.com/15476168/  (my debug script)
[00:24] <rharper> smoser: yes;  I think we should use the net.parse_eni() which dumps a dictionary
[00:24] <rharper> from there we can compare/contrast with a network_state
[00:24] <smoser> ok.
[00:24] <smoser> i'll getthis as close as i can
[00:24] <smoser> and then can you do that ?
[00:24] <rharper> alternatively, we need to write a network_state that can re-render network_config.yaml
[00:24] <smoser> i'll submit MP for your git in next 5 minutes.
[00:24] <rharper> prolly
[00:25] <rharper> but I'd like to get your thoughts on the config drive bits
[00:25] <rharper> that's what I'm trying to finish up right now
[00:25] <smoser> k
[00:25] <rharper> where to convert the network_data into network_config yaml
[00:26] <smoser> ah. ok
[00:26] <smoser> so you have the networkdata_raw (although its not *really* raw) as you json loaded it.
[00:26] <rharper> right
[00:26] <smoser> you can call it whatever you want.
[00:26] <smoser> sure keep it around in the datasource datasource.network_json
[00:26] <rharper> the same as vendordata_raw (which also was json loaded)
[00:27] <smoser> then, in datasource.network_config()
[00:27] <smoser> its a property
[00:27] <smoser> you can convert it there or anywhere really.
[00:27] <smoser> would probably be nice to have the code easily runnable with no class wrappers or something
[00:27] <smoser> def network_config(self):
[00:27] <smoser>    if not network_json:
[00:27] <smoser> err.
[00:27] <smoser>    if not self.network_json:
[00:27] <smoser>      return None
[00:27] <rharper> I was looking at also doing the _store_networkdata() in stages.py (also done for userdata/vendordata)
[00:28] <smoser>    return convert_network_json_to_netweork_config(sle.fnetwork_json)
[00:28] <smoser> dont worry about that i dont hthink
[00:28] <rharper> ok
[00:28] <smoser> just look at how nocloud does it.
[00:28] <smoser> execept you ahve the conversion stage.
[00:28] <smoser> userdata and vendordata are different
[00:28] <smoser> because the get parsed
[00:29] <rharper> ok
[00:29] <smoser> and that parsing may include otehr things
[00:29] <smoser> basically we preproces them once
[00:29] <smoser> and save the result.
[00:29] <rharper> ok
[00:30] <rharper> in the convert, don't we want to write out the network_config file?   somewhere in the cloud dir like the other files ?
[00:34] <smoser> rharper, http://paste.ubuntu.com/15476227/
[00:34] <smoser> could you apply that ?
[00:34] <smoser> you can write it in the cloud dir if you want.
[00:34] <smoser> i guess its more nicely available for ispection then.
[00:35] <rharper> smoser: I'll apply
[00:35] <rharper> smoser: ok
[00:35] <smoser> as long as its store in the object
[00:35] <smoser> the object gets pickled
[00:35] <smoser> and then restored for inspection
[00:35] <smoser> the files like userdata are only stored because things outside read them.
[00:35] <smoser> or could i guess.
[00:36] <smoser> oh. and also, fyi
[00:36] <smoser>  https://code.launchpad.net/~smoser/+recipe/cloud-init-daily
[00:36] <smoser> the bug i'll need to fix tomorrow is why i see that WARN for fallback
[00:36] <smoser> and now i turn into a pumpkin
[00:37] <smoser> rharper, thank you for all your help.
[00:37] <smoser> this is really comming along.
[00:37] <rharper> ok
[00:37] <rharper> smoser: do I need updated trunk.net1 to verify your patch?
[00:38] <rharper> looks like it
[00:42] <rharper> smoser: the curtin eni parser does handle source.d now (blake_r added that just a bit ago)
[01:05] <rharper> smoser: magicalChicken: ok,pushed your fixes to cloud-init-test master
[13:15] <smoser> rharper, do you have a branch up for config drive ?
[14:00] <rharper> smoser: not yet; ran out of time last night;
[14:00] <rharper> I did commit your change to cloud-init-test
[14:00] <smoser> yeah, saw that.
[14:01] <smoser> i'm chasing that fallback datasource issue.
[14:01] <smoser> once i get that, plan to get the kernel command line stuff
[14:02] <rharper> I'm gonna get a test-case in for configdrive;
[14:02] <rharper> that'll help verify the changes to configdrive datasource
[14:03] <smoser> yeah.
[14:03] <smoser> that will be very good.
[15:09] <smoser> magicalChicken, aroun d?
[15:14] <smoser> rharper, ping
[15:15] <rharper> smoser: here
[15:15] <smoser> https://code.launchpad.net/~smoser/cloud-init/trunk.net1-cmdline-ip/
[15:16] <smoser> rharper, so was lookoing at magicalChicken 's kernel command line there.
[15:16] <smoser> i'd like some tests, but other than that it seems generally reasonable
[15:17] <smoser> the thing that i want to ask about, is about using the 'network_state'
[15:17] <smoser> rather than network_config
[15:17] <smoser> magicalChicken did this on the fallback config also.
[15:17] <smoser> i wanted / needed *config* rather than network state.
[15:17] <smoser> so questions are
[15:18] <smoser> a.) what renders me a config from a network state? i'm guessing there is something
[15:18] <smoser> b.) do you thikn its preferable to work with one or the other directly
[15:22] <smoser> rharper, ^
[15:25] <smoser> i realize that the merge proposal above doesn't really help you look at it.
[15:26] <smoser> pull my branch and diff it versus trunk.net1 http://paste.ubuntu.com/15480216/
[15:27] <rharper> smoser: nothing yet renders a network_config from network_state ; I was looking for that myself but it's not written
[15:27] <rharper> smoser: I've worked with both;  I find the network_config to be more useful since it can be written out and shared
[15:28] <smoser> well, that and right now its network_config -> network state as one way
[15:28] <smoser> :)
[15:29] <rharper> right, but in general, we're employing conversions from *something* (ip, configdrive) to network_config
[15:29] <rharper> and then allowing distro.apply_net() to take config to state to eni
[15:29] <rharper> that seems like a reasonable path for consistency across cloud-init
[15:31] <smoser> 2 way mapping in NetworkState would be good though.
[15:32] <smoser> its actually confusing, because 'dump' creates a dictionary with a 'config' and a 'version' that then makes you think "oh, taht is network_state -> network_config'
[15:32] <rharper> smoser: yes;  we should write the state -> network_config to make it 2 way; that's a good way to ensure we don't miss something
[15:49] <rharper> smoser: question on the ConfigDriveReader,  it has a _fetch_available_version() call which does some os.listdirs and collects dir names;  it returns it in no particular order;  I seem to be suffering from unsorted errors;  sometimes it doesn't select latest first; I'm only populating the network_data.json under latest/  in the test-case;  this passes; but if it chooses something else (2012-08-03) it will fail to find ne
[15:49] <rharper> twork_data.json
[15:52] <smoser> so _fetch_available_version() returns one ?
[15:52] <smoser> and not necessarily the one you want.
[15:52] <smoser> yeah, i think then just fix it.
[15:53] <rharper> so I think _fetch_available_versions which refernces OS_VERSIONS , shouldn't that include OS_LATEST ?
[15:56] <rharper> ah, it had only passed when selected = OS_LATEST, and supported array was empty; then it defaulted to OS_LATEST which is where I had put the network_data.json
[15:56] <rharper> it seems to me that OS_VERSIONS should include OS_LATEST; but I'm a bit worried about this change since I don't have a lot of history with configdrive
[15:58] <smoser> rharper, by design OS_VERSIONS does not include OS_LATEST.
[15:59] <smoser> unless some special circumstance, you really dont want to ever use 'latest'.
[15:59] <rharper> ok, then I'll need to populate another dir in addition to latest
[15:59] <rharper> that's easy enough
[15:59] <smoser> we're given a versioned api essentially , and assuming we can deal with whatever is "latest" is going to fail at some point.
[16:06] <rharper> smoser: I need to fix the one unittest, but this is working otherwise: lp:~raharper/cloud-init/config-drive-network-data ; I;'ve got to run, will be back in a while
[16:07] <smoser> k. i'll review. thanks
[17:07] <vox_clamantis> smoser ... still having trouble using openstack metadata. would you mind looking at a snippet of my log?
[17:09] <smoser> sure. pastebin the cloud-init.log
[17:10] <vox_clamantis> thanks very much
[17:11] <vox_clamantis> btw to test changes ive been removing /var/lib/cloud/* and rebooting. is that a bad idea? takes much longer to reconfigure the template
[17:13] <vox_clamantis> http://pastebin.com/CLm5c70E
[17:13] <vox_clamantis> this is with "datasource_list: [ OpenStack ]"
[17:16] <smoser> vox_clamantis, thats generally clean.
[17:16] <smoser> i clear /var/log/cloud-init* also
[17:16] <vox_clamantis> y that too good
[17:17] <vox_clamantis> and i reset to dhcp so i get the route to metadata
[17:17] <smoser> you ran cloud-init init --local
[17:17] <smoser> right ?
[17:18] <smoser> that will nto fidn a openstack datasource, as it will only search local data
[17:18] <vox_clamantis> just a reboot
[17:18] <smoser> hm.
[17:18] <vox_clamantis> can i initiate a non-local run?
[17:18] <smoser> so , without reboot, you can just do
[17:18] <smoser> rm -Rf /var/lib/cloud /var/log/cloud-init-*
[17:19] <smoser> cloud-init init --local
[17:19] <smoser> cloud-init init
[17:19] <vox_clamantis> ok
[17:20] <smoser> Searching for data source in: []
[17:20] <smoser> Mar 23 12:59:21 lin-test [CLOUDINIT] util.py[WARNING]: No instance datasource found! Likely bad things to come!
[17:21] <smoser> er.. the searching for data sourcein: []
[17:21] <vox_clamantis> yeah
[17:21] <smoser> that seems wrong.
[17:21] <vox_clamantis> i was concerned about that
[17:21] <smoser> but that coudl be a bad message.
[17:23] <smoser> could you just let me in some wher e?
[17:23] <smoser> is that possible, i can probalby solve it in a couple minutes if you can
[17:23] <smoser> rather than asking you 40 questions
[17:23] <vox_clamantis> depends what you mean by let you in
[17:23] <vox_clamantis> lol
[17:24] <vox_clamantis> not to the network unfortunately
[17:26] <smoser> 0.7.4 is pretty old. i know it sucks for someone to tell you "can you try a newer version"
[17:26] <smoser> but....
[17:27] <waldi> and python 2.6. does cloud-init provide support for that?
[17:27] <vox_clamantis> i dont mind at all
[17:27] <vox_clamantis> i believe it is the latest from rhel6 epel
[17:28] <vox_clamantis> yep still there
[17:28] <vox_clamantis> id be happy to try anohter
[17:29] <vox_clamantis> 0.7.6?
[17:32] <vox_clamantis> just need to install some build tools i think
[17:41] <smoser> thats surely better.
[17:42] <smoser> cloud-init should work on 2.6. but it could be that.
[17:46] <vox_clamantis> thanks for the suggestions
[17:46] <vox_clamantis> and appreciate all the work you do on this project
[18:12] <vox_clamantis> ugh i cant even build it on rhel 6. maybe i best just go to configdrive
[18:33] <rharper> smoser: ok, fixed up that last unittest for config-drive network_data
[18:34] <smoser> ok. i'll read.
[18:36] <smoser> rharper, i think your KILO is LIBERTY
[18:37] <rharper> smoser: OK, I wasn't sure;  i"ll fix up
[18:37] <smoser> https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L76
[18:38] <rharper> k
[18:40] <rharper> smoser: do we yet have a unittest that combines configdrive + distro.apply_network() ?  I don't think I saw a distro.apply_network() yet;
[18:41] <smoser> that does apply_network ?
[18:41] <smoser> or apply_network_config
[18:42] <rharper> which ever renders the network_config as distro config file (our case eni)
[18:43] <rharper> hoping to have a unittest that pulls the json data from ConfigDrive, convert to network_config yaml, then run apply_network() which should call net.render_interfaces() which writes out an eni
[18:43] <rharper> for VM test, we will need to generate a configdrive iso file and feed the VM that (does the cloudds-local util do that? )
[18:44] <smoser> convert_networkdata_json
[18:44] <smoser> what does that do ?
[18:44] <smoser> rharper, cloud-localds does not do that at the moment.
[18:44] <rharper> converts the json into network_config yaml
[18:45] <rharper> this shows up as datasource.network_config (attr)
[18:45] <rharper> like in nocloud ds
[18:45] <rharper> I put a long description in the function itself
[18:47] <smoser> no..
[18:48] <smoser> convert_networkdata_jso
[18:48] <smoser> convert_networkdata_json
[18:48] <rharper> oh, in openstack helpers
[18:48] <smoser> right.
[18:48] <smoser> not convert_network_data
[18:48] <rharper> I assumed it was needed to do the json read , like vendor data
[18:48] <rharper> but maybe it's not needed at all
[18:49] <rharper> it looked to me like it ensured that we opened the data as list (json.load()) and returned the list that should be in the json data
[18:53] <smoser> hm.. ok. i think it should be really necessary.
[18:53] <smoser> we just want to laod whatever is there. if its bad, then its bad.
[18:54] <smoser> rharper, does network config have a 'hostname' ?
[18:54] <rharper> hrm
[18:54] <rharper> I don;'t think so since hostname wasn't set in eni
[18:54] <rharper> but nothing prevents us from adding it
[19:01] <rharper> smoser: you mean it should *not* be needed ?
[19:04] <smoser> right.
[19:04] <smoser> should not need that i think
[19:04] <rharper> k
[19:04] <smoser> right.
[19:04] <smoser> sorry
[19:05] <rharper> heh, we don't actually run get_data(); that's sorta mocked out
[19:06] <rharper> which makes me somewhat uncomfortable with knowing whether or not we needed that or not (though I think using util.load_json() should be fine as you say)
[19:12] <smoser> rharper, it shoudl be reasonably easy to *acutally* test.
[19:12] <smoser> kind of painful
[19:13] <smoser> but easy to add the config drive data as /dev/vdb on a instance
[19:13] <smoser> ie, serverstack then just dd that stuff that we got yesterday
[19:13] <smoser> except for with the fixed diredtory
[19:16] <rharper> smoser: ok
[19:16] <smoser> i do understand wanting to actually have unit test though
[19:18] <rharper> I might try my hand at a get_data() call on datasource
[19:18] <rharper> mock can help for most of it
[19:26] <rharper> smoser: if I supply a config drive, what should I specify on the kernel cmdline to have cloud-init use that?
[19:33] <smoser> rharper, it will just use it.
[19:33] <smoser> so what you can do is boot the system
[19:33] <smoser> then just dd the content to /dev/vdb
[19:33] <smoser> the ephemeral device
[19:33] <smoser> and then run
[19:33] <rharper> I don't need to specify ds= XXX or antthing like that
[19:33] <smoser> rm -Rf /var/lib/cloud /var/log/cloud-init=*
[19:33] <rharper> ah, and reboot
[19:33] <smoser> no. because its in the normal search
[19:33] <smoser> dont have to reboot
[19:33] <rharper> it'll re-run cloud-init
[19:33] <rharper> oh
[19:33] <smoser> cloud-init init --local
[19:33] <rharper> nice
[19:33] <rharper> ok
[19:33] <smoser> cloud-init init
[19:34] <smoser> that shoudl push it through the code your'e after.
[19:34] <rharper> nice
[19:34] <rharper> should I update something in cloud-utils to generate config-drive ?  that'd be useful for cloud-init-test
[19:35] <smoser> yeah, it woudl be useful. definitely we need it for the vmtest
[19:35] <rharper> ok
[19:35] <rharper> should I modify cloud-localds, or create something new cloud-config-drive ?
[19:35] <rharper> I assume a new command
[19:35] <rharper> bbiab, need to pickup kiddo
[19:36] <smoser> yeah, i think new command
[19:54] <smoser> magicalChicken, https://code.launchpad.net/~smoser/cloud-init/trunk.net1-cmdline-ip/+merge/289968
[19:54] <smoser> if you have time, take that and run with it.
[19:55] <smoser> that may be functioanl for ip= on cmdline.
[19:55] <smoser> based on a lot of your work
[19:55] <smoser> rharper, i have to run now.
[19:55] <smoser> will be in tomorrow
[19:55] <smoser> give your code a test. but largely it looked good to me.
[19:59] <magicalChicken> smoser: Sure, I'll get it merged tonight and make sure it works
[20:02] <rharper> smoser: cool
[20:53] <rharper> smoser: urg, so when I re-inited cloud-init, it decided to use DataSourceOpenStack instead of DataSourceConfigDrive since I'm running this in a OS cloud;  need to figure out how to specify order I supose
[21:14] <rharper> bah, and /dev/sr0 is populated with configdrive as well
[21:14]  * rharper launches without config-drive=1 
[21:57] <rharper> smoser: cool, got it parsing a config drive;  I need to generate a clean config-drive that doesn't include meta-data that dumps the content file (like the rackspace one does) and we should see or interfaces.d/50- file rendered
[22:09] <rharper> ok, now we're not writing out network_config from content;  but I don't see distro.apply_network() ; I wonder if I need to enable a different module
[22:19] <rharper> sweet, works!  I don't quite understand why it wont' apply when I run cloud-init init ( the only call to init.apply_network_config()) happens in stage5 and only if you pass --local parameter;  however, when I run cloud-init init --local; it's definitely not run either.  so as a hack, I dropped the if args.local check; and it writes it out fine;  so I'll leave the MP as-is and smoser you can help me figure out what I'm m
[22:19] <rharper> issing here