[00:02] smoser: around ? [00:02] here [00:02] looking at why i broke your test suite [00:03] theres two issues i'm seeing [00:03] a.) a WARN in the log says the fallback datasource got used, but idont know how (and other parts say NoCloud got used) [00:04] b.) it seems the nics aren't being renamed appoprirately. [00:04] do you have an easy way to get in to the test vm ? [00:09] ok. well, 'b' seems to be because we have a /etc/network/interfaces inside that doesnt' srouce /etc/network/interfaces.d [00:13] smoser: it's somewhat tricky to get it but: I usually drop the poweroff script in generate_payload_scripts(); this will leave the system up [00:15] the if you ps, you can get the /tmp/launch.XXXX/boot.img ; copy that out before you kill the vm and the qemu command line; then I replace -serial file.* with -serial telnet:localhost:2446,server,nowait , quote the --append, then add -monitor stdio -S; when you run it; you'll get monitor prompt on stdio, guest will be paused; you can setup your telnet to localhost on the port; then type 'cont' on the monitor; [00:18] smoser: I've got the configdrive openstack helper updated to search for the network_data.json, so we can fetch this, we use the same convert_to_json() logic and stuff that in configdrive.networkdata_raw; I see in stages where userdata_raw and vendordata_raw are processed (and written out); ideally I'd do that with networkdata and write out a networkdata.json ; I'm trying to find where best to convert the json networ [00:18] kdata into network config; it appears that I need to write out network_config.yaml where we write out userdata conf as well, and then the distro apply_net will do it's thing [00:19] oh. well the ps is easily fixable [00:19] rharper, CURTIN_VMTEST_KEEP_DATA_PASS=logs,collect [00:19] run with that, add 'disks' to it and you'll keep the disks too [00:19] I usually want the interactive nature [00:20] right. but i was just saying to keep the disk around [00:20] oh [00:20] so you dont have to race it. [00:20] yeah [00:20] ah, gotcha [00:20] or modify the power stage [00:20] yep [00:20] ok. so i understand most of the failure now [00:20] and have some changes. [00:20] but test_etc_network_interfaces [00:20] will have to be updated [00:21] as now i'm (by design) not writing to /etc/network/interfaces [00:23] http://paste.ubuntu.com/15476168/ (my debug script) [00:24] smoser: yes; I think we should use the net.parse_eni() which dumps a dictionary [00:24] from there we can compare/contrast with a network_state [00:24] ok. [00:24] i'll getthis as close as i can [00:24] and then can you do that ? [00:24] alternatively, we need to write a network_state that can re-render network_config.yaml [00:24] i'll submit MP for your git in next 5 minutes. [00:24] prolly [00:25] but I'd like to get your thoughts on the config drive bits [00:25] that's what I'm trying to finish up right now [00:25] k [00:25] where to convert the network_data into network_config yaml [00:26] ah. ok [00:26] so you have the networkdata_raw (although its not *really* raw) as you json loaded it. [00:26] right [00:26] you can call it whatever you want. [00:26] sure keep it around in the datasource datasource.network_json [00:26] the same as vendordata_raw (which also was json loaded) [00:27] then, in datasource.network_config() [00:27] its a property [00:27] you can convert it there or anywhere really. [00:27] would probably be nice to have the code easily runnable with no class wrappers or something [00:27] def network_config(self): [00:27] if not network_json: [00:27] err. [00:27] if not self.network_json: [00:27] return None [00:27] I was looking at also doing the _store_networkdata() in stages.py (also done for userdata/vendordata) [00:28] return convert_network_json_to_netweork_config(sle.fnetwork_json) [00:28] dont worry about that i dont hthink [00:28] ok [00:28] just look at how nocloud does it. [00:28] execept you ahve the conversion stage. [00:28] userdata and vendordata are different [00:28] because the get parsed [00:29] ok [00:29] and that parsing may include otehr things [00:29] basically we preproces them once [00:29] and save the result. [00:29] ok [00:30] in the convert, don't we want to write out the network_config file? somewhere in the cloud dir like the other files ? [00:34] rharper, http://paste.ubuntu.com/15476227/ [00:34] could you apply that ? [00:34] you can write it in the cloud dir if you want. [00:34] i guess its more nicely available for ispection then. [00:35] smoser: I'll apply [00:35] smoser: ok [00:35] as long as its store in the object [00:35] the object gets pickled [00:35] and then restored for inspection [00:35] the files like userdata are only stored because things outside read them. [00:35] or could i guess. [00:36] oh. and also, fyi [00:36] https://code.launchpad.net/~smoser/+recipe/cloud-init-daily [00:36] the bug i'll need to fix tomorrow is why i see that WARN for fallback [00:36] and now i turn into a pumpkin [00:37] rharper, thank you for all your help. [00:37] this is really comming along. [00:37] ok [00:37] smoser: do I need updated trunk.net1 to verify your patch? [00:38] looks like it [00:42] smoser: the curtin eni parser does handle source.d now (blake_r added that just a bit ago) [01:05] smoser: magicalChicken: ok,pushed your fixes to cloud-init-test master [13:15] rharper, do you have a branch up for config drive ? [14:00] smoser: not yet; ran out of time last night; [14:00] I did commit your change to cloud-init-test [14:00] yeah, saw that. [14:01] i'm chasing that fallback datasource issue. [14:01] once i get that, plan to get the kernel command line stuff [14:02] I'm gonna get a test-case in for configdrive; [14:02] that'll help verify the changes to configdrive datasource [14:03] yeah. [14:03] that will be very good. [15:09] magicalChicken, aroun d? [15:14] rharper, ping [15:15] smoser: here [15:15] https://code.launchpad.net/~smoser/cloud-init/trunk.net1-cmdline-ip/ [15:16] rharper, so was lookoing at magicalChicken 's kernel command line there. [15:16] i'd like some tests, but other than that it seems generally reasonable [15:17] the thing that i want to ask about, is about using the 'network_state' [15:17] rather than network_config [15:17] magicalChicken did this on the fallback config also. [15:17] i wanted / needed *config* rather than network state. [15:17] so questions are [15:18] a.) what renders me a config from a network state? i'm guessing there is something [15:18] b.) do you thikn its preferable to work with one or the other directly [15:22] rharper, ^ [15:25] i realize that the merge proposal above doesn't really help you look at it. [15:26] pull my branch and diff it versus trunk.net1 http://paste.ubuntu.com/15480216/ [15:27] smoser: nothing yet renders a network_config from network_state ; I was looking for that myself but it's not written [15:27] smoser: I've worked with both; I find the network_config to be more useful since it can be written out and shared [15:28] well, that and right now its network_config -> network state as one way [15:28] :) [15:29] right, but in general, we're employing conversions from *something* (ip, configdrive) to network_config [15:29] and then allowing distro.apply_net() to take config to state to eni [15:29] that seems like a reasonable path for consistency across cloud-init [15:31] 2 way mapping in NetworkState would be good though. [15:32] its actually confusing, because 'dump' creates a dictionary with a 'config' and a 'version' that then makes you think "oh, taht is network_state -> network_config' [15:32] smoser: yes; we should write the state -> network_config to make it 2 way; that's a good way to ensure we don't miss something [15:49] smoser: question on the ConfigDriveReader, it has a _fetch_available_version() call which does some os.listdirs and collects dir names; it returns it in no particular order; I seem to be suffering from unsorted errors; sometimes it doesn't select latest first; I'm only populating the network_data.json under latest/ in the test-case; this passes; but if it chooses something else (2012-08-03) it will fail to find ne [15:49] twork_data.json [15:52] so _fetch_available_version() returns one ? [15:52] and not necessarily the one you want. [15:52] yeah, i think then just fix it. [15:53] so I think _fetch_available_versions which refernces OS_VERSIONS , shouldn't that include OS_LATEST ? [15:56] ah, it had only passed when selected = OS_LATEST, and supported array was empty; then it defaulted to OS_LATEST which is where I had put the network_data.json [15:56] it seems to me that OS_VERSIONS should include OS_LATEST; but I'm a bit worried about this change since I don't have a lot of history with configdrive [15:58] rharper, by design OS_VERSIONS does not include OS_LATEST. [15:59] unless some special circumstance, you really dont want to ever use 'latest'. [15:59] ok, then I'll need to populate another dir in addition to latest [15:59] that's easy enough [15:59] we're given a versioned api essentially , and assuming we can deal with whatever is "latest" is going to fail at some point. [16:06] smoser: I need to fix the one unittest, but this is working otherwise: lp:~raharper/cloud-init/config-drive-network-data ; I;'ve got to run, will be back in a while [16:07] k. i'll review. thanks [17:07] smoser ... still having trouble using openstack metadata. would you mind looking at a snippet of my log? [17:09] sure. pastebin the cloud-init.log [17:10] thanks very much [17:11] btw to test changes ive been removing /var/lib/cloud/* and rebooting. is that a bad idea? takes much longer to reconfigure the template [17:13] http://pastebin.com/CLm5c70E [17:13] this is with "datasource_list: [ OpenStack ]" [17:16] vox_clamantis, thats generally clean. [17:16] i clear /var/log/cloud-init* also [17:16] y that too good [17:17] and i reset to dhcp so i get the route to metadata [17:17] you ran cloud-init init --local [17:17] right ? [17:18] that will nto fidn a openstack datasource, as it will only search local data [17:18] just a reboot [17:18] hm. [17:18] can i initiate a non-local run? [17:18] so , without reboot, you can just do [17:18] rm -Rf /var/lib/cloud /var/log/cloud-init-* [17:19] cloud-init init --local [17:19] cloud-init init [17:19] ok [17:20] Searching for data source in: [] [17:20] Mar 23 12:59:21 lin-test [CLOUDINIT] util.py[WARNING]: No instance datasource found! Likely bad things to come! [17:21] er.. the searching for data sourcein: [] [17:21] yeah [17:21] that seems wrong. [17:21] i was concerned about that [17:21] but that coudl be a bad message. [17:23] could you just let me in some wher e? [17:23] is that possible, i can probalby solve it in a couple minutes if you can [17:23] rather than asking you 40 questions [17:23] depends what you mean by let you in [17:23] lol [17:24] not to the network unfortunately [17:26] 0.7.4 is pretty old. i know it sucks for someone to tell you "can you try a newer version" [17:26] but.... [17:27] and python 2.6. does cloud-init provide support for that? [17:27] i dont mind at all [17:27] i believe it is the latest from rhel6 epel [17:28] yep still there [17:28] id be happy to try anohter [17:29] 0.7.6? [17:32] just need to install some build tools i think [17:41] thats surely better. [17:42] cloud-init should work on 2.6. but it could be that. [17:46] thanks for the suggestions [17:46] and appreciate all the work you do on this project [18:12] ugh i cant even build it on rhel 6. maybe i best just go to configdrive [18:33] smoser: ok, fixed up that last unittest for config-drive network_data [18:34] ok. i'll read. [18:36] rharper, i think your KILO is LIBERTY [18:37] smoser: OK, I wasn't sure; i"ll fix up [18:37] https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L76 [18:38] k [18:40] smoser: do we yet have a unittest that combines configdrive + distro.apply_network() ? I don't think I saw a distro.apply_network() yet; [18:41] that does apply_network ? [18:41] or apply_network_config [18:42] which ever renders the network_config as distro config file (our case eni) [18:43] hoping to have a unittest that pulls the json data from ConfigDrive, convert to network_config yaml, then run apply_network() which should call net.render_interfaces() which writes out an eni [18:43] for VM test, we will need to generate a configdrive iso file and feed the VM that (does the cloudds-local util do that? ) [18:44] convert_networkdata_json [18:44] what does that do ? [18:44] rharper, cloud-localds does not do that at the moment. [18:44] converts the json into network_config yaml [18:45] this shows up as datasource.network_config (attr) [18:45] like in nocloud ds [18:45] I put a long description in the function itself [18:47] no.. [18:48] convert_networkdata_jso [18:48] convert_networkdata_json [18:48] oh, in openstack helpers [18:48] right. [18:48] not convert_network_data [18:48] I assumed it was needed to do the json read , like vendor data [18:48] but maybe it's not needed at all [18:49] it looked to me like it ensured that we opened the data as list (json.load()) and returned the list that should be in the json data [18:53] hm.. ok. i think it should be really necessary. [18:53] we just want to laod whatever is there. if its bad, then its bad. [18:54] rharper, does network config have a 'hostname' ? [18:54] hrm [18:54] I don;'t think so since hostname wasn't set in eni [18:54] but nothing prevents us from adding it [19:01] smoser: you mean it should *not* be needed ? [19:04] right. [19:04] should not need that i think [19:04] k [19:04] right. [19:04] sorry [19:05] heh, we don't actually run get_data(); that's sorta mocked out [19:06] which makes me somewhat uncomfortable with knowing whether or not we needed that or not (though I think using util.load_json() should be fine as you say) [19:12] rharper, it shoudl be reasonably easy to *acutally* test. [19:12] kind of painful [19:13] but easy to add the config drive data as /dev/vdb on a instance [19:13] ie, serverstack then just dd that stuff that we got yesterday [19:13] except for with the fixed diredtory [19:16] smoser: ok [19:16] i do understand wanting to actually have unit test though [19:18] I might try my hand at a get_data() call on datasource [19:18] mock can help for most of it === devicenull_ is now known as devicenull [19:26] smoser: if I supply a config drive, what should I specify on the kernel cmdline to have cloud-init use that? === JasonF is now known as JayF [19:33] rharper, it will just use it. [19:33] so what you can do is boot the system [19:33] then just dd the content to /dev/vdb [19:33] the ephemeral device [19:33] and then run [19:33] I don't need to specify ds= XXX or antthing like that [19:33] rm -Rf /var/lib/cloud /var/log/cloud-init=* [19:33] ah, and reboot [19:33] no. because its in the normal search [19:33] dont have to reboot [19:33] it'll re-run cloud-init [19:33] oh [19:33] cloud-init init --local [19:33] nice [19:33] ok [19:33] cloud-init init [19:34] that shoudl push it through the code your'e after. [19:34] nice [19:34] should I update something in cloud-utils to generate config-drive ? that'd be useful for cloud-init-test [19:35] yeah, it woudl be useful. definitely we need it for the vmtest [19:35] ok [19:35] should I modify cloud-localds, or create something new cloud-config-drive ? [19:35] I assume a new command [19:35] bbiab, need to pickup kiddo [19:36] yeah, i think new command [19:54] magicalChicken, https://code.launchpad.net/~smoser/cloud-init/trunk.net1-cmdline-ip/+merge/289968 [19:54] if you have time, take that and run with it. [19:55] that may be functioanl for ip= on cmdline. [19:55] based on a lot of your work [19:55] rharper, i have to run now. [19:55] will be in tomorrow [19:55] give your code a test. but largely it looked good to me. [19:59] smoser: Sure, I'll get it merged tonight and make sure it works [20:02] smoser: cool [20:53] smoser: urg, so when I re-inited cloud-init, it decided to use DataSourceOpenStack instead of DataSourceConfigDrive since I'm running this in a OS cloud; need to figure out how to specify order I supose [21:14] bah, and /dev/sr0 is populated with configdrive as well [21:14] * rharper launches without config-drive=1 [21:57] smoser: cool, got it parsing a config drive; I need to generate a clean config-drive that doesn't include meta-data that dumps the content file (like the rackspace one does) and we should see or interfaces.d/50- file rendered [22:09] ok, now we're not writing out network_config from content; but I don't see distro.apply_network() ; I wonder if I need to enable a different module [22:19] sweet, works! I don't quite understand why it wont' apply when I run cloud-init init ( the only call to init.apply_network_config()) happens in stage5 and only if you pass --local parameter; however, when I run cloud-init init --local; it's definitely not run either. so as a hack, I dropped the if args.local check; and it writes it out fine; so I'll leave the MP as-is and smoser you can help me figure out what I'm m [22:19] issing here