| rharper | smoser: around ? | 00:02 |
|---|---|---|
| smoser | here | 00:02 |
| smoser | looking at why i broke your test suite | 00:02 |
| smoser | theres two issues i'm seeing | 00:03 |
| smoser | a.) a WARN in the log says the fallback datasource got used, but idont know how (and other parts say NoCloud got used) | 00:03 |
| smoser | b.) it seems the nics aren't being renamed appoprirately. | 00:04 |
| smoser | do you have an easy way to get in to the test vm ? | 00:04 |
| smoser | ok. well, 'b' seems to be because we have a /etc/network/interfaces inside that doesnt' srouce /etc/network/interfaces.d | 00:09 |
| rharper | smoser: it's somewhat tricky to get it but: I usually drop the poweroff script in generate_payload_scripts(); this will leave the system up | 00:13 |
| rharper | the if you ps, you can get the /tmp/launch.XXXX/boot.img ; copy that out before you kill the vm and the qemu command line; then I replace -serial file.* with -serial telnet:localhost:2446,server,nowait , quote the --append, then add -monitor stdio -S; when you run it; you'll get monitor prompt on stdio, guest will be paused; you can setup your telnet to localhost on the port; then type 'cont' on the monitor; | 00:15 |
| rharper | smoser: I've got the configdrive openstack helper updated to search for the network_data.json, so we can fetch this, we use the same convert_to_json() logic and stuff that in configdrive.networkdata_raw; I see in stages where userdata_raw and vendordata_raw are processed (and written out); ideally I'd do that with networkdata and write out a networkdata.json ; I'm trying to find where best to convert the json networ | 00:18 |
| rharper | kdata into network config; it appears that I need to write out network_config.yaml where we write out userdata conf as well, and then the distro apply_net will do it's thing | 00:18 |
| smoser | oh. well the ps is easily fixable | 00:19 |
| smoser | rharper, CURTIN_VMTEST_KEEP_DATA_PASS=logs,collect | 00:19 |
| smoser | run with that, add 'disks' to it and you'll keep the disks too | 00:19 |
| rharper | I usually want the interactive nature | 00:19 |
| smoser | right. but i was just saying to keep the disk around | 00:20 |
| rharper | oh | 00:20 |
| smoser | so you dont have to race it. | 00:20 |
| rharper | yeah | 00:20 |
| rharper | ah, gotcha | 00:20 |
| rharper | or modify the power stage | 00:20 |
| rharper | yep | 00:20 |
| smoser | ok. so i understand most of the failure now | 00:20 |
| smoser | and have some changes. | 00:20 |
| smoser | but test_etc_network_interfaces | 00:20 |
| smoser | will have to be updated | 00:20 |
| smoser | as now i'm (by design) not writing to /etc/network/interfaces | 00:21 |
| rharper | http://paste.ubuntu.com/15476168/ (my debug script) | 00:23 |
| rharper | smoser: yes; I think we should use the net.parse_eni() which dumps a dictionary | 00:24 |
| rharper | from there we can compare/contrast with a network_state | 00:24 |
| smoser | ok. | 00:24 |
| smoser | i'll getthis as close as i can | 00:24 |
| smoser | and then can you do that ? | 00:24 |
| rharper | alternatively, we need to write a network_state that can re-render network_config.yaml | 00:24 |
| smoser | i'll submit MP for your git in next 5 minutes. | 00:24 |
| rharper | prolly | 00:24 |
| rharper | but I'd like to get your thoughts on the config drive bits | 00:25 |
| rharper | that's what I'm trying to finish up right now | 00:25 |
| smoser | k | 00:25 |
| rharper | where to convert the network_data into network_config yaml | 00:25 |
| smoser | ah. ok | 00:26 |
| smoser | so you have the networkdata_raw (although its not *really* raw) as you json loaded it. | 00:26 |
| rharper | right | 00:26 |
| smoser | you can call it whatever you want. | 00:26 |
| smoser | sure keep it around in the datasource datasource.network_json | 00:26 |
| rharper | the same as vendordata_raw (which also was json loaded) | 00:26 |
| smoser | then, in datasource.network_config() | 00:27 |
| smoser | its a property | 00:27 |
| smoser | you can convert it there or anywhere really. | 00:27 |
| smoser | would probably be nice to have the code easily runnable with no class wrappers or something | 00:27 |
| smoser | def network_config(self): | 00:27 |
| smoser | if not network_json: | 00:27 |
| smoser | err. | 00:27 |
| smoser | if not self.network_json: | 00:27 |
| smoser | return None | 00:27 |
| rharper | I was looking at also doing the _store_networkdata() in stages.py (also done for userdata/vendordata) | 00:27 |
| smoser | return convert_network_json_to_netweork_config(sle.fnetwork_json) | 00:28 |
| smoser | dont worry about that i dont hthink | 00:28 |
| rharper | ok | 00:28 |
| smoser | just look at how nocloud does it. | 00:28 |
| smoser | execept you ahve the conversion stage. | 00:28 |
| smoser | userdata and vendordata are different | 00:28 |
| smoser | because the get parsed | 00:28 |
| rharper | ok | 00:29 |
| smoser | and that parsing may include otehr things | 00:29 |
| smoser | basically we preproces them once | 00:29 |
| smoser | and save the result. | 00:29 |
| rharper | ok | 00:29 |
| rharper | in the convert, don't we want to write out the network_config file? somewhere in the cloud dir like the other files ? | 00:30 |
| smoser | rharper, http://paste.ubuntu.com/15476227/ | 00:34 |
| smoser | could you apply that ? | 00:34 |
| smoser | you can write it in the cloud dir if you want. | 00:34 |
| smoser | i guess its more nicely available for ispection then. | 00:34 |
| rharper | smoser: I'll apply | 00:35 |
| rharper | smoser: ok | 00:35 |
| smoser | as long as its store in the object | 00:35 |
| smoser | the object gets pickled | 00:35 |
| smoser | and then restored for inspection | 00:35 |
| smoser | the files like userdata are only stored because things outside read them. | 00:35 |
| smoser | or could i guess. | 00:35 |
| smoser | oh. and also, fyi | 00:36 |
| smoser | https://code.launchpad.net/~smoser/+recipe/cloud-init-daily | 00:36 |
| smoser | the bug i'll need to fix tomorrow is why i see that WARN for fallback | 00:36 |
| smoser | and now i turn into a pumpkin | 00:36 |
| smoser | rharper, thank you for all your help. | 00:37 |
| smoser | this is really comming along. | 00:37 |
| rharper | ok | 00:37 |
| rharper | smoser: do I need updated trunk.net1 to verify your patch? | 00:37 |
| rharper | looks like it | 00:38 |
| rharper | smoser: the curtin eni parser does handle source.d now (blake_r added that just a bit ago) | 00:42 |
| rharper | smoser: magicalChicken: ok,pushed your fixes to cloud-init-test master | 01:05 |
| smoser | rharper, do you have a branch up for config drive ? | 13:15 |
| rharper | smoser: not yet; ran out of time last night; | 14:00 |
| rharper | I did commit your change to cloud-init-test | 14:00 |
| smoser | yeah, saw that. | 14:00 |
| smoser | i'm chasing that fallback datasource issue. | 14:01 |
| smoser | once i get that, plan to get the kernel command line stuff | 14:01 |
| rharper | I'm gonna get a test-case in for configdrive; | 14:02 |
| rharper | that'll help verify the changes to configdrive datasource | 14:02 |
| smoser | yeah. | 14:03 |
| smoser | that will be very good. | 14:03 |
| smoser | magicalChicken, aroun d? | 15:09 |
| smoser | rharper, ping | 15:14 |
| rharper | smoser: here | 15:15 |
| smoser | https://code.launchpad.net/~smoser/cloud-init/trunk.net1-cmdline-ip/ | 15:15 |
| smoser | rharper, so was lookoing at magicalChicken 's kernel command line there. | 15:16 |
| smoser | i'd like some tests, but other than that it seems generally reasonable | 15:16 |
| smoser | the thing that i want to ask about, is about using the 'network_state' | 15:17 |
| smoser | rather than network_config | 15:17 |
| smoser | magicalChicken did this on the fallback config also. | 15:17 |
| smoser | i wanted / needed *config* rather than network state. | 15:17 |
| smoser | so questions are | 15:17 |
| smoser | a.) what renders me a config from a network state? i'm guessing there is something | 15:18 |
| smoser | b.) do you thikn its preferable to work with one or the other directly | 15:18 |
| smoser | rharper, ^ | 15:22 |
| smoser | i realize that the merge proposal above doesn't really help you look at it. | 15:25 |
| smoser | pull my branch and diff it versus trunk.net1 http://paste.ubuntu.com/15480216/ | 15:26 |
| rharper | smoser: nothing yet renders a network_config from network_state ; I was looking for that myself but it's not written | 15:27 |
| rharper | smoser: I've worked with both; I find the network_config to be more useful since it can be written out and shared | 15:27 |
| smoser | well, that and right now its network_config -> network state as one way | 15:28 |
| smoser | :) | 15:28 |
| rharper | right, but in general, we're employing conversions from *something* (ip, configdrive) to network_config | 15:29 |
| rharper | and then allowing distro.apply_net() to take config to state to eni | 15:29 |
| rharper | that seems like a reasonable path for consistency across cloud-init | 15:29 |
| smoser | 2 way mapping in NetworkState would be good though. | 15:31 |
| smoser | its actually confusing, because 'dump' creates a dictionary with a 'config' and a 'version' that then makes you think "oh, taht is network_state -> network_config' | 15:32 |
| rharper | smoser: yes; we should write the state -> network_config to make it 2 way; that's a good way to ensure we don't miss something | 15:32 |
| rharper | smoser: question on the ConfigDriveReader, it has a _fetch_available_version() call which does some os.listdirs and collects dir names; it returns it in no particular order; I seem to be suffering from unsorted errors; sometimes it doesn't select latest first; I'm only populating the network_data.json under latest/ in the test-case; this passes; but if it chooses something else (2012-08-03) it will fail to find ne | 15:49 |
| rharper | twork_data.json | 15:49 |
| smoser | so _fetch_available_version() returns one ? | 15:52 |
| smoser | and not necessarily the one you want. | 15:52 |
| smoser | yeah, i think then just fix it. | 15:52 |
| rharper | so I think _fetch_available_versions which refernces OS_VERSIONS , shouldn't that include OS_LATEST ? | 15:53 |
| rharper | ah, it had only passed when selected = OS_LATEST, and supported array was empty; then it defaulted to OS_LATEST which is where I had put the network_data.json | 15:56 |
| rharper | it seems to me that OS_VERSIONS should include OS_LATEST; but I'm a bit worried about this change since I don't have a lot of history with configdrive | 15:56 |
| smoser | rharper, by design OS_VERSIONS does not include OS_LATEST. | 15:58 |
| smoser | unless some special circumstance, you really dont want to ever use 'latest'. | 15:59 |
| rharper | ok, then I'll need to populate another dir in addition to latest | 15:59 |
| rharper | that's easy enough | 15:59 |
| smoser | we're given a versioned api essentially , and assuming we can deal with whatever is "latest" is going to fail at some point. | 15:59 |
| rharper | smoser: I need to fix the one unittest, but this is working otherwise: lp:~raharper/cloud-init/config-drive-network-data ; I;'ve got to run, will be back in a while | 16:06 |
| smoser | k. i'll review. thanks | 16:07 |
| vox_clamantis | smoser ... still having trouble using openstack metadata. would you mind looking at a snippet of my log? | 17:07 |
| smoser | sure. pastebin the cloud-init.log | 17:09 |
| vox_clamantis | thanks very much | 17:10 |
| vox_clamantis | btw to test changes ive been removing /var/lib/cloud/* and rebooting. is that a bad idea? takes much longer to reconfigure the template | 17:11 |
| vox_clamantis | http://pastebin.com/CLm5c70E | 17:13 |
| vox_clamantis | this is with "datasource_list: [ OpenStack ]" | 17:13 |
| smoser | vox_clamantis, thats generally clean. | 17:16 |
| smoser | i clear /var/log/cloud-init* also | 17:16 |
| vox_clamantis | y that too good | 17:16 |
| vox_clamantis | and i reset to dhcp so i get the route to metadata | 17:17 |
| smoser | you ran cloud-init init --local | 17:17 |
| smoser | right ? | 17:17 |
| smoser | that will nto fidn a openstack datasource, as it will only search local data | 17:18 |
| vox_clamantis | just a reboot | 17:18 |
| smoser | hm. | 17:18 |
| vox_clamantis | can i initiate a non-local run? | 17:18 |
| smoser | so , without reboot, you can just do | 17:18 |
| smoser | rm -Rf /var/lib/cloud /var/log/cloud-init-* | 17:18 |
| smoser | cloud-init init --local | 17:19 |
| smoser | cloud-init init | 17:19 |
| vox_clamantis | ok | 17:19 |
| smoser | Searching for data source in: [] | 17:20 |
| smoser | Mar 23 12:59:21 lin-test [CLOUDINIT] util.py[WARNING]: No instance datasource found! Likely bad things to come! | 17:20 |
| smoser | er.. the searching for data sourcein: [] | 17:21 |
| vox_clamantis | yeah | 17:21 |
| smoser | that seems wrong. | 17:21 |
| vox_clamantis | i was concerned about that | 17:21 |
| smoser | but that coudl be a bad message. | 17:21 |
| smoser | could you just let me in some wher e? | 17:23 |
| smoser | is that possible, i can probalby solve it in a couple minutes if you can | 17:23 |
| smoser | rather than asking you 40 questions | 17:23 |
| vox_clamantis | depends what you mean by let you in | 17:23 |
| vox_clamantis | lol | 17:23 |
| vox_clamantis | not to the network unfortunately | 17:24 |
| smoser | 0.7.4 is pretty old. i know it sucks for someone to tell you "can you try a newer version" | 17:26 |
| smoser | but.... | 17:26 |
| waldi | and python 2.6. does cloud-init provide support for that? | 17:27 |
| vox_clamantis | i dont mind at all | 17:27 |
| vox_clamantis | i believe it is the latest from rhel6 epel | 17:27 |
| vox_clamantis | yep still there | 17:28 |
| vox_clamantis | id be happy to try anohter | 17:28 |
| vox_clamantis | 0.7.6? | 17:29 |
| vox_clamantis | just need to install some build tools i think | 17:32 |
| smoser | thats surely better. | 17:41 |
| smoser | cloud-init should work on 2.6. but it could be that. | 17:42 |
| vox_clamantis | thanks for the suggestions | 17:46 |
| vox_clamantis | and appreciate all the work you do on this project | 17:46 |
| vox_clamantis | ugh i cant even build it on rhel 6. maybe i best just go to configdrive | 18:12 |
| rharper | smoser: ok, fixed up that last unittest for config-drive network_data | 18:33 |
| smoser | ok. i'll read. | 18:34 |
| smoser | rharper, i think your KILO is LIBERTY | 18:36 |
| rharper | smoser: OK, I wasn't sure; i"ll fix up | 18:37 |
| smoser | https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L76 | 18:37 |
| rharper | k | 18:38 |
| rharper | smoser: do we yet have a unittest that combines configdrive + distro.apply_network() ? I don't think I saw a distro.apply_network() yet; | 18:40 |
| smoser | that does apply_network ? | 18:41 |
| smoser | or apply_network_config | 18:41 |
| rharper | which ever renders the network_config as distro config file (our case eni) | 18:42 |
| rharper | hoping to have a unittest that pulls the json data from ConfigDrive, convert to network_config yaml, then run apply_network() which should call net.render_interfaces() which writes out an eni | 18:43 |
| rharper | for VM test, we will need to generate a configdrive iso file and feed the VM that (does the cloudds-local util do that? ) | 18:43 |
| smoser | convert_networkdata_json | 18:44 |
| smoser | what does that do ? | 18:44 |
| smoser | rharper, cloud-localds does not do that at the moment. | 18:44 |
| rharper | converts the json into network_config yaml | 18:44 |
| rharper | this shows up as datasource.network_config (attr) | 18:45 |
| rharper | like in nocloud ds | 18:45 |
| rharper | I put a long description in the function itself | 18:45 |
| smoser | no.. | 18:47 |
| smoser | convert_networkdata_jso | 18:48 |
| smoser | convert_networkdata_json | 18:48 |
| rharper | oh, in openstack helpers | 18:48 |
| smoser | right. | 18:48 |
| smoser | not convert_network_data | 18:48 |
| rharper | I assumed it was needed to do the json read , like vendor data | 18:48 |
| rharper | but maybe it's not needed at all | 18:48 |
| rharper | it looked to me like it ensured that we opened the data as list (json.load()) and returned the list that should be in the json data | 18:49 |
| smoser | hm.. ok. i think it should be really necessary. | 18:53 |
| smoser | we just want to laod whatever is there. if its bad, then its bad. | 18:53 |
| smoser | rharper, does network config have a 'hostname' ? | 18:54 |
| rharper | hrm | 18:54 |
| rharper | I don;'t think so since hostname wasn't set in eni | 18:54 |
| rharper | but nothing prevents us from adding it | 18:54 |
| rharper | smoser: you mean it should *not* be needed ? | 19:01 |
| smoser | right. | 19:04 |
| smoser | should not need that i think | 19:04 |
| rharper | k | 19:04 |
| smoser | right. | 19:04 |
| smoser | sorry | 19:04 |
| rharper | heh, we don't actually run get_data(); that's sorta mocked out | 19:05 |
| rharper | which makes me somewhat uncomfortable with knowing whether or not we needed that or not (though I think using util.load_json() should be fine as you say) | 19:06 |
| smoser | rharper, it shoudl be reasonably easy to *acutally* test. | 19:12 |
| smoser | kind of painful | 19:12 |
| smoser | but easy to add the config drive data as /dev/vdb on a instance | 19:13 |
| smoser | ie, serverstack then just dd that stuff that we got yesterday | 19:13 |
| smoser | except for with the fixed diredtory | 19:13 |
| rharper | smoser: ok | 19:16 |
| smoser | i do understand wanting to actually have unit test though | 19:16 |
| rharper | I might try my hand at a get_data() call on datasource | 19:18 |
| rharper | mock can help for most of it | 19:18 |
| === devicenull_ is now known as devicenull | ||
| rharper | smoser: if I supply a config drive, what should I specify on the kernel cmdline to have cloud-init use that? | 19:26 |
| === JasonF is now known as JayF | ||
| smoser | rharper, it will just use it. | 19:33 |
| smoser | so what you can do is boot the system | 19:33 |
| smoser | then just dd the content to /dev/vdb | 19:33 |
| smoser | the ephemeral device | 19:33 |
| smoser | and then run | 19:33 |
| rharper | I don't need to specify ds= XXX or antthing like that | 19:33 |
| smoser | rm -Rf /var/lib/cloud /var/log/cloud-init=* | 19:33 |
| rharper | ah, and reboot | 19:33 |
| smoser | no. because its in the normal search | 19:33 |
| smoser | dont have to reboot | 19:33 |
| rharper | it'll re-run cloud-init | 19:33 |
| rharper | oh | 19:33 |
| smoser | cloud-init init --local | 19:33 |
| rharper | nice | 19:33 |
| rharper | ok | 19:33 |
| smoser | cloud-init init | 19:33 |
| smoser | that shoudl push it through the code your'e after. | 19:34 |
| rharper | nice | 19:34 |
| rharper | should I update something in cloud-utils to generate config-drive ? that'd be useful for cloud-init-test | 19:34 |
| smoser | yeah, it woudl be useful. definitely we need it for the vmtest | 19:35 |
| rharper | ok | 19:35 |
| rharper | should I modify cloud-localds, or create something new cloud-config-drive ? | 19:35 |
| rharper | I assume a new command | 19:35 |
| rharper | bbiab, need to pickup kiddo | 19:35 |
| smoser | yeah, i think new command | 19:36 |
| smoser | magicalChicken, https://code.launchpad.net/~smoser/cloud-init/trunk.net1-cmdline-ip/+merge/289968 | 19:54 |
| smoser | if you have time, take that and run with it. | 19:54 |
| smoser | that may be functioanl for ip= on cmdline. | 19:55 |
| smoser | based on a lot of your work | 19:55 |
| smoser | rharper, i have to run now. | 19:55 |
| smoser | will be in tomorrow | 19:55 |
| smoser | give your code a test. but largely it looked good to me. | 19:55 |
| magicalChicken | smoser: Sure, I'll get it merged tonight and make sure it works | 19:59 |
| rharper | smoser: cool | 20:02 |
| rharper | smoser: urg, so when I re-inited cloud-init, it decided to use DataSourceOpenStack instead of DataSourceConfigDrive since I'm running this in a OS cloud; need to figure out how to specify order I supose | 20:53 |
| rharper | bah, and /dev/sr0 is populated with configdrive as well | 21:14 |
| * rharper launches without config-drive=1 | 21:14 | |
| rharper | smoser: cool, got it parsing a config drive; I need to generate a clean config-drive that doesn't include meta-data that dumps the content file (like the rackspace one does) and we should see or interfaces.d/50- file rendered | 21:57 |
| rharper | ok, now we're not writing out network_config from content; but I don't see distro.apply_network() ; I wonder if I need to enable a different module | 22:09 |
| rharper | sweet, works! I don't quite understand why it wont' apply when I run cloud-init init ( the only call to init.apply_network_config()) happens in stage5 and only if you pass --local parameter; however, when I run cloud-init init --local; it's definitely not run either. so as a hack, I dropped the if args.local check; and it writes it out fine; so I'll leave the MP as-is and smoser you can help me figure out what I'm m | 22:19 |
| rharper | issing here | 22:19 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!