/srv/irclogs.ubuntu.com/2016/03/23/#cloud-init.txt

rharpersmoser: around ?00:02
smoserhere00:02
smoserlooking at why i broke your test suite00:02
smosertheres two issues i'm seeing00:03
smosera.) a WARN in the log says the fallback datasource got used, but idont know how (and other parts say NoCloud got used)00:03
smoserb.) it seems the nics aren't being renamed appoprirately.00:04
smoserdo you have an  easy way to get in to the test vm ?00:04
smoserok. well, 'b' seems to be because we have a /etc/network/interfaces inside that doesnt' srouce /etc/network/interfaces.d00:09
rharpersmoser: it's somewhat tricky to get it  but:  I usually drop the poweroff script in generate_payload_scripts();  this will leave the system up00:13
rharperthe if you ps, you can get the /tmp/launch.XXXX/boot.img ; copy that out before you kill the vm and the qemu command line;  then I replace -serial file.* with -serial telnet:localhost:2446,server,nowait , quote the --append, then add -monitor stdio -S;  when you run it; you'll get monitor prompt on stdio, guest will be paused;  you can setup your telnet to localhost on the port; then type 'cont' on the monitor;00:15
rharpersmoser: I've got the configdrive openstack helper updated to search for the network_data.json, so we can fetch this,  we use the same convert_to_json() logic and stuff that in configdrive.networkdata_raw;  I see in stages where userdata_raw and vendordata_raw are processed (and written out);  ideally I'd do that with networkdata and write out a networkdata.json ;  I'm trying to find where best to convert the json networ00:18
rharperkdata into network config;  it appears that I need to write out network_config.yaml where we write out userdata conf as well, and then the distro apply_net will do it's thing00:18
smoseroh. well the ps is easily fixable00:19
smoserrharper, CURTIN_VMTEST_KEEP_DATA_PASS=logs,collect00:19
smoserrun with that, add 'disks' to it and you'll keep the disks too00:19
rharperI usually want the interactive nature00:19
smoserright. but i was just saying to keep the disk around00:20
rharperoh00:20
smoserso you dont have to race it.00:20
rharperyeah00:20
rharperah, gotcha00:20
rharperor modify the power stage00:20
rharperyep00:20
smoserok. so i understand most of the failure now00:20
smoserand have some changes.00:20
smoserbut test_etc_network_interfaces00:20
smoserwill have to be updated00:20
smoseras now i'm (by design) not writing to /etc/network/interfaces00:21
rharperhttp://paste.ubuntu.com/15476168/  (my debug script)00:23
rharpersmoser: yes;  I think we should use the net.parse_eni() which dumps a dictionary00:24
rharperfrom there we can compare/contrast with a network_state00:24
smoserok.00:24
smoseri'll getthis as close as i can00:24
smoserand then can you do that ?00:24
rharperalternatively, we need to write a network_state that can re-render network_config.yaml00:24
smoseri'll submit MP for your git in next 5 minutes.00:24
rharperprolly00:24
rharperbut I'd like to get your thoughts on the config drive bits00:25
rharperthat's what I'm trying to finish up right now00:25
smoserk00:25
rharperwhere to convert the network_data into network_config yaml00:25
smoserah. ok00:26
smoserso you have the networkdata_raw (although its not *really* raw) as you json loaded it.00:26
rharperright00:26
smoseryou can call it whatever you want.00:26
smosersure keep it around in the datasource datasource.network_json00:26
rharperthe same as vendordata_raw (which also was json loaded)00:26
smoserthen, in datasource.network_config()00:27
smoserits a property00:27
smoseryou can convert it there or anywhere really.00:27
smoserwould probably be nice to have the code easily runnable with no class wrappers or something00:27
smoserdef network_config(self):00:27
smoser   if not network_json:00:27
smosererr.00:27
smoser   if not self.network_json:00:27
smoser     return None00:27
rharperI was looking at also doing the _store_networkdata() in stages.py (also done for userdata/vendordata)00:27
smoser   return convert_network_json_to_netweork_config(sle.fnetwork_json)00:28
smoserdont worry about that i dont hthink00:28
rharperok00:28
smoserjust look at how nocloud does it.00:28
smoserexecept you ahve the conversion stage.00:28
smoseruserdata and vendordata are different00:28
smoserbecause the get parsed00:28
rharperok00:29
smoserand that parsing may include otehr things00:29
smoserbasically we preproces them once00:29
smoserand save the result.00:29
rharperok00:29
rharperin the convert, don't we want to write out the network_config file?   somewhere in the cloud dir like the other files ?00:30
smoserrharper, http://paste.ubuntu.com/15476227/00:34
smosercould you apply that ?00:34
smoseryou can write it in the cloud dir if you want.00:34
smoseri guess its more nicely available for ispection then.00:34
rharpersmoser: I'll apply00:35
rharpersmoser: ok00:35
smoseras long as its store in the object00:35
smoserthe object gets pickled00:35
smoserand then restored for inspection00:35
smoserthe files like userdata are only stored because things outside read them.00:35
smoseror could i guess.00:35
smoseroh. and also, fyi00:36
smoser https://code.launchpad.net/~smoser/+recipe/cloud-init-daily00:36
smoserthe bug i'll need to fix tomorrow is why i see that WARN for fallback00:36
smoserand now i turn into a pumpkin00:36
smoserrharper, thank you for all your help.00:37
smoserthis is really comming along.00:37
rharperok00:37
rharpersmoser: do I need updated trunk.net1 to verify your patch?00:37
rharperlooks like it00:38
rharpersmoser: the curtin eni parser does handle source.d now (blake_r added that just a bit ago)00:42
rharpersmoser: magicalChicken: ok,pushed your fixes to cloud-init-test master01:05
smoserrharper, do you have a branch up for config drive ?13:15
rharpersmoser: not yet; ran out of time last night;14:00
rharperI did commit your change to cloud-init-test14:00
smoseryeah, saw that.14:00
smoseri'm chasing that fallback datasource issue.14:01
smoseronce i get that, plan to get the kernel command line stuff14:01
rharperI'm gonna get a test-case in for configdrive;14:02
rharperthat'll help verify the changes to configdrive datasource14:02
smoseryeah.14:03
smoserthat will be very good.14:03
smosermagicalChicken, aroun d?15:09
smoserrharper, ping15:14
rharpersmoser: here15:15
smoserhttps://code.launchpad.net/~smoser/cloud-init/trunk.net1-cmdline-ip/15:15
smoserrharper, so was lookoing at magicalChicken 's kernel command line there.15:16
smoseri'd like some tests, but other than that it seems generally reasonable15:16
smoserthe thing that i want to ask about, is about using the 'network_state'15:17
smoserrather than network_config15:17
smosermagicalChicken did this on the fallback config also.15:17
smoseri wanted / needed *config* rather than network state.15:17
smoserso questions are15:17
smosera.) what renders me a config from a network state? i'm guessing there is something15:18
smoserb.) do you thikn its preferable to work with one or the other directly15:18
smoserrharper, ^15:22
smoseri realize that the merge proposal above doesn't really help you look at it.15:25
smoserpull my branch and diff it versus trunk.net1 http://paste.ubuntu.com/15480216/15:26
rharpersmoser: nothing yet renders a network_config from network_state ; I was looking for that myself but it's not written15:27
rharpersmoser: I've worked with both;  I find the network_config to be more useful since it can be written out and shared15:27
smoserwell, that and right now its network_config -> network state as one way15:28
smoser:)15:28
rharperright, but in general, we're employing conversions from *something* (ip, configdrive) to network_config15:29
rharperand then allowing distro.apply_net() to take config to state to eni15:29
rharperthat seems like a reasonable path for consistency across cloud-init15:29
smoser2 way mapping in NetworkState would be good though.15:31
smoserits actually confusing, because 'dump' creates a dictionary with a 'config' and a 'version' that then makes you think "oh, taht is network_state -> network_config'15:32
rharpersmoser: yes;  we should write the state -> network_config to make it 2 way; that's a good way to ensure we don't miss something15:32
rharpersmoser: question on the ConfigDriveReader,  it has a _fetch_available_version() call which does some os.listdirs and collects dir names;  it returns it in no particular order;  I seem to be suffering from unsorted errors;  sometimes it doesn't select latest first; I'm only populating the network_data.json under latest/  in the test-case;  this passes; but if it chooses something else (2012-08-03) it will fail to find ne15:49
rharpertwork_data.json15:49
smoserso _fetch_available_version() returns one ?15:52
smoserand not necessarily the one you want.15:52
smoseryeah, i think then just fix it.15:52
rharperso I think _fetch_available_versions which refernces OS_VERSIONS , shouldn't that include OS_LATEST ?15:53
rharperah, it had only passed when selected = OS_LATEST, and supported array was empty; then it defaulted to OS_LATEST which is where I had put the network_data.json15:56
rharperit seems to me that OS_VERSIONS should include OS_LATEST; but I'm a bit worried about this change since I don't have a lot of history with configdrive15:56
smoserrharper, by design OS_VERSIONS does not include OS_LATEST.15:58
smoserunless some special circumstance, you really dont want to ever use 'latest'.15:59
rharperok, then I'll need to populate another dir in addition to latest15:59
rharperthat's easy enough15:59
smoserwe're given a versioned api essentially , and assuming we can deal with whatever is "latest" is going to fail at some point.15:59
rharpersmoser: I need to fix the one unittest, but this is working otherwise: lp:~raharper/cloud-init/config-drive-network-data ; I;'ve got to run, will be back in a while16:06
smoserk. i'll review. thanks16:07
vox_clamantissmoser ... still having trouble using openstack metadata. would you mind looking at a snippet of my log?17:07
smosersure. pastebin the cloud-init.log17:09
vox_clamantisthanks very much17:10
vox_clamantisbtw to test changes ive been removing /var/lib/cloud/* and rebooting. is that a bad idea? takes much longer to reconfigure the template17:11
vox_clamantishttp://pastebin.com/CLm5c70E17:13
vox_clamantisthis is with "datasource_list: [ OpenStack ]"17:13
smoservox_clamantis, thats generally clean.17:16
smoseri clear /var/log/cloud-init* also17:16
vox_clamantisy that too good17:16
vox_clamantisand i reset to dhcp so i get the route to metadata17:17
smoseryou ran cloud-init init --local17:17
smoserright ?17:17
smoserthat will nto fidn a openstack datasource, as it will only search local data17:18
vox_clamantisjust a reboot17:18
smoserhm.17:18
vox_clamantiscan i initiate a non-local run?17:18
smoserso , without reboot, you can just do17:18
smoserrm -Rf /var/lib/cloud /var/log/cloud-init-*17:18
smosercloud-init init --local17:19
smosercloud-init init17:19
vox_clamantisok17:19
smoserSearching for data source in: []17:20
smoserMar 23 12:59:21 lin-test [CLOUDINIT] util.py[WARNING]: No instance datasource found! Likely bad things to come!17:20
smoserer.. the searching for data sourcein: []17:21
vox_clamantisyeah17:21
smoserthat seems wrong.17:21
vox_clamantisi was concerned about that17:21
smoserbut that coudl be a bad message.17:21
smosercould you just let me in some wher e?17:23
smoseris that possible, i can probalby solve it in a couple minutes if you can17:23
smoserrather than asking you 40 questions17:23
vox_clamantisdepends what you mean by let you in17:23
vox_clamantislol17:23
vox_clamantisnot to the network unfortunately17:24
smoser0.7.4 is pretty old. i know it sucks for someone to tell you "can you try a newer version"17:26
smoserbut....17:26
waldiand python 2.6. does cloud-init provide support for that?17:27
vox_clamantisi dont mind at all17:27
vox_clamantisi believe it is the latest from rhel6 epel17:27
vox_clamantisyep still there17:28
vox_clamantisid be happy to try anohter17:28
vox_clamantis0.7.6?17:29
vox_clamantisjust need to install some build tools i think17:32
smoserthats surely better.17:41
smosercloud-init should work on 2.6. but it could be that.17:42
vox_clamantisthanks for the suggestions17:46
vox_clamantisand appreciate all the work you do on this project17:46
vox_clamantisugh i cant even build it on rhel 6. maybe i best just go to configdrive18:12
rharpersmoser: ok, fixed up that last unittest for config-drive network_data18:33
smoserok. i'll read.18:34
smoserrharper, i think your KILO is LIBERTY18:36
rharpersmoser: OK, I wasn't sure;  i"ll fix up18:37
smoserhttps://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L7618:37
rharperk18:38
rharpersmoser: do we yet have a unittest that combines configdrive + distro.apply_network() ?  I don't think I saw a distro.apply_network() yet;18:40
smoserthat does apply_network ?18:41
smoseror apply_network_config18:41
rharperwhich ever renders the network_config as distro config file (our case eni)18:42
rharperhoping to have a unittest that pulls the json data from ConfigDrive, convert to network_config yaml, then run apply_network() which should call net.render_interfaces() which writes out an eni18:43
rharperfor VM test, we will need to generate a configdrive iso file and feed the VM that (does the cloudds-local util do that? )18:43
smoserconvert_networkdata_json18:44
smoserwhat does that do ?18:44
smoserrharper, cloud-localds does not do that at the moment.18:44
rharperconverts the json into network_config yaml18:44
rharperthis shows up as datasource.network_config (attr)18:45
rharperlike in nocloud ds18:45
rharperI put a long description in the function itself18:45
smoserno..18:47
smoserconvert_networkdata_jso18:48
smoserconvert_networkdata_json18:48
rharperoh, in openstack helpers18:48
smoserright.18:48
smosernot convert_network_data18:48
rharperI assumed it was needed to do the json read , like vendor data18:48
rharperbut maybe it's not needed at all18:48
rharperit looked to me like it ensured that we opened the data as list (json.load()) and returned the list that should be in the json data18:49
smoserhm.. ok. i think it should be really necessary.18:53
smoserwe just want to laod whatever is there. if its bad, then its bad.18:53
smoserrharper, does network config have a 'hostname' ?18:54
rharperhrm18:54
rharperI don;'t think so since hostname wasn't set in eni18:54
rharperbut nothing prevents us from adding it18:54
rharpersmoser: you mean it should *not* be needed ?19:01
smoserright.19:04
smosershould not need that i think19:04
rharperk19:04
smoserright.19:04
smosersorry19:04
rharperheh, we don't actually run get_data(); that's sorta mocked out19:05
rharperwhich makes me somewhat uncomfortable with knowing whether or not we needed that or not (though I think using util.load_json() should be fine as you say)19:06
smoserrharper, it shoudl be reasonably easy to *acutally* test.19:12
smoserkind of painful19:12
smoserbut easy to add the config drive data as /dev/vdb on a instance19:13
smoserie, serverstack then just dd that stuff that we got yesterday19:13
smoserexcept for with the fixed diredtory19:13
rharpersmoser: ok19:16
smoseri do understand wanting to actually have unit test though19:16
rharperI might try my hand at a get_data() call on datasource19:18
rharpermock can help for most of it19:18
=== devicenull_ is now known as devicenull
rharpersmoser: if I supply a config drive, what should I specify on the kernel cmdline to have cloud-init use that?19:26
=== JasonF is now known as JayF
smoserrharper, it will just use it.19:33
smoserso what you can do is boot the system19:33
smoserthen just dd the content to /dev/vdb19:33
smoserthe ephemeral device19:33
smoserand then run19:33
rharperI don't need to specify ds= XXX or antthing like that19:33
smoserrm -Rf /var/lib/cloud /var/log/cloud-init=*19:33
rharperah, and reboot19:33
smoserno. because its in the normal search19:33
smoserdont have to reboot19:33
rharperit'll re-run cloud-init19:33
rharperoh19:33
smosercloud-init init --local19:33
rharpernice19:33
rharperok19:33
smosercloud-init init19:33
smoserthat shoudl push it through the code your'e after.19:34
rharpernice19:34
rharpershould I update something in cloud-utils to generate config-drive ?  that'd be useful for cloud-init-test19:34
smoseryeah, it woudl be useful. definitely we need it for the vmtest19:35
rharperok19:35
rharpershould I modify cloud-localds, or create something new cloud-config-drive ?19:35
rharperI assume a new command19:35
rharperbbiab, need to pickup kiddo19:35
smoseryeah, i think new command19:36
smosermagicalChicken, https://code.launchpad.net/~smoser/cloud-init/trunk.net1-cmdline-ip/+merge/28996819:54
smoserif you have time, take that and run with it.19:54
smoserthat may be functioanl for ip= on cmdline.19:55
smoserbased on a lot of your work19:55
smoserrharper, i have to run now.19:55
smoserwill be in tomorrow19:55
smosergive your code a test. but largely it looked good to me.19:55
magicalChickensmoser: Sure, I'll get it merged tonight and make sure it works19:59
rharpersmoser: cool20:02
rharpersmoser: urg, so when I re-inited cloud-init, it decided to use DataSourceOpenStack instead of DataSourceConfigDrive since I'm running this in a OS cloud;  need to figure out how to specify order I supose20:53
rharperbah, and /dev/sr0 is populated with configdrive as well21:14
* rharper launches without config-drive=1 21:14
rharpersmoser: cool, got it parsing a config drive;  I need to generate a clean config-drive that doesn't include meta-data that dumps the content file (like the rackspace one does) and we should see or interfaces.d/50- file rendered21:57
rharperok, now we're not writing out network_config from content;  but I don't see distro.apply_network() ; I wonder if I need to enable a different module22:09
rharpersweet, works!  I don't quite understand why it wont' apply when I run cloud-init init ( the only call to init.apply_network_config()) happens in stage5 and only if you pass --local parameter;  however, when I run cloud-init init --local; it's definitely not run either.  so as a hack, I dropped the if args.local check; and it writes it out fine;  so I'll leave the MP as-is and smoser you can help me figure out what I'm m22:19
rharperissing here22:19

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!