=== praneshp_ is now known as praneshp === harlowja is now known as harlowja_away === evilissimo|afk is now known as evilissimo [10:00] Hi there, I'm having some issues with cloud-init, I've built a Ubuntu desktop image with cloud-init and OpenStack as the source, but cloud-init fails to retrieve the metadata from the server as the network isn't configured yet. Any ideas how to get cloud-init to configure the network so that the server can be contacted? Running it after the OS has finished booting (and subsequently once the network is up) works fine. [12:37] Just noticed that the Azure data source tries to start "walinuxagent". This appears inconsistent to me as the upstream source provides "waagent" [12:38] shouldn't cloud-init stick to upstream provided names? [12:41] Also the code is setup to always use the "service" command. While distros switching to systemd have service rewired to do the "right" thing it would be nice to use systemctl where appropriate [13:22] CatKiller, hi [13:22] cloud-init runs its network metadata search when all configured 'auto' interfaces are set up in /etc/network/interfaces. [13:23] i suspect you just need to put 'auto eth0' and config it for dhcp there. [13:23] robjo, i'm not opposed to that, but service is a functional work around. [13:24] ie, usage of it was by design. [13:24] wrt walinuxagent or waagent, i'm not sure. i'm fine for patches to go looking for the right name. [13:25] smoser: so conditional code for name and command would be OK? [13:25] sure. i dont really care. [13:25] once we got azure functional, i have not looked back at it. and have been happier :) [13:26] yup, I usually look at this stuff only as long as I have to as well, will work on a patch [13:26] the worst hack in azure is bouncing the network interface in order to "publish" the hostname. [13:27] i'm guessing what i have there wont work on suse [13:27] OK, thanks for the heads up, will take a look at that as well [13:28] fwiw, the agent start command is configurable [13:28] via the datasource config [13:29] (and so is bounce command) [13:30] Yes I saw the "merge" call but I cannot put all the parts together just yet on how the configuration and everything else connects [13:31] smoser: Oh my god you're right, I know what's happening [13:31] Silly Ubuntu desktop comes with Network Managr [13:31] *Manager [13:32] Or nightmare manager as I like to call it [13:32] So cloud-init must be waiting from the network to be up from upstart which is probably not really happening with Network Manager [13:32] I'm going to remove it completely (unless there's a better option) [13:37] CatKiller, if it were me, i might go the route of grabbing a cloud image and apt-get install ubuntu-desktop [13:37] rather than building your own [13:37] smoser: I thought about this [13:38] smoser: I thought it would be bad [13:38] CatKiller, its not waiting for anything from network manager [13:38] but I just learned the hard way! [13:38] cloud-init's job only waits on "static networking" [13:38] I'm actually probably do what you suggested [13:38] which means "stuff configured 'auto' in /etc/network/interfaces" [13:38] It does make a lot of sense [13:39] Few distros use NetworkManager and no server ones would (which is usually what people start in the cloud) [13:39] smoser: I think I'll go the way you suggested with getting ubuntu-desktop in a cloud image [13:39] But thanks a lot for helping me understand what was going wrong [13:39] CatKiller, i might do something like: [13:39] get-cloud-iamge [13:39] it's good to know why something didn't work [13:40] mount-image-callback my.image -- chroot _MOUNTPOINT_ apt-get intsall ubuntu-desktop [13:40] that wont' "just work", but it will be pretty close [13:40] a couple gotchas: [13:40] a.) you'll have to resize the image to larger than 2G (maybe) [13:40] er... 1.4G [13:40] True, didn't think of that [13:40] b.) you'll have to disable services . i have that code somewhere, let me find it. [13:41] smoser: Thanks, that's great [13:41] (disable services so they dont start when you 'apt-get install' stuff) [13:42] CatKiller, https://code.launchpad.net/~smoser/maas/maas-ephemerals-v2 [13:43] the bin/maas-cloduimg2ephemeral is what we do to make the cloud images into "ephemeral" images. [13:43] the process is much the same [13:43] but we dont have to grow the disk. [13:43] smoser: Nice! [13:43] So I don't even have to boot the cloud image in KVM to configure it then? [13:44] CatKiller, well, i do the build process inside a kvm. [13:44] but thats neither here nor there. [13:44] we just do that for safety [13:44] ok but you don't need to actually boot the cloud-image to install packages etc [13:45] i'd be very open to patches to mount-image-callback for '--grow-first=4G' or something like that. [13:45] right. we dont boot it. [13:45] (which I had been doing to configure my existing cloud image, I booted it and configured it) [13:45] we chroot in and 'apt-get install' [13:45] smoser: If I can figure out a way to streamline the growing [13:45] I'll submit a patch [13:46] So your package is similar to "vmbuilder" right? [13:47] except it relies on Ubuntu's preconfigured cloud images instead of building one from scratch from what I can gathr [13:47] *gather [13:48] bbl [13:55] CatKiller, i really would never suggest to anyone that they build a cloud iamge from scratch [13:55] there are tools that do that, and i thik that they are silly [13:56] similarly i'd never suggest to anyone that they build their own linux kernel, python, glibc or php [13:56] people do that for you, use their work. [14:07] smoser: tried the config route for Azure and starting the agent [14:08] then "rm -rf /var/lib/cloud/*" [14:09] after running cloud-init -d {init, init --local, modules --mode=config} in that order on the command line there is no evidence of an attempt to start the agent? [14:10] Is this testable live or do I have to run a re-build upload etc. cycle? [14:10] The config entry looks as follows: [14:11] datasource: [14:11] Azure: [14:11] agent_command: ['service', 'waagent', 'start'] === zz_gondoi is now known as gondoi [14:21] robjo, it should be testable [14:21] pastbin /var/log/cloud-init.log ? [14:22] robjo, [14:22] init --local [14:22] needs to run first [14:23] (it clears the state) [14:23] then init [14:23] and init should have ran and found the azure datasource [14:23] OK, let me try again [14:24] you can configure off the other datasources to reduce cruft. [14:29] smoser: Makes total sense, I'll use the cloud image first [14:31] smoser: http://pastie.org/9326842 [14:34] robjo, /var/log/cloud-init.log should be much more verbose than that. [14:35] oh. and i see. seed=/var/lib/waagent [14:35] heres another fun bit of azure, robjo [14:35] you get that CDrom that has important data [14:35] but at some point in a reboot they just yank it from you [14:35] /var/log/cloud-init.log is empty, I pasted /var/log/cloud-init-output.log [14:35] robjo, you rpboably need to restart syslog [14:36] maybe remove that file and resart syslog. [14:36] or maybe on suse thats not hooked up right. but thtats a bug . [14:36] if its not. [14:36] hm.. [14:36] hm.. sorry for being dense. [14:37] there is defintiely more verbose output going somewhere. [14:41] smoser: http://pastie.org/9326911 [14:41] http://pastie.org/9326914 [14:42] http://pastie.org/9326917 [14:42] http://pastie.org/9326920 [14:42] Output in the terminal from each command, was too large for 1 paste [14:43] robjo, suse needs 'pastebinit' [14:43] its amazingly convenient. [14:43] pastebinit [14:43] or [14:43] some-command | pastebinit [14:43] /var/log/cloud-init.log is still empty even after restarting syslog [14:44] 2014-06-26 14:39:34,259 - stages.py[DEBUG]: Restored from cache, datasource: DataSourceAzureNet [seed=/var/lib/waagent] [14:45] it found that, so its not going to go through its earch [14:45] not really sure why it found it as init --local should have removed that link [14:45] so where is that cache? so I can get rid off it ;) [14:46] well its /var/lib/cloud/instance that is a link. [14:46] you can purge all of /var/lib/cloud [14:46] I did, before I ran the commands.....grmbl [14:46] rm -rf /var/lib/cloud/* [14:49] are you able to just let me in to poke really quick ? [14:51] yes, send me your public key [14:51] rjschwei@suse.com [14:53] robjo, https://launchpad.net/%7Esmoser/+sshkeys [14:53] another tool you need is 'ssh-import-id' :) [14:55] smoser: OK, try this: ssh azuser@sp3-ibs-try4.cloudapp.net [14:56] robjo, k. i'm in [14:58] robjo, for logging, cloud-init sends its log to /dev/log [14:59] which in ubuntu gets sent to /var/log/cloud-init.log because of [14:59] http://paste.ubuntu.com/7706306/ [15:00] so yours is just goign to /var/log/messages [15:00] as its not being captured specifically somewhere === gondoi is now known as zz_gondoi [15:00] OK, can certainly setup the rule, thanks [15:00] if you ust remove the [15:00] [ *log_base, *log_syslog ] [15:01] line in /etc/cloud.d/cloud.cfg.d/05_loging [15:01] then it will go straight to the file [15:02] so i made that change [15:02] and now ran [15:02] cloud-init init --local [15:02] and [15:02] cloud-init init [15:02] and see the log [15:03] you can see your [15:03] 'service', 'waagent', 'start' [15:03] the reason for the logging being as it is, is that generally, i wanted to use syslog [15:03] but early in boot syslog isnt necessarily available [15:03] so cloud-init tries to write to /dev/log [15:04] and if htat fails it writes to /var/log/cloud-init.log directly [15:04] then, the next time it comes up (cloud-init init) it probably has /dev//log [15:04] and it works. [15:06] note, the file above in that pastebin is a rsyslog format file. i dont knwo about your syslog (syslog-ng) [15:06] OK, makes sense. I'll fix the logging ni my image builds. And will just configure the datasource in the config file, then there is no need to have a bunch of ugly serach code in the azure datasource code [15:07] we have syslog-ng in sles 11, rsyslog in openSUSE and rsyslog will be in SLE 12, it's a mess, bute getting better :) [15:08] Thanks, one problem down :) [15:08] so just one other gotcha there. [15:08] on our walinuxagent [15:08] we do not start it by default [15:09] and we basically configure off all of its function [15:09] as it overlaps with cloud-init. so we have cloud-init start it only on azure. and neuter it heavily. [15:10] I am setting "Provisioning.Enabled=n" in the waagent.conf, that's the info I got from M$ [15:11] so should I or should I not enable the agent to start at boot? [15:13] i'm trying to remember [15:17] smoser: Does cloud-init (using configuration found on the Ubuntu cloud-image) run unattended security upgrades on boot, or is it something that needs to be defined at the metadataserver level? [15:19] CatKiller, by default it does not do that. [15:19] you'd enable that via user-data or some other way [15:19] you could patch that on when you re-build your images [15:20] smoser: ok, no problem. I'll patch that while rebuilding [15:20] Another thing, what are the "cloud_config_modules" defined in the cloud.cfg config file? [15:20] Are they programs that will run during the config stage [15:20] For instance I wanted to know what "package-update-upgrade-install" was doing but couldn't really tell [15:21] CatKiller, i'm sorry that documentation for such things sucks. [15:21] that ends up involking the 'cc_package_update_upgrade_install.py' file [15:21] which handles 'apt_update' and 'apt_upgrade' [15:21] but this is not anywhere well documented. [15:23] smoser: OK, I'll look in that file. Just a quick question though [15:23] all of the modules defined in the cloud.cfg will run, or does that just mean they're available? [15:23] and the metadata will tell which one needs to run [15:24] they run [15:24] the user-data can configure which ones run [15:24] yes. [15:24] by re-defining that list [15:25] smoser: But if it's not defined in the user-data (we use OpenStack here and the user data is *very* scarce) it'll run [15:25] what do you mean the user data is very scarse [15:26] modules generally do the right thing. [15:26] if you give htem config, they may act and have sane defaults [15:26] smoser: Well the metadata only provides hostname, authorized SSH keys but not much more [15:26] user-data [15:26] not meta-data [15:27] ah ok sorry my bad I confused the two [15:27] user provides user-data when they launch an instance [15:27] nova boot --user-data=your.user-data.file.txt [15:27] Ah ok I see. We don't provide any specific user data here as far as I know [15:28] right. so you get the default behavior. which should be perfectly fine. [15:28] OK, sounsd good === evilissimo is now known as evilissimo|afk [15:29] Another thing I'm not entirely sure about. In the config there is a "package-update-upgrade-install" module, and in "cc_package_update_upgrade_install.py" the handle function looks for "package_update" and "apt_upgrade" from the configuration [15:29] these are unrelated right? [15:30] they're synonyms [15:30] distro-generic and distro-specific. [15:31] So "apt-get update" and "apt-get upgrade" will run on boot (or is that only on *first* boot) with the default cloud-init config? [15:31] default is off [15:32] you can turn it on and it will run on first boot [15:32] that module only runs 'per-instance'. by default. [15:32] meaning once per instance-id [15:32] But it's in the config file though, so the user-data defines the behavior here? [15:32] you can change it to run per-always [15:32] you can put it in config also [15:32] and the user-data can still override [15:33] ah ok, so here it's in the config but user-data probably overrides. [15:33] Where can I find the user data in the system actually? [15:33] (the default ones) [15:34] there is no real location for defaults. [15:34] most of stuff is described [15:34] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt [15:34] and other files in http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/head:/doc/examples/ [15:35] thanks for those files [15:36] but in the specific of cloud.cfg's "package-update-upgrade-install" module, where can I figure out what will happen there? [15:36] I'm not really sure what this clause does [15:36] or where it is configured [15:37] I guess I'm having trouble linking those modules with what will actually happen when they run (their config files etc) [15:41] CatKiller, by default basically nothing happens there. [15:42] but it responds to some settings in [15:42] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt [15:42] (package_upgrade=true) [15:47] smoser: Ah ok, makes sense [15:47] smoser: And just a quick question so that I can understand this config file better [15:47] where are all these clause defined then in the code? [15:48] I grepped for "package-update-upgrade-install" without success [15:48] in the cloudinit Python code [15:48] (I wanted to figure out what config clause did what by looking at the source) [16:33] cloud_config_modules, cloud_init_modules and cloud_final_modules [16:33] are lists [16:33] they reference "config modules" [16:34] config_modules are loaded . they'r ein the source tree at cloudinit/config/cc_.py [16:34] where 'name-in-list' has '-' replaced with '_' [16:44] smoser: Ahhh ok I get it. Sorry you were trying to tell me this earlier and I had completely missed it. I thought the "cc_" file you told me to lookup handled all of the "config modules" [16:44] smoser: It's much much clearer now, thank you very much [16:44] cool [16:44] smoser: There's one thing that I'm not quite sure about yet [16:45] smoser: Let's imagine I don't want to pass custom "user-data" to the instance, where can I configure a "default" user-data within the image? [16:46] you cant explicitly provide default user-data. [16:46] user-data an be cloud-config syntax or other syntaxes. [16:46] but you can provide any config in /etc/cloud/cloud.cfg.d/.cfg [16:47] smoser: Ah great, so I could add a file /etc/cloud/cloud.cfg.d/my-config.cfg that would contain: [16:47] #cloud-config\npackage_upgrade: true [16:47] for instance [16:47] In that case, that config would be interpreted as a "cloud-config" file and the package_update should run on first boot [16:47] Does that sound right? [16:53] yes. [16:53] files in that dir do not need to start with #cloud-config [16:53] they can only be cloud-config [16:53] smoser: Good to know [16:53] smoser: In the absolute, it seems that I could add the package_upgrade: true clause at the top level of the /etc/cloud/cloud.cfg config file as well no? [16:54] looking at the cc_package.... script it seems that it'll get the flag from the config(s) files (presumably cloud.cfg included) [16:56] yeah. [16:57] config modules know no difference between "builtin" config and user-data [16:57] its all just config that they act on === harlowja_away is now known as harlowja [17:32] smoser hey, do u have any medium size ideas for cloud-init that i can do, need a diversion from the other projects :-P [17:33] is that medium sized in human terms [17:33] or in super-human harlowja terms [17:33] lol [17:33] harlowja terms i guess :-P [17:34] the stuff i have listed here is at: [17:34] https://blueprints.launchpad.net/ubuntu/+spec/servercloud-u-cloud-init [17:34] Utopic ? [17:34] whats that [17:35] 14.10 Utopic Unicorn [17:35] ah [17:35] went looking for mark's post on utopic [17:35] and saw: [17:35] http://www.markshuttleworth.com/archives/1342 [17:35] nice [17:36] http://www.markshuttleworth.com/archives/1363 [17:36] that one is utopic [17:36] i should forward that on here, [17:36] anywah... [17:36] for things.. [17:36] the 2 things you might be interested in: [17:36] a.) python3 [17:36] ack, that one again, lol [17:37] python3 hasn't gone away yet, lol [17:37] b.) more work on ci-tool [17:37] http://bazaar.launchpad.net/~smoser/cloud-init/ci-tool/view/head:/ci-tool [17:37] i'm not sure how i feel about ci-tool [17:37] oh. the other one... the query stuff. [17:37] you revved that at some point. [17:38] it'd be nice to have 2 json files in /run [17:38] smoser ya, good-ole-query stuff [17:38] i think thats the way i'd go now. [17:38] just 2 json files [17:38] one with non-sensitive data [17:38] and one with sensitive data [17:38] right [17:38] and try to have a sane format for things common to most clouds [17:39] :) [17:39] and allow the datasource to shove other stuff in its own place in the json [17:39] right [17:39] ci-tool; would that be needed with a more extensive query tool (or maybe they merge?) [17:39] into super-ci-query-tool [17:40] ci-tool 'seed' is the real function of ci-tool [17:40] 'seed', 'reset', 'set-ds' [17:40] oh man, marks page is now showing me 'Error establishing a database connection' [17:40] :) [17:41] smoser sure, ya, the seed stuff is nice to have [17:41] oh. [17:41] the other bug... [17:41] ? [17:41] the network interfaces stuff sucks [17:41] :) [17:41] yaaaa [17:42] do u want to try to see how the netcf stuff works? [17:42] i can try messing around there [17:42] maybe its 'ready' for primetime [17:43] *although we'd probably need both, if netcf isn't avaiable on given distro [17:44] omg, what is my password doin in http://bazaar.launchpad.net/~smoser/cloud-init/ci-tool/view/head:/ci-tool#L51 [17:44] haha [17:44] thats your password to? wow. coincidence. [17:44] too [17:45] :) [17:45] smoser in 14.10 unicorny no more 2.x python? [17:45] yeah, the netcf stuff. ive' come to needing that elsewhere too. [17:45] we really need to get openstack networking sorted [17:45] :-/ [17:45] to the way that amazon does it. [17:46] so that you can hotplug a NIC into the system [17:46] i believe with neutron u can do that [17:46] and then the system can hit the metadata service and get the interface config for that. [17:46] ya [17:46] and that itnerface config needs to be in some format [17:46] and thus... i was asking you again about netcf [17:47] :) [17:47] ya, let me see if i can get mark mcclain in here to chat about the openstack neutron networking thing [17:47] maybe after running xml2json on it [17:47] how we can get from here to there [17:47] smoser ya, xml == evil [17:47] lol [17:48] it really just isnt suitable for that reason [17:48] smoser exactly [17:48] harlowja: here [17:48] hi markmcclain1 [17:48] markmcclain1 hey, so we were just wondering a little bit about the future of hot-plugging, metadata in openstack, and how cloud-init will help out here [17:48] thought u might have some knowledge (that i don't have) [17:48] *hotplugging nics [17:49] you can plug the nics in [17:49] i have done that. [17:49] and they do show up [17:49] all that works. [17:49] i know what yahoo is doing here (with config drive, which won't work obviously here) [17:49] and then atually.... in the metadata servie i noticed one of them actually in a /etc/network/interfaces file after hotplugging. [17:49] config drive will die [17:49] and that will be ok [17:49] :) [17:50] ya, i'll have to fight the people here on that one to make it die (if possible) [17:50] i'm not too hung up on it. [17:50] mostly people hated the metadata service because it didn't work. [17:50] or the networking never worked to get the instance to it. [17:50] but now i think that those problems are less comon [17:50] but anyway.. [17:50] i hotplugged a nic [17:50] k [17:50] yeah so with hotplugging dhcp should work on that interface [17:50] and then an entry in /etcnetwork/interfaces styel file was in the interfaces location in the metadata service [17:51] but it was out of order :) [17:51] ie, the new nic was eth0 and the orig was eht1 [17:51] so its just silly. [17:51] thus need for some beter format for describing [17:51] and then enter xml netcf format :-P [17:52] *xml (cough) [17:52] yeah, i most certaily would make major mistakes if i invented my own language. [17:52] but really would not be interested in putting xml into a stack where it doesn't otherwise exist. [17:53] :) [17:53] xml is da best [17:54] markmcclain1 has there been any thought on moving to a more agnostic format in the community for this [17:54] not saying it has to be https://fedorahosted.org/netcf/ (but something similar in json or other would be nice) [17:54] I haven't heard too much yet about how to fix these issue [17:54] k [17:55] smoser so the mission, if u choose to accept it, is to fix this issue, this message will self-destruct in 5 seconds [17:55] boom [17:55] yeah. i'd like to do it. it generally needs doing i think [17:56] agreed [17:57] smoser would this be a cloud-init thing at that point, or something else? [17:57] something else that listens for hotplugs and then does stuff [17:57] network-cloud-init (or something) [17:57] i think i'd keep it in cloud-init. but try to make it separatable [17:57] k [17:58] basically, its udev hooks, code to get the data, and then code to act on it. [17:58] ya [17:58] any idea how that works for freebsd ? [17:58] harmw activate, lol [17:58]