#cloud-init 2013-12-02
<tmclaugh[work]> Hey, we're having an issue where our puppet section is failing to install Puppet from our repo.
<tmclaugh[work]> One person here feels it's possibly because networking isn't fully up...
<tmclaugh[work]> Isn't it safe to say that cloud-init will be running after networking is operable?
<Chocobo> does anyone see anything wrong with this configuration:  http://pastie.org/8523144
<Chocobo> What is the difference between "- UserName" and "- name: UserName
<smoser> tmclaugh[work], ubuntu ?
<tmclaugh[work]> smoser: CentOS
<tmclaugh[work]> does cloud-init perform any networking service restart?
<smoser> it really depends on the datasource, but for ubuntu all interfaces defined in /etc/network/interfaces are guaranteed to be up before packages will be installed.
<smoser> tmclaugh[work], why would it do a restart ?
<tmclaugh[work]> smoser: I don't know.  Just ruling it out.  I'm 99% certain that networking is initialized and fully operational before cloud-init starts and that it's not causing the network connection to blip
<tmclaugh[work]> to backup a few steps, out puppet package installation periodically fails.
<tmclaugh[work]> So someone proposed, "let's move the puppet installation to further down in the cloud-init order"
<smoser> Chocobo is gone
<smoser> but http://paste.ubuntu.com/6510271/ is the eanswer to his question
<smoser> basically, he provided broken yaml.
<tmclaugh[work]> He wants to move it from cloud_config_modules to cloud_final_modules
<smoser> tmclaugh[work], do you have multiple interfaces ?
<tmclaugh[work]> he's claiming that maybe by delaying the install Networking will be working.  Which makes no sense to me.
<tmclaugh[work]> nope, signle interface
<smoser> i can't speak definitevely on the init scripts for centos
<tmclaugh[work]> The host should have successfully DHCPed  before cloud-init starts.
<smoser> but for ubuntu, yes. your statement above is correct
<smoser> what is the datasource you're using ?
<smoser> (openstack or ec2 ?)
<tmclaugh[work]> and I'm 99% sure this is the case with CentOS
<tmclaugh[work]> ec2
<smoser> i suspect that if it were not the case on ec2, you'd not be the first to report this issue.
<tmclaugh[work]> Good, time for another argument with this person. :)
<Chocobo> Is there a way to test cloud configurations?  I am having a real problem with "users:"
<smoser> Chocobo, http://paste.ubuntu.com/6510271/
<smoser> running that will expose your problem
<smoser> you had invalid yaml
<Chocobo> smoser: Thanks...  also if I move "vmuser" to the first entry in users, I should be able to eliminate the "user: vmuser" line correct?
<smoser> 'user' affects the entry in user that is marked 'default: True'.
<smoser> i think.
<smoser> wait
<smoser> ignore that last statement
<smoser> from doc/examples/cloud-config-user-groups.txt:
<smoser>  # users[0] (the first user in users) overrides the user directive.
<smoser> so yeah:
<smoser> users:
<smoser>  - name: vmuser
<Chocobo> smoser: thanks, I will give that a shot.
<smoser> Chocobo, note, unfortutely this bit of code has been error prone :-(
<Chocobo> smoser: the "user:" creation is error prone?   
<smoser> well...w ait.
<smoser> the thing that is error prone is the inclusion of 'user:' into 'users'
<smoser> if you only use 'users'
<smoser> you should be ok
<Chocobo> ok, that is alright...  and it is ok to list a user that already exists (root for example)
<harlowja> error prone u say smoser, impossible, ha
<smoser> Chocobo, its ok to list a user that exists
<smoser> but it wont be modified
<smoser> ie, its gecos wont be changed.
<smoser> (or groups)
<Chocobo> smoser: How aout locking an account?
<smoser> i think that is only applied on creation
<Chocobo> smoser: ok, thanks.
#cloud-init 2013-12-03
<gondoi> utlemming or smoser: you guys around today?
<smoser> gondoi, here.
<smoser> whats up?
<smoser> gondoi, ^
<gondoi> smoser, sorry... quick team huddle brb
<gondoi> smoser, okay sorry about that
<gondoi> okay, i am looking for some help with setting up cloud-init the proper way
<gondoi> i'm having trouble finding docs on the config file syntax/options
<gondoi> also, we have images for different distribution releases so versions vary
<smoser> gondoi, well the best documentation on config is doc/example/
<smoser> and/or the ones that ship in 'config/
<smoser> '
<smoser> (in source)
<gondoi> okay, let me look over those
<smoser> gondoi, i'm completley open to improvements in documentation
<smoser> (or code!)
<smoser> :)
<gondoi> smoser, i don't see any examples of /etc/cloud/cloud.cfg, or the files in the .d directory
<gondoi> i'm trying to figure out the best working config for cloud-init itself, not necessarily user-data that it parses
<smoser> they're one and the same.
<gondoi> oh.....
<smoser> see 'config/cloud.cfg' as the one that ships with cloud-init (and config/cloud.cfg.d/05_logging.cfg)
<smoser> thats kind of a design goal of cloud-init
<smoser> whatever you could configure or do when you built the image
<smoser> you can do in user-data
<smoser> and for the most part that is true
<gondoi> smoser, how does backporting work for cloud-init?
<gondoi> for example, i think I found a bug, but if I fix it, i assume it will only land in 0.7.3 and above only
<smoser> gondoi, well for ubuntu, we can get the fix back into 12.04 (or other previous releases)
<smoser> for other distros, whatever their distro policy is.
<gondoi> okay
<gondoi> thanks smoser 
<gondoi> i'll get a launchpad issue opened to start with :D
<gondoi> then i'll try my hand at patching it
<harmw> harlowja: I'm reading over the _write_network stuff in distros/rhel.py
<harlowja> yo
<harmw> you still need to fix that?
<harlowja> i'd still like to get to a non-ubuntu input format
<harlowja> i haven't checked the status of netcfg in a while though
<harlowja> *netcf [https://fedorahosted.org/netcf/]
<harmw> ok, well I haven't verified if rhel_util.translate_network() works - my centos6 instance most certainly didn't configure eth0 accordingly so...
<harlowja> oh, it defintly works
<harlowja> at least good enough for y!
<harmw> i thought so, I probably left some required config 
<harmw> *out
<harmw> netcf looks nice btw
<harlowja> ya, its a little more distro-independent
<harmw> do you have an example of what cloud-init eventually passes to the translate_network() function?
<harlowja> sure
<harlowja> btw, are u using openstakc or ec2
<harmw> shouldn't be that hard to convert that debian-style blob to freebsd's /etc/rc.conf
<harmw> my cloud is openstack, though cloud-init says it took the ec2 datasource
<harlowja> k, thats why its not getting called then
<harmw> ah
<harmw> haven't debugged that part just yet, but thanks
<harlowja> the cfgdrive is the main provider of this data
<harlowja> http://paste.ubuntu.com/6515839/
<harmw> excellent,thanks
<harlowja> http://docs.openstack.org/grizzly/openstack-compute/admin/content//config-drive.html
<harlowja> http://paste.ubuntu.com/6515844/ 
<harlowja> from a grizzly instance at y! 
<harmw> i've read about configdrive, it looks nice
<harmw> can't cloud-init setup static networking on DHCP nets?
<harlowja> sure, if someone provides it that data
<harlowja> the ec2 datasource afaik doesn't have that data
<harlowja> if it did, then sure, it likely could
<harmw> hm hm
<harmw> ok
<harmw> apply_network() only exists in datasources ConfigDrive, NoCloud and OpenNebula
<harlowja> ya, do u know if the openstack ec2 metadata server even provides it?
<harlowja> i don't recall if it does
<harmw> dont think so
<harmw> well, there is a meta-data/local-ipv4 key
<harlowja> ya, thats not enough afaik to be useable by apply_network
<harmw> nope
<harlowja> i think the openstack ec2 could host network config
<harlowja> i think most people that use the ec2 metatadata sever though use dhcp
<harlowja> *but doesn't mean it has to be that way (of course)
<harmw> yea, and besides, the instance already got all required stuff through dhcp
<harmw> so reading that and writing it to sysconfig would seem viable
<harlowja> depends, the whole 169.blah.blah address stuff imho seems to magical (and hard to debug/reason)
<harlowja> configdrive u know pretty much what the source data was, where it is (via blkid) so it can be easily looked at post-boot...
<harlowja> $ blkid 
<harlowja> ...
<harlowja> ex: /dev/vdb: SEC_TYPE="msdos" LABEL="config-2" UUID="124A-8C96" TYPE="vfat"
<harlowja> and the 169 business means u have to run a external service (that can be DOS'd from user vms)
<harlowja> and u have to connect that 169 business via iptables/other 
<harmw> true, true
<harlowja> configdrive simpler imho
<harlowja> but to each there own i guess :-P
<harmw> :)
<harmw> hm, can I attach a configdrive to a running vm harlowja ?
<harlowja> not afaik, its created in nova, during the vm creation process, so if it wasn't created during that process, it won't exist
<harlowja> and settings in nova.conf turn it on/off
<harlowja> force_config_drive=true being the main one i think
<harmw> yea, thats to forcefully create a cfgdrive iirc
<harlowja> ya, all controlled by https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2516 afaik
<smoser> fwiw, i'll be fairly annoying on "change the network interface description code"
<smoser> harlowja, harmw
<smoser> from the openstack side.
<harlowja> agreed
<smoser> i'd accept patches for cloud-init to read one or the other 
<harmw> :)
<smoser> but i think it is in general a bad idea to change that on the hypervisor
<smoser> as then you're in a situation where either:
<smoser> a.) you have to know something about the guest in order to know what to provide it
<smoser> b.) you break existing guests
<harmw> agreed
<smoser> wrt config drive...
<smoser> i think looking forward it will go away.
<smoser> i think it was a stop gap solution for a time when quantum wasn't really there and many people had issues getting the metadata service set up correcdtly.
<harlowja> hmmm, idk
<smoser> the dynamic nature of a web service makes it so much more useful.
<smoser> example: 
<smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1153626
<smoser> ie, on amazon when you attach a new NIC, the metadata service then has information about what the instance should do with that NIC.
<smoser> amazon linux has a tool that then takes the udev event, hits the MD service, and runs 'ifup ethX' appropriately.
<smoser> all making magic work.
<harmw> that sure is nice
<smoser> (good magic in this case ... that can be confusing as other times I complain about magic).
<harmw> :)
<harlowja> ya, its the weird boundary of agents vs init things
<harlowja> *active agents
<harmw> how nice, freebsd respects the rfc in regards to receiving multiple routes through dhcp
<harmw> thus, it just (only) adds the static route to 169.x and not the default route
<gondoi> smoser, I have a question about the upstart process... on my rackspace cloud server, it appears the nova-agent in use is waiting for cloud-init to finish before configuring netowrking...
<gondoi> this is not good since chef can't be installed at that point
<gondoi> although, I can't find the upstart events that would prevent rc 2 from being fired
<gondoi> do you have a good handle on upstart events and their dependencies?
#cloud-init 2013-12-04
<smoser> gondoi, i do have a pretty good handle on upstart events and dependencies yes.
<harlowja> gondoi can u provide the upstart config that starts nova-agent, that might help
<harlowja> *since its likely a rackspace upstart config
<smoser> gondoi, is this something i could jsut start on rackspace ?
<smoser> ie , just public instances ?
<smoser> i haven't started instances there in quite some time
<harlowja> i can try also smoser, i deleted mine though since i got a damn bill for it
<harlowja> although i wasn't aware that rackspace images had cloudinit
<harlowja> *since they have that agent thingy
<harlowja> https://github.com/rackerlabs/openstack-guest-agents-unix
<smoser> gondoi, looking at an instance of saucy i started on rackspace
<smoser> in general sysvinit will *start* only after cloud-init has finished.
<smoser> so first glance, the reason for that (by design) is
<smoser>  cloud-inti starts on
<smoser> (cloud-init.conf)
<smoser> starts on
<smoser>  start on mounted MOUNTPOINT=/ and stopped cloud-init-nonet
<smoser> cloud-init-nonet is a a job who's only purpose is to block running of cloud-init until static-networking is up
<smoser> 'mounted MOUNTPOINT=/' is blocking event
<smoser> meaning /etc/init/rc-sysinit.conf (start on (filesystem and static-network-up) or failsafe-boot)
<smoser> will not run until that has finished.
<smoser> so nova-agent, which is set to run via sysvinit will not run until after cloud-init has run
<smoser> this is by design from cloud-init's perspective
<smoser> hm..
 * smoser wonders if gondoi was running a 'hvm' instance, because i just ran one, and it doesn't seem to get networking up.
<gondoi> smoser, sorry i had to leave yesterday
<gondoi> just catching up on your messages
<gondoi> what is an hvm instance?
<gondoi> i was using the performance class if that's what you mean
<smoser> in my rackspace control panel, there was an option to run 13.10 (ubuntu) in something called "pvhvm"
<smoser> which i suspect is xen HVM rather than PVM.
<gondoi> ohhh no i was not using that one
<smoser> but it never came up.
<smoser> that was an image type.
<smoser> (oddly an image type, not a instance type)
<gondoi> yeah i'm not sure about that one :D
<gondoi> let me see which one I was usning
<gondoi> using
<gondoi> i'm using Ubuntu 13.10 (Saucy Salamander) in ORD
<gondoi> yeah the start on and stop on for rc is confusing to me since it's just looking for a runlevel change
<smoser> gondoi, well its the 'rc-sysinit.conf' that emits that
<gondoi> smoser, i'm sorry i don't see the connection.. I believe that's how it works, i'm just trying to understand it... 'start on mounted ...' that blocks cloud-init right?
<smoser> cloud-init is 'start on mounted'
<smoser> which is "really early"
<smoser> and combined with the 'static-networking"
<smoser> forces a "networking up and root mounted rw" bottleneck
<smoser> so cloud-init blocks boot on itself
<smoser> after cloud-init is done, rc-sysinit.conf can run
<smoser> which will emit the 'runlevel'
<smoser> which will trigger rc.conf to run the sysvinit scripts for the given runelvel (2)
<gondoi> okay... i guess my confusion is on the filesystem state and fired events in upstart.. sorry
<gondoi> upstart is async right?
<smoser> yes.
<smoser> but rc is not.
<gondoi> or does it fire an event mounted MOUNTPOINT=/ and wait for the cloud-init task to finish before finishing all mounts and firing filesystem?
<smoser> and by design cloud-init forces a synchronous bottleneck
<smoser> mounted MOUNTPOINT= events are blocking
<smoser> and nothing happens until after 'mounted MOUNTPOINT=/'
<smoser> so that specific event is very globally blocking
<smoser> this is all by design
<smoser> so that cloud-init can do boothooks or config changes
<smoser> and those things not be race condition enducing
<smoser> inducing
<gondoi> yeah
<gondoi> thanks for the clarification... upstart is much more confusing than I ever expected it to be :/
<gondoi> the docs are great on upstart.ubuntu.com, but i failed to find this one nugget of information in trying to troubleshoot this
<smoser> gondoi, well, part of this is "upstart"
<smoser> and part is "ubuntu"
<smoser> ie, part of this is ubuntu-usage of upstart. not specifically upstart.
<gondoi> oh ....
<smoser> so thats why you didn't/don't find documentation on some of it on upstream upstart.
<gondoi> oh man, it just clicked.. I think that's why this all works with the rackspace debian image.........
<gondoi> is it blocking in debian too?
<smoser> probably blocking in debian, yes.
<gondoi> oh .. hmm
<smoser> i'd have to actually look at debian
<smoser> ie, do they use upstart there?
<smoser> on rackpace debian ?
<gondoi> oh... well.. i actually don't know the answer to that
<gondoi> i'd have to boot one and check it out
<smoser> if they're just using the sysvinit scripts, then ideally cloud-init forces a similar boot order
<gondoi> yeah
<smoser> (ie, it shoudl block just about everything other than '/' is mounted and network is up)
#cloud-init 2013-12-05
<mikehale> qq: will running a user-data script cause cloud-init to skip processing /etc/cloud/cloud.cfg?
<smoser> mikehale, no.
<smoser> what "processing" do you mean ?
<mikehale> well I put some mount directives in /etc/cloud/cloud.cfg but after running `cloud-init-cfg config-mounts` the mount I expected is not setup
<mikehale> maybe there is a guard somewhere to not do anything after the initial boot though
<smoser> 12.04 ?
<mikehale> 10.04 lucid, cloud-init 0.5.10
<smoser> oh. wow.
<smoser> old school
<smoser> from distant memory, if this is supported.... then there is probably a '.sem' file that you'd have to remove in order to make that run again.
<smoser> ie, it only intens to run it once er instance.
<smoser> per instance
<mikehale> k, thanks
<mikehale> strafe says this was read: /var/lib/cloud/sem/config-mounts.i-nnnn
<mikehale> s/strafe/strace/
<mikehale> cool, removing that file allows the directives to be processed
<smoser> mikehale, thanks for the trip down memory lane
<smoser> :)
<smoser> (really just kidding, if i had stuff working on 10.04, i probably wouldn't be looking to just move off it iether)
<harlowja> woah, 0.5
<harlowja> what existed in 0.5
<harlowja> :-P
<mikehale> smoser: we'd like to upgrade to the latest LTS, but it is still a relatively low priority, high risk thing for us
#cloud-init 2013-12-06
<harmw> smoser: (or whoever maintains DataSourceAltCloud.py)
<harmw> would this be a nice litle fix in terms of consistency?
<harmw> http://paste.ubuntu.com/6530317/
<harmw> I noticed this beeing done in one of the other modules
<smoser> (generally, i dont care for the 'which' business at all
<smoser> ie, execve does exactly that for me.
<smoser> which woudl check that the program is available, but theere is no reason to hard code paths
<smoser> that make sense ?
<harmw> totally
<smoser> hard coding paths just breaks stuff.
<harmw> agreed
<harmw> the solution just needs to be consequently applied, thats all :)
<smoser> yeah.
<harlowja> hi spandhe !
<harlowja> https://code.launchpad.net/~shraddha-pandhe/cloud-init/debug-module/+merge/198119/comments/459151 (fyi)
<harlowja> spandhe ^
<harlowja> thx smoser  :)
<spandhe> thanks smoser ! Going through the comments.
<smoser> harlowja, you think those make sense ?
<smoser> just more and more i'm trying to namepsace module config
<harlowja> sure
<harlowja> i think thats fine with me
<harlowja> +2
<harlowja> :)
<harlowja> spandhe if it doesn't make sense, feel free to ask :)
<smoser> in fact, if you think i make sense more than 1/2 the time, something is probably wrong.
<harlowja> hahaha
<spandhe> smoser: It does make sense :)
<harlowja> smoser http://lists.openstack.org/pipermail/openstack-dev/2013-December/021482.html :)
<smoser> i really think unified guest is just plain silly.
<smoser> err... unified guest agent.
<harlowja> possibly, haha
<smoser> there are hunderds of perfectly functional "agents" that run in a guest
<smoser>  * puppet
<smoser>  * chef
<smoser>  * sshd
<smoser>  * jujud
<harlowja> sure sure
<harlowja> guess it depends on the use-case, for that kind of stuff i agree
<harlowja> and one to rule them all, seems highly unlikely
<harlowja> *depends on what the agent is doing i guess*
<harlowja> *or what the agent wants to do
<smoser> what does "VMs are not accessible from controller." mean?
<harlowja> my guess is not accesible easily from control network (which i guess is where savanna runs)
<harlowja> so isolated control network from vm network, means savanna can't actually do that ssh
<harlowja> although not enabling ssh access seems odd
<smoser> or it maybe means that savanaa has no business being on the control network
<smoser> :)
<harlowja> right
<harlowja> so that might be what it means, not 100% sure either, my interpretation, ha
<smoser> no.
<smoser> i think your interpretation is right
<smoser> but what i'ms aying is "why was savanah not operating on the tenant network"
<smoser> which clearly has access to the tenant machines
<harlowja> sure, its a good question
<harlowja> probably complicates savanna deployments i would think
<harlowja> or would require savanna to open some connection to tenant network before it would do some work
<harlowja> *which i guess is setup hadoop
<harlowja> said temporary access might not be possible (almost requires something like oauth, which allows limited application access to work on behalf of a user for a given time)
<harlowja> so i guess then there idea of we'll just bypass all that with a agent :-P
<smoser> ewll, in this case i'm mainly just arguing that savanah has no business as part of openstack
<smoser> rather than "juju deploy hadoop"
<smoser> (which needs no such special access)
<smoser> or "ansible make me hadoop"
<harlowja> u won't get any disagreement from me there :-P
<harlowja> or create instnaces with tiny hadoop setup userdata script
<harlowja> :-/
<smoser> trove somewhat seems more valid.
<smoser> but really, i dont see any reason why you need a transport mechanism that is not ipv4
<smoser> it turns out lots of things use ipv4
<smoser> and its pretty reliable
<smoser> i think i'm even chatting with you over it!
<harlowja> for real?
<harlowja> haha
<harlowja> ipv4 -> nsa -> irc
<smoser> exactly.
<harlowja> ya, hehe, the amount of projects in openstack that something like juju already does is a little weird, i agree
<harlowja> juju, ansible, .... [500+ others]
<harlowja> anyways, its openstack, never know what it will do at anytime, haha
<smoser> right
<smoser> i dont think juju is special
<smoser> i just use it because thats what i'm told to use
<smoser> :)
<harlowja> of course
<harlowja> i don't think rolesdb, pogo (yahoo things) are special either
<harlowja> *which serve similar purposes as juju
<harlowja> machine gets a role [role being a list of packages that it will have, for ex: mysql_server, blah blah], pogo can be used to ensure role gets onto instance...
<harlowja> yada yada, ha
<harmw> smoser: https://code.launchpad.net/~harm-o/cloud-init/freebsd
<harmw> i've hacked up some code for fbsd
<harmw> perhaps you can take a peak at it some time :)
<smoser> harmw, i forget, have you signed the CLA ?
<smoser> harlowja, ^ https://code.launchpad.net/~harm-o/cloud-init/freebsd/+merge/198130
<harmw> I'm quite new to bazaar/launchpad when it comes to adding code and such, so i'm a bit clueless on what that means :)
<smoser> harmw, http://www.canonical.com/contributors
<smoser> you have to read the appropriate "individual contributors" or "entities" likn, agree, and then click on "signed online" link and sign it
<harmw> ah, the 'all your base are belong to use' stuff :)
<smoser> you'll be asked to sign in with your launchpad id.
<smoser> harmw, exactly.
<smoser> :-(
<smoser> harmw, did sed -n really not work ?
<harmw> nope
<smoser> is it just '-n' ?
<smoser> because its not 1987 any more
<harmw> let me check on that again
<harmw>  sed '/^[0-9]\+[.][0-9]\+[.][0-9]\+:/ {s/://; p; :a;n; ba; }' ChangeLog
<harmw> sed: 1: "/^[0-9]\+[.][0-9]\+[.][ ...": unexpected EOF (pending }'s)
<smoser> hm..
<smoser> bugs me when a utility from a "full blown unix" is less capable than the one from busybox
<harmw> hehe
<smoser> uptime is clever.
<harmw> yeah well, thats on the list of improvements
<harmw> since it's totally whack to include the library stuff like i did there
<harmw> should go in a more decent function of some kind
<smoser> yeah
<smoser> harmw, there is a bug for "use ip rather than route"
<spandhe> I want to skip some tests while running nosetests. How do I do that?
<smoser> is ip more portable ?
<smoser> spandhe, to run only on one file just run like:
<smoser>  nosetests tests/unittests/test_datasource/test_altcloud.py
<harmw> good one... its from iproute, afaik thats specificly linux
<spandhe> smoser: thanks!
<harmw> hm, -1 for adding a blank line
<smoser> harmw, hm..
<smoser> that looks really nice though over all
<harmw> thanks :)
<smoser> can you online resize disks on freebsd?
<smoser> ie, like i an on 3.8+ kernels on linux
<smoser> s/disks/partitions/
<harmw> it requires a reboot, eg. like centos 6 
<harlowja> woah, nicefreebsd
<smoser> well, on centos there is cloud-initramfs-growpart
<smoser> (ie, that works under drakut on centos in initramfs when the partition isnt mounted)
<harmw> indeed
<smoser> is there an analog to that in freebsd ?
<smoser> something we could make for the initramfs ?
<harmw> not that i know of
<smoser> is thre an initramfs ?
<harlowja> smoser now this will bring up the battle of does y! want to keep on enabling the usage of  freebsd :-P
<harlowja> if we allow vms with it, then freebsd never die, ha
<harlowja> *die in yahoo
 * smoser used *bsd probably last in like 2001.  that was before harlowja could ride a bike.
<harlowja> likely
<harmw> no initramfs on fbsd smoser 
<smoser> reboots suck
<harmw> true, though freebsd boots quite fast nonetheless :)
<smoser> harmw, we could make it reboot magically if necessary
<harmw> also possible
<smoser> ie, like cloudinit/config/cc_package_update_upgrade_install.py does
<smoser> if it grew the root, then just reboot, and things continue on where you last left off.
<harmw> ah, instant reboot on the spot?
<smoser> yeah.
<harmw> would be good enough
<smoser> that module there does that if a package upload marks 'reboot-required'
<smoser> s/upload/upgrade/
<harmw> yea, im going over the source now
<harmw> *reading
<smoser> i'm heading out for th eweekend.
<smoser> thanks again, harmw and spandhe .
<harmw> surething
<smoser> harmw, please read/sign the cla
<harmw> already did that
<smoser> ah. great.
<smoser> have a nice weekend all
<harmw> same to you
<spandhe> smoser: enjoy your weekend!
#cloud-init 2014-12-02
<john-davidge> Hello everyone o/
<john-davidge> I have a question. Is it possible to change the 'magic' IP (169.254.169.254) cloud init uses to access EC2-style meta data with a config option?
<JayF> I don't believe so; but IMBW
<john-davidge> Interesting. Specifically I'm interested in using an IPv6 address instead of the default. Perhaps this will require a change to cloud inti
<john-davidge> *init
<harmw_> ipv6 metadata? 
<harmw> how whre you planning that?
<JayF> Oooh. Fancy.
<JayF> I like IPv6, would be interesting to hear how you get it working :)
<harmw> ^ +1 :)
<john-davidge> so I'm testing for cloud environments running on IPv6-only networks, and I'm trying to figure out what changes (if any) are needed to get cloud-init working
<john-davidge> I'll do some more poking around and see what needs doing. Will keep you updated :)
<JayF> Well you can always use configdrive with cloud-init
<JayF> and just get rid of the metadata service
<JayF> if you wanted to just not have to worry about that piece of it today
<john-davidge> JayF: True, but I'm trying to cover all the bases for now.
<harmw> I'm running IP6 cloud atm
<harmw> with only metadata on V4
<harlowja_> john-davidge so u can change it; the problem is that it needs to get the config from somewhere to change that value in the first place ;)
<harlowja_> unless u have control over your images
<john-davidge> harlowja_: Does that have to be changed on an image-by-image basis, or can it be passed at startup via user-data or similar?
<JayF> well, unless you use configdrive
<JayF> the metadata service is how user_data gets to the node :)
<JayF> so you have a bit of a bootstrapping problem
<harlowja_> yup :)
<JayF> you have to make the default (last IP in v4 link local) work or patch your images to have a different cloud-init config
<harlowja_> john-davidge the issue is 'passed at startup via user-data' (how does this data get to the image if not by the metadata service)?
<john-davidge> Hmm, very true
<john-davidge> What is the liklihood that cloud-init could adopt an IPv6 default as well as the current IPv4 default?
<john-davidge> And does anyone know who I'd need to talk to about that?
<harlowja_> u talking to the right people i think ;)
<harlowja_> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceEc2.py is where the ec2 stuff happens
<harlowja_> and where said change would occur
<harlowja_> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceEc2.py#L42 could likely just have a ipv6 url, and it might 'just work'
<harlowja_> although idk what ipv6 default that should use :-P
<john-davidge> harlowja_: Thanks! Will poke around with it in the morning
<harlowja_> k
<john-davidge> harlowja_: Yes that's another debate!
<john-davidge> I was thinking the easist might be the v4-in-v6 of the current default
<john-davidge> Perhaps ::ffff:a9fe:a9fe
<harlowja_> maybe
<harlowja_> so then it'd be interseting to try that
<harlowja_> if u are stuck on ec2, that might be hard
<harlowja_> since u need somewhere that provides an ipv6 metadata service
<john-davidge> harlowja_: I'm on OpenStack, so I can do what I want! :D
<harlowja_> kk
<john-davidge> harlowja_: Will give it a go and report my findings tomorrow. 7pm here in the UK so I'm clocking off for the day
<harlowja_> kk
<harlowja_> cool
<john-davidge> Thanks for your help!
<harlowja_> may the force be with u :-P
<harlowja_> ha
<john-davidge> Always!
<tennis> hi... Where are the various files with "#cloud-config" supposed to live?
<harlowja_> define live?
<harlowja_> u can place them wherever u want
<smoser> o/ just wanted to say hello.
<smoser> tennis_, they live in /etc/cloud/cloud.cfg.d/whatever.cfg
<smoser> or you can put them in as user-data.
<tennis_> smoser: How does that relate to /etc/cloud/config.cfg?  That is, does /etc/cloud/cloud.cfg get ignored? 
#cloud-init 2014-12-03
<mhroncok> hi again
<mhroncok> any chance of this being merged? https://code.launchpad.net/~harlowja/cloud-init/py2-3
<tennis> smoser: U there? 
<smoser> here.
<smoser> you dropped yesterday.
<smoser> files are loaded and merged with later getting preference over newer
<tennis> smoser: Yup.  Sorry.  Life got in the way.  Thanks for responding. 
<smoser> /etc/cloud/cloud.cfg , then /etc/cloud/cloud.cfg.d/*.cfg in C locale sorted order
<tennis> smoser: ok.  Do I need to explicitly call  "curl http://169.254.169.254(etc)" to get my public keys, or is that accomplished somewhere automatically in cloud-init processing?
<tennis> smoser: I already have a key pair defined in my environment, but for some reason the init process doesn't put it into the "~/.ssh/authorized_keys" file.  Trying to figure out where it comes from.
<smoser> tennis, cloud-init does that.
<smoser> tennis, it should do that.
<tennis> smoser: Thats what I was thinking too.  But for some reason it does not seem to.  Can't get into my images because of it.  Can you point me to a minimalist example that should do that and notthing (or very little) more? 
<smoser> tennis, i'd 'backdoor' the images, giving you another way in
<tennis> smoser: ...and the best way to do that is ... ? (sorry,  yet another question :))
<smoser> tennis, ubuntu ?
<smoser> what i would use to backdoor an image is
<smoser> https://code.launchpad.net/~smoser/+junk/backdoor-image
<tennis> smoser: Ah, so its "baked in" to the image. 
<smoser> right.
<smoser> thats how i debug such things
<tennis> smoser: Thanks
<tennis> smoser: I'll use packer to drive the image setup with 'backdoor' included.  That should enable everything.
<tennis> smoser:  Not sure how backdoor-image would be called.  Is the 'target' meant to be an AMI? 
<tennis> smoser: basically, i'm wondering where it is called from and what the target should be.
<smoser> tennis, sorry. target is some disk image.
<smoser> or root filesystem.
<tennis> smoser: got it. 
<tennis> smoser: makes it complicated to use with an aws ami. Hmmm ... 
<smoser> well what would you want it to take ?
<tennis> smoser: Just saw your resp.  Can it be used on an active filesystem, or does it expect it to be an externally-mounted drive? 
<smoser> it will work on active filesystem, yes.
<smoser> block device or filesystem root
<smoser> ie, backdoor-image /
<smoser> or
<smoser> backdoor-image /dev/sda
<smoser> or 
<tennis> smoser: ah,ok  ...Thanks
<smoser> backdoor-image my.qcow2
<smoser> fwiw, its logic is just:
<smoser>  if -d $PATH; then
<smoser>    operate on path
<smoser>  elif -b $PATH; then
<smoser>    call mount-image-callback $0 _MOUNTPOINT_ "$@"
<tennis> smoser: I noticed you wrote util.py in cloud-init (nicely done, btw).  I'm getting "util.py[WARNING]: Failed loading yaml blob" errors.  How do I make the output verbose enough to tell me where the problem is?  Or, is there a way to validate the yaml with your code offline?
<tennis> smoser: ok. That makes sense.
<smoser> well, if you're feeding bad yaml
<smoser> then just:
<smoser>  python -c 'import yaml; yaml.load(open("my.yaml").read())'
<smoser> will likely give the same error
<tennis> perfect.  Thansk
<tennis> thanks
<tennis> Yup.  That found it.
<dryicerx> hello. Is there a accepted method for enabling services (upstart specific) via cloud init in a clean manner
<dryicerx> (or enabling/disabling services with a clean cloudinit way)
<tennis> smoser: How to debug "failed to shellify" errors? 
#cloud-init 2014-12-04
<xenol> hi, I am trying to setup cloud-init and I would like to setup root password, which is provided by SmartOS datasource as root_pw metadata. However, I do see anything in the source that sets up the root password
<xenol> is there a way how to achieve this?
<tennis> smoser: Trying to set an id other than ubuntu to be the system default. I notice that cloud-init creates a file called /etc/sudoers.d/90-cloud-init-users.  Can cloud-init generate this file with my alternative user id in it, or do I have to manually zap it like what is described here: http://alestic.com/2014/01/ec2-change-username
<xenol> can cloud-init set root password?
<smoser> if you just define it in the users, it might not know any better.
<smoser> tennis, you just provide the user you want in user-data
<smoser> yeah, as that blog post says.
<smoser> xenol, similar answer. 
<smoser> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config-user-groups.txt
<smoser> you might be able to do:
<smoser>  users:
<smoser>   - default
<smoser>   - name: root
<smoser>    passwd: foo
<xenol> smoser: I have password available by the metadata service and I would like to get it from there if possible
<smoser> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt#L565
<smoser> so cloud-init probably can't direclyt do it like that. 
<smoser> i'm not *opposed* to a feature ot pull it from the openstack metadata service
<smoser> but really thats a regression
<smoser> use ssh public keys
<xenol> so, I have a default cloud.cfg, which is slightly modified to suit my needs. If I want to add users, shoudl I move cloud.cfg contents with "users" block into metadata service?
<smoser> you mean to user-data
<xenol> I have defined datasource_list in my cloud.cfg. I suppose that should stay there and the rest moved to user-data?
<smoser> yeah,, generally thats more dynamic. although user can override what is in cloud.cfg through user-data
<tennis> smoser: Thanks!  Yeah, I have tiny script (like the blog post) which I run just before a reboot.  
<tennis> tennis: testing
<xenol> smoser: the passwd needs to be a hash, right? 
<kwadronaut> xenol: i agree with smoser, use ssh public keys, generate passwords.
<xenol> I am generating them creating virtual machines (I'm not giving the user the ability to input his password). I just to this because not every user has a SSH key and expects password
<xenol> This way I am able to offer both solutions
<tennis> smoser: re the usermod to change the default user.  How can that work for the default id if it is being called by the (current) default?
#cloud-init 2014-12-05
<smoser> tennis, the 'users' list defaults to just 'default'
<smoser> which is special
<smoser> and reads the 'default' from the operating system section
<smoser> but you can just change that in that section to whatever you want
<tennis> smoser: ok.  Thanks. :)
<smoser> just as a reminder, basically everything other than the datasource list can be patched or reset in user-data
#cloud-init 2015-12-04
<resmo> hi
<resmo> using cloudinit 0.7.5,  cloudstack as datasource, and facing an issue, root password won't be set using the password-server API of cloudstack
<resmo> any hints?
<Odd_Bloke> resmo: That's supported in more recent versions of cloud-init, so upgrade if possible. :)
<resmo> Odd_Bloke, ok then, thx!
<zilberstein> hello #cloud-init
<zilberstein> how to debug user-data ?
<zilberstein> cloud-init just ignores everything
<larsks> zilberstein: Any logs from cloud-init?  Is the user-data actually available where cloud-init is looking for it?
<harlowja> zilberstein find the log files
<harlowja> and go from there
<harlowja> *cloud-init log files
#cloud-init 2016-12-05
<another_larsks> smoser, I'm curious how soon you think 0.7.9 might drop...I see that there are a ton of changes to the systemd units since 0.7.8, and thinking about carrying patches vs. just waiting a bit.
<larsks> harlowja, systemd dependencies question:  if cloud-init-local.service is Before=NetworkManager, shouldn't it also be Before=networking.service?
<larsks> And similarly, if cloud-init.service is After=networking.service, shouldn't it also be After=NetworkManager.service?
<larsks> This may just be me being unfamiliar with the naming of ubuntu/debian systemd units...
<harlowja> nrezinorn ^
<harlowja> larsks there was dragons there that nrezinorn hopefully remembers
<larsks> harlowja, because I need to set deps correctly on the redhat legacy "network.service" as well as NetworkManager, and want to be consistent...
<harlowja> draggggggons
<harlowja> lol
 * harlowja poked nrezinorn internally, he should be able to answer
<nrezinorn> all i remember is that on the newer cloud-init the systemd files were wrong for RHEL 7
<mdorman_> doyou have the link to the bug or patch?  This was a while ago, iâll have tp page this stuff back into my brain
<nrezinorn> because its called network.service (rhel) vs networking.service (ubuntu)
<nrezinorn> thats all i remember, and i think i just made a specific file for rhel builds, but i dont think any of that is merged back upstream yet
<mdorman_> i think taht was the only difference, too
<nrezinorn> once i fixed the rpm build for network.service things just worked - for my super limited testing that i did lol - other things have taken my time away
<harlowja> larsks u trying to fix it for good i hope :-P
<harlowja> never again!
<harlowja> lol
<larsks> nrezinorn, where are your fixes?
<larsks> harlowja, I don't know.  Because rhel and ubuntu have different service names, we can't really have a single unit file.
<larsks> I think unit files should be part of the packages, not part of the upstream code, but smoser and I disagree on that point.
<harlowja> larsks right, we may need 2
<harlowja> larsks prob disagree with me also :)
<harlowja> i like it when code-bases can build themselves into packages
<harlowja> if we have 2 unit files, doesn't seem to bad
<larsks> harlowja, I think it's a terrible idea because it requires the upstream code maintainers to also be expert package maintainers for multiple distributions.
<larsks> Or they end up being experts in one distribution, and everyone else has to carry a list of patches a mile long.
<harlowja> sounds like package creation is to much a PITA then
<harlowja> and if it requires a expert then its not done right :-P
<harlowja> packaging software shouldn't require experts imho ;)
<larsks> Which is why it should just be left to the package maintainers.
<harlowja> (not in the 21st century)
<larsks> Why make upstream developers worry about those things?
<larsks> Let the upstream concentrate on quality code, rather than making sure they have the names of depdendencies correct.
<harlowja> i get that POV
<harlowja> it still feels like a messed up situation
<larsks> I dunno.  I mean, ubuntu/debian/redhat/suse/fedora/alpine/etc...everyone has there own packaging system.  I'm not sure it's possible to get everyone to settle on a single set of package metadata.
<larsks> ...until systemd introduces a packaging format, I guess.
<harlowja> sounds like we need a package trump for president
<harlowja> lol
<nrezinorn> make packages great again!
<larsks> So here's another idea...we remove *all* distribution specific dependencies out of the unit files, and those go into separate systemd drop-in files.  This lets the distro-specific stuff be managed without needing to duplicate the entire unit file(s).
<harlowja> k
<larsks> nrezinorn, you mentioned that you were working on the rpm packages...what did you end up with for service dependencies?  Are those rpms somewhere I can grab them and look?
<nrezinorn> larsks: i last poked things 2 months ago- and i dont have anything to share .  let me see if i can drop a file somewhere you can grab...you can poke it yourself (if i remember where i did all this stuff)
<nrezinorn> i cant find where i put the stuff i was doing lol, it may have been on a server that got wiped...so thats fun
<larsks> nrezinorn, thanks for looking!
<nrezinorn> i really only added the rhel files for systemd and editing some other thing, prob the makefile macros to feed in the rhel systemd files
<nrezinorn> there was also other things not working right lol
<nrezinorn> i looked under a few rocks, sorry i couldnt find it.
<larsks> harlowja, debian packaging question: what part of the packages/debian/* files causes the systemd unit files from the systemd/ directory to get copied into the right place?
<harlowja> eck, ask smoser not me :-P
<larsks> Darn.  smoser seems to be afk today.  Or at least af irc.
<nrezinorn> ill find it for you
<harlowja> i don't do that debian stuff, ha
<nrezinorn> https://git.launchpad.net/cloud-init/tree/packages/bddeb?id=166df605dc9864eb163007300db7a611feb309d6  something in that lol
<nrezinorn> jsut trace it all the way down to where its copying the systemd/ folder i guess :)
<nrezinorn> ah i see
<nrezinorn> so the debian packager just takes ther tarball of the source
<nrezinorn> so "it's just there"
<larsks> nrezinorn, ugh, right, we've abused setup.py as a general installation tool.  Thanks.  I keep forgetting that.
<nrezinorn> yeah im not a fan of that stuff lol
<suro> @harlowja: around?
<larsks> Ugh, so why does the spec file copy things in manually?  HEAD ASPLODE.
<nrezinorn> rpm spec?
<nrezinorn> it copies things becuase it has to be copied into the RPM BUILDROOT, thats how rpms work
#cloud-init 2016-12-06
<larsks> nrezinorn, but those files are already installed by setup.py.
<larsks> In this case, the spec file should just trust setup.py.
<larsks> harlowja, smoser: https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/312519 is my proposal for moving distribution-dependent service names out of the systemd unit files, as I was discussing with harlowja earlier today.
<smoser> hm.
<smoser> larsks, did you specifically have a problem ?
<smoser> while there is noise in those files for sure (such as Before=network.service). it doesn't cause any issues that i'm aware of.
<larsks> smoser, if you prefer to just list all the dependencies in the existing file (so Before=network.service networking.service NetworkManager.service), I'll submit that patch instead.
<smoser> what does that diff look like ?
<larsks> I like keeping the config files clean, but either solution works.
<smoser> i suspect its pretty small isn't it ?
<larsks> Yeah.
<larsks> smoser, speaking of service dependencies....cloud-init-local.service is Before=NetworkManager.service.  Should it also be Before=networking.service (on ubuntu)?
<smoser> larsks, i am pretty sure ubuntu's are right.
<larsks> Okay.  But why a dependency on NetworkManager but not on networking.service?  I am just trying to understand the dependencies.
<larsks> smoser, because I would like to make the deps under RHEL match as much as possible, but I don't understand the current model w/r/t networkmanager vs. networking.
 * larsks heads out for a bit; back later.
<smoser> right.
<smoser> larsks, before=network-pre
<smoser> is sufficent actually
<smoser> and *should* be sufficient for NetworkManager
<smoser> but netowrk manager has  bug
<smoser> i think there is an upstream bug for that
<smoser> yea, so NetworkManager should have
<smoser>  After=network-pre.target
<smoser> its buggy that it does not
<smoser> larsks, https://bugzilla.gnome.org/show_bug.cgi?id=761001
<larsks> smoser, awesome, thanks!
#cloud-init 2016-12-07
<mastier> Sorry, bothering you, but maybe some dev got the idea why the hell it's not working as expected
<mastier> http://stackoverflow.com/questions/40979937/cloud-init-datasource-setting-timeout-doesnt-work
#cloud-init 2016-12-08
<powersj> smoser: integration test merge is good to go. Other than adding new files, the only changes are to tox and added to RTD index. https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/312830
#cloud-init 2016-12-09
<mastier> no idea, anyone ? what can cause that changing timeout and max_wait for datasource doesn't work
<mastier> can you guide me how to debug it ?
<smoser> mastier, ?
<smoser> what was the question ?
<ffledgling> Hello, I was going through the examples on https://cloudinit.readthedocs.io/en/latest/topics/examples.html
<ffledgling> It seems like https://cloudinit.readthedocs.io/en/latest/topics/examples.html#adjust-mount-points-mounted uses an option called "nobootwait" and mentions that this should be used for volumes that may not be attached at instance boot
<ffledgling> It seems like that's not a valid option for fstab in general and was only supported by certain versions of ubuntu
<ffledgling> The correct option to use is likely "nofail", can the example be changed to reflect that?
<nacc> ffledgling: i assume that could be done with a merge request somewhere
<ffledgling> nacc: I think I need to modify the docs folder in https://git.launchpad.net/cloud-init
<ffledgling> But I wanted to confirm that that's not an intentional mention
<nacc> ffledgling: fyi, http://unix.stackexchange.com/questions/53456/what-is-the-difference-between-nobootwait-and-nofail-in-fstab
<nacc> ffledgling: so you could verify they are semantically the same (and if not, file a diffeent bug :)
<ffledgling> nacc: I saw that question, I can confirm that nobootwait does not work on fedora (which is how I started digging in the first place), and from the comments - "nobootwait is no longer a valid option in Ubuntu 16.04 (as of 2016-07-10 ...)" :)
<ffledgling> I don't think they're semantically the same either
<nacc> ffledgling: yeah, makes sense. I would suggest sending a merge request with explanation there and see what the maintainers say
<ffledgling> nacc: Will do
<ffledgling> Hi, I opened up a MR for change to docs - https://code.launchpad.net/~ffledgling/cloud-init/+git/cloud-init-1/+merge/312917
<ffledgling> I'm not entirely sure who to mark for review, if someone could take a look and give me feedback, that'd be great.
#cloud-init 2017-12-04
<blackboxsw> smoser: rharper I'm no longer op in this channel. can either of you change topic to Next status meeting Monday: 12/11
<rharper> blackboxsw:  /topic all yours
* blackboxsw changed the topic of #cloud-init to: Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj |  Next status meeting: Monday 12/11 16:00 UTC | cloud-init 17.1 released
<blackboxsw> teach a man to fish ....
<blackboxsw> tjx
<blackboxsw> thx
<smoser> someoow i can make it auto-op you. let me find
<blkadder> Hi all. Having problems partitioning a device: https://paste.ubuntu.com/26113667/ Wondering if it is me or an actual bugâ¦
<blkadder> Do I need to explicitly wipefs the device beforehand perhaps?
<smoser> blkadder: well, ideally you would not need to.
<blkadder> Thatâs what I would hope. :-)
<smoser> i'd consider it a bug in cloud-init, but yeah. thats what the issue is.
<smoser> maybe there is some way to say that ... let me check
<blkadder> Gracias.
<smoser> what is yit looks like if you can live with a gpt it might work
<smoser> table_type: <'mbr'/'gpt'>
<blkadder> k
<blkadder> Iâm trying to move /var so hope so.
<smoser> go ahead and file a bug though please
<smoser> ubuntu-bug cloud-init
<blkadder> Will do, thanks.
<smoser> will do it
<blkadder> Done.
<blkadder> gpt works, thanks.
<ybaumy> hi
<ybaumy> can somebody help me find an error in a cloud-init python config script? http://paste.ubuntu.com/26114214/ the error is at the bottom
 * rharper takes a peek
<ybaumy> for there were the default path variables wrong on centos
<ybaumy> s/for/first
<ybaumy> i changed those
<ybaumy> and then this error comes up
<rharper> I wonder if there's something up with the python2.7 puppet in centos then ?
<rharper> 6? 7 ?
<ybaumy> its 7 the latest patches
<ybaumy> i already wrote a post to the mailling list but no one replied yet. and i need that puppet working
<Wulf> with six.StringIO() as outputstream...
<Wulf> perhaps something wrong with six?
<blackboxsw> there was a BytesIO instead of stringIO
<blackboxsw> I think I already submitted a patch for that
<rharper> google is telling me python2.7 and with and StringIO may be problematic
 * blackboxsw lookgs
<blackboxsw> f831a874021f3d6d24cbe5639a176f416b5436a6 in master
<rharper> blackboxsw: ah, fixed upstream but maybe not yet in centos's cloud-init
<blackboxsw> not sure if that's in the centos version that is being run agains
<rharper> there are some fedora/rhel folks in here
<rharper> ybaumy: you may need to file a centos/rhel issue to get the upstream commit merged into the distro cloud-init
<ybaumy> i had trouble with django issues with katello/foreman and i installed the latest django version. would something like that be possible to fix this?
<ybaumy> i installed that with pip
<ybaumy> rharper: that will take forever i guess
<blackboxsw> yeah given the comment on line 77 of the pastebin, my patch for BytesIO isn't in that version
<rharper> yeah, I dunno what the general frequency of updates
<Wulf> I had a look at debian/sid, and python2 doesn't like with six.StringIO() either.
<ybaumy> ok then i will write an update to my mail to the mailling list
<blackboxsw> hrm, we have a copr daily build of master at  https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/ that might help in this case. :/ at least it'd have the cloud-init fix
<ybaumy> ah it might be in epel already
<ybaumy> let me check that
<ybaumy> hmm no not in the official
<ybaumy> yea
<ybaumy> :D
<ybaumy> works
<ybaumy> you still have to change pathes in py and compile to pyc
<ybaumy> thanks guys
<blackboxsw> good to hear
<blackboxsw> at least we don't have an upstream bug to work ;)
<rharper> ybaumy: what's the path changes ?
<rharper> that sounds like possible upstream bug or is this related to your pip ?
<ybaumy> ok last question before i quit for today. the certname is nocloud.kent-hanchett.1000.local but my hostname is kent-hanchett.1000.local. why is that? i added certname: "%i.%f" to the cloud-init config
<ybaumy> rharper: i dont think so. on the client i havent updates django yet only on the server
<ybaumy> i removed the %f
<ybaumy> i mean i
<ybaumy> now the hostname is correct
<dojordan> @blackboxsw I've addressed your comments, when you get a chance can you look again? https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/334341
<blackboxsw> dojordan: thanks, will do today. trying to unstick myself at the moment ;)
<ybaumy> rharper: path changes to /etc/puppetlabs/puppet and the /var/lib/puppet/ssl to /etc/puppetlabs/puppet/ssl
<ybaumy> rharper: this is the client from the official repo
<rharper> ybaumy: does that match what distro clients use?
<ybaumy> rharper: i dont know... the official katello install docu says i should use this client. so i did
<rharper> I think it could make sense to support both ;
<ybaumy> rharper: think so too.
<rharper> ybaumy: can you file a bug ? https://bugs.launchpad.net/cloud-init/+filebug and attach your changes to the path ? we can pick that up from there
<ybaumy> or make the path configurable
<ybaumy> in the cloud-init.cfg
<rharper> yeah
<ybaumy> rharper: will open a bug
<rharper> thanks
<ybaumy> np
<blackboxsw> does puppet.cfg announce that configuration path difference?
 * blackboxsw wonders if we can inspect that config change from puppet itself (either through cmdline or config file)
<ybaumy> https://bugs.launchpad.net/cloud-init/+bug/1736261 hope thats enough
<ubot5> Launchpad bug 1736261 in cloud-init "puppet.conf and puppet ssl path changed in official client" [Undecided,New]
<ybaumy> im off for today. thanks for your fast help guys. really brought me forward
<rharper> ybaumy: thanks again for the bug
<smoser> blackboxsw: http://paste.ubuntu.com/26114924/
<smoser> that seems to function for laujnch-softlayer
<smoser> lots of improvements still to do, but generally gets near there. and makes me more familiar with  launch-ec2 also
<blackboxsw> smoser: good deal, I was just pushing up changes to launch-ec2 for specific vpc/subnet/route_table fixes
<blackboxsw> checking it out
<blackboxsw> just pushed to qa-scripts everything I have ( just finished testing --proposed ec2 for the dhcpclient  sanbox fixes)
<blackboxsw> smoser: can you push a launch-softlayer to qa-scripts too. we can consolidate on next iteration
<smoser> blackboxsw: i'll push tomorrow
#cloud-init 2017-12-05
<smoser> blackboxsw: i pushed there.
<ybaumy> moin. how do i setup nocloud correctly? i added this datasource: NoCloud: to cloud-init.cfg but im getting an error on first boot. and on disable ec2 im getting an error too. http://paste.ubuntu.com/26116710/
<ybaumy> version is now cloud-init-17.1+46.g7acc9e68-1.el7.centos.noarch
<ybaumy> strangely this didnt happen yesterday. but now i created a new vmware image template and then added a host to foreman
<ybaumy> maybe i should use ansible to setup puppet and the rest
<ybaumy> i was using cloud-init with openstack and that was ok. but that datasource nocloud thing i dont get
<smoser> ybaumy: i'm not sure about the first issue with ec2 disable.  there is probably something in /var/log/cloud-init-output.log or /var/log/cloud-init.log (you could pastebin the whole of both would be helpful)
<smoser> for nocloud you'd need to give mreo data... not sure what would cause there to be no md (metadata) on that object.
<smoser> for setting up nocloud, i suggest using 'cloud-localds' http://paste.ubuntu.com/26118885/
<smoser> but basically nocloud is a attached disk in either iso or vfat that has a 'user-data' and 'meta-data' on them. meta-data is yaml formated, and expected to have 'instance-id'.
<ybaumy> smoser: i think i dont need all of that nocloud parameters. i only want to setup puppet and thats about it. i feel like im breaking a leg here for that to do.
<smoser> its nto a lot of parameters.
<smoser> instance-id
<smoser> i'm only suggesting using cloud-localds to get you a disk so you dont have to worry about impmlementation, and you can see a working example.
<ybaumy> ok
<robjo> smoser: blackboxsw rharper Could I please get some feedback on the chrony support? See mailing list message dated 11/14 "RFQ chrony support" which should of course have been "RFC"
<smoser> :-(
<smoser> yeah, that is fair
<ybaumy> smoser: how do i create a new instance-id for each cloned vmware image
<ybaumy> i guess they have to be unique
<ybaumy> hmm i could make a firstboot scripts that runs before cloud-init
<smoser> ybaumy: just attach different data in a disk, no ?
<smoser> i'm confused.
<smoser> instance-id=$(uuidgen)
<smoser> so i'm missing something for sure.
<ybaumy> im confused as well
<ybaumy> ok let me explain what i want to do
<ybaumy> i have a vmware template that is centos. then i want to automatically deploy this image with foreman and register it to the katello part
<ybaumy> therefor i need puppet integration
<ybaumy> the puppet part i still not working 100% but 90%
<ybaumy> so i thought using nocloud datasource is the best solution when running cloud-init
<smoser> ok. so i think youa ve a couple options
<smoser> a.) vmware has an "OVF" datasource that should mostly do what you're wondering about.
<smoser> i really have no familiarity with it  vmware though.
<smoser> b.) when you create a new "instance" (that is 'automatically deploy') you can create a second disk and attach it with the necessary NoCloud data
<smoser> c.) you can just put NoCloud data into each image hard coded in /var/lib/cloud/seed/nocloud-net. it will just never change, you'll have to find some other way to identify individual systems or give them specific instructions.
<smoser> the 'instance-id' is really just a bit of information specific to this instance.
<smoser> it allows cloud-inti to do things that should only be done on a "new instance".
<ybaumy> smoser: c) sounds th ebest option. since all the custom parts are run through puppet when it is setup
<smoser> but what does puppet use to determine what their role is
<smoser> if 100 things come and say "hi puppet master" all at the same time, how does it decide what is supposed to do what?
<ybaumy> foreman has hostgroups which is a customer and all those hostgroups have different parameters for puppet
<smoser> and hostgroups are based on ?
<smoser> hostnames ? which you're providing ?
<smoser> how does the instance know its hostname?
<ybaumy> so i just create a host put it in a hostgroup which inherits all parameters for that hostgroup
<ybaumy> hostname gets through dhcp on the first setup before customizing
<ybaumy> hostgroups are just groups with different attributes and parameters
<ybaumy> its really cool
<ybaumy> i will try c) now
<ybaumy> lets see how far i get
<smoser> are you using cloud-inti for anything else ?
<smoser> does it do anything for you?
<ybaumy> nope just puppet i need
<smoser> then you can just disable cloud-init entirely.
<ybaumy> ok and use what?
<smoser> nothign ?
<smoser> you said it doesnt do anything for you
<ybaumy> well there is the problem with proxy servers on subnets
<ybaumy> so i need a instance i can tell which proxy server the puppet.conf should hold
<ybaumy> proxy server=pupetmaster
<smoser> does foreman provide any endpoint that the node can query to learn about itself? other than the dhcp server ?
<ybaumy> but maybe you are right and should just script it msyself
<ybaumy> well there is remote ssh execution
<ybaumy> for example
<ybaumy> i could use ansible to register the host and setup puppet as well
<smoser> it seems like forman kind of needs a mechanism by which a node can query its parameters
<smoser> which is basically what a datasource is to cloud-init
<smoser> then the node would look and see what proxy settings it had and what it was supposed to do.
<ybaumy> thats what i thought yesterday
<smoser> other wise you're just deploying dozens of clones that differ only in hostname
<smoser> if you can provide hostnames you could manufacture them such that they indicate some information
<smoser> group1-name1
<ybaumy> there are clones as long as they havent contacted the puppetmaster
<smoser> and then in code inside key off 'group1' to know the hard coded proxy server.
<ybaumy> but you are right
<ybaumy> i should find another way to setup servername in puppet.conf
<ybaumy> this just doesnt work
<ybaumy> since cloud-init.cfg is not generated
<ybaumy> depending on the hostgroup
<smoser> you should convince foreman that they need a node-facing endpiont
<smoser> and then write a datasource for cloud-init to utilize that.
<ybaumy> it kind of lacks this feature i must saa
<ybaumy> say
<smoser> you need to have *some* id for a node
<ybaumy> https://theforeman.org/plugins/foreman_ansible/0.x/index.html i guess this what im trying now
<ybaumy> but in the end its the same problem
<ybaumy> i need a foreman_url
<ybaumy> which is the proxy puppetmaster in the end
<ybaumy> i have think about it
<ybaumy> i think i need a installtion network
<ybaumy> then its always the same
<ybaumy> server
<ybaumy> if i dont have a datasource
<ybaumy> there is no other way i can see
<smoser> robjo: https://hackmd.io/CwEwHARgbAzDDsBaArPOjhQMYEZEEMBOHQjQkeAUxgCYb9h8og==
<smoser> blackboxsw, rharper
<smoser> also. interested in feedback there.
<robjo> smoser: added some comments
<dojordan> @blackboxsw sorry to bump, but can you take a look when you get a chance? https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/334341
<blackboxsw> Good deal dojordan I have context on an azure deployment right now so I'll test it out and see if we can labd it today
<blackboxsw> 'land' rather
<dojordan> great, thanks! feel free to ping with questions
<dojordan> also note that the platform doesn't publicly support it yet so I've been testing the azure fabric changes in private test clusters. I have the logs from that if need be
<rharper> smoser: is there an easy way to modify the prposed-branche and merge-into for MP ?
<smoser> robjo: thanks. i responded theer.
<smoser> we can carry on here. also
<smoser> rharper: ?
<rharper> https://code.launchpad.net/~cloud-init-dev/cloud-init/+git/cloud-init/+merge/334733
<rharper> the targets need swapping
<smoser> :)
<smoser> rharper: so.. interestingly, i could 'resubmit' that proposal and swap the source and target
<smoser> which is not really re-submitting
<smoser> but i only saw that i could do that after i had already submitted another.
<smoser> so, commented there to that affect.
<rharper> ok, thanks
<rharper> I didn't know that either
<rharper> smoser: ntp discuss sounds fine, you've a few isc-dhcp where I think  you mean isc-ntp;
<robjo> more comment sin the hackmd.io document, but maye we should move the discussion here to be more interactive?
<smoser> ah. yeah. probably.
<smoser> robjo: yes that is fine.s orry
<robjo> To my last comment, my concern is primarily with the "custom image builders" and the expectation that is implied in "auto", i.e. it just works
<smoser> rharper: fixed rthose.
<robjo> If we use auto and there's no other mechanism then basically the cloud-init package on SUSE would have to supply a /etc/cloud/cloud.cfg.d/00-distro.cfg
<robjo> which of course the "cutom image builder" can mess up more easily than having something in the cloud-init code that handle "auto" in some deterministic way that is "distro default" compliant
<robjo> But maybe I am overthinking the problem
<smoser> robjo: in your scenario, wouldnt the user have likely had cloud-init installed from a package you maintain ?
<smoser> in which case it would have a setting not 'auto'.
<smoser> and would Depend on the right client
<robjo> Well it appears that lost of custom image builders decide to clobber the default cloud.cfg and thus I'd say the answer is no
<smoser> "I changed something and broke it"
<smoser> ?
<robjo> I see people reading teh doc, then put in their own cloud.cfg and set "ntp: enabled", done
<blackboxsw> right, I would expect this 'auto' problem only to show up on Sles/OpenSuse if the custom image builder is not installing stock sles cloud-init via yast, but rolling their own off of lp:cloud-init master
<rharper> wouldn't the distro object itself in code have a preference?  the system config is for overriding the built-in-code preference, no ?
<robjo> well the "in-code" preference only works if I have a "if distro.version == X:" block
<rharper> much like how we set locale by default, or other distro specific settings, default in the code, but allow system settings to override;  default net renderer is another like these where code has a default preference, and allowing system config to override
<smoser> 'auto' will generally do the right thing.  You're free to have 'auto' on Suse have different "right thing" than on Ubuntu.
<robjo> which means I have to read os-relese file and that gets me back to my testing issue :(
<smoser> But I think giving preference to already installed clients versus non-installed clients is important.
<smoser> we can fix your tests. i agree those tests are in need of re-factoring.
<smoser> blackboxsw or I can help with tests.
<robjo> agreed with the preference to installed clients, and maybe that in the end is the solution
<rharper> I'm not sure what's wrong with reading os-release?  if SLES vs OpenSuSE have different default clients for ntp; that can be accomodated in the distro classes, right?
<smoser> it would seem quite reasonable for the 'distro' object that is instantiated to "know" more about the speficis than it does.
 * blackboxsw loves mocks, yeah if we want some simple targeted mock ups or unit tests on a agreed upon direction, I'm game for helping out
<robjo> it's SLES vs. SLES, not SLES vs openSUSE
<smoser> :)
<rharper> ah
<robjo> SLES 12 -> ntp, SLES 15 -> chrony
<robjo> so sles.py gets loaded and tere I have to make a decision
<rharper> yes;  we're in similar situation Xenial will carry one preference, where as bionic will likely use something different
<smoser> i think reading os-release is fine.
<rharper> even in the presence of a default installed client (timesyncd in this case)
<smoser> in Ubuntu i'mi fine with carrying a package patch to maintain old behavior even if upstream cloud-init changes..
<rharper> I would thnk the distro object probably already has the system_info structure which has that detail any how, right ?
<smoser> and if someone builds their own package, or starts wiping out files installed by cloud-init or any other package when they build their image....
<smoser> i dont really see how that is my problem :)
<robjo> Ican build teh package such that on SLES 12 cloud-init delivers /etc/cloud/cloud.cfg.d/00-distro.cfg with system_info['ntp_client'] = ntp and on SLES 15 that's set to chrony, and that's fair, just pointing out that that was not necessary previously and is also reasonably easy to break
<smoser> make the change in /etc/cloud/cloud.cfg
<smoser> and if the user changes it or blows that file away, then it will default to auto
<smoser> and auto will do the right thing
<smoser> even including looking at /etc/os-release
<smoser> but pleease preferring installed clients over non-installed clients.
<robjo> rharper: system_info does not have distro version
<rharper> might not include release number
<rharper> platform.linux_distribution()
<rharper> ('Ubuntu', '16.04', 'xenial')
<rharper>  
<rharper> that seems pretty specific
<robjo> worse for me openSUSE and SLES have the same data, which is why handling flavor and distro on SLES/openSUSE is a three way ball juggling exercise ;)
<robjo> ah I was looking down the path of determining "flavor" and there we do not look at that information.
<robjo> So yes, platform.linux_distribution() would do the trick
<blackboxsw> smoser: rharper    hrm, so kvm-wise, I've used mount-image-callback to install a new cloud-init deb in the cloudimage I downloaded. Though it seems that the generator hasn't run because cloud-init is no longer configured as a listed systemd dependency in xenial in this image
<blackboxsw> should I have done something additional (beyond the mchroot dpkg -i new-cloud-init.deb)?
<rharper> no
<rharper> it's never listed as a dep
<rharper> all generators in /lib/systemd/generators/* are invoked, there I think is a link to the cloud-init one
<rharper> /lib/systemd/system-generators/cloud-init-generator
<rharper> systemd runs that, which in turn checks for the kernel parameters or disabled file, and if so, leaves things off; otherwise it'll run and import ds-identify stuff, which will result in cloud-init getting enabled via a dynamic link in multi-user.target.wants/
<rharper> in /run
<rharper>  /run/systemd/generator-early I think
<rharper> bbiab
 * rharper relocates
<smoser> blackboxsw: updated https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/333513
<blackboxsw> grabbing
<smoser> if you agree with my suggested change then we can pull
<blackboxsw> yeah that makes sense smoser will apply and land.
<blackboxsw> just ran into (and am fixing with a trivial) https://bugs.launchpad.net/cloud-init/+bug/1736600
<ubot5> Launchpad bug 1736600 in cloud-init "cloud-init modules -h documents unsupported --mode init" [Low,Triaged]
<blackboxsw> just finished testing ovf with msaika's changes
<blackboxsw> looks to work without impacted standard ovf behavior with one curiousity.
<blackboxsw> if I rm -rf /var/log/cloud* /var/lib/cloud; sudo reboot    on the kvm machine with the updated locally installed deb package cloud-init doesn't run by default, I had to cloud-init init --local; cloud-init init, cloud-init modules --mode config ... --mode final .
<rharper> blackboxsw: did your ovf test use no-cloud ?
<rharper> if so, you nuked your /var/lib/cloud/seed/no-cloud
<rharper> which means ds-identify won't enable cloud-init
<blackboxsw> hrm yes I think I did... and I nuked seed (which contained nothing)
<rharper> blackboxsw: aren't you working on a cloud-init clean to DTRT so we don't have to rm -rf /var/lib/cloud*
<rharper> but
<smoser> ugh
 * smoser sees
<smoser>  tests/cloud_tests/testcases/modules/salt_minion.py
<rharper> it's presense triggers cloud-init
<blackboxsw> ahhh
<smoser> and says what is going on there ?
<blackboxsw> oops
<blackboxsw> right
<smoser> oh. cloud_tests. silly me
<blackboxsw> and rharper yeah that branch cloud-init clean is minutes from landing
<blackboxsw> :)
<rharper> =)
<blackboxsw> then I don't have to worry about shooting myself in the foot
<rharper> I know smoser really wants to relocate seed to somewhere else outside of /var/lib/cloud
 * rharper relocates home
<rharper> bbiab
<blackboxsw> msaika is also using the seed directory for vmware's markefiles too
<blackboxsw> and that branch is emminent
<blackboxsw> https://code.launchpad.net/~msaikia/cloud-init/+git/cloud-init/+merge/330105
<smoser> ugh. markerfiles shoud go in /var/lib/cloud/data, no?
<smoser> was there  a reason for seed?
<smoser> powersj: or anyone who wanted
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/334780
<blackboxsw> strike that comment about /var/lib/cloud/seed ... looks like the marker files are just in /var/lib/cloud/<markerfile>
<powersj> smoser: thx I'll play with that next
<blackboxsw> smoser: still time to comment that. She's around all week (per pvt msg).
<blackboxsw> smoser: I'll suggest the path change
<blackboxsw> markefiles were  originally under / and I wanted something that cloud-init clean would still blow away
<smoser> oh i see.
<smoser> i have to read it to know what mnarker files are for. i forget in thatcontext
<smoser> i also approved https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330115
<smoser> blackboxsw: did you aggree/disagree with my comment on 333513 ?
 * blackboxsw wonders if we should  suggest the same makerpath changes for https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/334341 dojordan is doing something kinda similary with markerfiles for azure
<blackboxsw> smoser: right yeah vmware is using them user-provided customization scripts which run pre-and post-cloud-init network configuration in vmware's get_data() method
 * smoser is behind on reviews
<smoser> and hasto run now.
<dojordan> FWIW the marker file in my PR won't exist once the actual provisioning is completed. therefore by the time you can login and run clean, the file will be gone
#cloud-init 2017-12-06
 * powersj will fix integration test failures shortly
<Raboo> smoser, dpb1 just wanted to update you on the debugging. I haven't had time to turn on debug for cloud-init on trusty yet. But I have learned a bit since, it only gets stuck when I have a root ext4 and a data xfs partition. cloud-init works when I got one big root partition for instance. And i've been able to replicate this on a virtual machine. So it's going to be easier to debug.
<smoser> hum.
<smoser> i suspect that it is not really cloud-init that is hanging, but a dependency (local-filesystems) or something that is not being met
<rharper> Raboo: your image has a ext4 rootfs and an xfs partition ?  what's the partition layout?
<rharper> I just saw this the other day w.r.t xfs filesystem being resized, https://www.spinics.net/lists/linux-xfs/msg13555.html
<Raboo> rharper well what I do is i partition the disk, and dd the cloud image into the ext4 partition, grow the image into the partition, mount it, fix networking, fstab, grub and some stuff. Then reboot and could-init bootstraps the node with chef and stuff
<Raboo> rharper something like this https://hastebin.com/otowajicox.bash
<rharper> fancy! using hastebin
<Raboo> yeah smoser sent me a hastebin link the other day, so i figured i'ed try it
<rharper> nice
<Raboo> it's yours?
<rharper> no
<rharper> smoser had been using it since one can get raw output without "logging" in , like we have to the ubuntu one;
<rharper> now I just need pastebinit to have hastebin
<Raboo> it appears to have console tools
<rharper> Raboo: so, putting ext4 on the first partition should mean that resizefs on root will get ignored; as it isn;t the last one;   but maybe there is something going on that we missed w.r.t growpart/resizefs
<rharper> Raboo: did you have a cloud-init.log from the failed boot/hang ?
<Raboo> rharper i will change this config to DEBUG on line 33 https://hastebin.com/odahevojix
<Raboo> to see if I get more info, just not right now
<rharper> sure
<Raboo> but exact same partition layout works in xenial
<Raboo> and now I know that it got to do with the parition layout.
<rharper> where was the failing case if not xenial ?
<Raboo> just after ssh keys generation it gets stuck
<rharper> but what distro release?
<Raboo> ubuntu trusty
<rharper> ok
<smoser> rharper, blackboxsw https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1731868
<ubot5> Launchpad bug 1731868 in cloud-init (Ubuntu) "vmware reported as 'EC2', 'None'" [Undecided,New]
<smoser> that one..
<smoser> with comment from sabdfl
<blackboxsw> yep let's get a fix for that
<blackboxsw> eek
<smoser> we have no way there to determine "this is vmware" really. :-(
<rharper> now, where to get a Vsphere setup ?
 * blackboxsw thinks I might have one.
<blackboxsw> will dig
<rharper> well, vmware does similar things as kvm does to identify itself
<smoser> you can determine vmware, yes
<smoser> you can't determine easily that there is an ovf without a mount
<rharper> right
<smoser> from dsidentify
<smoser>  http://paste.ubuntu.com/26126298/
<rharper> man, pylint on tests has lots of stuff; gonna be a big one =)
<rharper> https://github.com/PyCQA/pylint/issues/1498
<powersj> smoser: do you want me to enable pylint for all of tests or start with integration tests?
<smoser> powersj: all would be good.
<smoser> did you see what was in  my tree ?
<smoser> here
<smoser> http://paste.ubuntu.com/26126566/
<powersj> ah no
<powersj> smoser: do we need tools/hacking.py?
<powersj> still using pep8 :\
<rharper> powersj: I'm working on smoser's pylint diff right now
<powersj> rharper: oh are you going through fixing things?
<rharper> yes
<powersj> then I shall stop
<powersj> thx :D
<rharper> np
<rharper> smoser helpfully handed the patch to you as well =)
<dojordan> @blackboxsw, sorry to bother but any update on my PR?
<blackboxsw> dojordan: I'll push comments here in a couple mins (was camping on them until I tested). Admitedly, I ratholed on another bug that came in.
<blackboxsw> "rat-holed" even a word? anyway, will wrap up your review now
<dojordan> haha not sure if its a word but sounds good
<blackboxsw> dojordan: couple comments added
<blackboxsw> back in a few
<blackboxsw> to wrap up the review
<dojordan> thanks, ill get started on those
<rharper> oi! pylint 1.7.1 doesn't like our add_patch, finally worked out how to ignore member checks with m_* which is how we designate  mock patch members
<smoser> ?
<smoser> doesnt like add_patch ?
<smoser> hm..
<rharper> yeah, says m_foo is non-member
<rharper> because add_patch does a set_attr
<rharper> smoser: so pylint can't know that the string inside add_patch() is going to get set that way (not statically I guess)
<smoser> rharper: https://github.com/PyCQA/pylint/issues/697
<rharper> related, but not the same
<rharper> https://github.com/PyCQA/pylint/issues/1498
<rharper> we actually get the E1101
<rharper> tests/unittests/test_datasource/test_ovf.py:191: [E1101(no-member), TestTransportIso9660.test_mount_cb_called_require_iso_false] Instance of 'TestTransportIso9660' has no 'm_mount_cb' member
<rharper> but it's reasonable to mark m_.* and mock_.* as generated-members; that's what our  CiTestCase.add_patch() does
<smoser> hm..
<smoser> rharper: well, the end result in thelink i provided is
<smoser> Closing this. I think it is not worth investing effort into something relying heavily on patching things and magic, as mock is doing it. if anyone wants to spend some effort fixing this, you might have a look at how astroid brain modules are implemented (https://github.com/PyCQA/astroid/tree/master/astroid/brain)
<rharper> I saw those modules too
<rharper> I mean, seems reasonable to do what I did, the lint is catching stuff (real issues)
<smoser> basically, it is "dont run pylint on code with mock"
<rharper> it's generally clean with mock
<smoser> ?
<rharper> it's the dynamic set_attr for our patch that is gitting hit
<smoser> i'm confused.
<rharper> the bulk of the errors, came out from our use of self.add_patch() in CiTestCase subclsses which specify a mock.patch dynamically in the setUp() method
<rharper> adding m_.*, mock_.* to .pylintrc to tell pylint to recognize any m_* members as 'generated members of a class' means it won't error E1101 on all of those mock'ed methods
<rharper> http://paste.ubuntu.com/26127428/
<smoser> http://paste.ubuntu.com/26127442/
<smoser> that is my run.
<rharper> write, grep no-member;
<rharper> those m_* go away if you update your .pylintrc
<rharper> urg, python2.7 ConfigParser has readfp, that's deprecated in py3; which uses read_file (but that's not in py27)
<smoser> ugh
<smoser> blackboxsw: fyi, don't use git merge --squash.
<smoser> unless you know of an arg that says to keep the author
<blackboxsw> whoa really?
 * blackboxsw checks logs of what I just landed
<smoser> as i stole credit for powers' commit 47016791
<blackboxsw> ahh smoser yeah I generally do a --squash then a commit --amend --author afterward before pushing (though I forgot 1 time)
<blackboxsw> d90318b21d0379e24337bcb92a0a90ebfa359c35 yeah here I stole robjo's commit ownership by accident :/
<blackboxsw> forgot the post-commit ammend that I usually do
<smoser> need a lander
<smoser> stupid humans
<smoser> incapable of simple tasks
<blackboxsw> yes. push button landing
<blackboxsw> or tarmac for git really.
<rharper> another pylint annoyance;  referencing class properties (which may trigger code) but with no assignment makes it unhappy, but if we d foo = class.prop; then flake says, you didn't do anything with foo
<blackboxsw> hrm can't get vsphere login working at the moment
<smoser> weee!
<smoser> blackboxsw, rharper hangout ? /wanted to catch up where we are.
<rharper> y
<blackboxsw> sure
<rharper> smoser: here's where I'm at on the linting branch, http://paste.ubuntu.com/26127753/
<rharper> that passes just about everything; save some things in tools/
<rharper> needs a flake fiix up for one two locations for #pylint disable I did
<smoser> http://paste.ubuntu.com/26128142/
<smoser> blackboxsw: ^
<smoser> and dpb1
<smoser> i have to walk out th edoor though. i'll be back in later and can polke it at it some more.
<blackboxsw> smoser: will give it some runs on vkm
<rharper> smoser: https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/334868
<rharper> that's the pylint merge, all clear here
<smoser> i'll try this in a bit. http://paste.ubuntu.com/26128501/
<smoser> rharper: and will read your mp in a bit
<smoser> dinner and such now
<smoser> thanks rrh
<rharper> np
<blackboxsw> smoser: your paste approach worked great on my local OVF kvm setup
<blackboxsw> I'm crazy. nevermind. tweaking the script a bit
<blackboxsw> still got disabled from ds-identify. checking things out now
<blackboxsw> ahh minor tweak needed
<blackboxsw> http://paste.ubuntu.com/26128864/
<blackboxsw> needed to define DI_BLKID_OUTPUT
#cloud-init 2017-12-07
<blackboxsw> smoser: pushed WIP branch. started adding unit tests https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/fix-ds-identify-vmware-ovf
<blackboxsw> ->dinner
<dojordan> hey folks, any idea about this pylint error? Cell variable section_name defined in loop
<smoser> dojordan: more context ?
<dojordan> for section_name in ('PlatformSettingsSection', 'PlatformSettings', 'PreprovisionedVm'):
<dojordan> sections = find_child(section, lambda n: n.localName == section_name)
<smoser> do you use section_name outside the loop ?
<dojordan> nope
<dojordan> ah think i may have found it
<dojordan> i was using sections outside the loop
<smoser> maybe chttps://stackoverflow.com/questions/25314547/cell-var-from-loop-warning-from-pylint
<smoser> some context there.
<smoser> if you run with '--output-format=parseable' you get the code nubmer (W0640)
<dojordan> haha i was on that page. but good idea
<smoser> the answer there is probably as good as i would be able to give you and it'd take me a minute to fully grock it too
<smoser> :)
<catmando> hey all
<catmando> question: i want to disable datasources completely: i've set ```datasource_list: ['None']``` but when an instance comes up it still takes a _very_ long time looking for metadata (e.g.: ``` Calling 'http://172.16.8.254/latest/meta-data/instance-id```)
<sinanp> Hi, how can I add peerdns=no to the ifcfg configuration file?
<ybaumy> smoser: remember we talked about datasources in foreman. there is a rpm ruby gem userdata for foreman
<ybaumy> smoser: its nocloud datasource
<ybaumy> smoser: but only for userdata
<ybaumy> smoser: i couldnt test it yet completly because i have problems with my templates
<sinanp> Shouldn't this create an user account including the provided password and public key? https://pastebin.com/J7J6cZ9X
<sinanp> It doesn't create any user...
<sinanp> smoser: Thanks! Have it working now. Any clue how to have peerdns=no in the ifcfg-eth0 file?
<smoser> sinanp: probably not able to.
<sinanp> Ah ok :) Thanks
<sinanp> And another problem :) Shouldn't "disable_root: 1" disable root login over SSH? It doesn't disable it...
<smoser> sinanp: it disables it with the keys that you gave.
<smoser> rharper: i didnt realize that we were *not* running pylint for python2 and python3
<smoser> we do that in curtin
<smoser> hm..
<rharper> right, we have a py27-pylint and py3-pylint in curtin
<rharper> vs. just 'pylint' in cloud-init
<rharper> I thought it was only python3 based due to the default env; but I didn't verify
<rharper> do you want me to fix up top to use py27-/py3 pylints ?
<rharper> with the exception of no py27 on cloud-tests ?
<smoser> ipm not even sure.
<rharper> and what do you want to do about tools ?
<smoser> which it woudl pick.
<smoser> i'll fix tools it cant be that hard.
<smoser> i'm just tsarted looking at your branch.
<smoser> i'll see.
<rharper> ok
<rharper> one was optparse
<rharper> I didn't really want to rewrite that to argparse; seemed hardly worth it, not knowing how often that tool is used
<rharper> the other one about a missing default parametter to the mime-type finder wasn't very obvious where we need to set a default value
<smoser> harlowja: tools/hacking.py ...
<smoser> why does that exist ?
<harlowja> sup dawg
<harlowja> let me see
<harlowja> i think it was before time
<smoser> hm..
<harlowja> before https://pypi.python.org/pypi/hacking came to existence
<harlowja> https://pypi.python.org/pypi/hacking was born
<harlowja> so that file can die
<harlowja> lol
<smoser> right
<smoser> and we have hacking in our flake8
<smoser> so we get it.
<smoser> thanks
<harlowja> yup
<harlowja> kill its
<smoser> rharper: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+ref/fix/cloud-test-add-pylint-and-fix
<rharper> whoa
<rharper> what's the trailing slash do ?
<smoser> i dont know. :)
<rharper> hehe
<smoser> i thikn it would then go searching for modules in that directory
<rharper> freaky
<smoser> versus treating that as a module
<smoser> try it though i'm pretty sure you can reproduce
<rharper> I'm sure I can, just kinda boggles my mind
<smoser> well, anyway, with those changes tip-pylint works also
<smoser> andd tools/ is clean
<smoser> or 'tools'' rather
<rharper> hehe
<rharper> pulling your branch now
<smoser> and drop the metadata bit
<smoser> i did not intend to get that to you
<smoser> it was in my working dir ithink
<rharper> oh, did you drop that already ?
<rharper> in your branch ? or do I still need to ?
<rharper> powersj was asking about that
<powersj> yeah it was in your diff yesterday
<smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/334780
<smoser> that was how i found out that integration tests were busted
<smoser> i was going to run integration test on m y changes
<smoser> and found it broken
<smoser> and t hen wentn off to try to get pylint going
<smoser> and then handed that off to you
<powersj> what a chain of events :)
<smoser> so i'd say drop that commit
<rharper> right, I just meant did you drop that delta out of the pylint branch in the thing you just  proposed to merge into my pylint branch ?
<smoser> and we can pull it in separately.
<smoser> just git rebase -i master
<smoser> and delete it
<smoser> then after your merge we'llg rab mine
<rharper> well, it's not a separate commit
<rharper> so I was going to find out which bits were unneeded and diff them out in a another commit and squash during rebase
<smoser> oh its not ?
<rharper> it's the "what bits do I need to drop" that I'm asking for
<smoser> :-(
<rharper> I don't know
<rharper> that's what I've been asking
<rharper> I'm generally not aware of which files were the "metadata" changes that powersj  and smoser are speaking of
<smoser> well, its in that mp there. probably will patch -R off if you wanted to
<rharper> ok
<rharper> that works
 * rharper patch -Rs
<smoser> git show 612f9896330e | patch -p1 -R
<smoser> one hunk failed
<smoser> i'll get it
<rharper> http://paste.ubuntu.com/26134634/
<rharper> I did: git diff -r master <path to both files>
<rharper> and then I can patch -p1 -R  <that diff> and it applies cleanly to revert those changes
<rharper> look sane ?
<dpb1> nice blog post bbsw: https://twitter.com/davidpbritton/status/938160996136271872
<smoser> i think so
<rharper> k, pushing again to my branch
<rharper> I commited the revert and squashed into my first commit
<rharper> ok
<rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/334868
<rharper> that looks better
<smoser> mention removal of tools/hacking.py (obsolete)
<smoser> and then i thikn happy
<smoser> assuming c-i is
<rharper> man, really don';t like cloud-images not having apt sources update data in them
<rharper> smoser: did we file a bug on that?
<rharper> apt install foo breaks until you apt update first
<rharper> on daily images , no less
<rharper> you noticed that ?
<smoser> ?
<rharper> I just launched daily bionic, sudo apt install tree
<rharper> fails, I have to apt update first
<smoser> well, youc ant really do anythign about bionic
<smoser> development release is basiclaly guaranteed to have out of date apt
<rharper> ok, on arful
<rharper> on xenial
<rharper> on zesty
<rharper> it's the new thing
<rharper> same in lxd images too
<smoser> i think that released iamges should have the frozen release pocket up to date in the image.
<rharper> launch a new one, lxc launch ubuntu-daily:xenial x1
<rharper> pretty sure it was broken there too
<rharper> smoser: I didn't squash your three commits; I'll do that and add their changes to the commit message, wait for ci and then push to master
<rharper> Tridactyl (Vimperator/Pentadactyl's successor).
<rharper> smoser: surely you've tried this in ff57+;  i really wanted to switch to ff, but the input-box editor is missing (using wasavi on chrome for that now)
<smoser> oh ther is one.
<smoser> i use it.
<smoser> let me find
<smoser> i have tridactlyl installed not really using it though
<smoser> https://github.com/jlebon/textern
<smoser> and 'xvim' is the editor i invoke
<smoser>  https://gist.github.com/smoser/1ee3f2b223717625c5f4
<smoser> ie, my editor configured is ['xvim', 'block']
<rharper> nice
<rharper> does that get your input box editing ?
<smoser> text area, i hit 'ctrl-I'
<rharper> nice!
<smoser> like i did before and gnome-terminal window pop up
<rharper> I couldn't get textern to work when I tried it a few weeks ago
<rharper> but I used gvim
<rharper> so maybe xvim will do the trick
<smoser> yeah i dont liek gvim
<smoser> gvim will work fine
<rharper> huh, not for my system, probably a win manager thingy
<smoser> i just dont like the mouse i think always messed me up
<smoser> on paste and things.
<rharper> got some xmonad here
<smoser> oh. i dont know. it does work for me. i dont remembered every thing i did. i just followed the doc it had
<smoser> rharper: so it looks like, on xenial the images have these
<smoser> oh man
<smoser> looks like haste is dead
<smoser> http://paste.ubuntu.com/26134814/
<smoser> so in /var/lib/apt/lists/ on xenial daily we have lists present for
<smoser>  xenial-updates mmain restricted
<smoser>  xenial main restricted
<smoser>  security main restricted
<smoser> no universe or multiverse or backports
<smoser> which are all going to be pulled when you type 'apt-get update'
<smoser> i've had discussions with rbasak about this before
<smoser> and i think the right thing to do would be to include only the "frozen" data
<smoser> ie,
<smoser>  xenial main restricted universe multiverse
<smoser> removing those files from the iamge just causes the user to have to download them and get the exact same thing as they could have been given in the image.
<smoser> for xenial-updates and security, it somewhat makes sense to have those out of date
<smoser> er... somewhat makes sense to not have those at all
<smoser> as they're basically guaranteed to be out of date.
<smoser> i think what we have now is probably just  happenstance or regression
<smoser> but one could probably argue that its best to have nothing in there
<smoser> which will force a user to 'apt-get update'
<smoser> because if they had only the 'frozen' bits, then they could 'apt-get install foo' and get a vulnerable 'foo'
<smoser> but that comes at a big cost
<smoser> on mirrors and data download and cpu decompression that.
<rharper> yeah
<rharper> well, it feels different, for better or for worse
<smoser> it may well have changed.
<smoser> i didnt check that
<rharper> daily images, for example are unlikely to get a security vulnerable one if they included up-to-date cached files
<smoser> and it feels wrong to me to have 'main' and 'universe' there but no multiverse or backports
<rharper> I think that's my gripe, if I'm already downloading a daily
<rharper> it shouldn't be out of date
<smoser> if the user is just going to get them on first 'apt-get update'
<smoser> (and gets them by default on an install)
<smoser> so.
<smoser> your tests failed in centos6
<smoser> the unittest there is to old to have assertRaisesRegex
<rharper> how does tox not find that ?
<rharper> oh, have to run tox *in* centos6
<rharper> that's py26, right ?
<smoser> yeah. and with system packages. yeah.
<rharper> urg
<smoser> i think easiest thing tpo do is just use the 'ignore' comment
<rharper> no pip unittest magic ?
<rharper> yeah
<smoser> oh. acutally
<smoser> oh.
<smoser> i think that unittest is builtin to python
<smoser> so we're using the one from python2.6 installed there.
<rharper> right, was wondering if we could somehow upgrade it; but I suspect not as easily as # pylint: disable=FOO
<smoser> well, helpers could patch smpething in
<smoser> oh. actually...
<smoser> http://paste.ubuntu.com/26134987/
<smoser> rharper: ^
<smoser> try that
<rharper> oh, heh
<rharper> what happens is that pylint will mark that line as using depricated method
<rharper> then we hae to mark #pylint: disabled=W1505 there
<rharper> now, we have two call sites
<rharper> we could add that and the disable, and then we'd be free to always use Regex
<rharper> thoughts?
<smoser> ?
<smoser> cant we just monkey patch it like that and use the new name ?
<rharper> we can, but we'll still get the warning, now on cloudinit/tests/helpers.py
<rharper> but I think that's better to do it in one place
<smoser> sure. but then its one place to coment and it has a text coment saying why
<rharper> prolly should move the parse_and_read() method to helpers too
<rharper> yes
<rharper> I'll do that as well for the ConfigParser
<smoser> ok. i got to go afk for a bit. biab
<rharper> k
<dojordan> @blackboxsw, can you take a look? I responded to the comments which i didn't fix, the rest are fixed
<dojordan> https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/334341
<smoser> rharper: http://paste.ubuntu.com/26135182/
<smoser> i think that atually works. i had '2' in the wrong place :)
<rharper> ah
<rharper> yeah, was just debugging that
<smoser> i think pylint wont even complain.
<rharper> so, how does pylint get executed in py27 ?
<rharper> on curtin we have those py27-pylint and p3-pylint targets which set base-python = ...
<smoser> it doesnt
<rharper> hrm
<rharper> do we not care then ?
<smoser> we can absolutely follow like curtin did/does
<smoser> and have explicit ones
<smoser> the thing i dont like about it is that... slow
<rharper> yeah
<rharper> well, in cloud-init we run .pylintrc with jobs=4
<rharper> so it does run in parallel
<smoser> still slow
<rharper> which is pretty nice
<rharper> yes
<smoser> i'm not opposed to it.
<smoser> but i dont think we *have* to do it right now
<smoser> i like to give future smoser some reason to think past smoser was an idiot
<smoser> tox -e pylint on diglett takes 1m28 seconds
<smoser> bso adding another of those just seems costly.
<smoser> without tools/ and tests/ it was only 30s ish
<rharper> what's the total tox run ?
<rharper> boo
<smoser> 2m26
<smoser> 1m30 of that to tox
<smoser> er.. to pyklint
<rharper> man, can't win;
<rharper> with the centos fix, tox -e pylint fails
<smoser> i think just for now do pylint on python3 only.
<smoser> ?
<rharper> even though unittest2.TestCase has both attributes
<rharper> if I comment out your patch
<rharper> it passes
<rharper> smoser: well, we can demote pylint to something like tip-pylint
<rharper> present, but not always run ?
<rharper> I think we don't lint on package build anyhow, right ?
<rharper> do we tox on package build ?
<rharper> or just unittest
<smoser> well we want to run pylint . i am fine with 1x that cost
<smoser> not 2x
<smoser> and yeah, we can add a py2-pylint that only runs on manual
<smoser> but i dont think we really need to.
<rharper> I still can't figure out why it's failing with your cent6 fix
<rharper> it doesn't make any sense; the fix-up path isn't taken at all
<rharper> https://github.com/PyCQA/pylint/issues/1653
<dojordan> :q!
<dojordan> (whoops)
<rharper> smoser: so I'd like to drop the cent6 2.6 pylint fix unless you can figure out how to fix that;  and just skip running pylint on the package builds;
<smoser> oh. its not actually pylint that complained
<smoser> it was a unit test complaining
<smoser> right?
<rharper> ah, lemme see
<rharper> yeah
<rharper>     self.assertRaisesRegex(util.ProcessExecutionError
<rharper>  /test_util.py", line 698, in test_subp_warn_missing_shebang
<rharper>     self.assertRaisesRegex(util.ProcessExecutionError
<rharper> the fix you had will work but pylint blows up on it
<rharper> and I can't figure it out
<rharper> so, I guess we leave it as Regexp and mark them disables
<rharper> what a pita
<smoser> http://paste.ubuntu.com/26135444/
<smoser> rharper: ignore the basepython thing in that patch
<smoser> and that runs for me (make unittest) in centos6
<rharper> wow
<rharper> how did you figure that one out?
<smoser> it was juts comp;laining that the thing was an integer
<rharper> but I don't like not using assertRaises*
<smoser> it still uses it.
<smoser> just the context_manager.exception.code
<smoser> is wht differs
<rharper> oh, my failure is on ubuntu
<rharper> with that the older assertRaisesRegexp fix
<rharper> tox -e pylint on my ubuntu system fails
<rharper> saying we're using deprecated methods
<rharper> Regex
<rharper> which 1) isn't deprecated  2)  hasn't been patched
<rharper> I hadn't yet run under centos again
<smoser> https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRaisesRegex
<rharper> was still trying to get a clean tox run on ubuntu first and added your older unittest2 fix for centos to test in a bit but; it won't pass pylint for me
<smoser> it may not be deprecated, but it deffinintely not renamed
<rharper> https://github.com/PyCQA/pylint/issues/1653
<smoser> that is odd.
<smoser> ugh
<rharper> yeah
<rharper> exactly
<smoser> rharper: http://paste.ubuntu.com/26135510/
<smoser> that i thikn is happy everywhre
<smoser> stupid
<smoser> that is on top of 5335e1632ddabdbad9ffb011720929ae4cd8ebe3
<smoser> oh boy
<smoser> hten i realize that Makefile (that runs 'nosetests' only runs on tests/ not on cloudinit)
<smoser> gah
<rharper> man, what a rat hole
<smoser> misplaced 2 again
<rharper> wow, what is that tricky bit ?
<smoser> http://paste.ubuntu.com/26135531/
<rharper> how does that help ?
<smoser> that worked in 'tox '
<rharper> I fail to see what tox found the first time
<rharper> so the "fix" is just as mystical to me too
<smoser> and it works in 'make unittest' in centos6
<smoser> the '_tricky' part there is complete crap
<rharper> k
<rharper> yeah
<smoser> what i think is happening is that pylint is looking for Regexp to go complaining
<smoser> and realiaes that the thing we're calling is == that other thing
<rharper> yeah
<smoser> but then now its not strictly equal
<rharper> lemme get both those in my branch now
<rharper> right
<smoser> all problems solved by anbother level of indirection :)
<smoser> oh. i was wrong.  'make unittest' *does* run with cloudinit
<smoser> so thats good.
<smoser> i thikn we're set here.
<smoser> you get that, push it, and  maybe we'll be happy!
<smoser> maybe
<smoser> and on that note, /me is out
<rharper> ok
<rharper> thanks
#cloud-init 2017-12-08
<smoser> rharper: do you reproduce https://jenkins.ubuntu.com/server/job/cloud-init-ci/598/console ?
<smoser> ok. that was on a different revision. you must have updated and p ushed. and it looks like run 599 is going now.
<smoser> well, i think this is good at this point.
<smoser> thanks.
<smoser> blackboxsw: for tomororw, need to update
<smoser>  https://bugs.launchpad.net/ubuntu/artful/+source/cloud-init/+bug/1733653
<ubot5> Launchpad bug 1733653 in cloud-init (Ubuntu Artful) "sru cloud-init (17.1-27-geb292c18) update to 17.1.46-g7acc9e86-0ubuntu1" [Medium,Fix committed]
<smoser> and /me to bed.
<sinanp> I have "disable_root: 1" in my cloud.cfg but I am still able to log in with root via SSH. Am I missing something?
<blackboxsw> I see disable_root: true in my /etc/cloud/cloud.cfg over here sinanp
<blackboxsw> I would have thought 1 should work too. but checking
<sinanp> I tried it with 1, true and True.
<sinanp> I see that /root/.ssh/authorized_keys is prefixed with: no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"username\" rather than the user \"root\".';echo;sleep 10"
<blackboxsw> as scott mentioned yesterday I think. that login as root only occurs for the keys you've specified in #cloud-config
<blackboxsw> so when sshing using other unauthorized keys, we wouldn't expect to see the message
<blackboxsw> hrm, actually I see that message when I log into my ec2 instance no matter what key I'm using
<blackboxsw> and I see that prefix in /root/.ssh/authorized_keys as well
<blackboxsw> thx for the bug https://bugs.launchpad.net/cloud-init/+bug/1737130
<ubot5> Launchpad bug 1737130 in cloud-init "disable_root: 1 but root can still log in" [Undecided,New]
<ybaumy> nayone here
<ybaumy> anyone
<ybaumy> i need help debugging script
<ybaumy> http://pastebin.centos.org/457036/  (<unknown>): could not find expected ':' while scanning a simple key at line 47 column 1
<ybaumy> what does that mean i cant find any errors in that line
<ybaumy> ok fixed that other one put i still have a problem running a script from a datasource
<ybaumy> where is the error http://pastebin.centos.org/457096/
<ybaumy> i tried it with -[ command ] and without
<ybaumy> but the runcmd is completly ignored
<robjo> smoser: time for more ntp/chrony fun?
<robjo> Added new comment to https://hackmd.io/CwEwHARgbAzDDsBaArPOjhQMYEZEEMBOHQjQkeAUxgCYb9h8og==?view
<ybaumy> can i make runcmd run again when booted or do i have reinstall?
<rharper> ybaumy: there is a frequency=always
<rharper> lemme see the docs
<ybaumy> cloud-init single --name cc_ssh --frequency always
<ybaumy> found it
<ybaumy> lets try
<ybaumy> cannot be user with init
<ybaumy> used
<rharper> if you dump your scripts into /var/lib/cloud/data/scripts/per-boot
<rharper> they'll get run each boot
<rharper> err,  here /var/lib/cloud/scripts/per-boot
<ybaumy> ok
<rharper> also, i see, you can invoke 'cloud-init-per <frequency> cmd
<rharper> so maybe that'll work with your runcmd
<rharper> one example in the docs looks like this
<rharper> bootcmd:
<rharper>     - echo 192.168.1.130 us.archive.ubuntu.com > /etc/hosts
<rharper>     - [ cloud-init-per, once, mymkfs, mkfs, /dev/vdb ]
<ybaumy> looks good
<ybaumy> will install a new vm now
<ybaumy> "/usr/lib/python2.7/site-packages/cloudinit/config/cc_phone_home.py" IOError: [Errno 2] No such file or directory: '/etc/ssh/ssh_host_dsa_key.pub'
<ybaumy> you  should consider that there no dsa keys anymore
<ybaumy> how does phone_home accept https url
<ybaumy> ic the problem is foreman
<blackboxsw> smoser: can you check my commit message at https://code.launchpad.net/~msaikia/cloud-init/+git/cloud-init/+merge/330105
<blackboxsw> or rharper
<blackboxsw> I reworded it a bit (and structured for git log length)
<blackboxsw> I'm running tox on that branch locally and will push once either of you get a chance to spot-check the adapted commit message
<rharper> blackboxsw: commit msg content is fine;  on the web the last line gets wrapped after 'scripts' and then has a newline;  I think that's just the web format
<blackboxsw> rharper: good deal thx. yeah in vi I've set tw=74 to ensure it's formatted properly
<rharper> cool
<blackboxsw> gah. scripts carried over
<blackboxsw> dumb
<smoser> yall good ?
<blackboxsw> smoser: about, what's two characters between friends (in the body of the commit)
<blackboxsw> I don't really want to --force but landed maitreyee's branch with a length at 76 chars on the 2nd to last line
<smoser> https://hastebin.com/raw/abizeqaviw
<smoser> blackboxsw: ^ that is 'go-test-proposed' for now.
<smoser> i jsut wanted to have that. so i can run that on diglett if you'd like and post to the bug.
<blackboxsw> saving it off locally and on our SRU trello card
<blackboxsw> and in ubuntu-sru/manual/nocloud-kvm log
<smoser> oh. yeah. ubuntu-sru.
<smoser> good idea
<blackboxsw> that'd be great smoser thanks if you can run it. I'll push the initial script reference to ubuntu-sru in the right file
<blackboxsw> I added a
<blackboxsw> I added a manual subdir there
<blackboxsw> just to collect logs we've manually attached to our SRU process bugs
<rharper> smoser: we added tox 1.7.1, in curtin we did 1.7.4 , was there a reason not to run 1.7.4 in cloud-init tox ?
<smoser> blackboxsw: your paste looks good to my ovf. i'll add that and then push.
<smoser> thanks!
<smoser> hm..
<smoser> oh. i bet that i had stale .tox/pylint-tip
<smoser> and that was teh version in there.
<smoser> i'm quite happy to move it to 1.7.4
<smoser> sorry
<rharper> np
<rharper> just noticed that when I committed the curtin change
<blackboxsw> pushed everything up to  ubuntu-sru
<blackboxsw> only remains getting smoser's nocloud-kvm runs into manual/nocloud-kvm-sru-17.1.46.txt
 * blackboxsw steps away for a few
<smoser> blackboxsw: xenial just finished.
<smoser> blackboxsw: so i have a xenial-nocloud-kvm.log and xenial-nocloud-kvm-results.d.tar.xz
<smoser> where do you want me to put those ?
<blackboxsw> smoser: https://github.com/cloud-init/ubuntu-sru/blob/master/manual/nocloud-kvm-sru-17.1.46.txt
<blackboxsw> and please attach to bug  https://bugs.launchpad.net/ubuntu/artful/+source/cloud-init/+bug/1733653
<ubot5> Launchpad bug 1733653 in cloud-init (Ubuntu Artful) "sru cloud-init (17.1-27-geb292c18) update to 17.1.46-g7acc9e86-0ubuntu1" [Medium,Fix committed]
<blackboxsw> then all we are waiting on is MAAS and solutions validation. and aging on pending-sru for 1 more day
<smoser> blackboxsw: ok. i'll get those. it is running zesty and took xenial 16 minutes-ish. so probably < 30 minutes till done.
<blackboxsw> saw the push of your content smoser thanks
<smoser> uploading now
<paulmey> Hi all, I was hoping to sneak https://bugs.launchpad.net/cloud-init/+bug/1722959 into the upcoming release
<ubot5> Launchpad bug 1722959 in cloud-init "Implement Key-Value Pair Telemetry for Azure" [Undecided,In progress]
<smoser> wow. telemetry
<paulmey> It's a new reporter, which is not enabled by default
<paulmey> lol
<blackboxsw> good timing paulmey, we were trying to clean up the review queue and will be talking about getting branches in the upcoming status meeting monday.
<paulmey> yeah I was the email that came down the list
<paulmey> also, is there something that still needs to happen on https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/334341 from Douglas' side?
<paulmey> ah now rereading that email... you wanted MP's merged by today... so that's pretty late for mine then. :-/
<blackboxsw> good deal. we are trying to blitz the queue today. I told folks here that I'd try to get though douglas' update today. I'm just a bit sluggish on that. wanted to get another azure run on that to make sure we haven't regressed instance behavior currently.
<blackboxsw> we'll see what there's time for. paulmey, can't rule it out yet (small changeset, might move fast)
<blackboxsw> initial comments, can you add test intent docstrings to those unit tests and import mock  from cloudinit.tests.helpers
<blackboxsw> minor style nits/comments posted https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/334989
<paulmey> will do
<paulmey> done
<smoser> robjo: ?
<smoser> https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334337 ?
<smoser> did you get an answer there?
<robjo> Not yet, sorry
<smoser> robjo: if you ahd to come up with a prefix for opensuse.org bugs
<smoser> "right now"
<smoser> what woudl you choose
<smoser> (ie, launchpad: LP: #XXXXXXX
<smoser> rhbz: https://bugzilla.redhat.com/show_bug.cgi?id=1194451
<ubot5> bugzilla.redhat.com bug 1194451 in cloud-init "[PATCH] Add dnf support to cloud-init" [Low,Closed: rawhide]
<robjo> We use "boo"  as in "Bugzilla,opensuse.org"
<smoser> :)
<smoser> boo it is.
<smoser> boo: https://bugzilla.opensuse.org/show_bug.cgi?id=1069635
<ubot5> bugzilla.opensuse.org bug 1069635 in Other "cloud-init packages differ for each build" [Normal,New]
<blackboxsw> couple comments posted to https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/334341
<blackboxsw> testing on azure next
<rharper> hehe, boo
<robjo> chrony support :) : https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334992
<robjo> this can wait until after 17.2 , but would love to have it in 18.1 with the assumption that 18.1 will be released sometime in March 2018
<blackboxsw> good plan. working out a unittest patch for https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333772 at the moment.
<blackboxsw> should have something shortly
<blackboxsw> baby steps for me when hitting robjo's queue :)
<smoser> blackboxsw: do you want to do a MP for me ?
<blackboxsw> sure
<smoser> for new upload to bionic ?
<blackboxsw> sure smoser will do
<blackboxsw> smoser: only has one commit not released right?
<blackboxsw>   * New upstream snapshot.
<blackboxsw>     - VMware: Support for user provided pre and post-customization scripts
<blackboxsw>       [Maitreyee Saikia]
 * blackboxsw checks bionic apt policy now to validate
<robjo> I don't think it was me: https://jenkins.ubuntu.com/server/job/cloud-init-ci/603/console help please
<robjo> Says: ERROR: InvocationError: '/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-ci/.tox/pylint/bin/python -m pylint cloudinit tests tools'
<blackboxsw> robjo: hrm.... /me looks at what we should be using in python going forward
<blackboxsw> cloudinit/distros/opensuse.py:171: [W1505(deprecated-method), Distro._set_default_timesync_client] Using deprecated method linux_distribution()
<smoser> hm..
<blackboxsw> so we could attempt http://pastebin.ubuntu.com/26141985/
<smoser> distro is not a ubiltin
<smoser> i think we need to build our own.
<robjo> yes, it would introduce a new dependency
<robjo> https://github.com/nir0s/distro
<blackboxsw> deprecate happens in python 3.7
<robjo> we do have packages in openSUSE and I suspect all other distros will have packages for this as well
<smoser> rharper: what are the kwargs you use for yaml.dump ?
<smoser> default_flow_style=false  i think ?
<rharper> print yaml.dump(y, default_flow_style=False, indent=4)
<blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/334996 smoser bionic upstream MP
<smoser> build-and-pushing, blackboxsw
<smoser> thanks.
<blackboxsw> thanks for the ubu-creds
<blackboxsw> gonna cash them in for a candy bar someday
<smoser> or an ubuntu cola
<smoser> https://www.amazon.com/Ubuntu-Fairtrade-Fluid-Ounce-Italian/dp/B018SSM756
<blackboxsw> nice. my world is now complete, if I drink it out of an ubuntu mug
<smoser> robjo: you have some merge conflicts showing
<smoser>  https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334338
<robjo> I'll rebase the branch
 * smoser has to go
<robjo> smoser all set
<blackboxsw> robjo just finished a suggested patch on https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333772 I set it work in progress for you to click back to needs review if you think it's acceptable.
#cloud-init 2019-12-06
<blackboxsw> nice powersj thanks for ubuntulog2
<powersj> that should do it
<meena> i think the changes that rharper requested are already in.
<meena> (and smoser approvedâ¦)
<smoser> that pull request was kinda crazy
<smoser> ending up in like 10 lines of code and 30 of test ;)
<powersj> Odd_Bloke, fyi I updated rtd to use github and confirmed the build works
<powersj> it was still pointing at our launchpad url
<powersj> https://cloudinit.readthedocs.io/en/latest/topics/security.html now shows up
<Odd_Bloke> powersj: Great!
<meena> aaaaaaand another one: https://github.com/canonical/cloud-init/pull/93
<meena> now to find out how to discourage https://bugs.launchpad.net/cloud-init/+bug/1855170 or document or whateverâ¦
<ubot5> Ubuntu bug 1855170 in cloud-init "system_info can change the distro" [Undecided,New]
<meena> or perhaps just ask Hetzner to remove that damn distro line from their # cloud-config
<blackboxsw> interesting. meena on that bug, I'd expect that the cloud-platform should not be setting an opinion on what the distro is expected to be from metadata checking the details on that
<blackboxsw> the "system_info: distro specification is generally expected from the image filesystem"
<meena> blackboxsw: especially since they allow to override the installation via ISO ð¤£
<blackboxsw> if you had cloud-init query --all on the platform that exhibited that info that'd be cool
<blackboxsw> it *should* show all metadata consumed by cloud-init (I think)
<blackboxsw> sudo cloud-init query --all
 * blackboxsw doesn't have a heztner account afaik
<meena> blackboxsw: https://gist.github.com/igalic/e637014696e7f0f8a69023a9fe873e15
<meena> ignore the userdata
<meena> cloud-init hasn't gotten that far yet xD
<blackboxsw> nice puppet facts :)
<blackboxsw> anyhow, ok vendor data is what's setting ubuntu as distro potentially?
<meena> s/potentially?//
 * blackboxsw needs to figure out a good oneliner  to unmime unjson that vendordata content
<rharper> blackboxsw: we could add a --decode flag to query ?
<meena> well, if i query only vendordata, you get something more useful
<meena> https://gist.github.com/d89bf0775ed9233987248d547929ac1f
<blackboxsw> nice paste meena thx
<blackboxsw> yeah I'd expect if you were launching a non-ubuntu instance, vendordata can't set distro: ubuntu that's busted
<blackboxsw> that sounds like a platform bug
<rharper> what's all in vendor data? user-data can disable vendor data
<meena> rharper: oh?
<meena> wait a damn minuteâ¦ where's the network config?!
<meena> oh, under network config
<meena> or notâ¦
<meena> on my other server, running a very old version of cloud-init, i can query ds.meta_data.network_config
<meena> but not on this one :(
<meena> i wonder why
<meena> i think i'm gonna have a lot more patching to do
<meena> some other time.
<meena> Good night folks o/~
<chillysurfer> on a bionic machine, trying to run cloud-tests and getting an error `No storage pool found. Please create a new storage pool`. any thoughts? i ran this example exactly: https://cloudinit.readthedocs.io/en/latest/topics/tests.html#treerun-and-treecollect (the tree_run)
<powersj> by default we try to use LXD, that certainly sounds like you don't have lxd setup
<chillysurfer> ah yeah that's probably it
<chillysurfer> actually looks like lxd pkg is already installed
<powersj> have you run lxd init?
<blackboxsw> Lxd unit then
<blackboxsw> Oops lxd init
<chillysurfer> looks like that might've done it
<chillysurfer> if i only want to run cloud-tests on azure, do i just comment out the ec2 config in platforms.yaml?
<chillysurfer> actually do i just set --build-platform to azurecloud then?
<chillysurfer> ah i bet that's what it is
<powersj> chillysurfer, that should do it
<chillysurfer> powersj: nice thanks
<powersj> we default the platform to lxd since it is fast and easier to run usually
<powersj> and doesn't cost money :D
<chillysurfer> i'm going to see if the read_file_or_url regression (that was already fixed) is caught with cloud-tests
#cloud-init 2019-12-07
<smoser> how do we make https://github.com/canonical/cloud-init/pull/75 re-run
<smoser> test faiure related to snap not started.
<powersj> smoser, details -> login to travisci -> on the job that failed there is a restart button
<powersj> i've kicked it off
<smoser> thanks
#cloud-init 2019-12-08
<aw-> hi, i'm wondering if there's any "dangers" of reading userdata directly from /var/lib/cloud/instance/user-data.txt - or is it safer to use "cloud-init query userdata" ? I ask because the cloud-init command is unbearably slow
<meena> aw-: if it is unbearably, then we should find out why.
<meena> i just saw goneri approved two of my patches, which i assume means he tested themâ¦ *nice*
 * meena distinctly remembers turning on their laptop to do a thing, but not what thing.
