#cloud-init 2014-04-14
<natorious> is there a reason why the include.txt example in the docs is not valid?
<harlowja> probably not a real reason but a mistake
<natorious> apparently cloud-init attempts to parse the non-user-data linked files and barfs.  
<harmw> god I hate building kernels
<harlowja> natorious can u open a bug, doesn't seem like a hard fix
<natorious> harlowja: sure
<harlowja> thx
#cloud-init 2014-04-15
<smoser> harlowja, https://github.com/coreos/coreos-cloudinit
<smoser> flattery
<harlowja> lol
<harlowja> there u go
<harlowja> although not a fan of forks :-/
<smoser> in go.
<smoser> i got to go
<smoser> later.
<harlowja> go go
<harmw> forking cloudinit...
<harmw> why!
<harmw> harlowja_away: I think they are scared by your presence, thats keeping them to contribute and made them fork instead :p
<harlowja_> harmw possibly, although maybe they just like go alot, lol
<harlowja_> or they don't want to include python in there distro thingy
<harlowja_> or who knows :-P
<harlowja_> maybe u scared them harmw 
<harlowja_> lol
<harlowja_> smoser http://hg.python.org/peps/rev/76d43e52d978 durn 
<harlowja_> does that change canonicals plan?
<smoser> oh wwo.
<smoser> well it makes life much nicer
<harlowja_> probably less pressure for python3 then i suspect?
<harlowja_> now that there is an extra 5 years
<smoser> well, i think we'll mostm likely still be almost completely python-free by 2016.4
<harlowja_> except that damn cloud-init
<harlowja_> lol
<harlowja_> python-free?
<harmw> yep, it costs like $155,95 untill april 2016
<smoser> :)
<harlowja_> damn
<harmw> well, thats what I was told by my local retailer
<harlowja_> do i get a bulk discount?
<harlowja_> 2 for 1?
<harmw> dunno, have to ask him
<harlowja_> so guido is your local retailer?
<harlowja_> python drug dealer
<harlowja_> lol
<harmw> :>
<harlowja_> guido give me some more of that python, for 155.95
<harlowja_> ha
 * harlowja_ i gotta figure out what happened to sean
<harmw> lol
<harlowja_> i saw him at lunch yesterday
<harmw> well, he's busy in #bsdmips :p
<harmw> occasionaly
<harlowja_> really?
<harmw> yup
<harlowja_> hmmm, gonna have to track him down there then, haha
<harmw> lol
<harlowja_> pull him by his ear in here
<harlowja_> get packages uploaded to ports
<harlowja_> smoser so when u gonna do some taskflow work?
<harlowja_> huh huh
<harlowja_> lol
#cloud-init 2014-04-16
<cppking> 0
<sonia_> Hi , need help with re-starting cloud-init without rebooting the VM
<sonia_> I could achieve it by deleting /var/lib/cloud/instance/  and running cloud-init modules --mode init/config/final
<sonia_> is there a way I can inject my own user-data in between to run that instead of originally provided ?
#cloud-init 2014-04-17
<Sonia_> Hi , need help in cloud-init ?
<kwadronaut> Sonia_: please state your question and stick around.
<kwadronaut> formulating your question often already helps understanding the problem
<kwadronaut> and people are not always behind their keyboards, so asking for the question, doesn't mean people stick around until it is there.
<Sonia_> sorry , yesterday I put my question and got no response till today morning, hence realized if there is no one in the chat , and if time is out , question gets deleted 
<Sonia_> my question on cloud-init is , how we can re-run cloud-init without rebooting the VM .....and in the same process give another user-data to it. (not the one initially executed by cloud-init)
<kwadronaut> ah i see it now if i scroll up...., you could add any local data in /var/lib/cloud/?
<Sonia_> I was doing following steps to re-run cloud-init :
<Sonia_> 1. rm -rf /var/lib/cloud/insance*
<Sonia_> 2. run cloud-init init , cloud-init modules --mode init/config/final
<Sonia_> these will run it with the same user-data which I supplied at the time of VM provisioning
<Sonia_> but in between if I try to give my own, doenst work
<harmw> you could also fetch the userdata using curl, just to make sure it got updated properly on the infra-side (beeit openstack or whatever platform you're running)
<Sonia_> I used config-drive here ,and user-data gets to the VM successfully and gets executed.
<Sonia_> when i re-run cloud-init , it again gets the user-data from config-drive 
<kwadronaut> i see a typo in your rm -rf, a t has gone missing in the instance
<kwadronaut> well, if you want to use different data/source, that needs to be changed too of course?
<smoser> Sonia_, you can use 'ci-tool' as a friendly-ish way to "seed" data.
<smoser> http://bazaar.launchpad.net/~smoser/cloud-init/ci-tool/view/head:/ci-tool
<smoser> it also has 'reset' which will clean out the state in /var/lib/cloud
<smoser> Sonia_, also, if you're just wanting to quickly iterate on testing things... lxc can provide a really nice way to do that.
<smoser> http://bazaar.launchpad.net/~smoser/cloud-init/ci-tool/view/head:/ci-tool
<smoser> oops.
<smoser> http://ubuntu-smoser.blogspot.com/2013/08/lxc-with-fast-cloning-via-overlayfs-and.html
<Sonia_> thanks smoser for the link , I will certainly look at the tool
<Sonia_> my use case is that , initially coud-init will work with user-data supplied during provisioning time 
<Sonia_> just that at later point of time , want to re-run cloud-init without rebooting the VM with a different user-data script.
<Sonia_> kwadronaut , sorry for the typo in the chat , during testing , 't' is there 
<smoser> Sonia_, why do you want to do that ?
<smoser> ci-tool would help you accomplish it, but curious.
<Sonia_> kwadronaut , so if I want a different data source after the VM is provisioned, can  I change it later point of time ?
<Sonia_> well , I dont want VM to do all the initial configurations .....running them again may wipe out many settings done
<Sonia_> smoser , shouldnt these steps work too ? 
<Sonia_> 1. clean out /var/lib/cloud 2. run cloud-init init 3.  replace the user-data script with out own in the instance/ directory 4. run cloud-init modules --mode init/config/final
<smoser> no. it probably wont work.
<smoser> i'm sorry, i can't really help now.
<Sonia_> no problem , I will try ci-tool
<Sonia_> smoser , is there a way to change the data source also ?
<smoser> sure. its configurable.
<smoser> in ubuntu 'dpkg-reconfigure cloud-init'
<smoser> dpkg-reconfigure or ci-toolw ill adjust etc/cloud/cloud.cfg.d/90_dpkg.cfg
<Sonia_> ok , I am on RHEL
<Sonia_> hope I can still change the configurations.
<kwadronaut> smoser: awwww. that lxc link is promising!
<smoser> kwadronaut, yeah, its nice.
<smoser> on ubuntu, you can also snapshot quickly with btrfs (or lvm if you wanted).
<smoser> the big feature really was the "clone hook" to allow you to pass in user-data on the clone rather than create.
<sauce> smoser you the man
<smoser> thanks
<smoser> :)
<Sonia_> smoser , I may need help with ci-tool 
<Sonia_> so first i ran ci-tool reset , which removed instance and instances/ directory for me.
<Sonia_> post that , I am trying to seed data using the command , "ci-tool seed -s nocloud /opt/my-user-data"
<Sonia_> then I run cloud init again by "ci-tool run"
<Sonia_> correcting my earlier command : "ci-tool seed /opt/SJuser-data"
<Sonia_> running ci-tool again doesnt execute the new user-data file , any mistake I am doing here in using the tool ?
#cloud-init 2014-04-18
<Sonia> smoser, let me know once you are back in this chat.
<Sonia> anyone else explored ci-tool ?
<smoser> Sonia, run isn't implmeented yet :-(
<smoser> but basically, yeah, you'd do the things you were doing before there.
<Sonia> Hi smoser , so if i run these commands : "ci-tool reset" followed by "ci-tool seed /opt/SJuser-data"  and then "cloud-init init" ...then it should work ?
<smoser> cloud-init init --local
<smoser> cloud-init init
<smoser> cloud-init config
<smoser> oops
<smoser> exec cloud-init modules --mode=config
<smoser> well, not exec
<smoser> cloud-init modules --mode=final
<Sonia> ok , let me try 
<Sonia> it worked , thanks Scott , I was missing this "cloud-init init -local"
<Sonia> so basically without ci-tool , one cannot seed own user-data 
<smoser> Sonia, well you can. you just have to do it carefully.
<smoser> ci-tool more selectively rm -Rf /var/lib/cloud
<Sonia> right , i will  try to do without ci-tool , thanks again !
#cloud-init 2015-04-13
<soumi> hey
<soumi> i have cloud init issue with centos5 image 
<soumi> have any one faced it earlier
<soumi> ?
<soumi> the issue is that, it doesn't add the ssh keys :(.. in console.log.. I get the below logs:
<soumi> found data source: DataSourceEc2
<soumi> 2015-04-09 09:54:52,315 - cc_ssh.py[WARNING]: applying credentials failed!
<soumi> clod-init package I am using is cloud-init-0.6.3-0.12.bzr532.el5
#cloud-init 2015-04-14
<Odd_Bloke> smoser: We've been asked to stop updating the hostname on reboot on Azure, but still set it at first boot (and when we bounce the connection).
<Odd_Bloke> smoser: We can do this using update_hostname; is there any way that cloud.cfg.d can stop update_hostname from running?
<Odd_Bloke> s/using/by disabling/
<smoser> Odd_Bloke, sure. you can remove it from the list of modules.
<smoser> but i thought it should only do it if you had not changed it.
<Odd_Bloke> smoser: This is the issue where Azure only want hostname set on first boot.
<Odd_Bloke> s/set/set to the value they provide/
<Odd_Bloke> smoser: So update_hostname does explicitly the opposite of what Azure want. :p
<Odd_Bloke> smoser: So can I pop something off the cloud_init_modules list in cloud.cfg.d (or the data source?), or would I have to redefine the entire list?
<Odd_Bloke> (Or just modify cloud.cfg)
<smoser> Odd_Bloke, you have to redefine the entire list.
<smoser> Odd_Bloke, but above, only set to the value they provide on first boot.
<smoser> i *think* htats the behavior you get.
<smoser> ie, boot, changes from ubuntu to some azure-provided name
<smoser> if the user modifies /etc/hostname subsequently, it shouldnt do this agian
<smoser> as it checks to see if the previous version is different than the current
<smoser> and if it is, then doesn't do it.
<Odd_Bloke> Hmm, I don't _think_ that's what I've been seeing.
<Odd_Bloke> But I might have been bitten by that race condition.
<Odd_Bloke> So I'll check again before I do anything else.
<smoser> hm.. i dont see it either.. wher eis that.. i know i did that somewher.e
<smoser> ah. update_hostname does that. but set_hostname does not.
<smoser> update_hostname is PER_ALWAYS
<smoser> set_hostname is PER_INSTANCE
<devicenull> https://bugs.launchpad.net/cloud-init/+bug/1424710
<devicenull> is there anything I can do to actually get this patch applied?
<Odd_Bloke> devicenull: Lennert says that he worked around this problem by installing a different version of cloud-init; does that mean that this patch has been applied downstream in Fedora/EPEL?
<devicenull> I don't see any relevant patches in the epel rpm for this
<devicenull> hah
<devicenull> the epel rpm "works" because it doesn't know about centos 7 using systemd
<devicenull> so it falls back to the old working method
<Odd_Bloke> <3
<Odd_Bloke> devicenull: Your best bet is probably to open up a merge proposal in to lp:cloud-init with your change as a commit.
<devicenull> ok
<smoser> devicenull, yeah, open a MP. i can pull it fairly quickly as it looks sane. i'd *like* a non-name-based approach to "does this use systemd"
<devicenull> will do
<harlowja> arg, smoser stackforge cloud-init is behind like by 100+ patches :(
<harlowja> suck
<harlowja> *behind bzr
<harlowja> on the 0.7.x branch...
<harlowja> http://paste.ubuntu.com/10823678/
<harlowja> poopy
 * harlowja wonder how we can get all those in without 155 openstack reviews, lol
<harlowja> oddly not all 155 just cleanily apploy
<harlowja> *apply
<harlowja> which is weird
<harlowja> smoser how did u make https://github.com/cloud-init/cloud-init initially?
 * harlowja doesn't make sense that those patches wouldn't apply (since they came from git<->bzr layer)
<harlowja> 0047-Remove-a-comment-turd.patch 
<harlowja> lol
<harlowja> 0051-Remove-debugging-turd.patch
<harlowja> lol
<smoser> i used bzr fast export / git fast import probably
<harlowja> hmmm, intersting
<harlowja> maybe i should try that vs https://github.com/felipec/git-remote-bzr (or simiar)
<harlowja> hmmm
<harlowja> weird, ya, still doesn't cleanly apply, oddness
<harlowja> http://paste.ubuntu.com/10823722/ seemingly weird that that happens at all...
<smoser> yeah, that is odd
<harlowja> def
<devicenull> actually, I'm looking at this again and I can't figure out how update_hostname is supposed to work
<devicenull> based on "LOG.debug("%s differs from %s, assuming user maintained hostname.","
<devicenull> I assume it's not supposed to update the hostname if it's different from what it expects (cloud-init set the hostname once, then the user changed it)
<devicenull> but that doesn't seem to be what the code does
<devicenull> is my assumption that it's not supposed to be overwriting user changes correct?
<harlowja> user changes being something out of band updating the hostname?
<harlowja> i forget what the code is doing, but maybe is what u sy
<harlowja> *say
<devicenull> yea
<devicenull> it looks like a bug tbh, I'll be looking into it more tomorrow
<harlowja> k
<harlowja> smoser arg, got closer
<harlowja> http://paste.ubuntu.com/10824510/
<harlowja> did we turn on the ccla check in that repo :-/
<harlowja> or need other way to make it ignore the committer...
<harlowja> but nearly got it to work, ha
#cloud-init 2015-04-15
<harlowja> https://github.com/harlowja/cloud-init/tree/0.7.x-fixed should be the repo we need resynced...
<harlowja> with all the missing 0.7.x stuffs
<harlowja> * https://github.com/stackforge/cloud-init/compare/0.7.x...harlowja:0.7.x-fixed
<Odd_Bloke> smoser: It does look like update_hostname is doing the right thing.
<Odd_Bloke> smoser: I think I was hitting the cloud-init/walinuxagent race before.
<smoser> Odd_Bloke, https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1444086 seems kind of serious.
<smoser> is someone looking at that ? 
<Odd_Bloke> smoser: I believe utlemming has been.
<utlemming> smoser: yeah, I
<utlemming> smoser: er, I'm looking at that
<yaronr_> hi guys
<yaronr_> trying to add a yum_repos in cloud-init. installation fails because repo returns 403
<yaronr_> interestingly, installation of packages that come from other repos fail as well
<ndonegan> It's going to be the server serving the repos that's returning 403
<ndonegan> So most likely you're pointing at the wrong directories for the repos, or there's some user or ip restriction on the repos.
<smoser> or, if the repos are on S3, then 404 are returned as 403
<smoser> (S3 does do that. ie, they tell you "you can't see that file", rather than 'that file doesnt exist')
<smoser> yaronr_, ^
<yaronr_> hmm
<yaronr_> baseurl: http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
<yaronr_> works on my browser..
<zer0c00l> howdy
<smoser> hi.
<kwaping> howdy
<harlowja> kwaping hey, so i'm working with openstack-infra to get https://github.com/harlowja/cloud-init/tree/0.7.x-fixed pushed in to stackforge
<kwaping> cool
<harlowja> and nm, https://github.com/stackforge/cloud-init/tree/0.7.x is up to date now
<harlowja> lol
<harlowja> sooo ya, thats done :-P
<kwaping> wooo
<kwaping> new source of truth?
<kwaping> so is the 0.7.x branch considered stable, and master is bleeding-edge?
<harlowja> sure
<harlowja> seems about right
<kwaping> ha
<kwaping> yeah, I like that scheme - was thinking about other options but that works for me
<harlowja> ya, doesn't need to be perfect
<kwaping> no, it needs to be perfect
<kwaping> jk
<harlowja> lol
<vila> hi there !
<vila> with a fresh vivid-server image and cloud-init 0.7.7 I end up with no ssh keys generated and suspicious messages in the console: https://pastebin.canonical.com/129649/
<vila> does that ring a bell ? 
<larsks> vila: that pastebin apparently requires on to log in to view?
<vila> ha right, sorry, bad habbit
<vila> http://paste.ubuntu.com/10828856/
<larsks> It looks like there is no sshd-keygen service on vivid, at least.  
<larsks> It looks like key generation happens as part of the openssh-server postinst script...which is a terrible idea, but there it is.
<vila> larsks: you mean sshd-keygen exists somewhere else ?
<vila> larsks: and openssh-server may need to be fixed for vivid ?
<larsks> vila: so, I'm not super familiar with the ubuntu openssh packages. *I* would fix this by having the sshd systemd service unit run a key-generation script prior to starting sshd.
<larsks> Someone more familiar with the ubuntu side of things can probably provide a better answer. 
<vila> ack, thanks
<vila> smoser, utlemming : any idea ?
#cloud-init 2015-04-16
<smoser> vila (who is gone) i've never seen that . that is odd.
<smoser> JayF, around ?
<smoser> see email
<larsks> smoser: my feeling is that the ubuntu ssh packaging is fundamentally broken: there's nothing that will generate new host keys if they are missing when attempting to start sshd.  As far as I can tell, key generation happens only in the package postinst script. I.e., I don't think this is a cloud-init problem.
<Odd_Bloke> larsks: I haven't been able to reproduce that issue, so I would tend to agree that it's not cloud-init's problem.
<smoser> larsks, you're right, that is the case.
<smoser> but it *is* cloud-init's problem here.
<Odd_Bloke> Would be good to check with vila precisely what he was doing, in case there's something going on that we don't know about.
<smoser> cloud-init needs to generate host keys (even if they exist) on first instance boot.
<larsks> smoser: See, I disagree.  I think that maybe cloud-init should have an option to *remove* the keys, but I think that whatever starts sshd should be generating the keys.
<smoser> so it is perfectly valid to rm /etc/ssh/ssh_*  in an image and then boot it.
<larsks> smoser: Well, right.  That's very common, and that's why I think the ubuntu ssh packaging is so broken.
<smoser> wll, that is definitely a path. 
<larsks> Because that's a very common situation (removing host keys as part of image prep)
<smoser> cjwatson had been averse to that.
<smoser> as that would mean the keys (if not present) are created with very little entropy.
<smoser> dropbear (often used with busybox) actually generates them on first connect
<smoser> which is guaranteed later than daemon start, and thus "better" from a perspective of available entropy.
<larsks> That seems like a fine solution, and substantially better than "not generating them at all and being broken" 
<smoser> mostly, i've not fought with cjwatson. he's the openssh package maintainer, and very capable :)
<larsks> I note that many cloud environments have a mechanism for feeding the entropy pool on virtual instances to help solve the low entropy issue...
<smoser> larsks, cloud-init tries to use said entropy also (ie, if it knows of it, it will seed /dev/random with it).
<larsks> Okay. So then in many environments low entropy shoulnd't necessarily be a problem.
<smoser> larsks, well, except that we have to set cloud-init to correctly run before sshd starts then.
<smoser> and also, you can't jsut say "its not a problem in some cloud environments" as any do not have said entropy utils
<larsks> cloud-init is going to be generating ssh keys at boot one way or the other, either because the ssh startup script does it or because cloud-init does it natively.  So this is a situation we already have...
<larsks> I guess a short-term bug -- that doesn't directly address this issue -- is that the cloud-init systemd unit has a dependency on sshd-keygen.service, which doesn't exist.
<Odd_Bloke> cloud-init actually specifies that it should run before sshd-keygen.service.
<larsks> Right.
<larsks> sshd-keygen.service does not exist (on vivid, at least)
<Odd_Bloke> Yeah.
<Odd_Bloke> It's pretty much just noise though, so I don't know that it's worth pushing for vivid.
<larsks> It is not operationally significant, it's true. 
<vila> smoser: \o/
<vila> Thanks for the fix, I was trying to write a test and found the right file, now I have the right example ;-D
<vila> (https://bugs.launchpad.net/bugs/1445143)
#cloud-init 2015-04-17
<Odd_Bloke> smoser: When do/should cloud-init bugs get marked as Fix Released in the cloud-init LP project?
<smoser> Odd_Bloke, i do it with release of the bug in a version.
<smoser> ie, when 0.7.6 releases all those go to fix-released.
<smoser> and ubuntu fix-released when ubuntu gets fixed.
<Odd_Bloke> smoser: Cool, so I don't need to pay attention and do the marking myself when appropriate?
<smoser> well, it wouldn't hurt :)
<smoser> its a pain
<Odd_Bloke> smoser: What are your thoughts on https://bugs.launchpad.net/cloud-init/+bug/1403617 ?
<Odd_Bloke> smoser: My comment lays out the decision we need to make.
<smoser> how do we not match with roject level keys now ?
<smoser> Odd_Bloke, ie "we already don't match the GCE docs in the way we handle project-level keys so this may be a foolish consistency."
<Odd_Bloke> smoser: I _think_ that we put all of the keys on the ubuntu user, even when they're defined against different users.
<Odd_Bloke> smoser: But I may not be recalling that correctly.
<smoser> how do they get defined for different users ?
<smoser> we do put them on the ubuntu user for user.
<smoser> sure
<Odd_Bloke> smoser: Keys come from GCE as a list of "<user>:<key>" strings.
<Odd_Bloke> smoser: Which GCE infers from the comment (e.g. "... dwatkins" will come as "dwatkins:...") in the user interface.
<Odd_Bloke> But you just pass in a mapping to the instance creation API.
<Odd_Bloke> smoser: We then trim the first half off before setting 'public-keys' in self.metadata (using the _trim_key function).
<smoser> ah. i see.
<smoser> i think you should override the per-project keys with per-instance if available.
<Odd_Bloke> So if I defined {'dwatkins': 'ssh-rsa foo', 'smoser': 'ssh-rsa bar'}, we'd get both 'ssh-rsa foo' and 'ssh-rsa bar' on the ubuntu user.
<Odd_Bloke> (And the Google scripts would create dwatkins and smoser users with the appropriate keys)
<smoser> can we know the difference between "no instance keys given" and "instance keys given as empty string"
<smoser> with the latter implying intent to have no ssh access
<Odd_Bloke> I'll have a look; I _suspect_ not, but I'll confirm.
<Odd_Bloke> Ah, we can; sshKeys is passed as a normal metadata attribute, and so if none are specified then the key isn't present.
<Odd_Bloke> Let me confirm that the web UI behaves the same as the CLI client.
<smoser> i'd just like to support that behavior
<smoser> Odd_Bloke, so my general feeling here is that it makes sense to try to do what the cloud vendor wants.
<smoser> however, i'm *more* interested in consistency of ubuntu across vendors
<smoser> than i am in ubuntu's consistency with other vms on a given vendor
<smoser> make sense?
<smoser> i care more about ubuntu than i do GCE
<Odd_Bloke> Yep, on the same page.
<Odd_Bloke> So you're saying you think that 'empty instance keys' is approximately equal to (e.g.) no key given when starting an EC2 instance?
<smoser> Odd_Bloke, yeah.
<smoser> i think so. right ?
<smoser> that could be achieved easily ehought though, by creating a key named "NOONE@NOWHERE"
<smoser> and promptly shredding the private key
<Odd_Bloke> Yeah.
<Odd_Bloke> Let me see what GCE does if I manually set an empty string through the API.
#cloud-init 2016-04-18
<mgagne> where can I find latest cloud-init sources? It looks like openstack/cloud-init is very outdated and links in channel topic are outdated.
<mgagne> looks like it's back to LP
#cloud-init 2016-04-19
<smoser> mgagne, sorry. bzr
<smoser> lp:cloud-init
<mgagne> yea, I figured. thanks!
<harlowja> mgagne  ya, were working on fixing that
<harlowja> i mean smoser is gonna save us all
<smoser> harlowja, right after you get the rhel networking sorted
<smoser> :)
<harlowja> lol
<harlowja> oh ya, i'll try to do something there this week
<smoser> :)
<smoser> found / hit this today: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1571761
<smoser> :-(
<smoser> and with that  /me disapears
<smoser> later gator.
<harlowja> hmmmm, interesting, later
<ReSam> good morning!
<ReSam> I'
<ReSam> I'm running MAAS and want to change my curtin_userdata preseed to install a few more things - but unfortunatley something always errors and then the exit code causes my deployment to fail
<ReSam> what is the best way to "ignore the exit code" completely?
<ReSam> I'm have lines like this in /etc/maas/preseeds/curtin_userdata:  puppet_02: curtin in-target -- sh -c 'dpkg -i /tmp/puppetlabs-release-pc1.deb'
<Odd_Bloke> smoser: Is the expected behaviour of the 'fallback' networking to present network interfaces as eth{0,1,...} rather than the systemd names?
<Odd_Bloke> rharper: ^
<Odd_Bloke> If it isn't that's fine, I just got the impression that maybe that's what's meant to happen.
<Odd_Bloke> And therefore want to know if I should be reporting if it isn't. :)
<Odd_Bloke> (Oh, actually, maybe I'm being thrown off by a red herring: systemd leaves Hyper-V interfaces named eth* and I looked at Azure first; confirmation of the above would still be appreciated :)
<rharper> Odd_Bloke: smoser  can confirm, but I don't think fallback does any renaming, so if you're booted without net.ifnames=0, and you have systemd, and fallback happens, you should see whatever systemd named the nic in the interfaces.d/50-cloud-init.cfg file
<Odd_Bloke> OK, cool.
<Odd_Bloke> smoser: rharper: So (assuming that to be true), do you have any suggestions for handling clouds that are sending network_config over in the meta_data.json, where content/0000 hard-codes eth0 as the expected network interface?
<rharper> if
<Odd_Bloke> (Other than going to the cloud and telling them to change what they're doing, of course ;)
<Odd_Bloke> (Though we will try that)
<rharper> if they are sending network_config via metadata service, we respect and use that
<rharper> including naming
<rharper> that's not fallback
<rharper> we convert network_data.json into internal network_config.yaml which is then parsed and applied as if you had passed in a network_config.yaml in user-data
<rharper> this is the configdrive stuff
<Odd_Bloke> Yeah.
<Odd_Bloke> Let me paste some things so we're on the same page.
<Odd_Bloke> rharper: http://paste.ubuntu.com/15929016/
<Odd_Bloke> So the cloud are just specifying an ENI file via meta_data.json and content/
<Odd_Bloke> 0000
<rharper> oh, this is network_config
<rharper> unfortunate naming collision smoser and I discussed
<rharper> Odd_Bloke: so, IIUC, cloud is providing an eni as-is;  however, it may not be aware of how nics will be named, right ?
<Odd_Bloke> Right.
<rharper> if they're sending ethX naming, and not also supplying an udev rule mapping file (70-persistent-net)
<rharper> or network_config.yaml; then it's a bit of a shot in the dark
<rharper> for Xenial (and wily), ensuring you boot with net.ifnames=0 will prevent nic naming to systemd style
<rharper> which should match what they send; but the cloud itself should transition to using network_config.yaml
<Odd_Bloke> rharper: Right; we're trying to get away from net.ifnames=0 as far as we can (because I suspect we will strongly regret having that delta in place for the next five years).
<rharper> agreed
<rharper> and the requirement is to specify what you want in network_config.yaml
<rharper> alternatively, they can emit the ConfigDrive format that OpenStack uses
<rharper> which cloud-init can convert to network_config.yaml
<rharper> which emits the rules files needed to do name mapping
<Odd_Bloke> rharper: Is there documentation that I can read/point them at?
<rharper> y
<rharper> one sec
<rharper> https://blueprints.launchpad.net/nova/+spec/metadata-service-network-info
<rharper> http://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html
<Odd_Bloke> rharper: So that's what they would put in /openstack/.../network_data.json, right?  How does network_config.yaml work?
<rharper> it goes in the ConfigDrive at openstack/<date>/network_data.json
<rharper> for network_config.yaml, lemme find the details
<Odd_Bloke> rharper: Thanks. :)
<smoser> Odd_Bloke, well, we need them to send the macs
<smoser> and openstack does.
<Odd_Bloke> smoser: From Liberty onward, right?
<Odd_Bloke> (Can I tell what version of OpenStack I'm running on easily?)
<smoser> the easiest way to do so is to look at the config drive.
<smoser> or metadata service.
<smoser> the version of the openstack isn't really exposed to the guest.
<smoser> to further make it painful you actually have to have this fix applied to your liberty cloud : https://bugs.launchpad.net/nova/+bug/1513267
<Odd_Bloke> So the most recent metadata version is 2013-10-17.
<smoser> Odd_Bloke, https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L72
<smoser> so they're ssomewhere between Havana and Liberty (Icehouse, Juno, or Kilo)
<Odd_Bloke> smoser: So one of Hava, Icehouse, Juno or Kilo?
<Odd_Bloke> Hah.
<smoser> its not unreasonabel to request the newer metadata really...
<smoser> if they want to provide network data to the guest, then that is how it is done, and any thing previous is just kind of crap.
<smoser> we could maybe support the path where they made a /etc/network/interfaces style file available to the guest
<Odd_Bloke> smoser: Assuming they aren't able to do network_config.json, is there any good path forward that might mean we can get something working on GA day?
<Odd_Bloke> (I'm assuming cloud-init changes are out of scope for Thursday :p)
<smoser> well, ga is almost certainly no.
<smoser> right.
<smoser> this is dreamhost ?
<Odd_Bloke> Yep.
<Odd_Bloke> Only their new region though.
<smoser> and each instance has static ip configuration expected ?
<Odd_Bloke> Their old region has a DHCP server.
<smoser> how did they think this is supposed to work?
<smoser> ie, if its static config, how does the host tell the guest what config to use
<Odd_Bloke> Did you see http://paste.ubuntu.com/15929016/ ?
<smoser> no sorry
<smoser> Odd_Bloke, ok. so yeah, thats the old way of declaring it and we could prbosbaly without too much effort make that work
<smoser> but not without some chagnes to cloud-init
<Odd_Bloke> smoser: No need to be sorry. :)
<Odd_Bloke> smoser: What would you envision those changes being, roughly?
<smoser> the config drive datasource would just have to look in that path for config information and the parse the ENI format
<smoser> and then apply it as networking.
<smoser> we do have an ENI parser in cloud-init
<smoser> and then a writer. so the pieces are mostly there.
<smoser> the hwaddress bit is actually *wrong* in openstack's use.
<smoser> ie, they're saying "eth0 is the nic with this mac address"
<Odd_Bloke> How would it infer that the config for eth0 should actually be applied to network_interfaces[0]?
<smoser> when ifupdown says "i shoudl change the nic named eth0 to use this mac address"
<smoser> how would it infer ?
<Odd_Bloke> Ohhh, I'd missed that they specified the MAC address.
<Odd_Bloke> So cloud-init would rename the NIC with the MAC address specified there to eth0 and then apply that ENI config in ENI.d/50-cloud-init.cfg (or whatever that file is called)?
<smoser> yeah.
#cloud-init 2016-04-20
<larsks> Anyone: is the difference between _read_hostname and _read_system_hostname supposed to be that the former returns the current (transient) hostname, and the latter return the persistent hostname?
<larsks> In particular, can _read_hostname be implemented pretty much for all distributions just by calling the `hostname` command, leaving the distro-specific stuff in _read_system_hostname (and, presumably, _write_hostname)?
<smoser> larsks thats a good question. i'd bzr blame to see if there are any hints.
<larsks> Not really.  It all stems from harlowja 's work in 11b9899c4eec7a73e5dc2ae1fa25184b2ca5c8e1.
<larsks> Hey, harlowja, around?
<larsks> Err, sorry.
<larsks> That's a git commit hash that won't do anybody any good.
 * larsks tries to figure out how the bzr reference...
<larsks> ugh, I don't get bzr.  If I clone cloud-init using git-bzr-remote (git clone bzr::lp:cloud-init) and run "git log", I find commit 11b9899 "System config niceness".  But if I clone it with bzr (bzr branch lp:cloud-init) and run bzr log, I can't find that revision.
<larsks> Not really important.  Just whining.
<harlowja> sorta :-P
<harlowja> ha
<harlowja> lol
<larsks> harlowja: can you elaborate on that "sorta" ?
<Odd_Bloke> <@harlowja> Sort of.
<harlowja> :(
<larsks> harlowja: Specifically, I want to rewrite the hostname handling so that the distro/__init__.py doesn't need to know anything about filenames (keeping that all contained to specific distro classes).
<larsks> But there are these weird relationships between _read_hostname and _read_system_hostname, and I want to make sure I"m doing the right thing.
<larsks> harlowja: I guess ping me when you're around again...
<harlowja> i'm medium around, trying to do some release stuff for oslo
<harlowja> larsks i think the weird relationship was because some operating systems give u the fqdn and some don't
<harlowja> but i might be forgetting :-P
<larsks> Bleargh.  Okay, I will muddle my way through, then, and see what happens.
#cloud-init 2016-04-21
<Odd_Bloke> rharper: smoser: Happy release day!  I've just realised/discovered that lxd images still write out eth0.cfg when they are launched; they use this template: http://paste.ubuntu.com/15963686/.  Does http://paste.ubuntu.com/15963681/ look like a reasonable replacement for that to drop in to /var/lib/cloud/seed/nocloud-net/network-config ?
<Odd_Bloke> (That seed directory is the same one they use for everything else)
<Odd_Bloke> My testing suggests that it is, but I wanted to get your eyes on it first. :)
<Odd_Bloke> (The interface we care about will always be called eth0)
<rharper> Odd_Bloke: the control field can always be auto; ie, do you want ifupdown to make sure the iface is 'up';
<smoser> Odd_Bloke, that does seem right.
<smoser> Odd_Bloke, fwiw, it does work right now for 'dhcp'
<smoser> cloud-init deletes the 'eth0.cfg' file that has the rendered template
<smoser>  http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/1210
<smoser> rharper, this  mioght have been the issue you were seeing that i coudl not reproduce.
<smoser> if you got 'manual' written there.
<Odd_Bloke> smoser: Even if cloud-init didn't delete it, it'd be fine for the dhcp case; it's the manual case that's the problem. :p
<rharper> manual case is for when you're doing link-local only networking
<rharper> hence the check in the lxd config get
<Odd_Bloke> Yeah, and then we don't want to dhcp, so fallback config would also be wrong.
<rharper> now, I think we want control: auto (which we didn't have before)
<rharper> Odd_Bloke: right, in the case one wants to use link-local, then we need subnet: manual  and control: auto
<rharper> I think that will have ifup "up" the iface; I'm not sure if that's enough for link-local to work; but it's likely better than ignoring it altogether unless some other part of lxd up's the iface
<rharper> and certainly better than attempting to dhcp on a the iface with no dnsmasq setup to respond
<rharper> IIUC the problem
<rharper> I should add a link-local set to curtin networking
<Odd_Bloke> https://bugs.launchpad.net/cloud-images/+bug/1573000 is the bug.
<rharper> ah
<rharper> indeed, so providing a config to cloud-init would prevent the writeout of the fallback dhcp
<Odd_Bloke> Yeah.
<smoser> where is your diff, Odd_Bloke ? the changes your proposing ?
<rharper> in the bug
<rharper> http://paste.ubuntu.com/15963681/ as well
<Odd_Bloke> smoser: There is no cloud-init diff, this is just about the network config lxd would write out to its seed.
<smoser> what should i write to get this ?
<smoser> er... how do i answer questions in dpkg-reconfigure lxd to get this
<rharper> disable the bridge
<rharper> I think you want to say no to lxdbr0 ?
<smoser>  http://paste.ubuntu.com/15965001/
<smoser> trying that
<Odd_Bloke> lxd ships in link-local mode.
<Odd_Bloke> So we are in the not-quite-working case in the images by default.
<Odd_Bloke> If you configure the bridge, then the lxd image and cloud-init write out configuration that agrees.
<Odd_Bloke> WHich is still a bug, but less noticeable.
<Odd_Bloke> You can easily fake one or the other with "-c user.network_mode=link-local" and "-c user.network_mode=" when launching a container.
<smoser>  http://paste.ubuntu.com/15965123/
<smoser> so that seems like it should work.
<Odd_Bloke> smoser: Yeah, I've tested it; it works as I expect.  Just checking if there are gotchas that I'm not seeing. :)
<smoser> Odd_Bloke, if you dont have a reason, i'd prefer 2 spaces for indentation levels rather than 4. but i dont have a good reason as to why.
<smoser> (in the template you're writing)
<Odd_Bloke> smoser: The rest of the lxd templating stuff is YAML using 4 spaces, so I'll stick with that.  I'll add you as a reviewer once I have something. :)
<smoser> its just obnoxiously wide. but consitency is better.
<Odd_Bloke> rharper: So were you earlier saying that I probably wanted 'control: auto' for both cases?
<smoser> what we really need though is for
<smoser> %{network_config.yaml}
<smoser> or soemthign to that affect.
<smoser> i geuss, need to declare the version that we want also
<smoser> but.
<zgredeq> Hello all, will someone give me a helping hand? tl;dr: cloud-init + multiple nics brings up only first interface. Metadata service is available and reachable (openstack).
<Odd_Bloke> rharper: (Did you see my earlier Q?  "So were you earlier saying that I probably wanted 'control: auto' for both cases?")
<rharper> Odd_Bloke: sorry, if I missed it; yes; control: auto means that network target will attempt to make sure any 'auto' iface is up before continuing;
<Odd_Bloke> OK, cool.
<rharper> what's not clear to me, without testing, is in the link-local case; what ifupdown does with auto
<rharper> basically ifquery will return the iface name if it's marked auto; and then I think ifup would need to touch the file in /run/network/$Iface.XXX to know that its up;  if ifup on a linklocal doesn;t do that
<rharper> then it'll fail and cause cloud-init not to continue (no network!)
<Odd_Bloke> rharper: OK, I'll investigate what happens. :)
<rharper> so we need to confirm that either manaul is fine (and the network comes up OK) or auto (and that ifup on a link-local) does the right thing
<rharper> cool
<Odd_Bloke> rharper: (I've seen that configuration there work as expected FWIW, but I'll see what auto does if you think it's more correct (if it works :p))
<rharper> Odd_Bloke: right, my thoughts as well
<smoser> hold on.
<smoser> you want auto
<smoser> control: auto
<smoser> on both.
<smoser> its very odd actually.
<smoser> i think there is a bug in ifupdown there.
<smoser>  iface eth0 inet manua
<smoser> wiuthout 'auto eth0' or 'allow-auto eth0'
<smoser> should not show the device as up
<smoser> what a mess.
<smoser>  http://paste.ubuntu.com/15966891/
<smoser> rharper, ^ . so if you put 'manual' then it comes up auto
<smoser> which i think i a bug
<smoser> the 2 calls i realize now are because i had an 'eth0.cfg' file , which we're fixing :)
<rharper> that sounds like a bug
<rharper> pretty sure manual means (don't up this yourself)
<smoser> yeah, so part of htat above is user fail
<smoser> i had multiple stanzas
<rharper> ah
<rharper> hehe
<smoser> actually it odd that auto eth0
<rharper> yes
<smoser> auto eth0
<smoser> auto eth0
<smoser> brings it up 3 times
<smoser> i think
<smoser> but ignore that.
<smoser> just putting this in:
<smoser> iface eth0 inet manual
<smoser> means that the interface *is* up (listed in ifconfig without -a)
<smoser> but no ifup hooks are called
<rharper> smoser: hangout
<smoser> yeah, so that said, we should declare it auto
<rharper> agreed, Odd_Bloke is testing that it DTRT for link-local
<Odd_Bloke> It looks like it DTRT with auto.
<smoser> i think you end up with link local with 'manual' with or without auto
<smoser> which i kind ofthink is a bug
<rharper> interesting
<smoser> rharper, opened https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1573112
<rharper> ok
<smoser> Odd_Bloke, https://code.launchpad.net/~daniel-thewatkins/cloudware/+git/cpc_build_tools/+merge/292557 just commented there. sorry didnt' look earlier
<smoser>  http://paste.ubuntu.com/15970544/
<Odd_Bloke> smoser: Oh, shoot, I copy-pasted from an old version on my test box.
<Odd_Bloke> smoser: Thanks. :)
<smoser> Odd_Bloke, what do you think about 'copy' ?
<smoser> i really think you dont want it
<smoser> as interfaces.tpl didnt have it.
<Odd_Bloke> smoser: Right, but all the other cloud-init bits _do_ have it.
<Odd_Bloke> smoser: So I was on the fence.
<smoser> and for 16.04 we should kill those upstart.X files
<Odd_Bloke> smoser: Right; let's get networking fixed first though. ;)
<smoser> what does 'copy' mean ?
<Odd_Bloke> smoser: I believe it means that the file will be regenerated when a container is copied to become a new container.
<smoser> that the rendered file should be coied when the image is copied (from inside the instance) or that the template shoudl be copied.
<Odd_Bloke> But I mostly cargo-culted it.
<smoser> hm....
<Odd_Bloke> smoser: It was more important when we were using the MAC address in there, but it turned out we didn't need it, which simplified matters.
<smoser> yeah, but only "mostly"
<smoser> :)
<smoser> well, with or without the mac, it would need to be re-rendered when you start a new image
<smoser> right ?
<Odd_Bloke> Right, because network_mode might have changed.
<smoser> right.
<smoser> so whatever means *that* is what we want
<Odd_Bloke> smoser: So in lxd/container_lxc.go, at line 2677
<Odd_Bloke> FAIL
<Odd_Bloke> At line 2702, the trigger is compared against the "when" list and if it's found the template is run.
<Odd_Bloke> So I believe having 'copy' there is what we want.
<Odd_Bloke> It looks like the triggers used are "start", "create", and "copy".
<Odd_Bloke> Bah, I accidentally cleaned up my test host.
<smoser> i still dont knwo what 'copy' means.
<smoser> start and create somewhat make sense to me
<Odd_Bloke> smoser: 'copy       - Copy containers within or in between lxd instances.'
<smoser> and i had strgraber ask you all to change to just 'create' rather than 'start'
<smoser> live?
<smoser> or images
<Odd_Bloke> I think that would be 'move       - Move containers within or in between lxd instances.'
<smoser> i still dont understand what it means. why would i care about rendering a file on 'copy'
<smoser> if i'm going to have to 'start' or 'create' it on the other side after i 'copy' it.
<Odd_Bloke> smoser: But you might not want to regenerate a template for _every_ start.
<smoser> that does make some sense, yes.
<smoser> (although i doubt you really want it as if your container goes down unintended it will get re-rendered , in addition to explicit 'stop' and 'start')
<smoser> but thats beside the point.
<smoser> we do not want 'start'. we want create.
<Odd_Bloke> Yep.
<smoser> hm..
<smoser> why is that mp private ?
<smoser> maybe i should not ask that in public :)
<smoser> but we probably need to ask stgraber what we want to do here.
<smoser> and we could/should do that in lxc-dev.
 * larsks wonders why there is a set_hostname method in the Azure data source.
<Odd_Bloke> smoser: Yeah, I was going to get stgraber to review as well.
<Odd_Bloke> But I'll pick this up tomorrow morning; I want to finish off the last release bits I have and then EOD. :)
<larsks> Odd_Bloke: it looks like you are responsible for large chunks of the azure data source? If that's true, got a moment for a couple of questions?
<Odd_Bloke> larsks: Drop me an email at daniel.watkins@canonical.com?
<larsks> Odd_Bloke: Sure, if that's your preference.
<Odd_Bloke> larsks: Yeah, I'll grab you on here once I've read through.
<Odd_Bloke> larsks: (I'm UK based, so I'm not really here ATM)
<Odd_Bloke> *disappears in a cloud of smoke*
<larsks> Odd_Bloke: I figured from your EOD comment earlier :)
#cloud-init 2016-04-22
<Odd_Bloke> smoser: I've updated https://code.launchpad.net/~daniel-thewatkins/cloudware/+git/cpc_build_tools/+merge/292557
<Odd_Bloke> smoser: Just waiting on your +1 in that MP now. :)
#cloud-init 2017-04-19
<rharper> smoser: http://netdevconf.org/2.1/session.html?hemminger
#cloud-init 2017-04-20
<smoser> rharper, no 'conclusions'
<rharper> smoser:  if I have a yaml file in /etc/cloud/cloud.cfg.d/50-foo.cfg ;   where does that show up object-wise?  sys_cfg ? datasource ?
<smoser> well, it deepnds.
<smoser> config modules are handed the entire config after merging in user-data
<smoser> datasources are handed the system config (config['system_info'])
<rharper> so in Init object, doing find_network_config;  it's not always clear to me what files feed into which obj, the command line is obvious but not sure how sys_config vs. ds config are composed
<smoser> thats wrong.
<smoser> datasources get the whole config.
<rharper> ok, but files in cloud.cfg.d  get slurped into what ... sys_cfg ?
<smoser> in the init object, they're part of self.cfg
<rharper> ok
<rharper> ok, perfect
<rharper> that's what I needed
<rharper> I'm missing a leading 'network' key in my .d cfg file
 * rharper will write up a section on how to feed network config in from various locations 
#cloud-init 2017-04-21
<smoser> powersj, so obviously my general inability to keep up with things is the cause of the problem.
<smoser> but https://code.launchpad.net/~syed1/cloud-init/+git/cloud-init/+merge/322024
<smoser> links to a bunch of failures that dont exist
<smoser> anything we could do about that ?
<powersj> Looking
<powersj> 1. Easiest is to trigger a rebuild or 2. I can up the amount of runs we keep even higher. That doesn't guarantee this won't happen again though.
<smoser> even the rebuild button doesnt work :)
<powersj> O.o
<smoser> cause it points to the same build
<smoser> which had the context of which branch was built and such
<powersj> hmmm
<powersj> increasing the number of jobs we keep may not even help without increasing it greatly. With 25 we only have 10 days worth
<powersj> I did manually kick a job off though
<smoser> its kind of pointless though really. i wonder we could reasonably post at least the failure portion of it to the log
<powersj> yeah
<smoser> blackboxsw, you were looking for a bitesize bug
<smoser> i dont have a bug (you can open one)
<smoser>  we use yaml
<smoser> http://stackoverflow.com/questions/27743711/can-i-speedup-yaml
<smoser> we dont use it extensively, and its even possible that loading the c code could have overhead for the small bits of yaml that we load...
<smoser> but i've wanted to look at that
<smoser> that make sense ?
<blackboxsw> smoser, sure reading that now.
<blackboxsw> yeah that makes sense
<blackboxsw> ok will file a bug about it as I test out whether there is a good performance imact.
<blackboxsw> *impact
<smoser> bug 1
<rharper> smoser: is there such a think as a daily-ppa but for fedora/centos cloud-init rpms ?
<rharper> larsks: ^^
<smoser> https://build.opensuse.org/ is the closest thing that i'm aware of
<rharper> maybe a lxd centos/fedora container calling build-rpm ?
<powersj> https://gist.github.com/smoser/19e65095b342e98fd4d6466e4d4fa1ac
<powersj> I've been using that locally to prototype running unittests and build tests for centos
<powersj> I think I made only a few slight changes
<rharper> powersj: nice!
<smoser> rharper, but honestly.. the suse build service migth be useful.
<smoser> i think it is basically the equivalent of ppas
<smoser> so we could have daily builds for rpm
<smoser> that anyone could easily use
<rharper> for fedora/centos  ?
<rharper> I assumed it was suse specific
<powersj> open build service does other distros
<rharper> cool
<smoser> rharper, all the touches to datasources/*.rst there...
<smoser> those just make targets for links ?
<rharper> smoser: I think so, I might need more context
<rharper> smoser: ah those touches were for anchors
<rharper> you're talking about the netconfig update
<rharper> you have to put a header in those files to allow link refs to work
<rharper> smoser: in particular, I wanted to document which network config was supported in which datasources
<rharper> and in the network doc link to that datasource documentation page
<rharper> since I was in the few that support network config; I wanted to touch all of them in case someone else wanted to use the refs later on
<smoser> rharper, yeah, i paste failed.
<smoser> meant to include link
<rharper> no worries, does the above make sense?
<smoser> yeah,  that sfine
 * smoser is out
<smoser> rharper, i commented on your mp
<smoser> it does look really good  thank you.
<rharper> ok
<rharper> sure
#cloud-init 2018-04-16
<mgerdts> @smoser - I pushed some updates for my merge proposal.  Not sure if I should have clicked "resubmit proposal".  Still learning the ropes with LP.  https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343118
<smoser> mgerdts: no. dont push 'resubmit' just git push over the top
<mgerdts> I did a git push.  Should I have squashed, then push -f?
<smoser> mgerdts: you dont have to squash. we squash when we merge.
<smoser> and generally, it is nice to  see commits that say "addressed feedback"
<smoser> ie, as you did it is really nice. thank you.
<mgerdts> :)
<mgerdts> The one thing I've not figured out is a portable way to deal with the byte comparisons, other than ord()
<mgerdts> I dislike having special code to help tests pass - there's probably a better way for me to do side_effects but I don't know what that is.
<smoser> mgerdts: ok. i can poke a bit anad see if i can improve that.
<mgerdts> thanks
<blackboxsw> looks like it's cloud-init status meeting time again
<smoser> o/
<blackboxsw> #startmeeting Cloud-init bi-weekly status meeting
<meetingology> Meeting started Mon Apr 16 16:01:49 2018 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<rharper> o/
<blackboxsw> Hi folks, welcome to cloud-init's bi-weekly status meeting. Feel free to interject at any time or bring up branches/bugs/questions over the next 30-60 mins. We'll have a number of folks around to get eyes and/or keyboards onto any problems.
<dpb1> o/
<blackboxsw> As always we'll go through the following topics (feel free to suggest others): Recent-changes, In-progress Development, and ~30 mins Office Hours)
<blackboxsw> #topic Recent-changes
<blackboxsw> a quick rundown of the hi level changes landed:
<blackboxsw> Prune integration test artifacts
<blackboxsw> Add support for LXD 3.0, fix pylxd integration test dependency
<blackboxsw> Fix Ubuntu proposed integration test CI job
<blackboxsw> Fix ec2 validation of instance-data.json network info
<blackboxsw> Do not retry optional userdata on 404 (LP: #1702160)
<blackboxsw> Add explicit cloud-init package dependency on isc-dhcp-client (LP: #1759307)
<ubot5`> Launchpad bug 1702160 in cloud-init "OpenStack datasource should not retry user-data on 404" [Medium,Fix released] https://launchpad.net/bugs/1702160
<ubot5`> Launchpad bug 1759307 in cloud-init (Ubuntu) "missing dependency on isc-dhcp-client (dhclient)" [Medium,Fix released] https://launchpad.net/bugs/1759307
<blackboxsw> additionally from most recent commits we have:
<blackboxsw> tools: Fix make-tarball cli tool usage for development
<blackboxsw> renderer: support unicode in render_from_file.
<blackboxsw> Implement ntp client spec with auto support for distro selection
<blackboxsw> Apport: add Brightbox, IBM, LXD, and OpenTelekomCloud to list of clouds.
<blackboxsw> tests: fix ec2 integration network metadata validation
<blackboxsw> tests: fix integration tests to support lxd 3.0 release
<blackboxsw> correct documentation to match correct attribute name usage.
<blackboxsw> cc_resizefs, util: handle no /dev/zfs
<blackboxsw> Last week rharper found and fixed a regression in zfs resize behavior that was blocking our ubuntu SRU
<blackboxsw> We have uploaded those fixes, as well as rharper's ntp spec changes (which should incorporate a number of robjo's opensuse/sles needs too)
<blackboxsw> anything else notable that I'm missing gentlemen?
<blackboxsw> If not, I'll jump to in-progress development
<blackboxsw> #topic In-progresss Development
<blackboxsw> So, on the ubuntu side of the house we are about to approve the cloud-init 18.2 SRU (Stable release update) into xenial and artful. Just one more validation run and we should be good to see 18.2.4 on xenial, artful. Ubuntu Bionic is already a few commits beyond that.
<blackboxsw> On Ubuntu as well we are beating the drop to the Bionic LTS (Long term release) feature/bug freeze.
<blackboxsw> This week marks the last week for use to get fixes into Bioinic images before that release is cut.
<blackboxsw> so we'll be heads down on any Bionic-specific changes that need to get in.
<blackboxsw> Feel free to checkout our trello board @
<blackboxsw> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
<blackboxsw> we track all tasks we are working on in public view there so if there are any questions you can ping one of us here about our development efforts
<blackboxsw> additional tasks that are in flight:  bash-autocompletion for cloud-init CLI (rhaper).  dropping ifconfig and route in favor of 'ip' (bbsw), and moving openstack datasource to cloud-init's local stage
<blackboxsw> (smoser)
<blackboxsw> also a couple of bugs to fix such as #1570997
<blackboxsw> #link https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1759406
<ubot5`> Ubuntu bug 1759406 in cloud-init (Ubuntu) "sru cloud-init (17.2-35-gf576b2a2-0ubuntu1~16.04.1 update to 18.2-4-g05926e48-0ubuntu1)" [Medium,Confirmed]
<blackboxsw> oops paste fail:
<blackboxsw> #link https://bugs.launchpad.net/bugs/1570997
<ubot5`> Ubuntu bug 1570997 in ssh-import-id (Ubuntu Xenial) "fail if HOME environment variable is not set" [Low,Fix committed]
<blackboxsw> I think that about wraps in-progress development, anything else that should be noted smoser rharper ?
<rharper> I think you covered it
<smoser> ssh-import-id is not relally at all related to cloud-init
<smoser> thanks blackboxsw
<blackboxsw> oops grabbed the wrong one, was thinking about this card
<blackboxsw> #link https://trello.com/c/JVaXSfpo/749-eol-fix-for-ssh-file
<smoser> right.
<blackboxsw> also of note, in some of our SRU testing we found time-tracking gaps in cloud-init analyze tracking on Azure. rharper put of a logging tracker fix to avoid those tracking gaps
<blackboxsw> #link https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/343123
<smoser> i'll point out one thing i just finished up with...
<smoser> for testing ubuntu, the https://github.com/cloud-init/ubuntu-sru/ has 'get-proposed-cloudimg' and 'lxc-proposed-snapshot'
<smoser> which now work more like each other.l and can do more than just upgrade cloud-init.
<blackboxsw> thanks smoser. Great tools to quicken dev-test cycles and make cloud-init development easier. That wraps up what we've been up to. We can probably move to the open forum for any discussions folks want to have
<blackboxsw> #topic Office Hours (next ~30 mins)
<blackboxsw> We'll be hanging out here for anyone who wants more eyes on a review, feature discussions or bug triage....
<smoser> mgerdts: i'm poking at the branch i think i shoudl be able to get something.
<mgerdts> I'm working on a bunch of fixes for things that have turned up on bhyve with SmartOS.  Since we are looking to transition from KVM to bhyve, we will need to provide updates at least as far back as xenial and probably trusty.  Is the process for this any more complicated than get the fixes in master, then cherry-pick the fixes into branches?
<mgerdts> thanks @smoser
<rharper> mgerdts: we preferrer not to cherry; rather we release master back to xenial via our SRU (Stable Release Update) process; however, we spend a lot of effort to not modify existing behavior on prevlous SRU releases;  so if the changes to support bhyve can be done in a compatible way (working with either) that'd be best;  worst-case, we patch in release specific bahvior into the release branch.
<mgerdts> Pretty much everything that I've got queued up is fully compatible.
<mgerdts> https://bugs.launchpad.net/cloud-init/+bug/1667735 implements proper protocol negotiation over the serial port.   The lack of this has caused problems with KVM at times too.
<ubot5`> Ubuntu bug 1667735 in cloud-init (Ubuntu Trusty) "cloud-init doesn't retry metadata lookups and hangs forever if metadata is down" [Medium,Confirmed]
<mgerdts> https://bugs.launchpad.net/cloud-init/+bug/1746605 adressess times when cloud-init and other software may be trying to use the metadata serial port at the same time.  This is purely a bug fix.
<ubot5`> Ubuntu bug 1746605 in cloud-init "DataSourceSmartOS needs locking" [Medium,Confirmed]
<mgerdts> I hit it when rc.local and cloud-init were both trying to get metadata.
<mgerdts> https://bugs.launchpad.net/cloud-init/+bug/1763480 makes it so that cloud-init doesn't stack trace and exit when there is no customer_metadata.  This is an unlikely case, but something that is hit when you are testing things that don't need ssh keys, etc.
<ubot5`> Ubuntu bug 1763480 in cloud-init "DataSourceSmartOS list() should always return a list" [Medium,Confirmed]
<mgerdts> https://bugs.launchpad.net/cloud-init/+bug/1763512 finishes off the partial implementation of sdc:routes support.  Previously, we didn't publish the required information to VMs, so it is fair to consider this a new feature.
<ubot5`> Ubuntu bug 1763512 in cloud-init "DataSourceSmartOS ignores sdc:routes" [Medium,Confirmed]
<mgerdts> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1763511 is probably the most incompatible change.  New ephemeral disks will get ext4 instead of ext3, which is needed for larger disks that seem to be getting more common.
<ubot5`> Ubuntu bug 1763511 in cloud-init (Ubuntu) "DataSourceSmartOS should default to ext4" [Undecided,New]
<mgerdts> I think that's all that I have in the works right now.  Do any of these sound like they would be problematic as far back as Xenial?
<rharper> new features are OK ,bug fixes are fine as well; I think that different filesystem is somewhat tricky
<smoser> i think i'm generally ok with the different filesystem.
<smoser> if the reason is that simply ext3 can't handle super-big
<rharper> yeah, that doesn't seem so user-visible w.r.t configuration;
<smoser> we could ensure being more backward compat if we checked the size of the disk and made an ext4 if > that-size
<smoser> then there'd less issue
<smoser> but more complication and future we'd be stuck with that
<rharper> yeah, it wouldn't have worked on ext3 then it would be fine to use ext4
<smoser> so i'd rather really just bite the bulleet
<rharper> it could be a metadata flag that the Datasource looks for
<smoser> rather than describing to pepole forever "well, if your  disk is < X you'll get ext3 otherwise ext4"
<blackboxsw> rharper: got time for a netplan global dns hangout?
<rharper> y
<blackboxsw> I want to make sure I'm reading the tea leaves right
<smoser> mgerdts: http://paste.ubuntu.com/p/5qdtFzY8w7/
<blackboxsw> ok rharper I'm in cloud-init hangout
<smoser> that makes tests pass. and i think the changes to the code path are right
<rharper> ok
<rharper> brt
<smoser> it is still hacked in a sense that the response only deals with fp.read(1) rather than possibly anything that read more than 1.
<cyphermox> blackboxsw: rharper: what's this about netplan global dns?
<rharper> cyphermox: converting network v1 syntax from maas into something that works with netplan which doesn't have "dns" unbound to any interfaces
<cyphermox> ok
<rharper> maas I believe has fixed this for 2.4.x
<rharper> they no longer will emit the type: nameserver  but legacy maas would have that, so we've a branch that stuffs them in reasonable places under defined interfaces which don't already have DNS values
<blackboxsw> cyphermox: just SRU validation w.r.t. https://bugs.launchpad.net/cloud-init/+bug/1750884
<ubot5`> Ubuntu bug 1750884 in cloud-init "[2.4, bionic] /etc/resolv.conf not configured correctly in Bionic, leads to no DNS resolution" [Medium,Fix released]
<cyphermox> all good
<blackboxsw> ooops, and /me forgot the end the epic meeting
<blackboxsw> thx rharper for the chat
<blackboxsw> #endmeetiung
<blackboxsw> #endmeeting
<meetingology> Meeting ended Mon Apr 16 18:11:33 2018 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2018/cloud-init.2018-04-16-16.01.moin.txt
<blackboxsw> e
<rharper> y
<mgerdts> smoser: everything checks out on my end with your change.  I've pushed an update to LP.  Thanks!
<blackboxsw> ok just finished SRU verification
<blackboxsw> officially.
<blackboxsw> I'm adding the verification-done tags now
<rharper> \o/
<smoser> mgerdts: still there?
 * mgerdts here
<smoser> ok. i'm on 62cafc5080ce3a21180770b65c43bb0964cdd70d of your (i think out of date branch)
<smoser> i paste-failed earlier
<smoser> http://paste.ubuntu.com/p/MtHPY4PKRb/
<smoser> thats what i have now. i have been working on it since.
<smoser> http://paste.ubuntu.com/p/Qq9YZkfJQY/
<smoser> that one with flake8 fixes
<smoser> http://paste.ubuntu.com/p/kbhVHvJHCs/ <-- and that with one fix
<mgerdts> ok, is it this last one I should be grabbing?
<smoser> yeah.
<smoser> i can paste files too if thats easier
<mgerdts> Is there a trick to make curl work with paste.ubuntu.com?
<smoser> not really
<smoser> here https://hastebin.com/raw/exukejojec
<smoser> https://hastebin.com/raw/fagotonixu
<mgerdts> I'm fine with (and prefer) patches.  It's just the download to my mac, scp where I need it to be, then run patch workflow that is tedious.
<mgerdts> curl | patch is much eaiser
<smoser> mgerdts: patch raw https://hastebin.com/raw/damujotohe
<smoser> sorry. went afk for a bit
<mgerdts> no worries.. I had it worked out already.  Was just reading through trying to make sense of it all.
<smoser> i think its fairly straight forward. the reader just returns the bytes it has. like BytesIO
<smoser> but allows the 'end_byte' to indicate an EOF. so as to get your short readds in there.
<mgerdts> Yeah, mostly so.  It looks as though it has more features than are strictly needed and I was just looking to see if there was something I was missing.
<smoser> i did the ShortReader rather than what i initially had because previously only 'read(1)' would work. if we changed the implementation it'd be busted.
<smoser> yeah... i knwo.
<mgerdts> I think we pretty much have to stick with read(1), as we don't have a way to put bytes that shouldn't have been read back.  But maybe that'
<mgerdts> s not a big deal
<smoser> yeah.
<mgerdts> tox is happy, and it still seems to work in a VM.
<mgerdts> I've pushed this change up to LP.
<mgerdts> thanks, again, for your help with this.
<mgerdts> from my read of the docs, I should be able to put cloud-config into cloud-init:user-data and that should override defaults.   But that seems not to work, at least for disk_setup and fs_setup.  Any idea where I might be going wrong?  https://hastebin.com/ovopajituh.php
<rharper> mgerdts: do you have your cloud-init.log ?
<mgerdts> https://hastebin.com/oxekuluhum.sql
<mgerdts> I guess it is sql because it found 'blob' in it somewhere.
<rharper> looks like yourr layout isn't correct
<rharper> you have a boolean and it should be a list of partition sizes
<rharper> oh, I see
<rharper> the single partition
<rharper>  Failed to find device during available device search.
<rharper> 2018-04-16 21:13:42,054 - cc_disk_setup.py[DEBUG]: Automatic device for /dev/vdb identified as None
<rharper> 2018-04-16 21:13:42,054 - cc_disk_setup.py[DEBUG]: No device aviable that matches request.
<rharper> it didn't find your /dev/vdb
<mgerdts> Doing a fresh run... one minute
<mgerdts> https://hastebin.com/wexarotilu.go
<rharper> looking
<mgerdts> changed layout and overwrite to false.  cloud-init clean, rm /var/log/cloud-init*
<mgerdts> then halted the vm, rolled-back the zvol under vdb to its initial state (all zeroes), then booted it.
<rharper> that looks good
<rharper> cc_disk_setup.py[DEBUG]: Creating file system ephemeral0 on /dev/vdb
<rharper> 2018-04-16 21:53:25,025 - cc_disk_setup.py[DEBUG]:      Using cmd: ['/sbin/mkfs.ext3', '/dev/vdb', '-L', 'ephemeral0', '-F']
<rharper> 2018-04-16 21:53:25,025 - util.py[DEBUG]: Running command ['/sbin/mkfs.ext3', '/dev/vdb', '-L', 'ephemeral0', '-F'] with allowed return codes [0] (shell=False, capture=True)
<rharper> 2018-04-16 21:53:25,402 - util.py[DEBUG]: Creating fs for /dev/vdb took 0.386 seconds
<mgerdts> but I specified xfs
<rharper> oh, fs isn't right though, you wanted xfs
<rharper> hrm
<mgerdts> It looks like I'm specifying it correctly but it's just being mishandled, right?
<rharper> I don't know what's getting passed to mkfs, but it appears that it's not the fs_setup entry
<mgerdts> 2018-04-16 21:53:25,011 - cc_disk_setup.py[DEBUG]: Partitioning disks: {'/dev/vdb': {'_origname': 'ephemeral0', 'overwrite': False, 'table_type': 'mbr', 'layout': False}}
<mgerdts> that comes some time after it shows that it picked up the config that I set in user-data
<rharper> hrm, lemme check something
<mgerdts> notice that it says mbr rather than gpt
<rharper> y
<rharper> this smells like user-data from DataSource isn't getting updated
<rharper> hrm, let's see if we can find it in object; on the instance: # python3 -c "from cloudinit import stages; import cloudinit; cloudobj = cloudinit.stages._pkl_load('/var/lib/cloud/instance/obj.pkl'); print(cloudobj.userdata_raw)"
<mgerdts> device_aliases: {'ephemeral0': '/dev/vdb'}
<mgerdts> disk_setup:
<mgerdts>     ephemeral0:
<mgerdts>          table_type: gpt
<mgerdts>          layout: False
<mgerdts>          overwrite: False
<mgerdts> fs_setup:
<mgerdts>     - label: ephemeral0
<mgerdts>       filesystem: xfs
<mgerdts>       device: ephemeral0.0
<rharper> I see your debug message
<rharper> but I want to see if it's recorded into the object; the 'ud' param in the datasource is kepted as self.userdata_raw
<rharper> that's referenced in the cloudify() stage which merges config before calling modules
 * rharper steps away for a bit 
<mgerdts> yay, figured it out
<mgerdts> needs to start with '#cloud-config`
<mgerdts> and with that, I'm stepping away too.
<mgerdts> thanks for your help rharper
#cloud-init 2018-04-17
<smoser> mgerdts: so for bug 1763511
<ubot5`> bug 1763511 in cloud-init (Ubuntu) "DataSourceSmartOS should default to ext4" [Undecided,New] https://launchpad.net/bugs/1763511
<smoser> just curious
<smoser> you have disks > 116 TiB
<smoser> oops. 16 TiB
<smoser>  ?
<mgerdts> Yeah, we are starting to need some big disks.
<mgerdts> :)
<mgerdts> Whether that is the optimal storage layout or not is another question.
<mgerdts> On the backend it is zfs, spread across many spindles
<smoser> that was my next question.
<mgerdts> I guess a TiB isn't what it used to be.
<smoser> 16TiB ought to be enough for anyone.
<mgerdts> VP of engineering here disagrees.
 * smoser was adjusting a quote of a former CEO at microsoft
<mgerdts> Yeah, a few words into my next reply, I caught that.  :)
<mgerdts> 16 TiB is the new 640 KiB
<mgerdts> Time to make a t-shirt.
<mgerdts> And refill the coffee cup
<mgerdts> smoser: now that I have your approval on a couple merge proposals, what's next for them?
<smoser> i'll get them pulled in today.
<mgerdts> great, thanks.  Just wasn't sure if there was any process step that I needed to complete first.
<paulgrmn> Has anyone been able to get the nocloud data source to work with the OpenStack qcow2 images provided by Debian? Do they not support nocloud?
<smoser> paulgrmn: https://asciinema.org/a/132009
<paulgrmn> I followed that, but can't get it to work with the latest images. I also don't see cloud-init looking for the NoCloud datasource on the console during virt-install.
<smoser> paulgrmn: hm.. well, it worked in 20170725
<smoser> i havent tried since. its possible they disabled that... but one woudl hope not
<paulgrmn> Yeah, that's what I was wondering --- if it was a regression or if I'm goofing up something.
<smoser> paulgrmn: you used https://cdimage.debian.org/cdimage/openstack/current/debian-9.4.3-20180416-openstack-amd64.qcow2
<smoser>  ?
<paulgrmn> I would assume my error if I saw it try "DatasourceNoCloud" and fail, but it doesn't seem to even look for it.
<paulgrmn> I used https://cdimage.debian.org/cdimage/openstack/current-9/debian-9-openstack-arm64.qcow2
<smoser> arm ?
<paulgrmn> Derp. No. amd64
<paulgrmn> https://cdimage.debian.org/cdimage/openstack/current-9/debian-9-openstack-amd64.qcow2
<paulgrmn> Which has the same MD5sum as the one you linked.
<smoser> alright. i just verified it "worked for me"
<paulgrmn> OK. Thanks a lot. Must be something I'm doing wrong.
<smoser> here.
<smoser> git clone https://gist.github.com/smoser/635897f845f7cb56c0a7ac3018a4f476 boot-test
<smoser> cd boot-test
<smoser> ./check-dependencies
<smoser> cloud-localds -v --network-config=network-config-v1.yaml            seed.img user-data.yaml meta-data.yaml
<smoser> ./boot ../debian-9.4.3-20180416-openstack-amd64.qcow2 seed.img -snapshot
<smoser> then i logged in as 'debian' and 'passw0rd'
<paulgrmn> Thanks for your help. I'll give it a try.
<smoser> h
<smoser> blackboxsw: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342428 didnt like your code.
<smoser> and rharper had mentioned he used timit on the socket.inet_ntoa
<smoser> i'd prefer  not to have 2 implementations of that
<smoser> (in annother of my coments there)
<smoser> regex...
<smoser> i'll be back in ~ 90 minutes and review more.
<blackboxsw> hab smoser , I need to rebase
<blackboxsw> pushing now
<blackboxsw> <smoser> and rharper had mentioned he used timit on the socket.inet_ntoa.   Yeah I addresssed that comment and pushed the common logic into cloudinit.net.network_state.net_prefix_to_ipv4_mask. So there is no cloudinit.netinfo.netdev_cidr_to_mask anymore
 * blackboxsw runs this branch on ec2 and openstack just to see that the output looks good
<rharper> nice
<smoser> blackboxsw: oh. good. thanks. sorry. i didnt notice.
<smoser> err.. read that wrong
<blackboxsw> +1, the branch kinda started to get bigger than I'd prefer. ohh well.
<smoser> its really not so bad. you did a good job
<smoser> blackboxsw: i posted some feedback there.
<blackboxsw> smoser: rharper wrapping up feedback. will have something up tonight on this thanks
<smoser> blackboxsw: i'll be back in to check later.
#cloud-init 2018-04-18
<rharper> smoser: shall I land https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/343122  ?
<blackboxsw> I'm for  it as mentioned, I think we can do a followup with something dynamic if needed.
<blackboxsw> I'd like us to continue spending time on the CLI, so there will be ample opportunity if needed to touch this again and make it dynamic at that point.
<blackboxsw> as it is for now it's fast and correct and 'done' so I'm all for that
<smoser> rharper: yeah.
<blackboxsw> we can borrow  some simple regex cli --help parsing that landscape-api CLI did for the next iteration, it won't be too bad to pull in.
<smoser> i consider myself more competant than one should be with shell scripting
<smoser> and compgen stuff just makes me cringe
<blackboxsw> seen here is another option. but yeah uses compgen too https://bazaar.launchpad.net/~landscape/landscape/trunk/view/head:/canonical/landscape/api/client/etc/bash_completion.d/landscape-api
<blackboxsw> and yeah, I'm all for something simple if we could get away with it
<blackboxsw> smoser: sorry I'm late on the review comments of ifconfig->iproute2 stuff, still massaging unit tests for the switch to look before you leap (i.e. which('ip') elif which('ifconfig') )
<blackboxsw> FTR, I agree with the approach to testing for the cmd before running it. instead of all the CalledProcessError handling
<smoser> blackboxsw: yeah, i knew that that was going to be a pain for your tests
<smoser> one way or another we shoudl decide "we are using 'ip'" and then use it
<blackboxsw> yeah right, instead of separate tests in each separate function (ip vs netstat    or  ip vs ifconfig)
<blackboxsw> yeah just a has_iproute2 would be easy, only one thing to mock then.
<blackboxsw> I might just make that change now, as it's too late to get this fixed and back to you for review tonight anyway
<blackboxsw> meh, n/m I retract that statement, as we also check which(netstat) and which(ifconfig) and have different error messages/failure paths to handle, so the consolidation doesn't buy much as we'd have to still handle ProcessExecutionErrors and log warnings if the 2nd command we run isn't present anyway
<blackboxsw> ahh well
<smoser> freebsd uses ifconfig i believe ?
<smoser> other than that i would be fine to assume that if you have netstat then you have ifconfig
<smoser> they';re' both from net-tools
<rharper> smoser: ok, pushing bash completion
<rharper> blackboxsw: Yeah, regex parsing is one way though I think the parsers are relegated to other files, so we can't just grep through cmd/main.py;  the recursive --help invokation may be more robust and we can do that at build-time so we have a static bash file.
<blackboxsw> +1 rharper we can create a ./tool to gen that static file from recursive --help invocations
<blackboxsw> good pt
<blackboxsw> could do it at build time. or just run it on a branch in progress prior to commit
<smoser> blackboxsw: have a minute ?
<blackboxsw> yes definitely
<blackboxsw> smoser: heading to hangout
<blackboxsw> rharper: smoser dropped Metric column from route info output to retain original behavior. Only differences in behavior to what currently exists in bionic when relying on ip instead of ifconfig/netstat are that we do the following:
<blackboxsw> 1. properly print full scope name for addrs as presented by ip
<blackboxsw> 2. drop trailing ':' in the device column
<blackboxsw> 3. print ipv6 addrs instead of '.'
<blackboxsw> 4. print an ipv6 routes table
<blackboxsw> smoser: rharper the diff on an ec2 instance: https://pastebin.ubuntu.com/p/KZWYPbCRvC/
<smoser> are we consistent with o urselves now ?
<smoser> between ifconfig and ip
<smoser> blackboxsw: my freebsd seems not working on server stack :-(
 * blackboxsw is trying freebsd on ec2
<rharper> blackboxsw: can you run ec2 with ipv6 addr so we can see the diff?  I didn;t think we displayed link-local ip6 addrs; and we don't split out v4 and v6 for netdev, but we do for route ?
<blackboxsw> rharper spinning one up now
<rharper> blackboxsw: w.r.t the interface to the various parse output methods;  as smoser has told me, if you don't want someone using the modules as an api, then you need to use _ prefix
<rharper> otherwise, those methods in netinfo *are* an api, and that means callers could pass non-strings to you
<blackboxsw> +1, I should change that
<rharper> if smoser doesn't mind, then I'm OK with not adding checks on the inputs, but it seems reasonable to me to set those as input=None, and then add if not input: input = ""
<rharper> or something safer;
<smoser> why set a default ?
<smoser> they require input
<rharper> one way of ensuring you don't blow up doing .split() on a non string for typical usage, None is more comment than passing in a dict;  or raise valueerror on the wrong type
<smoser> rharper: well, we really aren't providing an api. we're just not really doing that.
<rharper> smoser: then it I think blackboxsw will end up marking the methods with _
<smoser> we should, and ideally would. but just really are not.
<smoser> rharper: there is no reason to provide a default variable
<smoser> changing
<smoser>  def netdev_info_iproute(ipaddr_out):
<smoser> to
<smoser>  def netdev_info_iproute(ipaddr_out=None)
<smoser> is contrary to useful
<rharper> sure
<smoser> raising a TypeError if ipaddr_out is not a string is fine.
<blackboxsw> yeah +1 not defaulting as we don't want users calling netdev_info_iproute(). it needs something to parse
<blackboxsw> TypeError I balked at because our code is the only caller and it stuffs in util.subp output which is a known quantity of a string type
<smoser> right.
<smoser> so that is why i didnt' feel we need it.
<blackboxsw> so it's a case where I didn't feel users could likely induce a failure path by presenting an invalid type
<smoser> at this point we're not *really* providing an api.  no one that is using cloud-init as a python library (at thsi point) is surprised by that :)
<smoser> that isnt that we shouldnt. just that we're imature.
<blackboxsw> if we were, say, processing user-provided cloud-config information which we've loaded into a dict, then I'd be more concerned about checking types of data etc.
<smoser> yeah, i largely agree with that.
<smoser> but i'm also OK to just shove a _ on it too.
<blackboxsw> yeah I'll do that _ as I feel like for our own sanity it's nice to designate what we think should really have internal (to the module) callers
<blackboxsw> it's a good function/method documentation policy
<mgerdts> smoser: thanks for landing those
<blackboxsw> rharper: ec2 w/ ipv6 address
<blackboxsw> https://pastebin.ubuntu.com/p/96qtpRpdxV/
<rharper> thx
<rharper> so, the link-local is new, as is the lo in the routing v6 table
<rharper> should we clean those up?  unreachable seems unhelpful, on the fence on link-local ipv6 values, we're not showing link-local v4 stuff
<smoser> link local is new ?
<rharper> printing link-local values in the net dev info table
<rharper> yeah, look at the 17.2 vs 18.2
<blackboxsw> yeah it was previously just .
<blackboxsw> just absent I mean
<smoser> blackboxsw: so what are you doing with all that ?
<smoser> are we looking to make output identical to previous ?
<blackboxsw> smoser: rharper I just wanted to see if you guys had concerns about that behavior and if you think we should avoid printing link local info like line 29 of https://pastebin.ubuntu.com/p/96qtpRpdxV/
<blackboxsw> and if line 27 and 30 should really be like 8 and 10
<blackboxsw> do we want to continue to not process/report certain data, even though I fixed what I felt was a bug in the ommission
<rharper> I dunno; my use of that table is mostly restricted to seeing what networking as applied, typically the ipv4/v6 addresses, I don't think for clouds many users will ahve a second instance in the same network where they could reach it via link local
<smoser> i dont think line 29 is bad.
<blackboxsw> also the whole table line 40+ (ipv6 routes) is 'new'
<smoser> and i'm fine with 27 and 30 too
<smoser> i think they're generally improvements.
<smoser> blackboxsw: for the ipv6 output
<smoser> i think there is no reasno to show routes on interface lo
<smoser> (they're not seen on ipv4)
<blackboxsw> +1 will drop them
<blackboxsw> for SRU fodder, would we want to drop the ipv6 route table altogether
<blackboxsw> ?
<blackboxsw> or it's just more output that users could take advantage of (not really change in existing functionality presented by other tables)
<smoser> i think its fine. idont really believe any tools are likely scraping that.
<blackboxsw> cool sound good rharper ?
<rharper> blackboxsw: yeah, I agree with smoser
<smoser> blackboxsw: can i help you with anything ?
<blackboxsw> all good. just pushed all changes
<blackboxsw> was going to make deb one more time and test lxc and ec2 while waiting for ci
<blackboxsw> smoser: rharper all done
<rharper> nice
<blackboxsw> will pastebin one more pass
<smoser> rharper: i have updates to https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343246 and responses to your comments
<blackboxsw> done smoser rharper https://pastebin.ubuntu.com/p/dK8JBmdpNk/
<blackboxsw> looks good
<blackboxsw> I think we can land it and I'll put up a merge proposal for bionic if that sounds good?
<blackboxsw> https://jenkins.ubuntu.com/server/job/cloud-init-ci/1025/ in progress
 * blackboxsw will be back in 30 mins
<smoser> https://www.diffchecker.com/9sLtnCgT
<smoser> blackboxsw: i put a patch on there. https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/342428
<blackboxsw> waiting on ci
<smoser> blackboxsw: marked approved
<blackboxsw> smoser: sorry power loss. I'm back up, landing and publishing
<blackboxsw> and cutting a bionic upload MP
<blackboxsw> smoser: it'
<blackboxsw> s up https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/343562
#cloud-init 2018-04-19
<smoser> blackboxsw: on its up
<blackboxsw> thanks smoser for checking back in
<smoser> blackboxsw: if you're bored
<smoser>  https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343575
<blackboxsw> always bored
<blackboxsw> looking now
<blackboxsw> just pushed some changes to qa-scripts and was going to hit review queue now
<blackboxsw> smoser: yeah I had just made the arbitrary decision that people probably wouldn't duplicate their specific commands.
<blackboxsw> I'm +1 on the concept. looking over the changes and will land it if I don't have objections
<smoser> blackboxsw: real world... i would not be surprised to see
<smoser>  apt-get update
<smoser>  apt-get install package1
<smoser>  apt-add-repository
<smoser>  apt-get update
<blackboxsw> apt-get update
<blackboxsw> :)
<blackboxsw> yeah
<smoser> or even just
<smoser>  apt-get update
<smoser>  echo sometimes that fails
<smoser>  apt-get update
<smoser> :)
<blackboxsw> btw generally I'm kindof excited that schema warnings showed up somewhere in our consumers.
<blackboxsw> even it if resulted in a bug in the schema being too strict. That's the kindof problem I'd like to have (albeit infrequently). Too strict vs too premissive.
<blackboxsw> Bug #1764264
<ubot5`> bug 1764264 in cloud-init "bionic cloud-init 18.2 WARNING Juju's 'runcmd' stanza" [Medium,In progress] https://launchpad.net/bugs/1764264
<mgerdts> Is there any way to tell cloud-init to reconfigure networking on every boot?
<rharper> mgerdts: hrm; I don't think so; not without clearing the instance-id
<mgerdts> I ask because as we transition from kvm to bhyve we have been planning to use cloud-init for network configuration on all platforms.  Previously, those that didn't use cloud-init would rely on a dhcp server that is built-in to qemu.  This means that network config would get refreshed on every reboot (at least).
<mgerdts> This puts a bit of a wrinkle in the plan to rely solely on cloud-init.
<mgerdts> Is this something that would be hard to make into a tunable, presumably able to be overridden by user-data?
 * blackboxsw looks too. hrm.
<mgerdts> apply_network_config:
<mgerdts> 645         if (self.datasource is not NULL_DATA_SOURCE and
<mgerdts> 646                 not self.is_new_instance()):
<rharper> yes
<rharper> mgerdts: I think we would need some way to ask the ds if it wanted to do that
<rharper> we already ask is_new_instance; we could add some other check
<mgerdts> I'll code something up and shop it around for feedback.
<rharper> sure
<rharper> mgerdts: I don't think we have a user-data level way to toggle that; is it something the users could set such that the instance metadata would show that value? that would avoid having to create a user-data namespace for this setting; but leave control over whether network config is apply every boot or per-instance as something controllable but not seen outside of the SmartOS datasource
<blackboxsw> yea, was coming to the same conclusion. We currently surface network: config: disabled  in /etc/cloud/cloud.cfg or /etc/cloud/cloud.cfg.d files
<mgerdts> It could be created as a property in customer_metadata, which would make it a peer to user-data or put it into the sdc: namespace.  I need to discuss internally how we should handle this.
<blackboxsw> and we check that in cloudinit/net/__init__.py:is_disabled_cfg(). ... would something like that be an option for setting 'autorenew' or 'update-per-boot'?
<rharper> blackboxsw: but user-data can't update system config
<mgerdts> That brings up another question I fielded recently.  I was asked if my cloud-init work would make it so that networking could automatically reconfigure without taking a reboot.
<rharper> so, I think it would better be a metadata value that the Datasource can reason able, but requires cloud support to allow users to toggle the setting
<blackboxsw> rharper: right I'm wondering if system config update is an option (i.e. we surface a system_config knob that can be twiddled)
<rharper> mgerdts: we want to get there, I shared with you that doc  that discussed a cloud-init refresh-metadata  and cloud-init update-config --mode=network
<rharper> https://hackmd.io/M1Tae41PQBC7a9qMsurTJw
<rharper> mgerdts: but a non-reboot would look like that
<mgerdts> oh, nice
<rharper> there are some challenges w.r.t live update of networking; some things are easy, some things can leave you without connectivity if they fail
<rharper> but generally things like, adding a ip, adding a new interface, etc applying new config to unconfigured interfaces; that's doable without anything fancy
<rharper> having to bounce the primary interface is more troublesome
<rharper> there may be running services or other things
<mgerdts> I think the question I had was more specifically about routes.  Like you say, monkeying with live interfaces can be problematic.  By default, we have anti-spoofing enabled which further complicates live changes.  Right now, an IP change that takes effect on a reboot doesn't have any complicated synchronization issues between host and guest.
<mgerdts> If you are going to break all of your network connections, taking a reboot may not be that bad.
<rharper> right
<rharper> but incremental changes like adding a route is easy to do, hard to say it won't have an impact
<rharper> the rub comes with the OS level  networking service
<mgerdts> "what could go wrong?"  :)
<rharper> for 16.04, ifupdown requires a bounce,  18.04 with networkd, it's possible to "append" new config; though networkd is fickle enough that it woudl need some testing to ensure it behave sanely
<rharper> if we didn't rely on the OS, but had cloud-init do it itself, then it's more likely we could pick out the "new" config and apply just that (via iproute2 )
<rharper> though, generally we'd prefer to push that sort of functionality into the OS level network tool;  there are other users who may want to incrementally apply config outside of clouds
<akik> hi. i'm experiencing strange network timeout problems running cloud-init on centos 7 on azure
<akik> if i disable cloud-init from running, the network becomes responsive again
<akik> it's maybe some kind of race between azure linux vm agent and cloud-init
<akik> i've used Provisioning.UseCloudInit=y in /etc/waagent.conf
<blackboxsw> akik: not sure, not seeing that on ubuntu azure. you could try pulling our latest cloud-init 18.2 from https://copr.fedorainfracloud.org/coprs/g/cloud-init/el-testing/
<blackboxsw> I just published the  18.2 release there 10 days ago
<akik> blackboxsw: 18.2? i have cloud-init-0.7.9-9.el7.centos.6.x86_64 <- probably really old?
<akik> it's installed from centos updates repo
<blackboxsw> yeah it is a bit old, but not as old as the version numbers might imply. We switched cloud-init versioning scheme late in 2017. to YY.##.   The distros/cloud images don't update as frequently as we release upstream cloud-init.  https://lists.launchpad.net/cloud-init/msg00106.html
<blackboxsw> that message on our mailing list has the details of that versioning switch... looks like we made the move to year-based versioning September 2017 due to agreement at last year's cloud-init summit
<akik> i'd like to use stable instead of testing
<akik> as there are no instructions there. is it just installing one rpm package?
<blackboxsw> +1 akik, we will only update el-testing when cloud-init upstream releases a tested release. (like 18.1 18.2 18.3)
<blackboxsw> the cadence of those releases is about ~3 months
<blackboxsw> for those that live on the edge (and want to cope with occasional failures) we have a daily copr repo here https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/
<akik> oh ok so i download that "repo download" ?
<blackboxsw> akik: yeah it is just cloud-init
<akik> thanks so much
<akik> i'll try that el-testing first
<blackboxsw> for sure, it'd give you an idea at least if it's something that's been fixed by more recent code
<akik> i already made a support ticket to azure because i couldn't figure it out
<akik> No package python-jsonschema available.
<akik> ah maybe i don't have epel
<blackboxsw> yep, need epel :0
<akik> seems to have worked
<akik> i got this though "DataSourceAzure.py[WARNING]: Error communicating with Azure fabric; You may experience.connectivity issues." but some seconds later it ran the user-data i had supplied
<akik> thank you very much
#cloud-init 2018-04-20
<oz123> hi, I trying to figure what was uploaded to my server. where is cloud init stored on the machine after boot?
<mgerdts> In the event that anyone is trying to figure out how to fill their Friday (haha) I have a few merge proposals that are awaiting review.  https://code.launchpad.net/~mgerdts/+merges
<mgerdts> rharper blackboxsw following up from yesterday's discussion: https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712
<smoser> mgerdts: i'll look.
<mgerdts> thanks
<smoser> blackboxsw: the improvements you suggested to https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343609
<smoser> are now there.
<smoser> but i couldnt avoid one more... magic prefixing of the top level key.
<smoser> mgerdts: https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343577
<smoser> one comment there.
<smoser> i approved... but can you add that ?
<smoser> i guess add a docstring to TestJoyentMetadataClient
<smoser> and describe it there.
<blackboxsw> smoser: approved and landed your branch
<blackboxsw> I'm ok with minor magic  there :)
<smoser> mgerdts: around ?
<mgerdts> yep, just back from lunch.
<smoser> do we use cloud-init now in a container
<smoser> ?
<mgerdts> I don't think it is used in the ubuntu certified 16.04, haven't looked at later images.
<mgerdts> "I don't think" meaning I went there to test and found it wasn't in use
<smoser> k.  but we could presumably get there at this point ?
<mgerdts> there may be other images out there that use it, but it wasn't in use in the most recent one produced by canonical.
<smoser> oh nice.
<mgerdts> potentially
<mgerdts> I've not had a lot of luck (or concerted effort) in getting the attention of those that build the official images.
<mgerdts> smoser, I've fixed the magic 16 & 17, but not with docstrings.  lemme know if you are good with it.
<smoser> mgerdts: can you tell me what 'systemd-detect-virt' shows on a virtual guest?
 * smoser guesses kvm
<mgerdts> vm-other
<smoser> gracias
<mgerdts> what breaks if that changes?
<smoser> well, nothing. i was just looking at ds-identify.
<smoser> identifying smartos doesnt currently care about the virt type
<mgerdts> I suspect that someday systemd will learn about bhyve.
<smoser> but i was going to add a ds-identify unit test and ifgured i'd add a realistic mock
<smoser> thansk
<mgerdts> ah, ok. yw
<smoser> mgerdts: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343738
<smoser> and https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343739
<smoser> with those two in place, cloud-init master ran with just one WARNING on the container service.
<smoser> the warning was due to zfs and growpart.
<mgerdts> looking
<smoser> blackboxsw: mgerdts has 3 in
<smoser> Approved
<smoser> if yuou wanted to merge.
<smoser> i'm eod now.
<mgerdts> thanks smoser, have a good weekend
#cloud-init 2018-04-21
<smoser> mgerdts: sorry to bother you
<smoser> one comment on https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343576
<mgerdts> @smoser: running tests on fix now
<mgerdts> tests results are good, pushed a fix
<jcara_hx> https://www.youtube.com/user/l0de/live IS POPPIN HOT RIGHT NOW STILL GOING!! CALL 315-505-4666. IRC.EFNET.ORG #lrh
<jcara_hx> https://www.youtube.com/user/l0de/live IS POPPIN HOT RIGHT NOW STILL GOING!! CALL 315-505-4666. IRC.EFNET.ORG #lrh
<jcara_hx> caphrim007_ ahasenack gaughen hrybacki masuberu ch007m cpaelzer akik Saviq markus-k tribaal benj_ paulgrmn powersj mdt[m] intheclouddan NostawRm meetingology mgerdts smoser seba Evoken Beret frickler dpb1 natorious bemis_ voja aimeeu holser_ madorn apollo13 aisrael blackboxsw mpontillo paulmey logan- brunobronosky ivve ilianaw ubuntulog2 satsuma bjonnh kevinbenton_ Odd_Bloke boxrick rcj ubot5` ybaumy kukacz Majost AscII cornfeedhobo x58 mgagne bbradle
<neyder> hello everyone!, i can't gete my user.data get read by ubuntu image is there something wrong?
<neyder> this is my user-data https://pastebin.ubuntu.com/p/NKB7RBYb8j/
<neyder> this is my user-data https://pastebin.ubuntu.com/p/NKB7RBYb8j/
<neyder> it says that:
<neyder> util.py[WARNING]: Failed loading yaml blob
<neyder> util.py[WARNING]: Failed at merging in cloud config part from part-001
<neyder> https://pastebin.ubuntu.com/p/6rXfMsXJf4/ this is cloud-init output
#cloud-init 2018-04-22
<seba> sounds like your yaml is somewhat broken
<seba> have you tried validating it with either python or something like http://www.yamllint.com/ ?
<smoser> seba: yeah, i think you're right. i'd not seen yamllint.com before. i have 'yaml-dump' in my ~/bin/
<smoser> http://paste.ubuntu.com/p/YyqqwqKRvZ/
#cloud-init 2020-04-13
<mutantturkey> hello
<mutantturkey> is there a flag for cloud-init to only use specific modules?
<mutantturkey> right now i am playing whack-a-mole with cloud-init to prevent it from overwriting my various configs. However, I basically just need cloud-init's networking module.
<mutantturkey> or maybe the answer is 'provide my own cloud-init config'
<mutantturkey> that just imports one module
<mutantturkey> because mine right now is the stock one from digital ocean
<Odd_Bloke> mutantturkey: You can configure which modules cloud-init will execute by modifying /etc/cloud/cloud.cfg, but you can't do that after first boot (because that's too late, of course) so that would involve image modifications.  What is cloud-init doing that you would like to disable?  (There may be user-data you can pass to disable the particular undesirable behaviours you're seeing.)
<rick_h> 372795
<Odd_Bloke> Time to get a new phone.
<rick_h> pretty much
<rick_h> or yubikey
<rick_h> oops
<Odd_Bloke> :p
<blackboxsw> Odd_Bloke: rharper. so branching strategy for cloud-init ubuntu/daily/<series>  vs ubuntu/<series>  do we want to talk about concerns with the approach mentioned here https://hackmd.io/VbmtcZLyR4650aqqmfMMYg   or https://github.com/CanonicalLtd/uss-tableflip/pull/45 ?
<blackboxsw> It may be faster to chat about that in a virtual hangout to make sure we aren't exposed to gaps. but, generally it feels like PR #45  simplifies our cherry-pick process a bit
<Odd_Bloke> blackboxsw: Which is currently the canonical source of the proposed process?
<blackboxsw> Odd_Bloke: https://github.com/CanonicalLtd/uss-tableflip/pull/45/files#diff-d6b5a9d4866846cffb4df4461090dfbb
<Odd_Bloke> OK, so I think we can discard the HackMD entirely then?
<blackboxsw> Odd_Bloke: yes, we can. As I think the use-cases I mentioned there are handled by the proposal in PR #45
<Odd_Bloke> OK, cool.
<blackboxsw> I think the biggest set of open conversation on it was here https://github.com/CanonicalLtd/uss-tableflip/pull/45#discussion_r406505316
<blackboxsw> s/conversation/questions/
<rharper> Odd_Bloke:  blackboxsw: I have time now, meet in standup ?
<blackboxsw> sure I can swing on by
<Odd_Bloke> Oops, was AFK, OMW.
<mutantturkey> Odd_Bloke: well, I do use a custom image
<mutantturkey> that's so i can put things on like, users keys, and custom packages, its an entire dev environment in a single VM for my work stack, so it has lots of stuff on it
<mutantturkey> Odd_Bloke: what module seems to generate the files under /etc/network/interfaces?
<blackboxsw> rharper: I think you are right, we can git cherry-pick && git revert single cpick commitish properly I misinterpreted the output from the CLI. the revert is valid and applicable. it was a PEBCAK issue on my side thursday.   https://paste.ubuntu.com/p/QnFnvJHMGx/
<blackboxsw> Odd_Bloke: rharper per the above paste. I think we can use individial git cherry-pick <cpick_commit_from_devel>; (daily-branch) git revert <cpick_commit_from_devel>;
<blackboxsw> so no full merge necessary each time we cherry pic.
<blackboxsw> I can update the doc on PR #45 now for the minimal approach
<rharper> blackboxsw: even with changelog entries ?
<blackboxsw> rharper: that's without the changelog entries
<rharper> changelog is one file that could get in conflict
<rharper> with new commits to release that are not present in daily
<rharper> so maybe replay your test but with changelog commits to release before the recipe build  step such that you're merging daily into a branch that has release + changelog commit not in daily;
<rharper> and see if it blows up
<rharper> mutantturkey: cloudinit/net/eni.py writes to /etc/network/interfaces.d/50-cloud-init.cfg     we don't touch /etc/network/interfaces file directly
<rharper> mutantturkey: networking isn't a config module;  we don't yet have a "render-network-config" command cli that's does the right thing at this time (there are several places cloud-init looks for input for network-config);   but it has been requested on a few fronts;  I'll see about getting that on our next cycle features
<mutantturkey> rharper: so eni.py writes the config, cool. that config file is exactly what i want
<mutantturkey> rharper: if i remove everything elsefrom my /etc/cloud/cloud.cfg... will it still run eni.py and not the other modules?
<rharper> mutantturkey: eni will run independent of the config modules
<mutantturkey> okay
<mutantturkey> does anything else run independent of the config modules hten?
<mutantturkey> then?
<rharper> it's part of cloud-init stages (cloud-init init --local stage runs network rendering code)
<mutantturkey> my goal: get DO to provide the correct network information, but that's it
<rharper> DO as in digital ocean ?
<mutantturkey> yes exactly
<rharper> how is the information incorrect ?
<rharper> they provide network metadata IIRC
<mutantturkey> they provide exactly what i need yes. the /etc/network/interfaces.d/50-cloud-init.cfg works perfectly
<rharper> oh, I see;  you only need network-config
<mutantturkey> however, other things are done by cloud init that overwrite my config
<mutantturkey> i think so :-)
<rharper> yes; so if you make the init_modules and config modules and final modules lists empty; that should be rather sparse
<mutantturkey> is an empty /etc/cloud/cloud.cfg valid?
<rharper> yes but likley won't do what you want
<mutantturkey> okay.
<mutantturkey> i know my goal, i am not quite sure how to make cloud-init do what i want
<rharper> you want:  cloud_init_modules: [] , cloud_config_modules: []  and cloud_final_modules: []
<rharper> this will tell cloud-init to not run any of the config modules
<mutantturkey> okay
<mutantturkey> so something lik
<mutantturkey> echo "cloud_init_modules: []" > /etc/cloud/cloud.cfg
<mutantturkey> echo "cloud_config_modules: []" >> /etc/cloud/cloud.cfg
<mutantturkey> echo "cloud_final_modules: []" >> /etc/cloud/cloud.cfg
<rharper> I'm not sure what removing the system_info section will do
<rharper> likely break
<rharper> I'd put your config you have in a different file that will get merged
<mutantturkey> okay, that would be in
<rharper> so write that to /etc/cloud/cloud.cfg.d/99-disable-config-modules.cfg
<mutantturkey> /etc/cloud/cloud.d/
<rharper> yes
<mutantturkey> got it
<mutantturkey> thanks let me try that
<rharper> sure
<mutantturkey> (not that i dislike cloud init, its very cool)
<mutantturkey> i just am only needing it for one thing
<rharper> so what about resizing root and ssh host key clean/gen etc ?
<mutantturkey> hm
<mutantturkey> good point
<rharper> so, keep growpart, resizefs
<rharper> in cloud_init_modules
<mutantturkey> er how do i do multiline bash things
<rharper> for those;  and ssh if you want keygen
<mutantturkey> <<<EOF
<rharper> cat  > /myfile <<EOF
<rharper> ...
<rharper> ..
<rharper> EOF
<mutantturkey> right
<mutantturkey> do the numbers mean anything? 99, 50 etc?
<mutantturkey> is it order of operation?
<rharper> yes, order of merging
<rharper> so 99 will *clobber* previous keys in the dictionary by default
<mutantturkey> understood
<rharper> which is what you want in this case
<mutantturkey> cloud_init_modules: [growpart, resizefs]
<mutantturkey> just want to make sure that syntax is correct
<rharper> also, DO sends vendor-data which contains a lot of stuff, you may want to disable their vendor-data,  you can include "vendor_data: {enabled' False}"
<rharper> mutantturkey: that looks right
<mutantturkey> vendor_data: {enabled: False}"
<mutantturkey> ?
<rharper> mutantturkey: one thing you can do to test this out on existing instances is to write your config file out, and then: sudo cloud-init clean --logs;  then sudo cloud-init init --local;  sudo cloud-init init;  that runs the first two stages
<rharper> the platform can send additional cloud-config to your instance
<rharper> you don't have to accept it, but that turns it off
<rharper> you can check /var/lib/cloud/instance/vendor-data.txt to see if they sent you some;  ISTR DO had quite the bundle
<mutantturkey> rharper: could you review this quickly?
<mutantturkey> http://ix.io/2hTi
<rharper> y
<rharper> add #cloud-config to the first line of your config file you generate
<mutantturkey> awesome, done.
<mutantturkey> alright, i shall report back in a bit with my results!
<blackboxsw> rharper: one thing I don't understand is why we would want to cherry-pick the debian/changelog commitish too into daily for a revert from u/daily/devel? Since the changelog entry also increments debian/changelog version as well, a revert will mean that when we merge back into ubuntu/devel, we'll be down-reving cloud-init as built during daily images.
<blackboxsw> so this would mean that every daily deb build would be 1 subrevision lower than the version present (and released) in ubuntu/devel.
<blackboxsw> making it a bit harder to install and test a daily build vs the current released version of cloud-init
<mutantturkey> rharper: cool, i borked it up (whitspace after EOF, but let me build and run it again).
<mutantturkey> the other problem i currently have is that it doesn't seem to apply the new network information after cloud-init puts the information on the first boot. after the first boot it works just finewith thecorrecet info
<mutantturkey> though i wonder if thats debian doing something different, need to look ta logs
<punkgeek> I use this sample to config static ip on ubuntu 16.04 vm, But it doesn't work, https://paste.ubuntu.com/p/Hh8SkhkS9P/
<mutantturkey> rharper: yeet
<mutantturkey> No 'config' modules to run under section 'cloud_config_modules'
<mutantturkey> No 'final' modules to run under section 'cloud_final_modules'
<adrian27> hello, I have the following in /etc/cloud/cloud.cfg.d/01_runcmd.cfg
<adrian27>   runcmd:
<adrian27> but it is not executed. Any idea what am I doing wrong?
<adrian27> tanks
<sarnold> it looks like your paste may have been destroyed, better to use a pastebin site
<adrian27> thanks
<adrian27> its just those two lines
<sarnold> that's the thing, we only got one: https://paste.ubuntu.com/p/n93Dk5pBFB/
<rharper> mutantturkey: w.r.t network-config, we only write network-config once (per-instance)  ; if you're repeat testing on the same instance you want to use the cloud-init clean --logs (which wipes instance data and the log files)
<rharper> blackboxsw: I don't *want* daily to be down level; I was just worried that we'd see conflicts due to release and daily both writing to debian/changelog
<adrian27> he he, okay, here it is: https://paste.ubuntu.com/p/kRQMxmRBVf/
<sarnold> adrian27: try /usr/bin/echo or /bin/echo, depending upon where your echo executable is located
<sarnold> adrian27: (I have no idea if this'll help, but giving explicit paths to things often helps)
<blackboxsw> rharper: the way PR #45 is setup, we are only reverting the single commitish that introduced the debian/patches/cpick-* file. the changelog commit is separate, so if we don't cherry-pick it into ubuntu/daily/<series> then we essentially don't touch it, or risk merge conflicts. But we have the caveat of daily build recipe debian/changelog representing that a cherry-pick is applied, when it really isn't
<blackboxsw> BTW rharper Odd_Bloke I just force pushed all doc changes and script changes to use git revert instead of git merge ubuntu/devel -> ubuntu/daily/devel
<adrian27> sarnold thanks, I will try that
<rharper> blackboxsw: I see, so we don't conflict at all since we only revert the patch;   I'm not super worried about daily changelog
<blackboxsw> rharper: right, and I just double checked if we wanted to also revert the debian/changelog entries and yes it causes conflicts making it a far more manual process than we want :/
<adrian27> sarnold: it still does not work :(
<sarnold> adrian27: dang :( sorry. how about a touch command to create a file, instead? maybe stdout isn't going where you expect?
<adrian27> sarnold: hmm, good idea. Will be back
<adrian27> sarnold: nope, still does not work :(
<sarnold> adrian27: dang, sorry, I'm out of ideas
<adrian27> sarnold: thanks anyway :)
<adrian27> do you need to have the #cloud-config at the beginning in config files from  /etc/cloud/cloud.cfg.d/ ?
<rharper> adrian27: I don't think so but it can't hurt
<adrian27> can I see somewhere if cloud-init tried to execute my runcmd commands?
<rharper> adrian27: yes, cat /var/lib/cloud/instance/scripts/runcmd
<adrian27> rharper: well, that folder its empty. What does that mean? That it did not even tried to execute commands in https://paste.ubuntu.com/p/kRQMxmRBVf/
<adrian27> actually /var/lib/cloud/instance/scripts/ is empty
<rharper> it's hard to say, cloud-init collect-logs will produce a tarball with debugging info;  the log file will show whether it ran into issues with your userdata or something else
<adrian27> isn't that /var/log/cloud-init.log ?
<rharper> https://paste.ubuntu.com/p/8zb2qMmYbF/
<rharper> adrian27: it's more that just that, but take a look at that
<rharper> I'm not sure how you're re-running your test on your config but 1)  user-data needs a #cloud-config at the top  2) config modules typically only run once per instance so if you are calling cloud-config.service etc; it's skipping over the runcmd module as it's already run  3)  you can test with cloud-init single --   see the paste for an example;   when you're have it working, you can cloud-init clean --logs --reboot and see a "clean boot" run
<sarnold> rharper: oh handy, thanks
<rharper> sarnold: =)   -- yeah, we are in need of some cli refactor to make testing/validating config eaiser ;  we do have the schema subcommand but we've not completed writing schemas for all of the config modules
<rharper> blackboxsw: ok; yeah so touching changelog on both branches means merge conflicts ... that was expected;  so are we OK then with the "downlevel" changelog entry?  you also mentioned that the daily PPA version would not be newer than what's in the archive?
<blackboxsw> rharper: as PR 45 currently the pkg version generated  will be identical for packages generate from (ubuntu/devel) and (ubuntu/devel with ubuntu/daily/devel merged into it)
<blackboxsw> but the second merged case will have the cpick reverted (just no changelog dicc)
<blackboxsw> diff*\
<Odd_Bloke> I don't think the changelog is used in recipe builds.
<Odd_Bloke> Because the recipes generate their own version string
<blackboxsw> hahah, right
<blackboxsw> good pt
<Odd_Bloke> I thought of this at some point over the long weekend, forgot about it until just now. :p
<blackboxsw> right so it'll always be the  deb-version {latest-tag}-{revno}-g{git-commit}-0ubuntu1+{revno:ubuntu-pkg}~trunk
<blackboxsw> so, what would our preference be (that the unused debian/changelog in ubuntu/daily/devel contain an item that says "reverted" cpick? or is it a don't care (because we have a revert commit in the git log anyway
<rharper> don't care on daily build
<rharper> though maybe as Odd_Bloke pointed out, we'll suddenly find so many users really do look at daily ppa changelog ...
<rharper> I'd defer tackling it
<blackboxsw> ok, if we don't care there on daily build, I think #45 is in shape then, and only performs a single git  revert of the cpick file in the ubuntu/daily/devel branch.
<rharper> from my POV daily ppa is a way to verify that some bug works;  that's almost always an integration type of test
<rharper> blackboxsw: ok, let me re-review shell code
<rharper> I assume you added all of the git command || clean-up  bits I requested ?
<adrian27> rharper: thank you
<blackboxsw> rharper: I tried to mark resolved as I addressed each comment. I'll make sure I've pushed all those cleanup review comment resolutions
<rharper> adrian27: all good?
<rharper> blackboxsw: k
<blackboxsw> rharper: +1 on handling your review comments. I noticed a typo in one of doc my fixes and just pushed it now. 'a older series' instead of 'an older series'
<blackboxsw> the others look resolved
<adrian27> rharper: yes. you were right, rumcmd is run only once. I booted the instance, wrote the /etc/cloud/cloud.cfg.d/ file, and expected to see it running when rebooted
<adrian27> why cloud-init does not have a man page? :)
<rharper> adrian27: ok;  fair point on man page
<adrian27> well, you guys are doing a great job with cloud-init
<powersj> https://github.com/canonical/cloud-init/tree/master/doc/man
<rharper> adrian27: thanks for the kind words ;  powersj tells me we should have a man page ... so double checking on that
<blackboxsw> rharper: powersj hah!
<blackboxsw> time to bundle it in the deb package
<blackboxsw> or see where we reverted that
<blackboxsw> hah it *is* there on latest, and bundled
<Odd_Bloke> `man cloud-init` doesn't work in my focal EC2 instance (launched last week).
<blackboxsw> cloud-init 19.3-68-gc78fddea introduced it
<powersj> https://github.com/canonical/cloud-init/pull/101
<powersj> so apparently that needs to happen somewhere else?
<blackboxsw> I have it on focal lxc images
<Odd_Bloke> From the archive?
<Odd_Bloke> Because `man cloud-init` is failing in this container I just launched.
<rharper> Odd_Bloke: me too
<rharper> daily focal lxd does not have man page
<blackboxsw> Odd_Bloke: /me relaunches. interesting. it was on 20.1
<blackboxsw> and failed on latest current build
<blackboxsw> 20.1-10-g71af48df-0ubuntu3 for failed case... and my success case was locally built
<Odd_Bloke> Yeah, locally built will be fine.
<Odd_Bloke> Assuming you did it from master.
<blackboxsw> yeah right ;/
<Odd_Bloke> Because that PR only updated the packaging used in master.
<blackboxsw> +1. thanks for the bug adrian27 :)
<blackboxsw> :q
<blackboxsw> wrong term :/
<blackboxsw> ok addressed https://github.com/canonical/cloud-init/pull/152 comments rharper as well thanks for the schema review
<blackboxsw> ahh and thanks for the review on uss-tableflip
<Goneri> blackboxsw, https://github.com/canonical/cloud-init/pull/298 :-)
#cloud-init 2020-04-14
<punkgeek> Is there any way to config vms without iso, such as using python libvirt?
<Odd_Bloke> punkgeek: How would you want to use libvirt to configure them?
<blackboxsw> Odd_Bloke: per https://github.com/canonical/cloud-init/pull/314/files pytest fixtures. do you think we need to worry about xenial/bionic support of capsys/caplog as we did with ua-client?
<blackboxsw> I forgot if I asked this question yesterday
<Odd_Bloke> blackboxsw: What do you mean by "worry about"?
<Odd_Bloke> (I did comment on your PR with a comment about caplog.)
<punkgeek> Odd_Bloke: I saw a sample in ovirt that import cloud-init configuration. But I want this tool in python libvirt https://red.ht/2K1gBkn
<Odd_Bloke> And `capsys` is available in xenial.
<Odd_Bloke> punkgeek: Have you tried using that example?  Does it fail for you?  (This isn't an area I'm familiar with, apologies in advance for dumb questions. :)
<Odd_Bloke> blackboxsw: Sorry to expand on my previous comment: we should discuss caplog (and potentially using the separate python3-pytest-catchlog package on xenial), but I think that's orthogonal to documenting the current state of matters.
 * blackboxsw just reads the responses on my schema PR. I was wondering if we had shortcomings we'd have to be concerned with given that caplog wasn't available on xenial. but you've answered that for me. we can use pytest-catchlog potentially 
<blackboxsw> Odd_Bloke: sorry, was just reading the other context. and agreed it is orthogonal.
<blackboxsw> ~30 until cloud-init status meeting
<otubo> Anyone can review this oneliner PR really quick? https://github.com/canonical/cloud-init/pull/315
<otubo> Really sorry for the hard ping on this, it's a super silly fix that was breaking on our tests and I need to backport it by tomorrow EOD :-D
<otubo> blackboxsw, Odd_Bloke ^^
<blackboxsw> otubo: sure. so what is that whitespace?
<blackboxsw> ohh shuffling placement
<blackboxsw> ok looking
<blackboxsw> reviewing
<Odd_Bloke> I'm about to be AFK for a while, but I'd like us to add a test that would have caught this.
<Odd_Bloke> (Not necessarily as part of the PR that fixes it, as you have time pressure on that.)
<otubo> Odd_Bloke, sure! I'd be glad to include a test for that, like the day after tomorrow :-)
<Odd_Bloke> ^_^
<blackboxsw> otubo: whats the redhat bugzilla for this bug here? https://bugzilla.redhat.com/show_bug.cgi?id=1794664
<ubot5> bugzilla.redhat.com bug 1794664 in cloud-init "[RHEL8] swapon fails with "swapfile has holes" when created on a xfs filesystem by cloud-init" [High,Assigned]
<blackboxsw> I'm just wondering as I want to cross reference it in the commit message that we land when merging
<otubo> blackboxsw, yeah, I was just thinking about that.
<otubo> blackboxsw, can you add the BZ (that's the one, BTW) upon merge? Or I should do something else, like a v2?
<blackboxsw> otubo: I can set the squash merged commit message at merge time
<blackboxsw> so no need for a fix on your end
<blackboxsw> rharper: do you recall what prefix we use in the footer for RH bugzillas?  I'm not seeing any RH: <BUG_ID> in commit messages, but I though we had something.
<blackboxsw> at least for Suse
<otubo> blackboxsw, actually the correct BZ is 1772505
<blackboxsw> ahh thanks otubo
<otubo> that one was for RHEL8, but the fix is actually for RHEL7
<rharper> blackboxsw: we can use whatever is useful for otubo  ... I don't think we have any convention
<otubo> rharper, blackboxsw nothing special for me, BZ#<number> is normally what we do internally
<rharper> blackboxsw: sometimes there's an LP bug which has a mirror link to an downstream bz
<rharper> bug #1781781
<ubot5> bug 1781781 in curtin "/swap.img w/fallocate has holes" [Medium,Confirmed] https://launchpad.net/bugs/1781781
<otubo> blackboxsw, do we need Odd_Bloke for that or we can have a review/merge from rharper ? :-)
<blackboxsw> otubo: no I'll merge
<blackboxsw> otubo: this commit message ok with you?https://paste.ubuntu.com/p/RDXMwXxMmt/
<blackboxsw> RBZ: prefix for redhat bugzilla?
<otubo> blackboxsw, RHBZ is better I guess
<blackboxsw> +1
<otubo> blackboxsw, that's actually my fault. I got to the bottom of my backlog and found out this issue due to tomorrow.
<blackboxsw> no worries otubo, do you want to spend time adding the unit test today or land it and propose the unit test changes  in a separate PR?
<otubo> blackboxsw, and thank you very much for the commit :-) If we ever have a cloud-init summit this year, remember me to pay you a beer :-)
<otubo> blackboxsw, can I add the test once I get this out? Like tomorrow would probably be ok.
<blackboxsw> merged otubo . ok will fast track a unit test PR once available.
<blackboxsw> thanks
<otubo> blackboxsw, thanks a lot :)
<blackboxsw> no problemo
<blackboxsw> #startmeeting cloud-init status meeting
<meetingology> Meeting started Tue Apr 14 16:17:46 2020 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
<meetingology> Available commands: action commands idea info link nick
<blackboxsw> hey folks, time for cloud-init community status meeting
<powersj> \o/
<otubo> Right on time :-)
<blackboxsw> welcome again to our bi-weekly status meeting feel free to interject comments, questions, suggestions during this status meeting. Generally it is an opportunity for upstream to provide a frequent platform communication and drop-in discussion when a couple of upstream devs are available.
<blackboxsw> #chair Odd_Bloke smoser rharper powers
<meetingology> Warning: Nick not in channel: powers
<meetingology> Current chairs: Odd_Bloke blackboxsw powers rharper smoser
<blackboxsw> #chair powersj
<meetingology> Current chairs: Odd_Bloke blackboxsw powers powersj rharper smoser
<blackboxsw> our IRC channel topic carries the next planned status meeting for those that wish to participate. All are welcome to interject or drive converstation topics here
<blackboxsw> any objections to same time on Apr 28th?
<powersj> nah that's a good day given we want to cut 20.2 then
* blackboxsw changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting April 28 16:15 UTC | 20.1 (Feb 18) | 20.2 (Apr 28) | https://bugs.launchpad.net/cloud-init/+filebug
<blackboxsw> ok topic set for next meetin
<blackboxsw> our last meeting minutes can be found on github at
<blackboxsw> #link https://cloud-init.github.io/status-2020-03-31.html#status-2020-03-31
<blackboxsw> The topics we generally cover in this meeting are the following: Previous Actions, Recent Changes, In-progress Development, Community Charter, Upcoming Meetings, Office Hours (~30 mins).
<blackboxsw>  #topic Previous Actions
<blackboxsw> none listed from last meeting
<blackboxsw> #topic Recent Changes
<blackboxsw> git commits landed in tip since Mar 31: https://paste.ubuntu.com/p/VptqRBfVfJ/  found by git log --since 03-31-2020
<blackboxsw> changes sport doc updates, otubo's cc_mount fix , better url handling for regions which  contain underscores in their name  and openbsd fixes from Goneri for passwd locks.
<blackboxsw> thanks for the contributions this round folks!
<blackboxsw> #topic In-progress Development
<blackboxsw> upstream is currently focused on getting in bug fixes, dropping remnants of py2 in tooling  reviewing active PRs to get cloud-init in shape for the upcoming 20.2 release
<blackboxsw> community notice: as mentioned in the channel topic, Apr 28th is our upstream release date for 20.2
<blackboxsw> community notice: we ask that pull requests or bugs that need resolution for 20.2 be up for review by Friday April 24th so there is time to review and merge those fixes.
<blackboxsw> active pulls indended for the release should be up in github at https://github.com/canonical/cloud-init/pulls
<blackboxsw> *intended* rather
<blackboxsw>  #topic Community Charter
<blackboxsw> his section is generally reserved to discuss any general community goals for cloud-init, at last cloud-init summit  we defined those goals as:
<blackboxsw> - datasource doc fixes
<blackboxsw> - json schema validation for each cloudinit/config/cc_*py modules
<blackboxsw> there are feature bugs created for these tasks at https://bugs.launchpad.net/cloud-init/+bugs?field.tag=bitesize
<blackboxsw> #topic Office hours (next ~30 mins)
<blackboxsw> During office hours a couple of upstream devs will have eyes on this channel. Any questions, comments, branch reviews are fair game for discussion.
<blackboxsw> In leiu of active discussions, developers will be grooming the active pull request review queue  to unblock branch authors.
<punkgeek> Odd_Bloke: No it doesn't work. why there is no way to config cloud-init in kvm virtualization rather than iso, like vm xml file?
<blackboxsw> I'm getting through a belated review on https://github.com/canonical/cloud-init/pull/298
<blackboxsw> punkgeek: like seeding ovf-env.xml? https://github.com/canonical/cloud-init/blob/master/doc/sources/ovf/README
<rharper> punkgeek: you might want to look at virt-install , they recently have added support for providing cloud-config to VMs;  virt-install is a wrapper around creating VMs utilizing libvirt as a backend, https://athinapl.home.blog/2019/08/25/gsoc-2019-cloud-init-configuration-for-virt-manager-virt-install/
<Goneri> punkgeek, or you can take a look at virt-lightning
<Goneri> punkgeek, it's basically a CLI to use libvirt+cloud-init
<blackboxsw> Goneri: review done. sorry for the delay on such a minor set of change requests https://github.com/canonical/cloud-init/pull/298/files#
<blackboxsw> https://github.com/CanonicalLtd/uss-tableflip/pull/45 I think comments are resolved
<blackboxsw> I think we are at about the turn of the hour for cloud-init status. I'm going to review https://github.com/canonical/cloud-init/pull/305 next
<blackboxsw> thanks all for tuning in. see you next time
<blackboxsw> #endmeeting
<meetingology> Meeting ended Tue Apr 14 17:12:01 2020 UTC.
<meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2020/cloud-init.2020-04-14-16.17.moin.txt
<meena> it's tuesday?
<meena> i can't believe it's tuesday, when every day feels like tuesday
<blackboxsw> hahaha!
<blackboxsw> heya meena , welcome. :)
<blackboxsw> just completed another review for goneri https://github.com/canonical/cloud-init/pull/305/files#diff-1708fc6fbf7cc4ca958a7adab7ad615eR23
<Odd_Bloke> meena: Thanks for the review on that doc PR!
<blackboxsw> rharper: and json schema PR is good for followup review https://github.com/canonical/cloud-init/pull/152
<blackboxsw> yes meena thx for all the reviews on nearly every PR that is up.
<blackboxsw> rharper: this looks like bogus network configuration from a dhcp server lease response doesn't it?
<blackboxsw> 2020-04-07 02:51:55,948 - dhcp.py[DEBUG]: Received dhcp lease on eth0 for 10.54.62.43/255.255.255.255
<blackboxsw> 2020-04-07 02:51:55,949 - __init__.py[DEBUG]: Attempting setup of ephemeral network on eth0 with 10.54.62.43/32 brd 10.54.62.127
<rharper>  no
<rharper> do you have the full lease dump ?
<blackboxsw> how can eth0 access broadcast  addr  10.54.62.127 if subnet netmask is /32?
<rharper> I don't need to broadcast;  typically you'll also have a static route set
<blackboxsw> rharper: I don't just logs @ https://launchpadlibrarian.net/473747419/cloud-init.log
<rharper> we dump the lease parts though I thought
<blackboxsw> it's ephermeral dhcp response constructs/log messages as it parses tmp dhclient lease during initial setup.
<blackboxsw> per the openstack bug from a few days ago
<blackboxsw> https://bugs.launchpad.net/cloud-init/+bug/1871323
<ubot5> Ubuntu bug 1871323 in cloud-init "cloud-init fails to add default route during _bringup_static_routes" [Undecided,Incomplete]
<rharper> can we ask for the lease file?  either it's a bogus lease; (I'm not sure about that) or we're parsing it in correctly
<blackboxsw> seeing statements like this from cloud-init seem invalid : Running command ['ip', '-family', 'inet', 'addr', 'add', '10.54.62.43/32', 'broadcast', '10.54.62.127', 'dev', 'eth0']
<blackboxsw> yeah I can  get that.
<blackboxsw> will request the tmp lease and provide reproducible instructions to ge tit
<blackboxsw> I think it's a bogus dhcp config on their network... but just guessing here
<rharper> blackboxsw: well, let's confirm we're parsing it correctly, we could be putting the /32 on there ourselves
<rharper> is this recent cloud-init or older release on non-Ubuntu ?
<rharper> we added a redhat dhclient rfc parser last fall
<rharper> blackboxsw the lease is there
<rharper> option rfc3442-classless-static-routes 32,10,54,62,1,0,0,0,0,32,169,254,169,254,10,54,62,1,0,10,54,62,1
<blackboxsw> 19.4.33, so latest xenial I think.
<blackboxsw> ohh right the lease is there in the initial bug file.
<blackboxsw> which specifies:
<blackboxsw>   fixed-address 10.54.62.43;
<blackboxsw>   option subnet-mask 255.255.255.255;
<blackboxsw> is that invalid given that the router is at    option routers 10.54.62.1;
<blackboxsw> as subnet-mask should be something like 255.255.255.0 or .128 or something to give it visibility to the router right?
<blackboxsw> specifically, if their router is 10.54.62.1 the smallest possible subnet-netmask they can have is /26 to allow 10.54.62.43 to contact 10.54.62.1. so a netmask of 255.255.255.192
<blackboxsw> will pose the question on the bug and see if I'm bungling  something there
<powersj> otubo, have you seen the comments on this https://github.com/canonical/cloud-init/pull/70#pullrequestreview-393312656
#cloud-init 2020-04-15
<otubo> powersj, I'm gonna take a look. Thanks!
<andras-kovacs> huh... I didn't know Azure can be this terrible... I made a VM template with cloud-init installed in it, but disabled. My intention was to start to use it with Azure's own telemetry-collector whatnot than change to cloud-init as soon as possible.
<andras-kovacs> But it says "[ProvisionError] cloud-init appears to be running, this is not expected, cannot continue."
<andras-kovacs> I found a bug and the reason of it is/are probably the dhclient + NetworkManager scripts, but still...
<andras-kovacs> I believed it can't be worse after the fixed-size IDE connected vhd files. But actually, it can.
<andras-kovacs> https://github.com/Azure/WALinuxAgent/issues/806
<andras-kovacs> I was wondering... why do they need to re-invent the wheel?
<andras-kovacs> Even more funnier this waagent doesn't even log it's error on error level to journal. :D
<StyXman> is it possble that if I deploy a user, setting its password  and marking it as expired, the password is set *after* it was marked as expired, so this password set overwrites the expiration date? BTW, there is no expriration date in /etc/shadow, only last change date and max password age
<StyXman> here's my config: https://dpaste.org/4g89
<StyXman> in fact it's a bug: the image I'm using has a adduser who ignores --expirredate and there seems to be a function to go arund such bugs, but it doesn't seem to be used
<Odd_Bloke> StyXman: Are you saying it's a bug in the image you're using, or you think it's a bug in cloud-init?
<StyXman> https://bugs.launchpad.net/cloud-init/+bug/1872995
<ubot5> Ubuntu bug 1872995 in cloud-init "expiredate is not working if also setting a password." [Undecided,New]
<otubo> Odd_Bloke, powersj by the way, just found another bug on the swap file stuff. But won't make it to the release cut this time. Can't make upstream/backport in less than an hour.
<Odd_Bloke> otubo: OK, hm, should we back out the original change so we can reland it with these fixes applied?
<otubo> Odd_Bloke, powersj I'll post another PR soon, together with a unittest.
<otubo> Odd_Bloke, that part is correct, the assignment should be after the try/except. The bug is at create_swapfile()
<otubo> cmd = ['fallocate', '-l', '%dM' % size, fname] ---> is treating size as %d when it is %s
<otubo> errmsg = "Failed to create swapfile '%s' of size %dMB via %s: %s" ---> here too
<otubo> Odd_Bloke, I'll fix it and include the unittest by tomorrow. No need to drop anything right now.
<powersj> otubo, thank you!
<mutantturkey> so i have only enabled a few specific modules from cloud init, but another thing cloud-init is doing is setting the 'hostname'
<mutantturkey> i don't actually want it to do so, is there a way to prevent it from overwriting my hostname i set in my custom image?
<blackboxsw> mutantturkey: the hostname gets set early in boot by cloud-init to fix a number of platforms that need hostname set before the initial dhcp request..... I'll get you where that's done.
<blackboxsw> and we'll see if it can be disabled
<mutantturkey> is it    199 ^[[0;32m    digitalocean: /usr/lib/python3/dist-packages/cloudinit/config/cc_set_hostname.py^[[0m
<mutantturkey> sorry the output got munged
<mutantturkey> i did find a "echo "preserve_hostname: true" > /etc/cloud/cloud.cfg.d/99_hostname.cfg" but i wasn't sure if that was accurate
<blackboxsw> mutantturkey: it's in cloudinit/cmd/main.py
<blackboxsw> mutantturkey: https://github.com/canonical/cloud-init/blob/master/cloudinit/cmd/main.py#L698-L713
<blackboxsw> so it's set only if user-data or meta-data on the cloud provides that value
<blackboxsw> but yes, ultimately calls cc_set_hostname
<andras-kovacs> could you point me to the right direction to find a cloud.cfg which works in Azure? I mean which can set the hostname, etc. from the meta data.
<blackboxsw> mutantturkey: you can check if DO provides that hostname in metadata with `cloud-init query --all`
 * andras-kovacs sent a long message:  < https://matrix.org/_matrix/media/r0/download/matrix.org/SZJZxEKrgcFXnlWGZLCUSoed >
<andras-kovacs> but cloud-init query -a gives [] nulland ""
<mutantturkey> blackboxsw: cloud-int query --all: error argument subcommand: invalid choice 'query'
<mutantturkey> cloud-init -v: 15:58 <@blackboxsw> but yes, ultimately calls cc_set_hostname
<mutantturkey> sorry, /usr/bin/cloud-init 18.3
<andras-kovacs> but I can read the meta data with curl... hmmm..
<andras-kovacs> mutantturkey: cloud-init query -a
<andras-kovacs> mutantturkey: I like your name. Do you watch SouthPark?
<Odd_Bloke> andras-kovacs: Where is your current cloud.cfg coming from?
<mutantturkey> andras-kovacs: no, i mean i have seen it, but i dont know the reference
<Odd_Bloke> andras-kovacs: mutantturkey: 18.3 predates `cloud-init query` being available, I believe.
<mutantturkey> Odd_Bloke: seems so
<mutantturkey> i am fairly certain that the hostname is set by cloud init, as it sets it to the first part name of the DO box. (x.y.z: it is set to x)
<mutantturkey> and the logs confirm
<mutantturkey> Running command ['hostname', xzy]
<andras-kovacs> Odd_Bloke: from the package, the default. So I need to play around with it. It worked so good with openstack for the first try. But Azure is really different. Do you have a link or sg. for a working sample config? I saw the one in the doc but I got confused because of the __builtin__ and dhclient_lease_file part.
<mutantturkey> yeah; 2020-04-14 17:59:09,190 - util.py[DEBUG]: Read 51 bytes from /var/lib/cloud/data/set-hostname
<andras-kovacs> I don't know yet how to define the the lease file, because I use NetworkManager and it looks like /var/lib/NetworkManager/dhclient-5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03-eth0.lease
<Odd_Bloke> andras-kovacs: "the package" on what distro?  It works out of the box on Ubuntu, and I haven't heard of any other substantial issues.
<andras-kovacs> RedHat 7.8. I could proceed further, but now I see: DataSourceAzure.py[WARNING]: Could not crawl Azure metadata: No Azure metadata found.
<Vitto> Hello, I am a novice programmer and I am trying to use the cloud init functionality via Scaleway API but cannot seem to get it to work. I have a working cloud init shellscript that does its job when launched via web dashboard. Instead, when I try to submit it via API it seems that the cloud init boot does detect the script. Is anyone able to help
<Vitto> me?
<rharper> hi Vitto  one possible issue is how the API attaches the user-data to their instance ... some platforms require the user-data content (cloud-config yaml) to be base64 encoded ... is there any API documentation on scaleway that migth help ?
<johnsonshi> Hi guys, I have a question regarding trying to specify my own systemd unit to start in the middle of cloud-init.service and cloud-config.service. The question is here at: https://paste.ubuntu.com/p/tR6vMm6SGd/
<Vitto> rharper this is the user data section of the scaleway API https://developers.scaleway.com/en/products/instance/api/#user-data-35f60e, it does not mention anything regarding base64 encoding, I'll ask them directly in the meanwhile.
<mutantturkey> so documentation says hostname can be specified via hostname key, in my cloud-init ustom config would that be
<mutantturkey> hostname: myname
<mutantturkey> that worked, and since i know the expected hostname, my problem is solved. dope
<Odd_Bloke> blackboxsw: rharper: https://github.com/canonical/cloud-init/pull/316 should fix swap file creation in master; I'd appreciate a speedy review so we can discuss cherry-picking it to fix focal before Final Freeze (tomorrow :p).
<Odd_Bloke> otubo: Apologies to perhaps duplicate your work; today is the last day to get fixes into the next Ubuntu release on day one, so this is urgent for us. :)
<blackboxsw> Odd_Bloke: +1 grabbing
<blackboxsw> Odd_Bloke: nit comment  https://github.com/canonical/cloud-init/pull/316#pullrequestreview-394115233
<blackboxsw> Odd_Bloke: I see in the failure path case on cc_mounts:create_swapfile that if create_swap fails, it deletes fname, but then create_swapfile proceeds to     util.chmod(fname, 0o600)
<blackboxsw> which should no longer exist
<blackboxsw> is it worth create_swap raising in the util.ProcessExecutionError failure path?
<blackboxsw> just after the warning and del_file
<Odd_Bloke> blackboxsw: I'm just trying to apply the minimal fix that gets previously-working configurations to continue working.
<Odd_Bloke> (I'm not saying there isn't work to be done to improve the failure mode, but that's not RC.)
<blackboxsw> good pt. so for that, we can avoid the failure path test case (because it exposes this error).  I was trying to write up a failure_path test to get coverage on the errmsg  template as well
<blackboxsw> so we are good with happy_path test only
<johnsonshi> Hi guys, does including both "After=cloud-config.target" and "Before=cloud-config.service"
<johnsonshi>  in a custom systemd unit ensure that the custom systemd unit is started after cloud-init.service finishes but before cloud-config.service starts?
<Odd_Bloke> blackboxsw: Responded to your comment; will also work to improve the tests here.
<Odd_Bloke> (But I don't think we should block landing this on improved tests; it's completely broken right now, I can't make it worse with this change. :p)
<blackboxsw> johnsonshi:  maybe After=network-online.target cloud-config.target and Before cloud-config.service
<blackboxsw> Odd_Bloke: +1 on your PR with that in mind that we will have a followup
<Odd_Bloke> blackboxsw: Ack, thanks.  (Can you mark that in GH so I can land it?)
<blackboxsw> Odd_Bloke: merged
<johnsonshi> blackboxsw: Thanks for the that tip! Hopefully this time the custom systemd unit should always run in the correct order. Thanks!
<rharper> johnsonshi: I think your initial config is sufficient;   cloud-config.target already has After on cloud-init.service; and cloud-config.service runs after cloud-config.target;
<blackboxsw> Odd_Bloke: rharper https://github.com/canonical/cloud-init/pull/318
<blackboxsw> cherry-pick release to review
<Odd_Bloke> I'm not 100% sure how to replicate that locally currently, but that LGTM.
<blackboxsw> Odd_Bloke: sorry added cmdline steps
<blackboxsw> to the PR description
<Odd_Bloke> blackboxsw: +1
<Odd_Bloke> blackboxsw: https://github.com/canonical/cloud-init/pull/319 is the best I can do for additional tests; the sad path is actually broken more deeply than just the variable substitution (as you pointed out), so I'll leave it for now.
<blackboxsw> thanks Odd_Bloke will look it over before tomorrow
<blackboxsw> community notice: [ubuntu/focal-proposed] cloud-init 20.1-10-g71af48df-0ubuntu4 (Waiting for approval)
<blackboxsw> we are hoping to release that swap file creation into Ubuntu Focal before LTS  release is final
#cloud-init 2020-04-16
<otubo> Odd_Bloke, blackboxsw I was actually going to send a PR today for this fixes :-) But thanks anyway! Sorry for not being online at the time you requested the review. I was multitasking on meetings and other fixes yesterday.
<otubo> Anyone interested in discussing this particular bug? https://bugzilla.redhat.com/show_bug.cgi?id=1803928
<ubot5> bugzilla.redhat.com bug 1803928 in cloud-init "[RHEL8.2] Race condition of starting cloud-init and NetworkManager" [High,New]
<otubo> Not sure if that hits upstream as well, perhaps it does.
<otubo> I'll do the rubber duck here until someone reads the log :-D
<otubo> So if I understood correctly, cloud-init-local unit will call cloudinit/cmd/main.py which should start *before* NetworkManager
<otubo> cloud-init will perform the network configuration and *only then* NetworkManager starts and brings the interfaces up
<otubo> So question 1) Why is there the pretty log on cloudinit/cmd/main.py to print all network info, if that's responsible only for applying the configuration (networmanager should bring that up later)?
<otubo> And question 2) on the bugzilla, there's 2 scenarios the one that cloud-init "starts" (prints net info) "in the middle" of NetworkManager and that when it fails. The second is when cloud-init prints net info after NetworkManager does its stuff and that's when it works.
<otubo> In the meanwhile I'll just try to get access to the VM and check some logs.
<StyXman> user expiredate vs chpasswd expire. comments?
<Odd_Bloke> StyXman: Without more context, none. :)
<StyXman> Odd_Bloke: part of the context is the bug I opened yesterday
<StyXman> but basically, I can't make the password expire with either method
<StyXman> in both cases I'm also setting the password; in the bug's case, the password is set *after* the expiration is set, so ir resets it
<StyXman> and I'm gessing somehitng similar happens with chpasswd
<StyXman> hmm, is the yaml processed sequentially?
 * StyXman moves chpasswd to the end of the file
<Odd_Bloke> otubo: No worries!  As I said, we had a real time pressure else we'd have waited a whole day for it.  That said, I think there are more fixes required to that code: the create_swap function swallows ProcessExecutionError but the body of create_swapfile uses ProcessExecutionError to indicate that it should attempt dd after fallocate.  As a result, the fallback will never happen (I believe).
<Odd_Bloke> StyXman: The processing is not sequential; the order of processing is defined in /etc/cloud/cloud.cfg.
<StyXman> Odd_Bloke: ok
<otubo> Odd_Bloke, Understood, I'll take a look at that right after I fix that BZ I posted above. Looks like that's a little more urgent. Boy I need some PTO
<Odd_Bloke> blackboxsw: rharper: OK, so the fix we cherry-picked yesterday didn't mock util.get_mount_info which returns None on the Launchpad builders (rather than any valid content at all), causing test failures.  https://github.com/canonical/cloud-init/pull/319 does mock that and I just did a test PPA build with it included that built successfully:
<Odd_Bloke> https://launchpad.net/~daniel-thewatkins/+archive/ubuntu/temp2/+build/19168108  So a review of that would be appreciated, and then we can have another go at a focal upload.
<Odd_Bloke> blackboxsw: https://github.com/canonical/cloud-init/pull/320 has successfully built locally, and is building in a PPA now.
<Odd_Bloke> blackboxsw: If you review it (and approve, obvs), I'll wait for the PPA build to succeed and then upload.
<kqkq95> Hi al ! someone is using cloud-init with vmware under centos8 ?
<blackboxsw> Odd_Bloke: approved for you to build and push #320
<blackboxsw> not often kqkq95  but if you ask a specific question some folks may have an answer
<blackboxsw> people drop in frequently with vmware/centos questions
<Odd_Bloke> blackboxsw: Thanks!
<kqkq95> blackboxsw : I am trying to configure my vmware with cloud-init template, I use foreman to deploy my servers (they are compatible with cloud-init), I followed this doc: https://docs.theforeman.org/guides/build/doc-Provisioning_Guide/index-foreman.html#Provisioning_Virtual_Machines_in_VMware_vSphere-Provisioning_with_cloudinit_and_userdata_templates
<kqkq95> Except that when my vm starts, I have errors in the logs: https://hastebin.com/retipezonu.sql any ideas ?
<kqkq95> ?
<rharper> kqkq95: sorry, blackboxsw stepped out for a bit;   looking at your trace;  it's hard to say what the problem is;  OVF has had some challenges with the different customization modes and versions of cloud-init;
<rharper> kqkq95: there are also a number of vmware related fixes that are in newer release of cloud-init;
<kqkq95> rharper: ok no problem, i use cloud-init v18.5.7  and vsphere 6.5, how can i debug ? maybe documentation ?
<blackboxsw> sorry back
<rharper> kqkq95: I would suggest trying to run a newer cloud-init; let's see what we have in our daily yum repo ... https://copr.fedorainfracloud.org/coprs/g/cloud-init/cloud-init-dev/
<rharper> that has 20.1;   we also have centos7 19.4, https://copr.fedorainfracloud.org/coprs/g/cloud-init/el-testing/
<rharper> those will have a number of OVF related fixes; which may resolve your issue.    We don't have access to VMware infrastructure making testing/verification not possible;   it's been a rough spot for quite some time unfortunately; we reliant on downstream and VMware to test and fix their datasource.
<blackboxsw> rharper: kqkq95 from your share hastebin, it looks like the only datasource searched is only (DataSourceNoCloudNet) on line 111 ..... instead of DataSourceOVF detection. Typically whe we SRU test we are deploying in a manner that detects OVF
<blackboxsw> kqkq95: here's one of our output logs and scripts that show DataSourceOVF being detected https://github.com/cloud-init/ubuntu-sru/blob/master/manual/vmware-sru-18.5-45-g3554ffe8-0ubuntu1.txt
 * blackboxsw reads back on your guide you linked 
<kqkq95> ok
<blackboxsw> kqkq95: you have a bogus seedfrom configure
<blackboxsw> kqkq95: you have a bogus seedfrom configured
<blackboxsw> https//deploy.phys.pack/userdata/
<kqkq95> why ?
<blackboxsw> is not a valid url that's why you get the message 2020-04-15 19:41:59,845 - DataSourceNoCloud.py[DEBUG]: Seed from https//deploy.phys.pack/userdata/ not supported by DataSourceNoCloud [seed=None][dsmode=net]
<kqkq95> this my url
<blackboxsw> not https//deploy....  should be https://deploy...
<blackboxsw> missing the ':'
<blackboxsw> silly one character always gets ya ;)
<kqkq95> eu
<kqkq95> '=D
<kqkq95> i test again
<kqkq95> I did a lot of testing
<blackboxsw> so I found this by looking for the debug message "Seed from " in cloudinit code base. and found this https://github.com/canonical/cloud-init/blob/master/cloudinit/sources/DataSourceNoCloud.py#L166-L171
<blackboxsw> which pointed me at potentially invalid protocol being the issue.
<blackboxsw> it would have been a better/real message if cloud-init emitted a warning saying ignoring your seedfrom because it isn't one of the supported protocols: [ ("http://", "https://", "ftp://") ]
<kqkq95> i try a new deployment
<kqkq95> blackboxsw: ok so I corrected the url, but I have a new error: https://hastebin.com/apuvuyafuc.sql
<blackboxsw> hrm kqkq95 do you have access to the system's interactive serial console? it looks like cloud-init isn't finding the expected */metadata URL and it's getting a 404 on line 261 after 11 failed retries.
<kqkq95> blackboxsw: yep of couse
<kqkq95> blackboxsw: why he returns me a 404
<blackboxsw> dunno, something on your foreman system (deploy.phys.pack) isn/userdata/meta-data is inaccessible. you'd probably have to test a system console that
<blackboxsw> isn't surfacing userdata/meta-data.
<blackboxsw> sorry about that. I hit enter mid sentence.   something on the system deploy.phys.pack isn't responding to route userdata/meta-data. You might have to check the deploy.phys.pack system to see what's truly configured there
<blackboxsw> but, that's not really a cloud-init problem. cloud-init nocloudnet expects to see both a route <your_url>/user-data and  <your_url>/meta-data from whatever seedfrom you provide it https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html
<blackboxsw> so if those routes don't exist on the host/ip you provid in seedfrom cloud-init rejects it
<kqkq95> blackboxsw: ok indeed, by making a curl from my server deployed to the url https://myforemanurl/userdata/user-data, it responds well 404 instead of showing me my template, I will dig ...
<kunalbansal> Hey I needed some hep regarding the release versioning for cloud-init. Is this the right channel?
<powersj> kunalbansal, possibly
<blackboxsw> kunalbansal: ask away
<kqkq95> blackboxsw: ok i found that was coming from my haproxy front servers by making a curl on my foreman server directly it works! thank you :)
<blackboxsw> good to hear kqkq95
<blackboxsw> another satisfied customer ;)
<kunalbansal> How is minor version for cloud-init is decided. Like right now we get cloud-init-18.5-3-centos when we do "yum install cloud-init-18.5". Is it possible that in future, cloud-init-18.5-4 would be downloaded or 18.5 has been closed.
<kqkq95> blackboxsw: ;)
<blackboxsw> kunalbansal: release versioning is just time/year-based and we update the upstream version of cloud-init around 4 times per year
<blackboxsw> kunalbansal: our current release is 20.1  (the first release of 2020 released on Feb 18th). and 20.2  is scheduled for Apr 28 (in the topic of this IRC channel)
<kunalbansal> Does that mean we would not have a release for 18.5 anymore?
<kunalbansal> and releases always go in a forward session and no bug fixes in previous ones.
<kunalbansal> And the releases that would be coming out would be 20.x versions
<kunalbansal> or forward
<Odd_Bloke> kunalbansal: Upstream, we generally do not backport bug fixes to existing releases.  However, the "-3" part of that version comes from CentOS, and they may choose to do their own backporting.  You would really need to talk to them to understand the update policy you should expect.
<Odd_Bloke> (And presumably they _have_ done some sort of backporting, as -3 would be a strange number to choose for your first version. ;)
<kunalbansal> Yeah that was the confusion i was having. Thanks for clearing that up.
<blackboxsw> sorry I got pulled away before I could properly answer the actual question. my bad .
<blackboxsw> thx Odd_Bloke
<kunalbansal> Thank you that really helps. Do you know a channel where I can get this information.
<blackboxsw> kunalbansal: otubo may have a reference or pointer for some CentOS  related release info. I haven't seen a definition/policy for ongoing intended ports for CentOS. As a note, we (upstream cloud-init) just dropped python 2* support December 2019. So it's a relatively recent change that will adversely affect the ability to backport changes from tip of cloud-init upstream into CentOS7 if that is the environment in
<blackboxsw> which you are running. So, maybe there is a CentOS distro plan for how to handle backports for bug fixes etc.
<blackboxsw> kunalbansal: one path to go is if you have a specific feature or bug for centos you could try filing a bug there and see where the response goes from CentOS folks https://bugs.centos.org/main_page.php
#cloud-init 2020-04-17
<andras-kovacs> Odd_Bloke: Thanks for the info! I made in Azure an Ubuntu 18.04 with official cloud-init support so looking those configs closer I cloud make my own Red Hat 7.8 template work also (with waagent and cloud-init together!)
 * andras-kovacs sent a long message:  < https://matrix.org/_matrix/media/r0/download/matrix.org/VjXqdileLrLUzHDfVYukbNAv >
<andras-kovacs> Or should I just forget about it because all the other things work?
<Odd_Bloke> andras-kovacs: So I suspect that we should handle the absence of that file gracefully, but need more details.  Could you file a bug and attach the `cloud-init collect-logs` tarball to it, please?
<smoser> anyone else using launchpad-chroot ? blackboxsw it think you do
<smoser> i'm trying to use sbuild-it -> sbuild
<smoser> and the sources.list is getting busted by 90apt-sources
<blackboxsw> smoser: I do via build-package script
<smoser> right. same thing.
<smoser> you build focal ?
<smoser> build-apckage calls sbuild
<blackboxsw> will give it a go
<blackboxsw> smoser: we just pushed up a change yesterday that fixed a unittest leak that was breaking ppa builds FTBS etc.
<blackboxsw> but should be unrelated to aptsources... checking
<smoser> this is just launchpad-croot being proken
<smoser> brooken
<blackboxsw> I get you :)
 * blackboxsw waits on sbuild
<smoser> you use sbuild-launchpad-chroot ?
<smoser> do others still use that
<smoser> the problem is that in the launchpad 'livecd.ubuntu-base.rootfs.tar.gz' that gets downloaded
<smoser> bioinc has just one line in /etc/apt/sources.list:
<smoser>  http://archive.ubuntu.com/ubuntu/ bionic main restricted universe multiverse
<smoser> and focal has a "full /etc/apt/sources.list" (just like in a container or whatever)
<smoser> and /etc/schroot/setup.d/90apt-sources parses that wrong
<smoser> in the 'APT_POCKETS' part
<blackboxsw> hrm smoser I may have fallen out of sbuild-launchpad-chroot usage. I: 01launchpad-chroot: [focal-amd64] Processing config
<blackboxsw> I: 01launchpad-chroot: [focal-amd64] Doesn't exist.
<blackboxsw> digging into what that means for my focal sbuild
<smoser> rel=focal; sudo sbuild-launchpad-chroot create --architecture=amd64 --name=$rel-amd64 --series=$rel
<Odd_Bloke> I tried to start using launchpad-chroot earlier in the cycle and it didn't work (so I just kept doing what I was doing already).
<blackboxsw> smoser: can reproduce the problem now
<blackboxsw> https://paste.ubuntu.com/p/QcFRJB53xt/
<smoser> blackboxsw: https://bugs.launchpad.net/ubuntu/+source/sbuild-launchpad-chroot/+bug/1872163
<ubot5> Ubuntu bug 1872163 in sbuild-launchpad-chroot (Ubuntu) "focal chroots broken due to change in sources.list" [Undecided,Confirmed]
<smoser> i filed https://bugs.launchpad.net/ubuntu/+source/sbuild-launchpad-chroot/+bug/1873507
<ubot5> Ubuntu bug 1872163 in sbuild-launchpad-chroot (Ubuntu) "duplicate for #1873507 focal chroots broken due to change in sources.list" [Undecided,Confirmed]
<smoser> i dont really udnerstand. i thought launchpad itself used launchpad-sbuild-chroot
<blackboxsw> smoser: thanks for the links and bugs since I broke my own build system by switching back to sbuild-launchpad-chroot. I've fixed it locally :)
<blackboxsw> pastbin coming and I'll put up a patch for it
<blackboxsw> smoser: # See focal for how to upgrade to
<blackboxsw> # newer versions of the distribution.
<blackboxsw> deb http://archive.ubuntu.com/ubuntu/ focal main restricted
<blackboxsw> # deb-src http://archive.ubuntu.com/ubuntu/ focal main restricted
<blackboxsw> whoa
<blackboxsw> sorry about that
<blackboxsw> https://paste.ubuntu.com/p/vJ25FrJGVj/
<blackboxsw> since it's multiline now sbuild-launchad-chroot needs to be able to also ignore empty lines
<blackboxsw> which was what was killing things
<smoser> did that work ?
<blackboxsw> yes
<blackboxsw> putting up an MP
<smoser> my patch worked too
<smoser> :)
<smoser> yours does work generically ?
<blackboxsw> smoser: here's what it appends https://pastebin.ubuntu.com/p/FnyjzQD6TW/
<blackboxsw> is that bogus to have -proposed on universe etc?
<blackboxsw> ahh shoot I forgot the part that avoids dups
<smoser> no thats fine.
<smoser> and desired
<smoser> i honestly think its probably unintended that the sources.list file is there
<blackboxsw> I think so too. But it would be nice to also allow multiple lines if intended
<blackboxsw> here's the update with avoiding duplicates
<blackboxsw> actually we don't need the duplication reduction, as the APT_COMPONENTS will also reset additional apt config lines all to "main universe" anyway, so if we did have separate lines specifying different components, they are all going to get rewritten anyway to the same component sets. So duplication is inevitable without a rewrite of the whole script, which is not worth it in my mind
<blackboxsw> put up the following PR  https://code.launchpad.net/~chad.smith/ubuntu/+source/sbuild-launchpad-chroot/+git/sbuild-launchpad-chroot/+merge/382529
#cloud-init 2020-04-18
<mydog2> hi
