=== shardy is now known as shardy_mtg
=== shardy_mtg is now known as shardy
[15:54] <Odd_Bloke> smoser: rharper: Does https://bugs.launchpad.net/cloud-images/+bug/1740176 look at all familiar as a cloud-init bug?
[15:54] <ubot5> Launchpad bug 1740176 in cloud-images "Disksize is only 2GB, expected in 10GB" [Undecided,Confirmed]
[15:54] <Odd_Bloke> We haven't triaged it yet, but thought it might look like something you know how to handle.
[16:07] <smoser> Odd_Bloke: did you recreate that ?
[16:16] <Odd_Bloke> smoser: I think Tribaal did.
[16:17] <Tribaal> well, some user reported when using the artful Vagrant image
[16:17] <Tribaal> and I could reproduce it
[16:17] <dpb1> Tribaal: Uppercase T?
[16:18] <Tribaal> dpb1: says the guy with a "1" appended to his nick :)
[16:18] <dpb1> touche!
[16:18] <Tribaal> dpb1: I thin klowercase was already taken
[16:19] <dpb1> good ol irc
=== Tribaal is now known as tribaal
[16:19] <tribaal> there! :)
[16:19] <dpb1> hi tribaal
[16:19] <dpb1> welcome
[16:27] <blackboxsw> tribaal: yo. we were just talking about that bug
[16:28] <tribaal> oh?
[16:28] <blackboxsw> smoser: thought it might be reltaed to https://bugs.launchpad.net/cloud-images/+bug/1726818
[16:28] <ubot5> Launchpad bug 1726818 in linux (Ubuntu) "vagrant artful64 box filesystem too small" [High,In progress]
[16:28] <blackboxsw> and happy new year BTW
[16:28] <smoser> yeah, i marked as a dupe. it certainly smells that way.
[16:28] <tribaal> that looks like a total dupe indeed
[16:29] <smoser> tribaal: it would be good to know if this is present in bionic
[16:29] <tribaal> smoser: give me a sec, should be easy to try
[16:30] <smoser> as it really needs to be fixed in bionic.
[16:30] <tribaal> +100
[16:31] <smoser> tribaal and then, it looks like there is a 4.14 in proposed.
[16:31] <smoser> i suspect that this is probably still present in bionic in the 4.13 that is in the release pocket
[16:31] <smoser> but has a chance of being fixed in -proposed 4.14
[16:31] <smoser> https://launchpad.net/ubuntu/+source/linux
[16:35] <tribaal> smoser: indeed, bionic is affected as well (well, as of the 20180101 image)
[16:38] <tribaal> smoser: I'll enabled -proposed in the machine and reboot - could you remind me what to nuke to make cloud-init think it's running for the first time? /var/lib/cloud/*?
[16:39] <smoser> tribaal: cloud-init clean
[16:39] <tribaal> smoser: TIL! Sweet
[16:42] <smoser> tribaal: i'd wonder though if the image might be 'dirty' at that point
[16:42] <smoser> i dont know how easy it is to supply your own image to vagrant
[16:42] <smoser> but you might be able to modify
[16:42] <smoser>  https://github.com/cloud-init/qa-scripts/blob/master/scripts/get-proposed-cloudimg
[16:42] <smoser> or basically do what it does
[16:43] <smoser> better to create yourself a "clean" image
[16:43] <tribaal> smoser: yeah, I could just build a vagrant image with -proposed enabled but that takes much longer
[16:43] <smoser> blackboxsw:
[16:43] <smoser> root@b2:~# cloud-init clean
[16:43] <smoser> ERROR: Could not remove instance: Cannot call rmtree on a symbolic link
[16:43] <blackboxsw> boo!
[16:43] <blackboxsw> will work up a fix
[16:43] <tribaal> (and I'm almost EOD)
[16:43] <smoser> exits non-zero too, and that is clearly not a error.
[16:44] <smoser> tribaal: well, essentially i suspect you may be able to do
[16:45] <smoser>   mount-image-callback your.orig.image --system-resolvconf -- /bin/bash
[16:45] <smoser> then inside, just enable proposed, apt-get update, apt-get install linux-image (or whateve rpackage that is)
[16:45] <smoser> then exit
[16:50] <tribaal> the dirty way didn't produce the clear case, so I'll have to do something like this, yes (or just build an image with -proposed enabled)
[17:01] <blackboxsw> smoser: how'd you reproduce that cloud-init clean. I launched a bionic container, by default it doesn't hit this symlink error
[17:02] <blackboxsw> I assumed bionic because your instance was named b2
[17:02] <blackboxsw> in either case, I can fix it easily enough. but wondered how we got there
[17:19] <powersj> blackboxsw: https://paste.ubuntu.com/26314129/
[17:20] <blackboxsw> +1 powersj I  have a fix, just adding unit tests, will
[17:20] <blackboxsw> +1 powersj I  have a fix, just adding unit tests, will try to reproduce here too
[17:27] <smoser> blackboxsw: hm.
[17:28] <smoser> i just fresh launched bionic-daily container
[17:31] <blackboxsw> csmith@uptown:~$ lxc --version
[17:31] <blackboxsw> 2.0.11
[17:31] <blackboxsw> heh. powersj
[17:31] <blackboxsw> xenial
[17:31] <blackboxsw> for the loss
[17:32] <powersj> except I wouldn't expect that to make a difference in this case :\
[17:32] <powersj> right? should only be the version of cloud-init in the image
[17:32] <blackboxsw> one would think. I'm testing from zesty
[17:33] <smoser> hm.
[17:34] <blackboxsw> https://pastebin.ubuntu.com/26314191/
[17:34] <blackboxsw> yeah from zesty, still no problem on my side
[17:34] <blackboxsw> anyway we can easily test for link and unlink if needed
[17:35] <smoser> this is weird
[17:35] <blackboxsw> but not quite sure why that shows up in some cases
[17:35] <smoser> i reproduced again, but then not
[17:36] <powersj> 299e803c9fe1 | no     | ubuntu 18.04 LTS amd64 (daily) (20180101)
[17:36] <powersj> that's the bionic image I used
[17:36] <smoser> http://paste.ubuntu.com/26314202/
[17:38] <blackboxsw> https://pastebin.ubuntu.com/26314219/
[17:38] <blackboxsw> seemingly the same, but I get stuccess (I would've expected it to always fail
[17:38] <blackboxsw> on instance symlink
[17:38] <smoser> yeah. hmm.
[17:39] <blackboxsw> I'm adding a debug print
[17:40] <smoser> blackboxsw: i am guessing.
[17:40] <smoser> but i suspect thatlistdir()
[17:40] <smoser> listdir('.')
[17:40] <smoser> is not any specific order
[17:41] <smoser> and that when it does instances first its ok but when instance first it is not
[17:41] <smoser> or reverse
[17:43] <blackboxsw> yeah, I could have sorted() the dir list and then we would've always seen it. I bet becaause is_dir returns false when a symlink target is already broken
[17:43] <smoser> http://paste.ubuntu.com/26314255/
[17:43] <blackboxsw> yeah that's the fix I have
[17:44] <blackboxsw> same one. just wanted to know why
[17:44] <smoser> sorted is arbitrary
[17:44] <blackboxsw> it seems dirlist is indeterminate apparently. as you see the issue only sometimes
[17:44] <smoser> yeah, its not guaranteed sorted.
[17:45] <smoser> it is just traversing the dirent
[17:45] <smoser> and even if it was, its arbitrary that 'instnace' would sort before 'instances'
[17:47] <blackboxsw> https://pastebin.ubuntu.com/26314266/
[17:48] <blackboxsw> I would've thought sorting would have put instanec before instances too, but yeah the dirent iterartor isn't sorted
[17:48] <blackboxsw> ok anyway pushing the fix and unit test
[17:49] <blackboxsw> I wouldn't thought that even a sorted list would have '
[17:49] <blackboxsw> I wouldn't thought that even a sorted list would have 'instances' > '
[17:49] <blackboxsw> 'instance'
[17:49] <blackboxsw> but maybe that's arbitrary too as you point uot
[17:50] <blackboxsw> sorry was typing on a different keyboard.
[17:56] <smoser> blackboxsw: sorting would fix it i think, but just seems arbitrary from the perspective of it will start to fail again if we had a link named zz to aa
[17:58] <blackboxsw> right, yeah it was a fragile fix to sort(and would have been wrong)
[17:58] <blackboxsw> because it ignored the problem (that we weren't handling symlinks)
[17:58] <blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/335671 has the fix
[17:58] <blackboxsw> didn't realize I was doing something a bit different than you suggestion.
[17:58] <blackboxsw> utils util.is_link instead
[17:58] <blackboxsw> s/utils/using/
[18:13] <dojordan> @blackboxsw - happy new year! is there anything blocking checking in my PR to master? https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+ref/azure-preprovisioning
[18:17] <blackboxsw> dojordan: I think there was a side discussion I had with smoser that we might hit an issue with systemd unit timeouts. if we attempt block indefinitely in a polling loop in cloud-init's unit systemd might timeout at 5 minutes??? which could cause the behavior you are looking for to fail.
[18:18] <blackboxsw> I'm not sure about the 5 minute auto-timeout in systemd, lemme find a reference to see if I can dig up a doc on it
[18:18] <dojordan> I've tested in 16.04 with polling for much longer than that, and systemd didn't kill cloud init or anything
[18:19] <dojordan> what is the service name?
[18:21] <blackboxsw> cloud-init.service I believe
[18:23] <dojordan> doug@dojordandev:~$ systemctl show cloud-init.service -p TimeoutStopUSec TimeoutStopUSec=infinity
[18:23] <dojordan> thats why is worked :)
[18:24] <blackboxsw> or cloud-init.targetahh there you go
[18:25] <blackboxsw> ooops
[18:25] <blackboxsw> ahh I mean
[18:26] <blackboxsw> looks like 'we' (cloud-init) don't explicitly set that timeout to inifinity, but me OS-specific setting
[18:26] <blackboxsw> ohh wait
[18:26] <blackboxsw> TimeoutSec=0
[18:27] <dojordan> that infinity timeout was on the azure 16.04 LTS image
[18:27] <blackboxsw> looks like we set that for the systemd/cloud-final.service.tmpl
[18:27] <dojordan> got it
[18:28] <blackboxsw> ok this might not really be an 'issue' then, though it *may* be worth us explicitly configuring that inifinity timeout on azure images... I'm not certain.
[18:28] <dojordan> is 0 == infinity?
[18:30]  * blackboxsw thinks, as TmieoutSec == setting for both TimeoutStart and timeoutStop
[18:30] <blackboxsw> but trying to confirm
[18:30] <blackboxsw> sd_notify(3)).
[18:30] <blackboxsw> TimeoutSec=
[18:30] <blackboxsw> A shorthand for configuring both TimeoutStartSec= and TimeoutStopSec= to the specifie
[18:30] <blackboxsw> from https://www.freedesktop.org/software/systemd/man/systemd.service.html
[18:31] <smoser> dojordan: i think there is still general issues with timeouts
[18:31] <blackboxsw> though I don't see a reference that "0" == 'infiinity' for Timeout(Stop/Start)Sec
[18:32] <dojordan> it just says "Pass "infinity" to disable the timeout logic." on the timeoutstopsec
[18:32] <smoser> hm..
[18:32] <dojordan> but are we currently doing that...
[18:32] <smoser> buti dont know how other things in boot handle it
[18:32] <smoser> so cloud-init-local or cloud-init.service may have TimeoutSec set correctly
[18:33] <smoser> maybe i'm worng.
[18:33] <dojordan> doug@dojordandev:~$ systemctl show cloud-init.local -p TimeoutStopUSec TimeoutStopUSec=1min 30s
[18:33] <smoser> but i swear if've seen boot just go on when i didn't think it should.
[18:33] <dojordan> so local is showing 1:30, but service is showing infinite
[18:34] <smoser> dojordan: well you typo'd there.
[18:34] <smoser> cloud-init.local is not a service
[18:34] <smoser> cloud-init-local.service
[18:34] <dojordan> d'oh....
[18:35] <dojordan> "cloud-init-local.service" == TimeoutStopUSec=infinity
[18:42] <smoser> dojordan: so how does /var/lib/waagent/poll_imds get written?
[18:42] <smoser> newerubuntu and cloud-init i think do not rely on waagent at all
[18:42] <smoser> and i'm not really interested in adding such a dependency back
[18:42] <dojordan> there is no dep on waagent
[18:43] <dojordan> we can change the path - it should probably be in the instance directory
[18:43] <smoser> what creates it ?
[18:43] <dojordan> the azure data source
[18:43] <smoser> oh. ok. yeah. i see that. the waagent thrrew me off.
[18:44] <dojordan> also, I confirmed why the timeout is infinity - since we are using type=oneshot the timeout is disabled
[18:44] <smoser> i suspect that 0 did mean infinity at some point
[18:45] <smoser> and probably still does
[18:45] <smoser> REPROVISION_MARKER_FILE, why do we need that?
[18:46] <dojordan> the marker file (/var/lib/waagent/poll_imds) is needed in case the VM reboots for whatever reason before it is reused by a customer
[18:46] <smoser> rather than internal state or something.
[18:46] <dojordan> we report ready to the fabric which means we detach the provisioning ISO
[18:46] <smoser> why would it reboot?
[18:46] <dojordan> hardware, software updates, etc
[18:46] <dojordan> underlying platform
[18:47] <dojordan> and we don't write the Ovf since we don't want to persist any azure specific data since the real ovf will come from the customer
[18:47] <dojordan> we will be in this polling loop for a while
[18:47] <dojordan> and if there is an unexpected reboot we want to keep polling when the vm comes back up
[18:49] <smoser> it seems odd that the platform would choose to reboot such a machine
[18:49] <dojordan> in azure, all of our VMs are backed by remote storage. so when hardware issues occur, we simply move the VM to a new machine since the data is persisted and reboot it
[18:51] <dojordan> by data i mean the os vhd
[18:52] <smoser> seems like maybe you could just kill machines in this state. as they're not owned by anyone while maybe everything "should work" if you just kill a machine while booting and then re-start it, i suspect that in reality there are lots of issues.  but thats not really important here.
[18:55] <paulmey> smoser: true, but that would require some rearchitecting on a different level which is not going to happen any time soon...
[18:58] <dojordan> the same behavior is true today for all VMs. if the reboot before cloud-init finishes we just move thme
[19:30] <dojordan> @smoser these are all valid points. the underlying platform is not perfect, and there are cases today where we don't get an ACK from our remote storage layer, then the VM will be busted. I think the important thing about the marker file is that allows us to keep the pre provisioned vms around for longer which enables us to have a higher hit rate for reuse and therefore increase boot performance. the availability won't be any w
[19:37] <smoser> dojordan: i responded on mp there.
[19:37] <smoser> sorry for taking so long to take a look at your proposal
[19:37] <smoser> dojordan: please don't take offense. over all, you've done a good job.
[19:38] <dojordan> no worries, appreciate the feedback. Will address the PR comments later today
[20:03] <blackboxsw>  pushed a couple changes to the review-mps script in qa-scripts repo for landing branches
[20:03] <blackboxsw> slowly building it into something useful/working
[20:03] <blackboxsw> thx for the review btw. landed
[20:07] <ivve> heya guys, im doing a really ugly hack with write_files but been getting lots of trouble getting write_files to work at all, i have a hard time writing two files as well. first example https://hastebin.com/otubucuqin.pl , second example https://hastebin.com/egoxufopab.js
[20:08] <ivve> like the simplest example with only a path + content works... other than that i have a really hard time getting it to execute
[20:09] <ivve> is my syntax way off?
[20:10] <blkadder> You might find it easier just to base64 it.
[20:11] <ivve> ok
[20:11] <ivve> so - encoding: b64 ?
[20:11] <ivve> and then just content: |
[20:11] <blkadder> One sec.
[20:12] <blkadder> write_files:
[20:12] <blkadder>    - encoding: b64
[20:12] <blkadder>      content: T3JpZ2luOiByZXBvLnFiaXMuY28KTGFiZWw6IHJlcG8ucWJpcy5jbwpDb2RlbmFtZTogeGVuaWFsCkFyY2hpdGVjdHVyZXM6IGkzODYgYW1kNjQgc291cmNlCkNvbXBvbmVudHM6IG1haW4KRGVzY3JpcHRpb246IHFiaXMgcmVwbwpTaWduV2l0aDogZGVmYXVsdCAK
[20:12] <blkadder>      path: /tmp/distributions
[20:12] <blkadder>      owner: root:root
[20:12] <blkadder>      permissions: '0644'
[20:12] <blkadder> Sorry that may not have come through very well.
[20:12] <ivve> no worries
[20:13] <ivve> so i have to encode it then
[20:13] <blkadder> https://paste.ubuntu.com/26315006/
[20:13] <blkadder> Yes base64 -w0
[20:13] <blkadder> Copy/paste
[20:13] <ivve> alright
[20:16] <blkadder> I dont’ thinkl your second example will work.
[20:17] <blkadder> http://cloudinit.readthedocs.io/en/latest/topics/examples.html
[20:17] <blkadder> It’s very picky about where you put “-“ and spaces, etc.
[20:23] <ivve> aye
[20:26] <ivve> doesn't seem to like multiple files at all :P
[20:26] <ivve> i got it to work once or twice
[20:26] <ivve> this isn't working either
[20:26] <ivve> :(
[20:29] <smoser> ivve:  i suspect that you have general yaml issue.
[20:30] <ivve> aye, i keep getting [   18.299202] cloud-init[861]: 2018-01-03 20:29:00,205 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: '#cloud_config...'
[20:30] <smoser> oh. yeah.
[20:30] <smoser> cloud-init will ignore that
[20:30] <smoser> if it does not start with '#cloud-config'
[20:30] <ivve> it does
[20:31] <smoser> or is not declared as cloud-config with multipart
[20:31] <smoser> just add '#cloud-config' as first line.
[20:31] <ivve> oh shit
[20:31] <ivve> _ not -
[20:31] <ivve> :D
[20:31] <ivve> oh man
[20:31] <smoser> ah. yeah. funnyt.
[20:31] <ivve> i used a heat template first and just shortcutted
[20:32] <ivve> added # and removed the :
[20:32] <ivve> jeez what an idiot i am
[20:33] <smoser> ivve: whenever someone shows me something like the hastebin there, the first thing i do is use 'yaml-dump'
[20:33] <smoser>  http://paste.ubuntu.com/26315078/
[20:33] <smoser> json output is much more clear and identifies errors to a human more clearly.
[20:34] <smoser> (you didn't mess up that way, this is just fyi)
[20:34] <ivve> aye its a good pointer, thanks
[20:35] <ivve> however i can't read json even if my life depended on it
[20:35] <ivve> well it works now
[20:35] <ivve> i guess i will be encoding stuff now
[20:35] <ivve> it help writing stupid expect scripts :P
[20:36] <blkadder> Use IntelliJ
[20:36] <blkadder> It’s a life saver.
[20:38] <blkadder> And it has a vi/vim mode too which is nice.
[20:41] <ivve> stack completed, music to my ears :P
[20:41] <ivve> thanks a bunch guys
[20:42] <blackboxsw> while it's a little late in the process, your machine with cloud-init  installed can run 'cloud-init devel schema -c your-configfile.yaml     to validate the yaml https://pastebin.ubuntu.com/26315122/
[20:42] <smoser> :)
[20:43] <blackboxsw> it at least gives you a quick once over on the yaml file once you've discovered something didn't work as you expected
[20:44] <blkadder> Is that in mainline now?
[20:44] <blackboxsw> yep. should be in xenial and greater
[20:44] <blackboxsw> and on trukn
[20:44] <blackboxsw> and on trunk
[20:44] <blkadder> Cool.
[20:45] <ivve> aye its a good pointer as well and i did think about it but never came to that point since i just added a file and stuff stopped working
[20:45] <blackboxsw> needs a lot of work as it'll eventually support all attributes of each cloud-config module and --annotate
[20:45] <blackboxsw> to report specific errors
[20:45] <ivve> runcmd: also doesn't like " or ;
[20:45] <ivve> or : for that matter
[20:47] <blackboxsw> my new year's resolution is to make "cloud-init schema" a first-class citizen with support for reporting schema errors in all 54 cloud-config modules
[20:47] <blkadder> ivve: https://paste.ubuntu.com/26315137/
[20:47] <blkadder> Someone here helped me with that… :-)
[20:47] <blackboxsw> we'll see if that holds (hopefully better than the "exercising daily" new year's resolution)
[20:48] <blkadder> Yay exercise.
[20:49] <blkadder> I highly recommend: https://stronglifts.com/5x5/ Only 3 or 4 days a week. No two hours at the gym. :-)
[20:50] <ivve> ah yes ofc you can type it that way to get it right
[20:50] <ivve> however as you are pointing out help is needed to write it and read it :P
[20:50] <blkadder> Well that gives you some syntax to pattern off of.
[20:50] <blkadder> Now that I have that I can generally manage on my own.
[20:51] <ivve> is just preferred using write_files to write the script and runcmd: - bash/expect /path/to/script
[20:51] <ivve> its just a hack for a demo, nothing proper
[20:51] <ivve> i'd use ansible for proper stuff