[00:17] <redcavalier> Hey, I've ran into another weird cloud-issue issue on our centos instances in our Openstack setup.
[00:19] <redcavalier> This time, there's just this one VM which, for some reason, considers its cloud-init cache corrupt on every reboot.
[00:21] <redcavalier> So basically, it ignores its cache, deletes it and re-execute all metadata/userdata every reboot.
[00:26] <redcavalier> This seems to be unique to this one instance though, so I'm not sure what to check beside what I've already check. I think something may be interfering.
[00:27] <smoser> redcavalier: a /var/log/cloud-init.log would be the starting point.
[00:28] <smoser> for debugging
[00:29] <redcavalier> Yea, that'S how I know it'S discarding its cache. I can post it on a pastebin
[00:48] <redcavalier> smoser, http://pastebin.centos.org/385126/ . Line 25 bothers me especially, as the cache gets deleted every reboot.
=== jpmethot is now known as redcavalier
[14:51] <blackboxsw> bummer, powersj, the disabled fastestmirror plugin wasn't the magic bullet I'd hoped. still hit subprocess.CalledProcessError: Command '['yum', 'install'... on https://jenkins.ubuntu.com/server/job/cloud-init-ci/415/console   :(
[14:54] <powersj> :(
[15:25] <powersj> btw is smoser morning the loss today?
[15:25] <powersj> too soon? ;)
[15:38] <smoser> powersj: yeah, in mourning this morning.
[15:41] <smoser> now my only hope is in the Houstan Astros.  I can't imagine anyone ever wants to see the Yankees win, and I'm kind of sore on the dodgers right now.
[15:45] <powersj> heh I missed a 'u'
[15:45] <smoser> powersj: so looking at https://jenkins.ubuntu.com/server/job/cloud-init-integration-a/171/consoleText
[15:45] <smoser> i remember we had this in curtin
[15:45] <smoser> its hard to tell easily what test failed
[15:46] <powersj> yeah because of the large amount of text that gets spit out
[15:46] <powersj> I usually search for "FAIL:"
[15:46] <smoser> well, yes, but also because
[15:46] <smoser>  test_no_warnings_in_log (tests.cloud_tests.testcases.get_suite.<locals>.tmp) ... FAIL
[15:46] <smoser> doesnt tell me anything
[15:46] <smoser> i thought in curtin we did something so that it would list the class being run
[15:47] <powersj> well it is telling you the test that fails :)
[15:47] <powersj> that there are no warnings in the log
[15:48] <powersj> We did do that in Curtin and I even thought we already did that in our tests... let me check
[15:50] <powersj> hmm you are right, this is not obvious where that test lives
[15:52] <powersj> oh wait, we do have the change
[15:52] <powersj> in base.py:shortDescription(self)
[15:52] <powersj> """Prevent nose from using docstrings."""
[17:21] <smoser> powersj: so i download artifacts from jenkins
[17:22] <smoser> is there a way i can run 'verify' with those ?
[17:23] <powersj> smoser: python3 -m tests.cloud_tests verify -h
[17:23] <smoser> http://paste.ubuntu.com/25780059/
[17:23] <powersj> point to it via the --data-dir variable
[17:23] <smoser> thats my failed attempt^
[17:24] <powersj> give it to /cloud-init/results
[17:24] <powersj> sorry to be more specific --data-dir=/tmp/artful-171/archive/cloud-init/results
[17:27] <smoser> thanks
[17:34] <smoser> so why doesnt shortDescription work
[17:34] <smoser> :-(
[17:35]  * smoser nothaving a good day
[17:35] <smoser> oh.
[17:36] <smoser> cause we dont run nose
[19:04] <smoser> powersj, blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332585
[19:05]  * blackboxsw thinks that was related to you testing my branch
[19:05] <blackboxsw> just a 'feeling'
[19:06] <blackboxsw> :)
[19:11] <powersj> +1'ed
[19:12] <smoser> blackboxsw: were you fixing the ntp warning ?
[19:12] <blackboxsw> smoser: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/332540
[19:12] <blackboxsw> was debating about fixing the lxd warning in that branch
[19:12] <blackboxsw> if you want
[19:12] <blackboxsw> or separate branch
[19:13] <smoser> i have http://paste.ubuntu.com/25780634/
[19:13] <smoser> that i was going to grab in a "actually fix all warnings" merge
[19:13] <smoser> i can fix the ntp also, but if you had a branch that fixed it i woudl not
[19:13] <blackboxsw> yeah if you have a separate branch for that you're good, or take mine to supplement yours
[19:14] <blackboxsw> smoser: yeah my branch fixes ntp plus adds a unit test that'll show these integration test errors to us earlier once jsonschema is defined for a module
[19:14] <blackboxsw> it didn't address the lxd cfg warning throuhg
[19:14] <blackboxsw> though
[19:16] <smoser> blackboxsw: i have that one fine.
[19:17] <smoser> so if you're fixing jsonschema warning in
[19:17] <smoser>  tests/cloud_tests/testcases/modules/ntp.yaml
[19:17] <blackboxsw> yep fixed per the above branch
[19:17] <smoser> then we'll just grab yours, then mine. then i think we'll be happy on no failures from warnings
[19:17] <blackboxsw> yeah for sure
[19:17] <smoser> but /me doesn't kniow why ididnt see these when i ran...
[19:17] <smoser> i really promise ir an!
[19:17] <smoser> I ran
[19:17] <blackboxsw> we didn't use to have a warning validation test did we?
[19:18]  * blackboxsw looks over the git logs again.
[19:19] <blackboxsw> yeah, maybe you were referring to when you landed 41152f10ddbd8681cdac44b408038a4f23ab02df
[19:19] <smoser> blackboxsw: ... confused.
[19:19] <blackboxsw> I certainly am. I'll take another good sir
[19:19] <smoser> never mind
[19:19] <smoser> you fixsed it right
[19:19] <smoser> i was confused that modules/ntp.yaml had empty dicts for pools and servers
[19:19] <smoser> but you fixed that ;)
[19:20] <blackboxsw> right-o
[19:32] <smoser> blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332587
[19:32] <smoser> that has the other fixes
[19:33] <smoser> i think enough to get a integration test fully run
[19:33] <blackboxsw> reviewing (will have a minor test patch for that branch)
[19:33] <blackboxsw> landed the ntp thanks
[19:33] <blackboxsw> & thx powersj
[19:38] <smoser> im going to grab https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332585 after ci ACKs it
[19:39] <blackboxsw> yes sorry, distracted on the followup
[19:39] <blackboxsw> approved
[19:39] <smoser> powersj had already done it, so i wasnt going to wait on you :)
[19:39] <powersj> :)
[19:40] <smoser> Your code has been rated at 10.00/10
[19:40] <smoser> i love that.
[19:40] <smoser> 2 significant digits of perfection.
[19:40] <smoser> or is that 4
[19:41] <smoser> either way. i'm like the Mary Lou Retton of python coding.
[19:41] <blackboxsw> hahah
[19:44] <smoser> powersj: https://github.com/canonical-server/jenkins-jobs/blob/master/cloud-init/integration.yaml
[19:44] <smoser> in that, i think we're making the same sort of error as we were in curtin at one point.
[19:45] <powersj> which error :)
[19:45] <smoser> we're running trunk c-i against distro version of the code
[19:45] <smoser> err.. trunk level of integration test with distro version of code
[19:46] <powersj> I'm taking trunk, and building it using an sbuild of a particular release
[19:46] <blackboxsw> smoser: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332587 updated
[19:46] <powersj> there should be no test mismatch there
[19:46] <powersj> if I were, however, to take the daily build of cloud-init and run it against the trunk tests then I would agree
[19:47] <powersj> daily build of cloud-init for a particular release* to be even more specific
=== blackboxsw changed the topic of #cloud-init to: is Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj |  Next status meeting: Monday 10/16 16:00 UTC | cloud-init 17.1 released | quotes:  <@smoser> either way. i'm like the Mary Lou Retton of python coding.
[19:47] <blackboxsw> :)
[19:47] <smoser> yeah, you're right.
=== blackboxsw changed the topic of #cloud-init to: is Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj |  Next status meeting: Monday 10/16 16:00 UTC | cloud-init 17.1 released
[19:51] <smoser> hm.. so why did i not see that failure.
[19:51] <smoser> https://jenkins.ubuntu.com/server/job/cloud-init-ci/419/console
[19:52] <blackboxsw> not behind a proxy?
[19:52] <blackboxsw> where fastestmirror plugin is disabled?
[19:53] <smoser> i thought we disabled it.
[19:53] <smoser> where was that mp ?
[19:53] <blackboxsw> hrm: Loaded plugins: fastestmirror
[19:53] <blackboxsw> that mp landed yesterday.
[19:53] <blackboxsw> digging it up
[19:53] <blackboxsw> https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/332511
[19:54] <blackboxsw> strange to see fastestmirror showing up on that run as I thought it was behind a proxy (and as such fastestmirror should be disabled)
[19:54] <blackboxsw> or maybe I'm just focusing on the wrong problem
[19:55] <powersj> your branch doesn't have my fix?
[19:55] <blackboxsw> checking your branch out to make sure it has the fix
[19:56] <powersj> it doesn't
[19:56] <powersj> https://git.launchpad.net/~smoser/cloud-init/tree/tools/run-centos?h=fix/citest-show-class-in-failures
[19:57] <blackboxsw> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332585 doesn't yeah : )
[19:57] <blackboxsw> rebase for the win
[19:58] <blackboxsw> too bad that canadian fastestmirror will be sad
[20:02] <smoser> blackboxsw: thanks.
[20:19] <smoser> https://jenkins.ubuntu.com/server/job/cloud-init-ci/422/ is about to post on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332585
[20:20] <smoser> and then happy i'll pull that
[20:24] <smoser> blackboxsw: did you sort the maas/-proposed thing ?
[20:28] <blackboxsw> nope smoser I'm trying to figure out what gives with https://bugs.launchpad.net/cloud-init/+bug/1684869 and why it doesn't generate a 'proposed' image for me
[20:28] <ubot5> Ubuntu bug 1684869 in cloud-init (Ubuntu Artful) "growing root partition does not always work with root=PARTUUID=" [Medium,Fix released]
[20:28] <blackboxsw> was trying to reproduce the resize failure locally
[20:31] <smoser> hm..
[20:35] <smoser> blackboxsw: https://git.launchpad.net/~smoser/cloud-init/+git/sru-info/plain/bin/get-proposed-cloudimg just "worked for me"
[20:35] <smoser> http://paste.ubuntu.com/25781060/
[20:36] <blackboxsw> weird I think I'm being dumb.
[20:36] <blackboxsw> the zesty image I downloaded didn't upgrade cloud-init
[20:36] <blackboxsw> checking logs
[20:37] <smoser> oh. yeah. it would not i guess.
[20:37] <smoser> or possibly
[20:37] <smoser> as cloud-init got kicked maybe ?
[20:37] <smoser> hm.. no i get it there.
[20:37] <blackboxsw> pulling latest get-proposed-cloudimg to check
[20:39] <smoser> hm.. why is sfdisk --part-uuid not working
[20:39] <blackboxsw> uefi images on xenial right?
[20:39] <smoser> oh. no. what a pita
[20:40] <smoser> xenial are mbr
[20:40] <smoser> fudge
[20:40] <blackboxsw> https://bugs.launchpad.net/cloud-init/+bug/1684869/comments/8
[20:40] <ubot5> Ubuntu bug 1684869 in cloud-init (Ubuntu Artful) "growing root partition does not always work with root=PARTUUID=" [Medium,Fix released]
[20:40] <blackboxsw> :)
[20:42] <blackboxsw> ok my get-proposed-cloudimg was stale
[20:47] <blackboxsw> ok my network in the kvm I created is borked. Could not resolve host: entropy.ubuntu.com or anything else, which is why my proposed cloud-init package didn't update
[20:48] <smoser> blackboxsw: testing this
[20:49] <smoser>  http://paste.ubuntu.com/25781127/
[20:49] <smoser> dns issues scare me
[20:49] <smoser> artful guest ?
[20:52] <blackboxsw> artful host, zesty guest
[20:52] <blackboxsw> trying on another box to be certain
[20:52] <smoser> how did you make guest ?
[20:53] <blackboxsw> strange it's working on another box. ok will copy it in
[20:53] <smoser> oh. inside the mounted image you were failing dns ?
[20:55] <smoser> my test got throug, following that bug and / got rezied on xenial.
[20:55] <smoser> tryinng now with zesty
[20:55] <blackboxsw> smoser: yes inside the image
[20:56] <blackboxsw> and inside the image is cloud-init 0.7.9
[20:56] <blackboxsw> not 17.1 as I had expected to see
[20:57] <blackboxsw> 0.7.9-233-ge
[20:57] <blackboxsw> yeah
[20:57] <smoser> ok. yeah. so you have to (and i made this mistake) boot the -proposed
[20:57] <blackboxsw> ok I performed the steps I had used in previous SRU https://bugs.launchpad.net/cloud-init/+bug/1684869
[20:57] <ubot5> Ubuntu bug 1684869 in cloud-init (Ubuntu Artful) "growing root partition does not always work with root=PARTUUID=" [Medium,Fix released]
[20:57] <blackboxsw> ahh gotcha right
[20:59] <smoser> blackboxsw: so modify that template on the bug like:
[20:59] <smoser> proposed=${raw%.*}-proposed.img
[21:00] <smoser> qemu-img create -f qcow2 -b $proposed disk.img 10G
[21:00] <smoser> but i did that just now for zesty and worked fine. verified cloud-init 17.1.... in the guest
[21:02] <blackboxsw> oooooh
[21:02] <blackboxsw> oops
[21:02] <blackboxsw> thanks
[21:08] <blackboxsw> ok, made it to the finish line. see the resize succeed on 17.1
[21:08] <blackboxsw> zesty
[21:08] <blackboxsw> soooo, our test case doesn't validate https://bugs.launchpad.net/cloud-init/+bug/1725067   for some reason
[21:08] <ubot5> Ubuntu bug 1725067 in cloud-init (Ubuntu) "cloud-init resizefs fails when booting with root=PARTUUID=" [Medium,Triaged]
[21:09] <blackboxsw> and bbsw is out of his depth
[21:09] <smoser> blackboxsw: doesnt validate ?
[21:15] <blackboxsw> I need knowledge smoser. got 10 mins before your weekend?
[21:16] <smoser> blackboxsw: http://paste.ubuntu.com/25781261/
[21:16] <smoser> yeah, we can chat 10  minutes
[21:16] <smoser> that "works for me" to recreate failure actually in xenial
[21:17] <blackboxsw> ok, I was wondering what I did wrong as I saw resizes happening on 17.1 zesty I thought
[21:17] <smoser> i'm in hangout
[21:17] <smoser> and yeah, it does seem to work fine for zesty
[21:18] <blackboxsw> sorry joining
[21:18] <blackboxsw> was running your script
[21:33] <smoser> blackboxsw: for irc logs
[21:33] <smoser> https://github.com/cloud-init/qa-scripts/blob/master/scripts/get-proposed-cloudimg
[21:34] <smoser> https://github.com/cloud-init/ubuntu-sru/blob/master/bugs/lp-1684869/recreate.sh
[21:34] <blackboxsw> yeah thanks, stinks to close the hangout window  when you didn't grab all the links discussed
[21:38] <smoser> blackboxsw: ok. so the difference between xenial and zesty
[21:38] <smoser> on zesty, *something* is making /dev/root exist
[21:38] <smoser> but on xenial
[21:39] <smoser> $ ls -l /dev/root
[21:39] <smoser> ls: cannot access '/dev/root': No such file or directory
[21:39] <smoser> zesty
[21:39] <blackboxsw> ahh interesting... ok
[21:39] <smoser> $ ls -l /dev/root
[21:39] <smoser> brw------- 1 root root 8, 1 Oct 20 21:39 /dev/root
[21:39] <blackboxsw> right I'm with you
[21:40] <blackboxsw> saw the failure on xenial now thanks
[21:40] <smoser> yeah, so fix at this point should be pretty straight forward knowing what we did originally and such.
[21:40] <smoser> have a good one.
[21:40]  * smoser out
[21:45] <blackboxsw> yeah thanks again you too
[21:57] <smoser> blackboxsw: https://jenkins.ubuntu.com/server/job/cloud-init-ci/426/console
[21:57] <smoser> that is from https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332587
[21:57] <blackboxsw> no fastestmirror
[21:57] <blackboxsw> hrm
[21:57] <smoser> yeah
[21:57] <smoser> its definitely not that coee
[21:58] <smoser> it'd be nice ot have that in
[21:58]  * smoser hits 'rebuild'
[21:59] <smoser> you're welcome to just pull that though if you want...
[21:59] <smoser> especially with powersj approval
[21:59] <smoser> hopefully ci at 427 will agree
[21:59] <smoser>  https://jenkins.ubuntu.com/server/job/cloud-init-ci/427/console
[21:59] <smoser> (straight rebuild)
[21:59] <smoser> but with that, trunk should pass c-i again .
[22:00] <smoser> or rather the nightly integration test
[22:00] <powersj> it only failed MAAS tests
[22:00] <powersj> for that merge I don't care :) about those
[22:00] <powersj> push it :D
[22:03] <blackboxsw> I'll push it in