| redcavalier | Hey, I've ran into another weird cloud-issue issue on our centos instances in our Openstack setup. | 00:17 |
|---|---|---|
| redcavalier | This time, there's just this one VM which, for some reason, considers its cloud-init cache corrupt on every reboot. | 00:19 |
| redcavalier | So basically, it ignores its cache, deletes it and re-execute all metadata/userdata every reboot. | 00:21 |
| redcavalier | This seems to be unique to this one instance though, so I'm not sure what to check beside what I've already check. I think something may be interfering. | 00:26 |
| smoser | redcavalier: a /var/log/cloud-init.log would be the starting point. | 00:27 |
| smoser | for debugging | 00:28 |
| redcavalier | Yea, that'S how I know it'S discarding its cache. I can post it on a pastebin | 00:29 |
| redcavalier | smoser, http://pastebin.centos.org/385126/ . Line 25 bothers me especially, as the cache gets deleted every reboot. | 00:48 |
| === jpmethot is now known as redcavalier | ||
| blackboxsw | bummer, powersj, the disabled fastestmirror plugin wasn't the magic bullet I'd hoped. still hit subprocess.CalledProcessError: Command '['yum', 'install'... on https://jenkins.ubuntu.com/server/job/cloud-init-ci/415/console :( | 14:51 |
| powersj | :( | 14:54 |
| powersj | btw is smoser morning the loss today? | 15:25 |
| powersj | too soon? ;) | 15:25 |
| smoser | powersj: yeah, in mourning this morning. | 15:38 |
| smoser | now my only hope is in the Houstan Astros. I can't imagine anyone ever wants to see the Yankees win, and I'm kind of sore on the dodgers right now. | 15:41 |
| powersj | heh I missed a 'u' | 15:45 |
| smoser | powersj: so looking at https://jenkins.ubuntu.com/server/job/cloud-init-integration-a/171/consoleText | 15:45 |
| smoser | i remember we had this in curtin | 15:45 |
| smoser | its hard to tell easily what test failed | 15:45 |
| powersj | yeah because of the large amount of text that gets spit out | 15:46 |
| powersj | I usually search for "FAIL:" | 15:46 |
| smoser | well, yes, but also because | 15:46 |
| smoser | test_no_warnings_in_log (tests.cloud_tests.testcases.get_suite.<locals>.tmp) ... FAIL | 15:46 |
| smoser | doesnt tell me anything | 15:46 |
| smoser | i thought in curtin we did something so that it would list the class being run | 15:46 |
| powersj | well it is telling you the test that fails :) | 15:47 |
| powersj | that there are no warnings in the log | 15:47 |
| powersj | We did do that in Curtin and I even thought we already did that in our tests... let me check | 15:48 |
| powersj | hmm you are right, this is not obvious where that test lives | 15:50 |
| powersj | oh wait, we do have the change | 15:52 |
| powersj | in base.py:shortDescription(self) | 15:52 |
| powersj | """Prevent nose from using docstrings.""" | 15:52 |
| smoser | powersj: so i download artifacts from jenkins | 17:21 |
| smoser | is there a way i can run 'verify' with those ? | 17:22 |
| powersj | smoser: python3 -m tests.cloud_tests verify -h | 17:23 |
| smoser | http://paste.ubuntu.com/25780059/ | 17:23 |
| powersj | point to it via the --data-dir variable | 17:23 |
| smoser | thats my failed attempt^ | 17:23 |
| powersj | give it to /cloud-init/results | 17:24 |
| powersj | sorry to be more specific --data-dir=/tmp/artful-171/archive/cloud-init/results | 17:24 |
| smoser | thanks | 17:27 |
| smoser | so why doesnt shortDescription work | 17:34 |
| smoser | :-( | 17:34 |
| * smoser nothaving a good day | 17:35 | |
| smoser | oh. | 17:35 |
| smoser | cause we dont run nose | 17:36 |
| smoser | powersj, blackboxsw https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332585 | 19:04 |
| * blackboxsw thinks that was related to you testing my branch | 19:05 | |
| blackboxsw | just a 'feeling' | 19:05 |
| blackboxsw | :) | 19:06 |
| powersj | +1'ed | 19:11 |
| smoser | blackboxsw: were you fixing the ntp warning ? | 19:12 |
| blackboxsw | smoser: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/332540 | 19:12 |
| blackboxsw | was debating about fixing the lxd warning in that branch | 19:12 |
| blackboxsw | if you want | 19:12 |
| blackboxsw | or separate branch | 19:12 |
| smoser | i have http://paste.ubuntu.com/25780634/ | 19:13 |
| smoser | that i was going to grab in a "actually fix all warnings" merge | 19:13 |
| smoser | i can fix the ntp also, but if you had a branch that fixed it i woudl not | 19:13 |
| blackboxsw | yeah if you have a separate branch for that you're good, or take mine to supplement yours | 19:13 |
| blackboxsw | smoser: yeah my branch fixes ntp plus adds a unit test that'll show these integration test errors to us earlier once jsonschema is defined for a module | 19:14 |
| blackboxsw | it didn't address the lxd cfg warning throuhg | 19:14 |
| blackboxsw | though | 19:14 |
| smoser | blackboxsw: i have that one fine. | 19:16 |
| smoser | so if you're fixing jsonschema warning in | 19:17 |
| smoser | tests/cloud_tests/testcases/modules/ntp.yaml | 19:17 |
| blackboxsw | yep fixed per the above branch | 19:17 |
| smoser | then we'll just grab yours, then mine. then i think we'll be happy on no failures from warnings | 19:17 |
| blackboxsw | yeah for sure | 19:17 |
| smoser | but /me doesn't kniow why ididnt see these when i ran... | 19:17 |
| smoser | i really promise ir an! | 19:17 |
| smoser | I ran | 19:17 |
| blackboxsw | we didn't use to have a warning validation test did we? | 19:17 |
| * blackboxsw looks over the git logs again. | 19:18 | |
| blackboxsw | yeah, maybe you were referring to when you landed 41152f10ddbd8681cdac44b408038a4f23ab02df | 19:19 |
| smoser | blackboxsw: ... confused. | 19:19 |
| blackboxsw | I certainly am. I'll take another good sir | 19:19 |
| smoser | never mind | 19:19 |
| smoser | you fixsed it right | 19:19 |
| smoser | i was confused that modules/ntp.yaml had empty dicts for pools and servers | 19:19 |
| smoser | but you fixed that ;) | 19:19 |
| blackboxsw | right-o | 19:20 |
| smoser | blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332587 | 19:32 |
| smoser | that has the other fixes | 19:32 |
| smoser | i think enough to get a integration test fully run | 19:33 |
| blackboxsw | reviewing (will have a minor test patch for that branch) | 19:33 |
| blackboxsw | landed the ntp thanks | 19:33 |
| blackboxsw | & thx powersj | 19:33 |
| smoser | im going to grab https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332585 after ci ACKs it | 19:38 |
| blackboxsw | yes sorry, distracted on the followup | 19:39 |
| blackboxsw | approved | 19:39 |
| smoser | powersj had already done it, so i wasnt going to wait on you :) | 19:39 |
| powersj | :) | 19:39 |
| smoser | Your code has been rated at 10.00/10 | 19:40 |
| smoser | i love that. | 19:40 |
| smoser | 2 significant digits of perfection. | 19:40 |
| smoser | or is that 4 | 19:40 |
| smoser | either way. i'm like the Mary Lou Retton of python coding. | 19:41 |
| blackboxsw | hahah | 19:41 |
| smoser | powersj: https://github.com/canonical-server/jenkins-jobs/blob/master/cloud-init/integration.yaml | 19:44 |
| smoser | in that, i think we're making the same sort of error as we were in curtin at one point. | 19:44 |
| powersj | which error :) | 19:45 |
| smoser | we're running trunk c-i against distro version of the code | 19:45 |
| smoser | err.. trunk level of integration test with distro version of code | 19:45 |
| powersj | I'm taking trunk, and building it using an sbuild of a particular release | 19:46 |
| blackboxsw | smoser: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332587 updated | 19:46 |
| powersj | there should be no test mismatch there | 19:46 |
| powersj | if I were, however, to take the daily build of cloud-init and run it against the trunk tests then I would agree | 19:46 |
| powersj | daily build of cloud-init for a particular release* to be even more specific | 19:47 |
| === blackboxsw changed the topic of #cloud-init to: is Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 10/16 16:00 UTC | cloud-init 17.1 released | quotes: <@smoser> either way. i'm like the Mary Lou Retton of python coding. | ||
| blackboxsw | :) | 19:47 |
| smoser | yeah, you're right. | 19:47 |
| === blackboxsw changed the topic of #cloud-init to: is Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 10/16 16:00 UTC | cloud-init 17.1 released | ||
| smoser | hm.. so why did i not see that failure. | 19:51 |
| smoser | https://jenkins.ubuntu.com/server/job/cloud-init-ci/419/console | 19:51 |
| blackboxsw | not behind a proxy? | 19:52 |
| blackboxsw | where fastestmirror plugin is disabled? | 19:52 |
| smoser | i thought we disabled it. | 19:53 |
| smoser | where was that mp ? | 19:53 |
| blackboxsw | hrm: Loaded plugins: fastestmirror | 19:53 |
| blackboxsw | that mp landed yesterday. | 19:53 |
| blackboxsw | digging it up | 19:53 |
| blackboxsw | https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/332511 | 19:53 |
| blackboxsw | strange to see fastestmirror showing up on that run as I thought it was behind a proxy (and as such fastestmirror should be disabled) | 19:54 |
| blackboxsw | or maybe I'm just focusing on the wrong problem | 19:54 |
| powersj | your branch doesn't have my fix? | 19:55 |
| blackboxsw | checking your branch out to make sure it has the fix | 19:55 |
| powersj | it doesn't | 19:56 |
| powersj | https://git.launchpad.net/~smoser/cloud-init/tree/tools/run-centos?h=fix/citest-show-class-in-failures | 19:56 |
| blackboxsw | https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332585 doesn't yeah : ) | 19:57 |
| blackboxsw | rebase for the win | 19:57 |
| blackboxsw | too bad that canadian fastestmirror will be sad | 19:58 |
| smoser | blackboxsw: thanks. | 20:02 |
| smoser | https://jenkins.ubuntu.com/server/job/cloud-init-ci/422/ is about to post on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332585 | 20:19 |
| smoser | and then happy i'll pull that | 20:20 |
| smoser | blackboxsw: did you sort the maas/-proposed thing ? | 20:24 |
| blackboxsw | nope smoser I'm trying to figure out what gives with https://bugs.launchpad.net/cloud-init/+bug/1684869 and why it doesn't generate a 'proposed' image for me | 20:28 |
| ubot5 | Ubuntu bug 1684869 in cloud-init (Ubuntu Artful) "growing root partition does not always work with root=PARTUUID=" [Medium,Fix released] | 20:28 |
| blackboxsw | was trying to reproduce the resize failure locally | 20:28 |
| smoser | hm.. | 20:31 |
| smoser | blackboxsw: https://git.launchpad.net/~smoser/cloud-init/+git/sru-info/plain/bin/get-proposed-cloudimg just "worked for me" | 20:35 |
| smoser | http://paste.ubuntu.com/25781060/ | 20:35 |
| blackboxsw | weird I think I'm being dumb. | 20:36 |
| blackboxsw | the zesty image I downloaded didn't upgrade cloud-init | 20:36 |
| blackboxsw | checking logs | 20:36 |
| smoser | oh. yeah. it would not i guess. | 20:37 |
| smoser | or possibly | 20:37 |
| smoser | as cloud-init got kicked maybe ? | 20:37 |
| smoser | hm.. no i get it there. | 20:37 |
| blackboxsw | pulling latest get-proposed-cloudimg to check | 20:37 |
| smoser | hm.. why is sfdisk --part-uuid not working | 20:39 |
| blackboxsw | uefi images on xenial right? | 20:39 |
| smoser | oh. no. what a pita | 20:39 |
| smoser | xenial are mbr | 20:40 |
| smoser | fudge | 20:40 |
| blackboxsw | https://bugs.launchpad.net/cloud-init/+bug/1684869/comments/8 | 20:40 |
| ubot5 | Ubuntu bug 1684869 in cloud-init (Ubuntu Artful) "growing root partition does not always work with root=PARTUUID=" [Medium,Fix released] | 20:40 |
| blackboxsw | :) | 20:40 |
| blackboxsw | ok my get-proposed-cloudimg was stale | 20:42 |
| blackboxsw | ok my network in the kvm I created is borked. Could not resolve host: entropy.ubuntu.com or anything else, which is why my proposed cloud-init package didn't update | 20:47 |
| smoser | blackboxsw: testing this | 20:48 |
| smoser | http://paste.ubuntu.com/25781127/ | 20:49 |
| smoser | dns issues scare me | 20:49 |
| smoser | artful guest ? | 20:49 |
| blackboxsw | artful host, zesty guest | 20:52 |
| blackboxsw | trying on another box to be certain | 20:52 |
| smoser | how did you make guest ? | 20:52 |
| blackboxsw | strange it's working on another box. ok will copy it in | 20:53 |
| smoser | oh. inside the mounted image you were failing dns ? | 20:53 |
| smoser | my test got throug, following that bug and / got rezied on xenial. | 20:55 |
| smoser | tryinng now with zesty | 20:55 |
| blackboxsw | smoser: yes inside the image | 20:55 |
| blackboxsw | and inside the image is cloud-init 0.7.9 | 20:56 |
| blackboxsw | not 17.1 as I had expected to see | 20:56 |
| blackboxsw | 0.7.9-233-ge | 20:57 |
| blackboxsw | yeah | 20:57 |
| smoser | ok. yeah. so you have to (and i made this mistake) boot the -proposed | 20:57 |
| blackboxsw | ok I performed the steps I had used in previous SRU https://bugs.launchpad.net/cloud-init/+bug/1684869 | 20:57 |
| ubot5 | Ubuntu bug 1684869 in cloud-init (Ubuntu Artful) "growing root partition does not always work with root=PARTUUID=" [Medium,Fix released] | 20:57 |
| blackboxsw | ahh gotcha right | 20:57 |
| smoser | blackboxsw: so modify that template on the bug like: | 20:59 |
| smoser | proposed=${raw%.*}-proposed.img | 20:59 |
| smoser | qemu-img create -f qcow2 -b $proposed disk.img 10G | 21:00 |
| smoser | but i did that just now for zesty and worked fine. verified cloud-init 17.1.... in the guest | 21:00 |
| blackboxsw | oooooh | 21:02 |
| blackboxsw | oops | 21:02 |
| blackboxsw | thanks | 21:02 |
| blackboxsw | ok, made it to the finish line. see the resize succeed on 17.1 | 21:08 |
| blackboxsw | zesty | 21:08 |
| blackboxsw | soooo, our test case doesn't validate https://bugs.launchpad.net/cloud-init/+bug/1725067 for some reason | 21:08 |
| ubot5 | Ubuntu bug 1725067 in cloud-init (Ubuntu) "cloud-init resizefs fails when booting with root=PARTUUID=" [Medium,Triaged] | 21:08 |
| blackboxsw | and bbsw is out of his depth | 21:09 |
| smoser | blackboxsw: doesnt validate ? | 21:09 |
| blackboxsw | I need knowledge smoser. got 10 mins before your weekend? | 21:15 |
| smoser | blackboxsw: http://paste.ubuntu.com/25781261/ | 21:16 |
| smoser | yeah, we can chat 10 minutes | 21:16 |
| smoser | that "works for me" to recreate failure actually in xenial | 21:16 |
| blackboxsw | ok, I was wondering what I did wrong as I saw resizes happening on 17.1 zesty I thought | 21:17 |
| smoser | i'm in hangout | 21:17 |
| smoser | and yeah, it does seem to work fine for zesty | 21:17 |
| blackboxsw | sorry joining | 21:18 |
| blackboxsw | was running your script | 21:18 |
| smoser | blackboxsw: for irc logs | 21:33 |
| smoser | https://github.com/cloud-init/qa-scripts/blob/master/scripts/get-proposed-cloudimg | 21:33 |
| smoser | https://github.com/cloud-init/ubuntu-sru/blob/master/bugs/lp-1684869/recreate.sh | 21:34 |
| blackboxsw | yeah thanks, stinks to close the hangout window when you didn't grab all the links discussed | 21:34 |
| smoser | blackboxsw: ok. so the difference between xenial and zesty | 21:38 |
| smoser | on zesty, *something* is making /dev/root exist | 21:38 |
| smoser | but on xenial | 21:38 |
| smoser | $ ls -l /dev/root | 21:39 |
| smoser | ls: cannot access '/dev/root': No such file or directory | 21:39 |
| smoser | zesty | 21:39 |
| blackboxsw | ahh interesting... ok | 21:39 |
| smoser | $ ls -l /dev/root | 21:39 |
| smoser | brw------- 1 root root 8, 1 Oct 20 21:39 /dev/root | 21:39 |
| blackboxsw | right I'm with you | 21:39 |
| blackboxsw | saw the failure on xenial now thanks | 21:40 |
| smoser | yeah, so fix at this point should be pretty straight forward knowing what we did originally and such. | 21:40 |
| smoser | have a good one. | 21:40 |
| * smoser out | 21:40 | |
| blackboxsw | yeah thanks again you too | 21:45 |
| smoser | blackboxsw: https://jenkins.ubuntu.com/server/job/cloud-init-ci/426/console | 21:57 |
| smoser | that is from https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/332587 | 21:57 |
| blackboxsw | no fastestmirror | 21:57 |
| blackboxsw | hrm | 21:57 |
| smoser | yeah | 21:57 |
| smoser | its definitely not that coee | 21:57 |
| smoser | it'd be nice ot have that in | 21:58 |
| * smoser hits 'rebuild' | 21:58 | |
| smoser | you're welcome to just pull that though if you want... | 21:59 |
| smoser | especially with powersj approval | 21:59 |
| smoser | hopefully ci at 427 will agree | 21:59 |
| smoser | https://jenkins.ubuntu.com/server/job/cloud-init-ci/427/console | 21:59 |
| smoser | (straight rebuild) | 21:59 |
| smoser | but with that, trunk should pass c-i again . | 21:59 |
| smoser | or rather the nightly integration test | 22:00 |
| powersj | it only failed MAAS tests | 22:00 |
| powersj | for that merge I don't care :) about those | 22:00 |
| powersj | push it :D | 22:00 |
| blackboxsw | I'll push it in | 22:03 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!