| smoser | blackboxsw: ping | 03:08 |
|---|---|---|
| smoser | hey | 03:08 |
| blackboxsw | pong | 03:08 |
| smoser | i was just about to accept your merge | 03:08 |
| smoser | i was going to just put a comment in | 03:08 |
| smoser | except UrlError as urle: | 03:08 |
| blackboxsw | smoser: sorry was tweaking the description to be more appropriate | 03:08 |
| smoser | message = str(urle) | 03:08 |
| smoser | # older versions of requests may not get the url | 03:08 |
| smoser | # into the message. | 03:08 |
| smoser | and fix up the commit message. | 03:08 |
| smoser | yeah. | 03:08 |
| smoser | and that'd be it. | 03:08 |
| blackboxsw | ahh sounds good. want me to push | 03:08 |
| smoser | then i'im good. | 03:08 |
| smoser | i'll comment and you can take it. | 03:08 |
| blackboxsw | +1 will do | 03:08 |
| smoser | just approved with changes on the mp | 03:09 |
| blackboxsw | smoser: I was gonna land https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333575 | 03:09 |
| smoser | please grab | 03:09 |
| smoser | thanks | 03:09 |
| smoser | i'm fine with that too | 03:09 |
| blackboxsw | I can wait til morning on that 2nd branch though | 03:09 |
| blackboxsw | ok will do tonight | 03:09 |
| smoser | fix c-i first though | 03:09 |
| blackboxsw | agreed | 03:09 |
| smoser | ie, the urlerror message first | 03:09 |
| smoser | just to have less broken tips | 03:09 |
| smoser | thanks | 03:09 |
| smoser | and i'm out. | 03:10 |
| smoser | later | 03:10 |
| blackboxsw | thanks have a good one | 03:10 |
| === shardy is now known as shardy_lunch | ||
| === shardy_lunch is now known as shardy | ||
| smoser | oh for petes sake jenkins | 15:37 |
| smoser | blackbox fixed cloud-init but then jenkins cries | 15:37 |
| smoser | https://jenkins.ubuntu.com/server/job/cloud-init-ci-nightly/161/console | 15:37 |
| blackboxsw | smoser: can we rerun nighlty? | 16:50 |
| blackboxsw | to get a good value | 16:50 |
| blackboxsw | powersj: smoser: I'll fixup qa-scripts/scripts/launch-ec2 for bionic while smoser is working a unit test for fallback_nic on upgrade | 16:51 |
| blackboxsw | launch-ec2 on my end was working from xenial :/ I'll fixit up on bionic now | 16:51 |
| smoser | $ echo raw support for rharper | haste -r | 16:53 |
| smoser | https://hastebin.com/raw/zurabikoko | 16:53 |
| rharper | \o/ | 16:53 |
| rharper | can you hastebin your haste tool ? | 16:53 |
| smoser | blackboxsw: https://hastebin.com/akazezaroh | 16:54 |
| smoser | thats what i have so far | 16:54 |
| blackboxsw | thx smoser will pull that in | 16:54 |
| blackboxsw | and handle other issues | 16:54 |
| smoser | blackboxsw: it seems wierd that boto doesnt expose 'InvalidGroup.NotFound' | 16:59 |
| smoser | or any of those. | 16:59 |
| smoser | or even 'code' on the Error or seomthing | 17:00 |
| blackboxsw | Yeah that seems broken. string parsing in the error message is not appropriate | 17:00 |
| blackboxsw | not an appropriate design decision | 17:01 |
| blackboxsw | I wonder if there's a structure I can import. | 17:01 |
| blackboxsw | I'll look at boto3 modules | 17:01 |
| blackboxsw | ahh recent python3-boto3 in bionic has some exception goodnees | 17:06 |
| blackboxsw | smoser: powersj just pushed qa-scripts/scripts/launch-ec2 for bionic | 17:44 |
| blackboxsw | de823f2..58c9a97 | 17:45 |
| blackboxsw | will test the failed upgrade | 17:45 |
| smoser | blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+ref/bug/1732917-fix-fallback-interface | 17:47 |
| smoser | see what you think about that | 17:47 |
| smoser | i dont think we actually *need* the setter | 17:47 |
| smoser | (though i could add one) | 17:47 |
| blackboxsw | smoser: spelling inteface | 17:49 |
| blackboxsw | other than the typo on the class variable '_fallback_inteface' it should work. | 17:52 |
| smoser | blackboxsw: https://hastebin.com/kivunetopa | 17:53 |
| smoser | (hean, and i fixed those and pushed --force) | 17:53 |
| smoser | tox now passes | 17:54 |
| smoser | blackboxsw launch_ec2 is really nice. | 17:54 |
| blackboxsw | per your latest 'haste' I think boto3 on bionic may have an official exception for that. I'll try (changing my keypair | 17:55 |
| blackboxsw | bummer, still botocore.exceptions.ClientError: An error occurred (InvalidKeyPair.NotFound) when calling the DescribeKeyPairs operation: The key pair 'cloud-init-integration-chad' does not exist | 17:56 |
| blackboxsw | ok taking your try/except changes | 17:56 |
| blackboxsw | and thanks | 17:56 |
| blackboxsw | I want to add the ipv6 setup support and hit the blog with it in hand. | 17:56 |
| blackboxsw | let's get past this upgrade SRU bump | 17:57 |
| smoser | blackboxsw: i laucnhed an instance, ssh'd to it | 17:57 |
| smoser | and then it disappeared | 17:58 |
| smoser | oh. you terminated it for me ? | 17:58 |
| smoser | seems like keep_alive should be default :) | 17:59 |
| blackboxsw | smoser: yeah --keep-alive | 18:01 |
| blackboxsw | sorry | 18:01 |
| blackboxsw | could surface kill-it :) | 18:01 |
| blackboxsw | I'll change that param | 18:01 |
| blackboxsw | --destroy :) | 18:02 |
| smoser | ugh | 18:08 |
| smoser | launched instance | 18:08 |
| smoser | typed 'apt-get update' | 18:08 |
| smoser | 0% [Connecting to security.ubuntu.com (2001:67c:1560:8001::14)] | 18:08 |
| smoser | hung | 18:08 |
| blackboxsw | same | 18:10 |
| blackboxsw | other apt repos worked | 18:10 |
| blackboxsw | all local to amazon though | 18:11 |
| blackboxsw | works on 0.7.7 | 18:13 |
| smoser | so thats bad news | 18:14 |
| smoser | the others resolve to ipv4 though maybe ? | 18:14 |
| blackboxsw | yet why would 0.7.7 work | 18:15 |
| smoser | well, we get ipv5 address | 18:17 |
| smoser | so something notices that and returns the ipv6 address for the security.ubuntu.com | 18:17 |
| smoser | and then we do not have outbound connectivity i guess | 18:18 |
| blackboxsw | upgrading from 0.7.7 xenial (with working apt connectivity) -> 17.1 | 18:19 |
| smoser | that should be fine, no ? | 18:19 |
| blackboxsw | checking to be sure | 18:19 |
| smoser | blackboxsw: did you recreate this ? | 18:20 |
| smoser | the failure in that bug | 18:20 |
| blackboxsw | not yet smoser | 18:21 |
| blackboxsw | trying to though | 18:21 |
| smoser | i launched instance | 18:21 |
| smoser | upgraded | 18:21 |
| smoser | rebooted | 18:21 |
| smoser | no WARN | 18:21 |
| smoser | ugh | 18:22 |
| blackboxsw | no error on my side. on upgrade path. | 18:23 |
| blackboxsw | trying specifically from 0.7.9~233 | 18:23 |
| blackboxsw | for my next pass | 18:23 |
| smoser | hm. | 18:31 |
| blackboxsw | hmm is right, from 0.7.9 -> 17.1 (upgrade without clean) I reboot without error | 18:33 |
| blackboxsw | as cloud-init doesn't re-run | 18:33 |
| blackboxsw | hrm. checking that bug traceback again | 18:34 |
| smoser | blackboxsw: perhaps this is not on ec2 | 18:39 |
| smoser | he never says he is. | 18:40 |
| smoser | definitely datasource got used. but nots ure. | 18:40 |
| blackboxsw | smoser: wierd comment from him | 18:41 |
| blackboxsw | https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1732917/comments/1 :( hmm | 18:41 |
| ubot5 | Launchpad bug 1732917 in cloud-init (Ubuntu) "17.1 update breaks EC2 nodes" [Undecided,New] | 18:41 |
| blackboxsw | he says the failure happens when restarting the cloud-init? but goes away when restarting the node? | 18:41 |
| blackboxsw | I'm misreading that | 18:41 |
| blackboxsw | I'm just not really sure what that's saying | 18:42 |
| smoser | yeah. | 18:43 |
| blackboxsw | ohh maybe running cloud-init init or something? | 18:43 |
| smoser | we could see what happens on openstack if we set it to use the Ec2 datasoruce | 18:43 |
| smoser | blackboxsw: good news is that this isnt as serious as it seemed at first | 19:00 |
| blackboxsw | yeah, I'm just trying to see if maybe a complex networking setup would cause this? | 19:03 |
| blackboxsw | not sure | 19:03 |
| smoser | we need to identify the issue witih ipv6 too | 19:03 |
| smoser | the hang on security.ubutnu | 19:03 |
| blackboxsw | stepping away for 20 | 19:03 |
| blackboxsw | gotta help w/ lunch | 19:03 |
| smoser | blackboxsw: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/333905 | 19:27 |
| blackboxsw | ugh | 19:28 |
| smoser | its not too big a deal i dont think | 19:29 |
| rharper | smoser: blackboxsw: do you have logs from the upgrade cloud-init reboot ? | 19:31 |
| smoser | http://paste.ubuntu.com/25982990/ | 19:32 |
| rharper | thx | 19:32 |
| smoser | there is a /var/log/cloud-init | 19:32 |
| blackboxsw | http://paste.ubuntu.com/25982994/ | 19:33 |
| blackboxsw | 2late | 19:33 |
| smoser | line 109 is intersting there. | 19:33 |
| smoser | oh. thats tmp file deletion | 19:33 |
| smoser | lucky it didnt fail | 19:33 |
| rharper | smoser: this is both original and upgraded in the same file ? | 19:34 |
| rharper | smoser: your paste looks like launch, reboot, upgrade, reboot ? is that right ? | 19:36 |
| blackboxsw | 2017-11-17 18:32:43,838 - stages.py[DEBUG]: cache invalid in datasource: DataSourceEc2 | 19:37 |
| blackboxsw | first time i;ve seen this | 19:37 |
| rharper | smoser: actually, I'm really confused; | 19:37 |
| rharper | 2017-11-17 18:17:36,206 - handlers.py[DEBUG]: finish: modules-final: SUCCESS: running modules for final | 19:37 |
| rharper | 2017-11-17 18:19:06,783 - util.py[DEBUG]: Cloud-init v. 17.1 running 'single' at Fri, 17 Nov 2017 18:19:06 +0000. Up 102.29 seconds. | 19:37 |
| smoser | rharper: that was launch with old, upgrade, reboot, possibly more reboots. | 19:38 |
| rharper | 0.7.9 finished at 18:17 | 19:38 |
| rharper | then there's a single mode ? | 19:38 |
| rharper | after upgrade ? | 19:38 |
| rharper | I would have expected the reboot | 19:38 |
| smoser | i dont thin ki ran that. | 19:38 |
| rharper | well | 19:38 |
| rharper | your log shows you did | 19:38 |
| * rharper looks at blackboxsw | 19:38 | |
| smoser | oh | 19:38 |
| smoser | chad might have run that for me. | 19:39 |
| blackboxsw | --proposed reboots | 19:39 |
| rharper | same in changes | 19:39 |
| rharper | 2017-11-17 18:31:21,520 - util.py[DEBUG]: Cloud-init v. 17.1 running 'single' at Fri, 17 Nov 2017 18:31:21 +0000. Up 277.24 seconds. | 19:39 |
| rharper | 2017-11-17 18:31:21,521 - stages.py[DEBUG]: Using distro class <class 'cloudinit.distros.ubuntu.Distro'> | 19:39 |
| rharper | what's 17.1 single mode doing ? | 19:39 |
| smoser | hm. | 19:40 |
| blackboxsw | yeah not sure there | 19:40 |
| smoser | oh | 19:40 |
| rharper | don't we just boot; sudo apt update && sudo apt install cloud-init && reboot ? | 19:40 |
| smoser | its the upgrade | 19:40 |
| smoser | let me see. | 19:40 |
| rharper | what is upgrade doing ? | 19:40 |
| rharper | 2017-11-17 18:31:21,523 - cc_apt_pipelining.py[DEBUG]: Wrote /etc/apt/apt.conf.d/90cloud-init-pipelining with apt pipeline depth setting 0 | 19:40 |
| rharper | 2017-11-17 18:31:21,523 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False) | 19:40 |
| rharper | 2017-11-17 18:31:21,523 - util.py[DEBUG]: Read 14 bytes from /proc/uptime | 19:40 |
| rharper | 2017-11-17 18:31:21,524 - util.py[DEBUG]: cloud-init mode 'single' took 0.063 seconds (0.06) | 19:40 |
| rharper | I guess it's fixing up the apt conf ? | 19:40 |
| rharper | but, that reloads the on-disk object prior to reboot | 19:41 |
| smoser | debian/cloud-init.postinst | 19:41 |
| blackboxsw | which might be what breaks apt to security,ubuntu? | 19:41 |
| smoser | it only changes pipelining | 19:41 |
| smoser | i'm guesing that code should be version-fixed in some way | 19:43 |
| smoser | no | 19:43 |
| smoser | but that is just noise | 19:45 |
| rharper | ok | 19:45 |
| rharper | just walking through the log | 19:45 |
| rharper | I didn't expect that | 19:45 |
| smoser | yeah, we should probably fix that | 19:45 |
| rharper | ah | 19:46 |
| rharper | on ec2, instance is always invalid | 19:46 |
| rharper | I have a branch, but didn't finish it, to read instance_id from sys/dmi | 19:46 |
| rharper | so we never read the cache at local time | 19:47 |
| blackboxsw | ahh o | 19:47 |
| smoser | ? | 19:47 |
| rharper | we need to capture the system_uuid to compare | 19:48 |
| smoser | right. we always re-discover. because there is no check | 19:48 |
| smoser | yeah | 19:48 |
| rharper | we don't do that | 19:48 |
| rharper | so, the local cache check says, it;s invalid | 19:48 |
| rharper | that's expected at this point (we always do this on ec2) | 19:48 |
| blackboxsw | hmmm is it possible that get_fallback_nic returns None on some platforms | 19:50 |
| rharper | but it would have blown up in local mode, no ? | 19:52 |
| rharper | if I'm reading the bug log right, ti was stage init (versus init-local) | 19:52 |
| blackboxsw | yeah it should have fallen apart in init-local | 19:55 |
| blackboxsw | right if we were Ec2 proper, we wouldn't actually get to the DatasourceEc2 | 19:56 |
| rharper | well, Local exits on non-ec2 | 19:56 |
| blackboxsw | we would've already detected DatasourceEc2Local and not run init-network | 19:56 |
| smoser | yeah | 19:56 |
| smoser | thats why i asked if he was on Amazon | 19:56 |
| smoser | i dont think it is | 19:56 |
| smoser | and that we can try on serverstack | 19:57 |
| rharper | you can force it to rn ec2 even on Openstack ? instead of the OpenstackDS ? | 19:57 |
| blackboxsw | if we run dpkg-reconfigure cloud-init we can force it right | 19:57 |
| blackboxsw | just uncheck OpenStack | 19:57 |
| blackboxsw | I *think* | 19:58 |
| * blackboxsw fires up my vpn | 19:58 | |
| blackboxsw | ok creating a xenial instance and will attempt the upgrade | 20:01 |
| blackboxsw | ok clean reboot on 0.7.9 openstack instance w/ OpenstackDatasource gets me a warning banner | 20:22 |
| blackboxsw | and upgrading/rebooting doesn't hit that traceback about fallback_nic on the obj.pkl because Ec2 claimed invalid obj.pk and recreated it. | 20:30 |
| blackboxsw | so Openstack images limited to Ec2Datasource can't reproduce this on upgrade path | 20:31 |
| blackboxsw | Openstack-ec2datasource: ✔ | 20:32 |
| blackboxsw | here are the logs as that's a bit complex | 20:33 |
| blackboxsw | here are the logs http://paste.ubuntu.com/25983308/ | 20:33 |
| blackboxsw | and for the record dpkg-reconfigure cloud-init did allow me to unset OpenstackDatasource on an openstack instance | 20:35 |
| smoser | sure. and it should. | 20:36 |
| blackboxsw | just felt I needed to affirm my "I *think*" comment | 20:36 |
| blackboxsw | smoser: I'm testing your sandbox dhcpclient branch | 20:37 |
| blackboxsw | will approve shortly | 20:37 |
| rharper | blackboxsw: hrm; so we don't yet have a plausible path where we reload an EC2 datasource | 20:41 |
| blackboxsw | yeah not that I can figure currently | 20:41 |
| rharper | what about AliYun ? | 20:42 |
| rharper | it can run at local and net (DEP_FILESYSTEM, DEP_NETWORK) | 20:43 |
| rharper | and it will get the .fallback and the network config properties, but EC2Local won't run | 20:44 |
| rharper | which I think get's us the path we're on; that the variable defaults to None, and no path to set it to a fallback value that's not None | 20:44 |
| rharper | smoser: do you have a aliyun account ? | 20:45 |
| smoser | rharper: no. idont think so. | 20:45 |
| blackboxsw | maybe we need to spitball, but smoser your patch seems like it would fix this path, however we got there | 21:00 |
| smoser | blackboxsw: yeah. i think so too. :) | 21:01 |
| smoser | and we need the better save too | 21:01 |
| === Hazelesque_ is now known as Hazelesque | ||
| blackboxsw | so smoser yeah with public ipv6 configuration, I can't get to security.ubuntu | 21:05 |
| blackboxsw | as in, if I dhcp6, apt timesout | 21:05 |
| smoser | blackboxsw: is it possible that our security group is just set up incorrectly ? | 21:07 |
| smoser | not allowing outbound ipv6 | 21:07 |
| blackboxsw | ahh very | 21:08 |
| smoser | blackboxsw: that does somewhat still identify a regression | 21:09 |
| rharper | blackboxsw: I'm happy with the smoser patch; and I suppose that we can't yet find a path to the failure should mean that the impact is narrow; but it's rather frustrating that it;s not obvious how we hit that path | 21:09 |
| smoser | but its not really one we could do something about | 21:09 |
| smoser | we can't easily enable ipv6 when it was enabled in the metadata and then not have the system use it. | 21:09 |
| smoser | blackboxsw: rharper chat ? | 21:15 |
| rharper | y | 21:20 |
| smoser | https://hangouts.google.com/hangouts/_/canonical.com/cloud-init?authuser=0 | 21:20 |
| blackboxsw | Yeah lost network there in amin | 21:25 |
| blackboxsw | approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/333905 | 21:34 |
| smoser | blackboxsw: old deb | 22:07 |
| smoser | wget https://launchpad.net/ubuntu/+archive/primary/+files/cloud-init_0.7.9-233-ge586fe35-0ubuntu1~16.04.2_all.deb | 22:07 |
| blackboxsw | thx smoser | 22:07 |
| smoser | ddpkg install that deb | 22:09 |
| smoser | rm -Rf /var/lib/cloud /var/log/cloud-init | 22:09 |
| smoser | reboot | 22:09 |
| smoser | apt-get install cloud-init | 22:09 |
| smoser | cloud-init init | 22:10 |
| smoser | then i tried to fix with my deb (dpkg -i) | 22:10 |
| smoser | and run cloud-init init | 22:10 |
| smoser | again | 22:10 |
| smoser | 2017-11-17 22:05:41,781 - DataSourceEc2.py[WARNING]: unexpected metadata 'network' key not valid: None | 22:10 |
| blackboxsw | ok success | 23:27 |
| blackboxsw | functional branch is at | 23:27 |
| blackboxsw | https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/fix-ec2-fallback-nic | 23:27 |
| blackboxsw | needs tests | 23:28 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!