[00:00] sinzui, so in the openstack provider attempt, juju fires up a vivid instance, and i can see the nova console log that it's booted and ready with whatever userdata was passed to it. but there is a key mismatch. [00:00] sinzui, with the maas deploy, i get the same symptom (key issue), and it's much harder to get console output. [00:01] er rather, maybe not a key mismatch, but definitely a key issue. [00:01] beisner, I don't think key issues are series issues [00:01] sinzui, all those woes go away when I set default-series: utopic or trusty in the environments.yaml. [00:03] beisner, I think you found a bug :) [00:04] sinzui, ok, what can i do/collect to raise a helpful/meaningful bug on this? [00:04] ie. is there a way to get more verbosity from juju bootstrap? [00:05] beisner, the cloud-init-output log and maybe a machine-0 log if it gets that far [00:05] sinzui, unit 0 never comes alive according to juju [00:05] and juju debug-log is no help at that stage [00:06] beisner, I often ssh into the machine the moment I see the ip is open and tail the /var/log/cloud-init-output.log [00:07] sinzui, ah cool. i'll dive in a bit more. appreciate the guidance. [00:07] ericsnow: just realized I forgot to hit publish on that review, you have a review from me now. [00:07] wwitzel3: thanks [00:09] ericsnow: mostly minor, I found it all easy to follow, no comments about the service stuff since it is pretty generic boiler platey and we talkeda bout it before [00:09] wwitzel3: cool [00:20] thumper: you around? [00:28] sinzui: so we should be able to get 1.22 out now right? since the precise upgrade issue is only 1.23? [00:32] wallyworld, no, because we never got a pass for 1.22 [00:32] sinzui: assuming ci becomes happy [00:32] wallyworld, 1.22-beta4 has NEVER passed CI [00:32] :-( [00:32] wallyworld, if it does pass, then I release [00:33] let's hope this one works [00:41] thumper: ping [00:44] * thumper is here now [00:47] thumper: can i grab 5 when you are free? [00:48] wallyworld: sure, I'll be done chugging lunch in about 5 minutes [00:48] np [00:48] it's a banana protein smoothy [00:48] mmmmm [00:48] 5min for smoothie? [00:49] thy* [00:49] thumper: shall we meet in the 1:1 hangout ? [00:49] davecheney: yep, how about in 11 minutes? [00:49] thumper: go talk to wallyworld first [00:49] ta [00:49] i'll see ou in the hangout in whenever [00:53] wallyworld: our 1:1? [00:53] yup [00:58] sinzui: i just looked at the dashboard and the latest 1.23 run had local-upgrade-precise-amd64 passing [00:58] have you upped the timeout already? [01:13] wallyworld, I did, that was my proof [01:14] sinzui: logs are needed to help see where the time is going [01:15] i can't see any linekd to the dashboard [01:15] wallyworld, I can give you the failures, but the passes might be more informative since I also gave the slaves more time to collect [01:15] s/slave/services [01:16] sinzui: could you attach both to the bug for me? [01:16] wallyworld, I will see what I can do. I don't want to make them public if the contain credentials [01:16] sinzui: make the bug private? [01:18] wallyworld, I cannot because that will hide the critical blockers [01:18] sinzui: oh, maybe send privately? [01:18] wallyworld, this wouldn't be awkward if the credentials for reports.vapour.ws had not also failed this weekend too === kadams54-away is now known as kadams54 [01:25] wallyworld, did you see smoser's ruling on apt-get dist-upgrade [01:25] sinzui: one sec, just finishing standup [01:43] sinzui: read scott's bug comments, we should be ok - we don't use proposed pocket with precise [01:43] that i know of [01:49] menn0, thumper is this doc up to day??: [01:49] https://docs.google.com/a/canonical.com/document/d/1jsuoTbXZbj3wtoXCpc5MGFVvwuofYx3hLw2eNoXEmE0/edit#heading=h.aby6yid7wq2d [01:49] * menn0 looks [01:50] alexisb: yes, except that now we have a working proof of concept of logging to MongoDB and have run scaling tests (see your email) [01:50] wallyworld, we never use proposed [01:51] alexisb: I have been working on turning the POC into production code but whether it gets merged is somewhat dependent on whether the performance hit is deemed ok [01:51] menn0, yep understood, I just need a way to capture the work for logging that can be shared with those that are interested [01:51] wallyworld, but juju does do apt-get upgrade, which didn't give us an updated cloud-utils. apt-get install then did a remove [01:52] alexisb: ok. let me know if you need to know more. [01:52] ie answer the question for "why is logging for JES important and requires work" [01:52] nope I think that gets me what i need [01:52] thanks [01:52] sinzui: so maybe the fix to put cloud-utils and cloud-image-utils on the one line is not required anymore [01:53] wallyworld, it is required because we haven't changed anything else to ensure removals cannot happen [01:53] ok [01:53] wallyworld, sinzui: you guys seem to be discussing the same part of the code that I just filed a bug about. (bug 1424892) [01:53] Bug #1424892: rsyslog-gnutls is not installed when enable-os-refresh-update is false

[01:54] menn0: nah, different [01:54] menn0, yes [01:54] this is the deb packaging issue whcich affected cloud-utils [01:54] * thumper sees that menn0 has answered all of alexisb's questions [01:54] coffee time then... [01:54] menn0: your issue is the behaviour of the flags to disable apt [01:55] menn0, apt-get update finds the new packages (in cloud-* example) but apt-get update will not install them because update is not permitted to install new deps! [01:55] menn0, but apt-get dist-upgrade can install new deps [01:55] menn0: when upgrading, do you recall if the state server rejects connections until all nodes in the env are deemed to have upgraded? [01:56] wallyworld: not quite ... let me quickly look at the code [01:59] wallyworld: a state server will accept API connections once it itself has upgraded [01:59] wallyworld: but state servers always upgrade first, before other nodes [01:59] menn0: ok, ta. i'm looking into the CI blocker where precise upgrades tie out [01:59] there's a bucket load of mongo connection failures that go on and on [02:01] but state server probably not at fault if it accepts connections after it has finished [02:01] wallyworld: do you have some logs handy? [02:02] menn0: i'll forward an email. sinzui had to increase timeout from 10 mins to 20 to make CI local precise upgrades pass. but just precise [02:02] wallyworld: that is certainly odd. [02:02] indeed [02:02] precise runs a slightly different mgo version [02:02] that's all i can think of off hand [02:03] wallyworld, hp's swift isn't publish failed. I am getting out the hammers [02:03] what's that about hp? [02:04] wallyworld, timouts uploading [02:04] oh joy [02:05] wallyworld, I switch the job to not rebuild. just try to publish what the previous job made [02:05] ok === kadams54 is now known as kadams54-away [02:14] wallyworld, sinzui - back for a bit, raised a bug on that vivid thing. should be readily reproducible but holler if there are any ?s. https://bugs.launchpad.net/juju-core/+bug/1424900 [02:14] Bug #1424900: Bootstrapping Vivid: ERROR failed to bootstrap environment, Permission denied (publickey), ci-info: no authorized ssh keys fingerprints found for user ubuntu

[02:14] thank you [02:19] sinzui: menn0: with those logs, i see the 1st 4 minutes spinning up the state server and machines 1,2, then the state server upgrade completes in about minute, then we see a tonne of connection terminated errors lasting several mintes, so it seems there's an issue wit the worker nodes upgrading [02:19] wallyworld: yep, i'm looking at those logs now [02:19] wallyworld: the machine-0 logs are all perfectly normal [02:20] wallyworld: but the machine-1 logs indicate that the agent never restarted into the new version [02:20] menn0: in machine 1 lo, i see a 3 minute gap fetching tools [02:20] wallyworld: it sees the need to upgrade but never seems to reboot [02:21] menn0: the test may have timed out before machine 1 could restart [02:21] take off the 3 minutes to fetch tools [02:21] menn0, its about timing. I was on the machine when machine 1 was shutdown because of a timeout. it was upgrading. so I hacked the job on the precise slave to give it 20 minutes to see a pass. [02:21] and it probably would have been ok [02:22] but 20 mins is just crazy [02:22] menn0, all machines normally upgrade in less that 60 seconds. so 18 minutes is scarry [02:22] sinzui: this is what i see as the issue [02:22] 2015-02-23 18:36:07 INFO juju.worker.upgrader upgrader.go:201 fetching tools from "https://10.0.0.191:17070/environment/329350b1-edf2-4d62-8156-7338a12d3808/tools/1.23-alpha1.1-precise-amd64" [02:22] 2015-02-23 18:39:52 INFO juju.utils http.go:66 hostname SSL verification disabled [02:22] that almost 4 minute gap [02:22] looking at the timestamps, the machine was going VERY slowly [02:23] 30s between seeing the need to upgrade and then /starting/ to download the tools [02:23] menn0, yep. I rebooted the machine too. It is fast when I use it [02:24] sinzui, wallyworld: the agent is still starting up when it sees the need to upgrade. the timings between the various workers starting up is rather wide, like the system was crawling. [02:25] yes [02:25] it just seems machine 1 is very, very slow [02:25] menn0, it wasn't/isn't [02:27] sinzui: it might not be, but it just looks that way based on the logs [02:28] menn0, I also cleared /var/cache/lxc/cloud-precise [02:28] i'm not sure what to do now to diagnose further - if it really is just precise, that makes it very hard to reason about [02:28] we are still waiting for the same job to run with 1.22 to compare [02:29] i guess we see how 1.22comes out [02:29] wallyworld: we could spin up a precise instance on ec2 [02:30] we could, i might do that [02:33] wallyworld, sinzui: as an example, you can see the difference the "slow down" when you look at the deployer worker in the logs [02:34] wallyworld, sinzui: before the API disconnection (due to the state server upgrading itself), the deployer worker starts 1s after it's parent worker (api-post-upgrade) [02:35] wallyworld, sinzui: at the bottom of the logs it starts 5.5 mins after it's parent worker [02:35] wow [02:35] wallyworld, sinzui: that's pretty strange [02:36] that is indeed the difference we feel watching it [02:37] menn0, the upgrade jub just ran with 1.22. [02:37] hmmm. maybe something is trashing the disk? or eating to cpu? [02:37] wallyworld: that's what i'm thinking. and that thing could even be something in Juju itself. [02:37] 2015-02-24 02:35:45 INFO juju.cmd.juju upgradejuju.go:214 started upgrade to 1.22-beta4.1 [02:37] and status shows it complete at [02:37] 2015-02-24 02:36:09 [02:37] sinzui: what instance spec is being used for the precise tests? [02:38] sinzui: that's the kind of timing I would have expected [02:38] but is the above for the state server? or a machine? [02:38] ah [02:39] the cmd [02:39] menn0, the precise slave has 8G ram with 4 vcpu [02:40] menn0, at this moment it has lots of resources free, but it was busy last hour building and testing [02:40] menn0, there was nothing for us to cleanup when we started investigating. the machine was fast for us [02:41] sinzui: the precise slave is dedicated to running the juju test in question? [02:41] wallyworld, it is [02:43] sinzui: i'm looking at jenkins. are the latest successes because of the extended timeout? [02:43] menn0, the 2 master ones are [02:44] sinzui: kk [02:44] menn0, the 1.22 passed normally [02:46] sinzui: 18mins is just nuts [02:46] menn0, sure, but not for maas. I wondered if recent changes needs a new deps and extra work for precise [02:47] wallyworld: are you spinning up a precise instance? i'd like to poke around as the upgrade happens [02:47] menn0: just resetting by source tree [02:47] my [02:47] sinzui: perhaps bit I can't imagine what would cause this [02:47] but [02:55] sinzui: one thing that could be helpful is if the env had debug logging turned on. [02:56] sinzui: it starts of in debug but switches to info for the root logger [02:56] 2015-02-23 18:32:05 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "=DEBUG" to "=INFO;unit=DEBUG" [02:56] menn0, I can do that now that ci is locked down [02:56] menn0, I can switch it now, then we wait for 1.23 to test [02:57] wallyworld: meant to say... all those API disconnect messages in the machine-0 logs are just due to the repeated "juju status" polling that the test script does. that's fairly normal. [02:57] menn0, debug is in place [02:57] sinzui: awesome [02:58] menn0: i have a bootstrapped precise instance - did you want me to add your ssh public key? [02:58] wallyworld: please [02:59] * thumper tries for focus for an hour [02:59] if it is urgent, text me [02:59] * thumper doesn't expect anything urgent [03:00] menn0: 54.82.35.127 i'll start a precise worker machine [03:02] menn0: i'm a tool - i bootstrapped 1.23 not 1.21 [03:02] ffs [03:02] it's so hard to get good help these days... [03:03] forgot to type /usr/bin/juju [03:03] sigh [03:14] wallyworld: has that host gone away? I can't connect to it now. [03:15] menn0: yeah, almost done starting a 1.21 host, sorry [03:15] it's been slow to come up [03:17] menn0: machine 0 is 54.158.193.40 [03:17] machine 1 is 54.204.193.53 [03:19] wallyworld: I thought you were just going to test with the local provider on a single precise machine [03:19] menn0: i was curious to see how precise in general went [03:20] but we can do both [03:20] wallyworld: is my key there? [03:20] yep [03:20] do you want to use machine as host for a local env [03:20] machine 0 [03:21] mongo would need to be stopped [03:21] i should just fire up a new machine [03:21] wallyworld: I was using the wrong key... i have a personal one and a canonical one. fixed now [03:23] wallyworld: how about I fire up the machine for the local test [03:23] ok [03:29] menn0: upgrade on aws precise was fast, so it has to be a resource contention issue [03:29] wallyworld: ok. a useful data point. i'm just setting up this other instance now. [03:29] sinzui: SSD or magentic storage? [03:31] menn0, I think the latter. the machine is in Hp [03:31] sinzui: cool. i'll go with that. [03:41] wallyworld: it's 54.190.88.226. installing 1.21 now. [03:41] ok, does it have my ssh key imported? [03:42] wallyworld, all the mass deploy jobs are still failing. [03:42] hmmm [03:43] i wonder what's different compared with clouds [03:43] the same scripts should work on both [03:43] is there a url with logs? [03:46] wallyworld, no, we cannot get logs for maas because they have unresolvable dns [03:46] oh joy, this will be fun to solve [03:46] wallyworld, I am attempting it get into the maas to find the names, for the unit, them use virt-viewer to connect directly to the console [03:47] I dont' have creds though for maas 1.7 and 1.8 though [04:28] wallyworld: well i'm seeing a problem with the upgrade on precise but it doesn't look the same as what happen in the logs from sinzui [04:29] wallyworld: machine-0 fails to upgrade because of: [04:29] "set AvailZone in instanceData" failed: failed verification of local provider prerequisites: [04:29] cloud-image-utils must be installed to enable the local provider: [04:29] sudo apt-get install cloud-image-utils [04:29] wallyworld: shouldn't the upgrade step handle that itself? [04:30] that's expected [04:30] that package has to be on the host machine [04:30] as do one or two others [04:30] hmmm, maybe [04:30] but in general [04:30] the local provider checks for needed packages and prints those messages if they are not there [04:30] I installed juju-local so shouldn't this already have been taken care of [04:31] ? [04:31] wallyworld: maybe it got uninstalled by another operation? [04:31] cloud-image-utils isn't a prereq of juju local, maybe it should be? [04:32] wallyworld: i'll work around this for now so that I can get to the actual problem that CI is seeing [04:32] i think in trusty cloud-utils includes cloud-image-utils [04:32] i can't wait for precise to go away, but still 2 years keft [04:33] wallyworld, I cannot copy this crap gui's cloud-init-output [04:33] sigh [04:34] is there an obvious error? [04:35] wallyworld, I can say it is the same error as before where the apt-line is garbled [04:35] i'm trying to spin up maas locally but the bootstrap node refuses to transition from Deploying [04:35] that doesn't make sense as there should be no quotes there now [04:35] and why just maas and now AWS or HP etc [04:37] and it died as I almost got the log [04:37] wallyworld, as per the bug [04:37] util.py[WARNING]: Failed to install packages: ['bridge-utils', 'curl cpu-checker bridge-utils rsyslog-gnutls cloud-utils cloud-image-utils'] [04:38] sigh [04:38] * sinzui double checks commit [04:38] seems bridge-utils is added twice which is messing it up [04:38] wallyworld, are are testing your commit [04:39] yes on aws [04:39] works perfectly [04:39] wallyworld, apt doesn't care about duplicates on the command line [04:39] sure, but adding twice seems to mess up juju's rendering of the cmd line [04:40] sinzui: can i get access to your maas? [04:40] if i can't reproduce, i can't fix [04:40] wallyworld, sure but it so poorly documented I cannot offer much [04:41] maybe pm or email the ip of the controller and auth key [04:48] wallyworld, sinzui: i haven't been able to repro this precise upgrade timeout issue yet but I still have a few ideas [05:01] sinzui: ah, i see where bridge-utils is added twice - it's hard coded in the maas provider [05:02] that's why it is failing on maas [05:02] wallyworld, really> [05:03] wallyworld, jog hack the local machine to capture logs from machine-0. This isn't too helpful since the failures are machines 1 and 3 [05:03] sinzui: yes, on machine 0 we run the scripts ourselves, not cloud init [05:07] sinzui, wallyworld: ok, i'm stumped on how to reproduce this precise issue [05:07] sinzui: it may be the quickest thing to do is to go back to installing one package at a time now that cloud-image has been moved into the correct repo [05:07] sinzui, wallyworld: every upgrade works quickly for me [05:08] wallyworld, yes, with the caveat that we must also install clout-utils to force the upgrade [05:08] sinzui: yep, i'll install cloud-utils along with the other ones we expect [05:09] wallyworld, or we do dist-upgrade (but I think that is risky for today) [05:09] yeah, i'll keep it simple [05:09] menn0: i think juju might be ruled out - something must be thrashing the machine though [05:10] wallyworld: either that or i'm missing some aspect of the test setup [05:10] i'm about to EOD [05:11] ok, thanks for looking [05:15] sinzui: do i need to do apt-get install --target-release precise-updates/cloud-tools cloud-utils ? [05:16] or can i leave off the --target-release bit [05:16] i think i need it right? for precise? [05:25] sinzui, wallyworld: of course I just realised that I was testing with a 1.23 that was a little behind the times and didn't include some of the recent commits that might be contributing to the issue [05:25] * menn0 facepalms [05:25] sinzui, wallyworld: it might be worth repeating what i've done... [05:25] hard to get good help :-) [05:25] touche [05:25] i gotta fix this other one first :-( [05:26] but I really need to EOD or my wife is going to get pissed [05:29] np, leave it to us [05:50] hi wallyworld [05:53] wallyworld, I was just looking at the MaaS test results for 1.22 revision 79e5ea8a and still see the failure mentioned in bug 1424695. [05:53] Bug #1424695: maas cloud-init cannot download agent from state-server

[05:54] wallyworld, it's looks like you through that commit should have fixed that bug? [05:54] thought even [05:58] wallyworld, looks like maybe you dropped for a bit, did you see my comment above? [05:58] jog: no [05:58] wallyworld, I was just looking at the MaaS test results for 1.22 revision 79e5ea8a and still see the failure mentioned in bug 1424695. [05:58] Bug #1424695: maas cloud-init cannot download agent from state-server

[05:59] looks like you expected that commit to resolve that quoted package string issue? [05:59] jog: yes, sadly maas fails where the other clouds are ok, so i've marked bug as in progress again. i don't have a working maas to test with [06:00] wallyworld, you have access to finfolk.internal ? [06:00] no, i tried and couldn't get in [06:16] * jog wonders what happened to core vmaas setup on gremlin.internal that was setup during the Brussels sprint. [06:22] jog, well, we were supposed to use it for ipv6 work of maas and juju, but we didn't need to [06:22] jog, IIRC the maas guys used it for qa stuff (or was it finfolk?) [06:23] dimitern, finfolk is used by juju-qa but I have an extra env setup for debugging for anyone on core that needs it [06:24] jog, right, that's good to know then :) [06:42] dimitern: hi, i'm a little concerned that the vet warnings can be suppressed. shouldn't we be fixing the warnings? those provisioner ones are annoying :-) [06:43] wallyworld, they should've been fixed yesterday by jw4 [06:43] wallyworld, but maybe it bounced due to the blocker [06:44] wallyworld, nope - it did bounce - https://github.com/juju/juju/pull/1654 [06:45] wallyworld, and these appear for go1.4 only, which we don't officially support yet ;) [06:45] wallyworld, go vet in 1.4 is dumber than 1.2 - "%q" is reported for a type with String() method [06:56] wallyworld, I'm not sure how much you've seen of my comments [06:56] dimitern: my stupid connection to free node keeps dropping, so none sorry :-( [06:57] wallyworld, I thought so :) [06:57] wallyworld, basically go vet 1.4 is dumber than go vet 1.2 (which is still the official go version we're using) - "%q" for a type with String() is reported [06:58] wallyworld, and there's this PR which bounced due to the blocker - https://github.com/juju/juju/pull/1654 which fixes the warnings [06:58] wtf, go vet go dumber [06:58] what are they thinking [06:58] i'm on go 1.3.x [06:58] they were *not* thinking :) [06:59] dimitern: with the blockers - we tried and couldn't reproduce the precise uprade one today [06:59] we contend it's a machine thrashing issue on the test vm [06:59] wallyworld, the slowdown or the quoted packages? [07:00] slowdown' [07:00] with the packages one, i changed the behaviour but maas still failed (only maas, not aw etc) [07:01] so i'm looking to revert the apt behaviour to be as per 1.21 since as of today or yesterday the cloud archive has been updated [07:01] wallyworld, hmm.. maas took longer than usual to upgrade according to the bug reports [07:01] and we don't need to install cloud-utils and cloud-image-utils together [07:01] oh, we were working on the assumption of local taking too long, that's what cutis told us [07:02] one thing occurred to me - for precise, are we making sure we add the cloud-tools pocket? [07:03] dimitern: yeah, that's added [07:03] there's the MaybeAddCloudToolsWhatever in cloudinit that gets called for a given series - but IIRC it was put in an if block with enableAotUpdate [07:03] dimitern: and i just confirmed, doing apt-get install cloud-utils by itself breaks [07:03] but apt-get install cloud-utils cloud-image-utils works [07:03] but setting that up breaks on maas [07:03] wallyworld, due to cloud-init getting removed? [07:03] yeah [07:04] can you confirm cloud-image-utils comes from the cloud-tools pocket on precise? [07:04] it has to be fixed there [07:05] dimitern: actually, it may be that we are only adding the cloud-tools pocket when bootstrapping [07:05] that would explain things [07:05] wallyworld, that's wrong if we do [07:05] wallyworld, I'll have a look what triggers it [07:09] dimitern: i have a branch which i was going to propose pn gh to revert the apt behaviour to be more like 1.21, but with cloud-utils added (it needs to be installed to get the right version). bootstrap is fine, but adding a new machine fails. it may be fue to cloud-tools pcoket not being added except for at bootstrap. but i have to go out to a Foo Fighters concert, will be back later. here's the branch. https://github. [07:09] com/wallyworld/juju/tree/revert-apt-install-method [07:09] wallyworld, sure, I'll look into it [07:09] the above branch reverts the behaviour of multiple packags on the one apt-get line [07:09] wallyworld, enjoy the concert ;) [07:09] ty [07:10] dimitern: if you run up a machine, apt-cache policy cloud-utils should show 0.27 [07:10] on bootstrap and a worer node [07:10] worker [07:10] and cloud-image-utils should be there too [07:10] wallyworld, ok, will check both [07:10] i'll check back later in a few ours [07:11] tyvm [07:11] np [08:58] is gwacl's primary project site still launchpad.net/gwacl? [09:10] dimitern: did you see william today? [09:10] jam, not yet [09:20] wallyworld: did you want to discuss ensure-ha --to ? [10:32] natefinch: /wave [10:32] jam: howdy [10:32] jam: gimme 2 minutes to go get my coffee? [10:33] k [10:40] natefinch: I don't hear you [11:20] morning [11:59] natefinch_: still up? [11:59] perrito666: I am here... got up for an early meeting [12:00] perrito666: probably will have to go soon as the kids are stirring [12:01] natefinch_: I cannot help but notice you are OCR today and I have a very small change here http://reviews.vapour.ws/r/995/ [12:03] perrito666: ship it! [12:04] tx [12:04] I should have known that the trick was to find you half asleep [12:05] 3 blocking bugs? oh what have I done to deserve this [12:05] doh [12:10] * dimitern nailed the cause of the blocker [12:11] * dimitern hates cloud-init more than before now [12:22] fwereade: greetings [12:22] jam, heyhey [12:23] perrito666: doesn't removing restore break compat with older servers? [12:27] fwereade: so I commented on your last request. I was wondering if we would still want the server to give the official time. [12:28] jam, so the server would always return what you asked for? [12:29]