[00:04] axw: hiya, i'd love a second lgtm on http://reviews.vapour.ws/r/302 if possible [00:12] jw4, what was the last thing you saw me say? [00:12] jw4, last thing I saw you say was (figuratively) [00:18] perrito666: ping [00:20] bug 1384001 was marked as Fix Committed in juju core, except it's not. It's only fix committed in core when the dependencies file is updated [00:20] Bug #1384001: Juju doesn't retry hard enough when destroying MAAS environments

[00:21] fwereade: I wandered off when your autocomplete quit working [00:21] fwereade: :) [00:22] fwereade: I guess I don't see any way around serializing sequence() if we want to ensure ordered sequence numbers [00:23] fwiw, I found that the unit sequence number is using a similar but less stringent mechanism using 'unitseq' right on the service doc [00:23] perrito666: ignore me [00:23] my godeps failed [00:23] sorry [00:30] * jw4 eod-ish [01:01] wallyworld: will take a look in a sec [01:01] np, ty === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 [01:53] of all the api client tests, which one is a good example of how to mock test an api client? I don't want to follow the wrong example. === kadams54 is now known as kadams54-away [02:16] wallyworld: I just came back from dinner, I am pretty sure I committed that and juju bot thinks likewise [02:17] perrito666: yeah, forget i even mentioned it, my master was out of date and godeps didn;t work, so toally my mstake [02:17] that is why you changed the state of the bug I assume :) [02:18] ok then, going to sleep, see you all tomorrow [02:19] perrito666: see ya. i did change the bug back to waht it shold be [02:20] cmars: sadly, in juju/api/client, I am not aware of a test suite which doesn't run up the full stack. it's todo to not do that, but i don't thinks it's been done yet anywhere :-( [02:30] menn0: wired up the step: http://reviews.vapour.ws/r/310/ [02:30] waigani: will take a look shortly [02:31] waigani: np [02:35] wallyworld: here's part of the upgrade steps rework: http://reviews.vapour.ws/r/311/ [02:35] part 2 even [02:37] menn0: ok, ta. just about to go to lunch, will look when i get back [02:37] wallyworld: np [02:40] waigani: change looks good [02:40] menn0: thanks [03:25] menn0: one query in the review [03:26] wallyworld: looking [03:35] wallyworld: just responded [03:35] looking [03:37] menn0: maybe make it so that if Tag != "", then run steps for fromVersion? But if we are already fully on fromVersion, seems unnecessary to run steps again? [03:37] wallyworld: that's how it works already I believe [03:38] hmmm, ok. if thats how it works now, then we shouldn't change the behaviour in this branch perhaps. but seems wrong [03:39] wallyworld: this branch changes the behaviour in terms of the target version [03:39] wallyworld: those 2 extra tests just confirm already existing behaviour [03:39] wallyworld: it makes sense because 1.21-alpha/beta < 1.21 [03:39] yeah, that appears true. i was just surpised to see it running steps unnecessarily [03:40] yes when Tag != "", run the steps [03:40] wallyworld: so if you're upgrading past 1.21 the steps for 1.21 should get run [03:40] yes, but not if you're upgrading from 1.21 [03:40] wallyworld: I wouldn't say it's running steps unnecessarily. more steps may have been added between alpha/beta and final. [03:41] yes, and in those cases, Tag != "" [03:41] and we don't support upgrading from dev versions [03:41] but anyway, if it's existing behaviour, no need to hld up this branch [03:41] wallyworld: ok [03:41] just surprised me is all [03:42] ie i don't agree with it :-) [03:42] wallyworld: I think the behaviour is ok but whatever :) [03:42] wallyworld: I'll make that test change you suggested and re-push [03:42] well, so long as the steps are idempotent, it diesn't matter [03:42] but there's a cost to running unnecessary steps [03:43] just added a ship it, thanks for extra 1.22 change [03:46] wallyworld: the problem is I guess, that we don't know which steps were run when coming from a alpha/beta release [03:46] yes, but we don't officially support upgrading from alpha/beta [03:46] wallyworld: I'm not adding 1.22 to the test data btw. I just changed the releases to be 1.20-alpha/beta to 1.21 to take advantage of what's already there. [03:46] yep, +1 [03:48] wallyworld: thanks. will merge now. [03:48] thanks for fixing [04:09] wallyworld: FYI fwereade has asked me to look at the relation scope issue for subordinate relations [04:09] wallyworld: where they should be scoped to the machine, not the principal service [04:09] wallyworld: apparently it's affecting Landscaope [04:09] Landscape [04:09] wallyworld: ring any bells? [04:10] menn0: vaguely [04:10] menn0: as yes [04:11] they want to add a single global chamr on a machine [04:11] wallyworld: he asked me check with you regarding the priority of this [04:11] wallyworld: that's the one [04:11] menn0: if you ask landscape, very high [04:11] we did rule it out of cope this cycle [04:11] but [04:11] if you had bandwidth to fix, that would be amazeballs. but it will be big [04:12] wallyworld: he's asked me to look because i've just been in that code to do the env UUID upgrade [04:12] and not just landscape want it, wil be very useful [04:12] ok, can you then see if you can spec out what the changes will be, so it can be estimated and scoped? [04:13] I don't strictly have the cycles. thumper wants me focused on the multi-environment work [04:13] ok [04:13] in that case, leave it, as we already told them it would not be in scope this cycle [04:13] i'll do the initial thinking and scoping and see how it looks [04:13] ok, ta, anything would be good as ti will be more than we have now [04:18] good afternoon wallyworld, FYI bug 1364410 has become consistently reproducible again in Juju CI [04:18] Bug #1364410: API server fails to start with "address already in use" in MachineSuite tests

[04:18] hi [04:19] i wasn't aware of that bug, thanks for letting us know [04:20] jog: sadly, looking at the bug report, if it is the issue i think it is, it will not be easy to fix [04:20] it was a blocking CI regression bug in early September but sinzui demoted it after it became intermittent and we were able to get a blessed build revision [04:21] any fix will likely be a lot of work from what i now remember after reading the bug, as it changes the way tests are set up [04:21] so i'm not sure off hand how timely a fix can be [04:23] i'll take a look, just got to finish some wip [04:23] wallyworld, ok currently the last 10 ppc64el unit test runs have failed [04:23] i thought ppc64el was non voting? [04:24] it was non-voting for awhile but once the compiler test was worked around/disabled we set it back to voting [04:24] maybe not [04:24] ah ok [04:24] i guess a call needs to be made if we want to ship alpha3 this week [04:25] maybe this test needs to be disabled as well, so we can at least gain coverage from the rest? Depending on how risky this issue is to users? [04:26] if a fix is not coming soon, disabling the entire suite is not ideal [04:27] i.e. disable == make it non-voting [04:28] if it's only on ppc64el, maybe we can skip to get alpha3 out, and then unskip once that happens [04:37] wallyworld, unit tests are passing for other arches but there are a number of other non-unit test failures preventing a blessed revision [04:38] jog: just local upgrades though i think? [04:38] and a fix is currently going through CI now [04:39] I also see a functional-ha-backup-restore and kvm-upgrade-trusty-amd64 failure... I'll go see if I can find out why [04:41] ok, ta [04:41] axw: a small one http://reviews.vapour.ws/r/313 [04:41] ty :-) [04:46] wallyworld: done [04:46] ty [05:02] jog: i think i know why local upgrade tests are failing [05:02] juju --show-log set-env -e local-upgrade-trusty-amd64 tools-metadata-url= [05:03] tools-metadata-url is deprecated, replaced by agent-metadata-url [05:03] config will be migrated on start up, but old attr name is not valid in set-env [05:03] so test script needs to be tweaked === kadams54 is now known as kadams54-away [05:11] wallyworld, hmm... I thought we could not depreciate something like that and would need to support both tools-metadata-url and agent-metadata-url until the major version changes? [05:11] we do support both [05:11] in config [05:12] the approach is to print a warning when deploying an environment, and copy across the value for people [05:12] thus if you have an environments.yaml with the old attr, it will be migrated [05:12] when you deploy your environment [05:13] we've not supported set-env in that way before [05:14] I see... you said that above... so it's just set-env you think we need to update in the test scripts [05:14] not sure if that's something (set-env) we need to do or not [05:14] just set-env yes [05:14] for end customers, tools-metadata-url is only ever read only [05:15] they could update it, but there's no reason to [05:15] if stakeholder strongly feel we need to add deprecated handling to set-env, we could do that [05:16] it just hasn't come up before as an issue [05:18] I could see it being an issue if enterprise user scripts break. CI certainly has scripting that's setting tools-metadata-url [05:20] those same scripts are used for <1.21, so we'll need to add some checking [05:23] wallyworld: I'm going to take a look at #1387421, then I'll look at the scp bugs [05:23] Bug #1387421: cmd/juju: add help on placement directives (zones, maas host name)

[05:23] ok. i'd be surprised in any enterprise user set tools-metadata-url after bootstrapping. but other depecated attrs, maybe they do [05:23] axw: ok, sounds good. i'm about to look into the critical blocker to see if there's any palatable fix [05:24] okey dokey [05:29] jog: if i'm not around later past my EOD, could you ask curtis about his opinion on the set-env handling of deprecated attributes. it's not something we've supported in the past but if we need to we can add it [05:29] wallyworld, I think he's out Friday [05:30] ok, we can talk next week [05:30] maybe if i get time i can look to add it [05:31] If I had to bet though, risk of breaking external customers in the same way this breaks CI will make it an issue :-/ [05:31] except that external customers will not set tools-metadata-url [05:32] but yeah, for other maybe; i'd have to check to see what ones have been deprecated [05:32] I know we're doing things a bit different, so maybe I'm wrong [05:32] for customers tools-metadata-url is static [05:32] set once in yaml [05:33] main use is for private clouds, and those are i think almost always openstack, and suck customers would use keystronr catalog instead i would think [05:38] jog: i'm looking at http://juju-ci.vapour.ws:8080/job/run-unit-tests-trusty-ppc64el/ as linked in the bg - i can't see test runs with evidence of that same bug [05:38] i've looked at a few console logs, there's failures, but they are openstack reated [05:38] or upgrade related [05:39] not related to the api server port issue [05:42] wallyworld, I could have mis-diagnosed it as the same... I initially opened bug https://bugs.launchpad.net/juju-core/+bug/1387936 and it has a log with the failure we're seeing [05:42] Bug #1387936: Unit tests on ppc64el trusty failing with streams resources not found

[05:42] jog: there errors look like the control hooks aren't running, which is indicative of a compiler issue [05:43] jog: that bug you just pasted isn't a bug [05:44] juju will look for simplstream data in several places, and with debug on, it logs the places it tries but is unsuccessful [05:44] this helps developers diagnose issues if people complain their system won't bootstrap [05:45] the compiler issue i mention above in the past has been a gccgo vs golang go issue [05:45] they handle stack frames differently [05:45] so some low level stuff fails on gccgo [05:46] jog: on the basis of the above, it would be great if we could unblock landings from bug 1364410 [05:46] Bug #1364410: API server fails to start with "address already in use" in MachineSuite tests

[05:49] ok... I thought all the compiler issues had been worked around. mgz would working with various people on that... I can drop the severity to high for now. [05:50] jog: i have no answer for why the hooks aren't being run - in the past it's been a compiler issue [05:50] ok, it's set back to high [05:50] ty :-) [05:59] jog: i'll start the work to handle deprecated settings in set-env, so when it lands you can keep the CI scripts unchanged [06:01] wallyworld, that would be awesome and less risky, in case of possible customer scripting... It's 11pm here I'm about to turn in, anything else before I go? [06:01] jog: nah, thanks for your help [06:01] thanks to you as well [06:02] FYI, I moved the functional-ha-backup-restore test from HP to AWS and it passed, so possibly HP cloud flakiness [06:09] wallyworld, would you please comment in bug https://bugs.launchpad.net/juju-core/+bug/1387936 with your suspicions about the compiler? [06:09] Bug #1387936: Unit tests on ppc64el trusty failing

[07:42] finally re azure.. instance level public ips.. http://msdn.microsoft.com/library/azure/dn690118.aspx [07:45] axw, fwiw re storage, gce just added its first local disk availability .. not by itype.. by api allocation at creation (81usd per month per dev) https://cloud.google.com/compute/docs/local-ssd [07:47] hazmat: 81 USD??? [07:47] sounds like a lot [07:47] so they never had ephemeral storage before? [07:54] axw, no [07:56] axw, they always did net-storage afaics, they don't have sizing options (its per gb on local ssd) and color of black (model t ref) is 375gb [07:57] so I see. reasonably hefty [08:15] morning all [09:03] morning [09:29] TheMue, morning [10:01] TheMue: dimitern: ok if I'm late to standup by ten minutes? [10:02] voidspace, np [10:02] thanks [10:09] dimitern: TheMue: omw [10:16] davecheney, ping [10:27] gsamfira: hey [10:28] hazmat: ack [10:32] gsamfira: dimitern asked if reboot deals with debug-hooks in the same way it does juju-run? [11:04] is fwereade around today or off still? [11:04] mattyw, I'm here [11:04] mattyw, what can I do for you? [11:09] morning all [12:10] axw: thanks for scp fixes [12:11] fwereade: since you are ocr, can i pretty please get a review on a critical bug fix for alpha3 http://reviews.vapour.ws/r/317/ ? [12:11] wallyworld___: nps [12:12] wallyworld___, just toasting a sandwich, on it in 5 [12:12] sure, thanks [12:20] wallyworld___, LGTM, maybe one test tweak [12:20] fwereade: ty [13:29] man in this coutry its easier to get uranium that a port replicator for my lenovo model [13:45] perrito666: sorry, we'll have to move our 1:1, I have to go catch an escaped chicken [13:45] natefinch: np [13:46] best reason ever, btw [13:55] I live to amuse. [13:56] natefinch: nah, no pics nor vids of you chasing a chicken, thats not fun [14:00] perrito666: closest I can come (my daughter, not me): https://fbcdn-sphotos-g-a.akamaihd.net/hphotos-ak-xpa1/v/t1.0-9/10303745_10152646984367028_7439232690755188047_n.jpg?oh=4e262e18b2e2723819b2bb8e2b250b53&oe=54E3659E&__gda__=1423914888_2c43b0a7691dfd8982b5deb1773a1595 [14:01] is that the runnaway chicken? [14:01] nah, just one she picked up a while back. [14:01] I have surprisingly few pictures of the chickens :) [14:02] natefinch: she smiles too much I dont think she knows what will happen to that chicken in the future [14:02] perrito666: we are actually due for a "harvest" in a week or so [14:03] yum [14:03] a good things about chicken is that they loose the cute factor when they are ready to eat [14:04] Yeah, the only hard part is the initial act of going from chicken the bird to chicken the meat. Really, once the feathers are off, they're not so much different than what you see in the grocery store. [14:05] perrito666: moonstone? [14:05] well once you tags.Strip("") they kind of no longer look like chickens anyway [14:06] haha yeah [14:26] natefinch, does your team have time to take a look at https://bugs.launchpad.net/juju-core/+bug/1388073 [14:26] Bug #1388073: set-env no longer accepts empty values

[14:26] we need to get that resolved asap [14:27] alexisb: ok [14:27] thanks [15:11] natefinch, alexisb - I'm fixing #1388073 [15:11] Bug #1388073: set-env no longer accepts empty values

[15:13] tasdomas, thanks! [15:22] tasdomas: okay, I want to yell at you [15:23] why cd17b6f3 [15:24] mgz, because we had that same piece of code pasted over 4 different places? [15:24] you wrote it wrong again! [15:25] straight up regresses bug 1379208 [15:25] Bug #1379208: TestAddMetrics fails on gccgo

[15:42] Hey I keep having power outages, running to greener pastures now [15:51] tasdomas: see bug 1387936 [15:51] Bug #1387936: Unit tests on ppc64el trusty failing

[15:51] has link to one of the test runs [15:55] mgz, in comment #3 of that bug, (2) Is Ian's change for a critical bug fix or is it something we can revert? [15:57] alexisb: I'm still trying to tease out if it's an accidental merge issue or an intended change, there are like 70 files touched, which makes a straight revert annoying [15:58] I'll add more details in a second [15:58] mgz, ack, let core know how we can help [15:58] yay, back online [15:58] not sure if it's permanent or temporary [15:58] and thanks mgz for triaging! [15:58] they're still futzing with the line [15:59] voidspace, are you one of those that gets nervous when they are disconnected [15:59] alexisb: might have been the most productive hour of the day ;-) [15:59] alexisb: although I compulsively click things when I'm thinking [15:59] lol [15:59] alexisb: and it's distracting when the clicks do nothing... [16:22] fwereade: ping [16:23] katco, pong [16:24] fwereade: what i think is a reasonable draft of that spec is up [16:24] katco, lovely, thanks [16:24] fwereade: wallyworld didn't see anything too egregious, so i think we're ready for your +1 [16:24] fwereade: (or not as the case may be) :) [16:25] katco, brilliant, I *might* get to it tonight but I'm afraid I'm off to a halloween thing at laura's school imminently [16:25] fwereade: how dare you care about your family! [16:25] katco, ;p [16:25] fwereade: totally understand :) have fun and HAPPY HALLOWEEN! :D [16:26] katco, laura is dressed as a dinosaur :) [16:26] fwereade: oh dear. pictures!! [16:26] fwereade: mine is a ladybug :) [16:26] aww :) [16:26] alexisb: is yours going as a tractor or something similar? :) [16:27] katco, he is going as a lion as he loves the "roar" [16:27] alexisb: aw :) [16:28] katco: have you seen the ladtbug girl books? [16:28] natefinch: i have not! link please!! [16:28] http://www.ladybuggirl.com/ [16:29] natefinch: ok you are DIRECTLY responsible for me not being productive for the next bit of time. [16:29] katco: great art and good morals and strong girl lead [16:29] natefinch: love it [16:29] katco: yep :) [16:32] katco: Lily, our eldest: https://fbcdn-sphotos-d-a.akamaihd.net/hphotos-ak-xpf1/v/t1.0-9/p526x296/10624919_10152847646912028_8733903205136776490_n.jpg?oh=1ba16b1062ac2694854a3a2c57886f6a&oe=54E5B3A7&__gda__=1423915697_a6cde75ffa599528b4531dab195d58d5 [16:32] natefinch: oh my she is DARLING! [16:33] is that... a donut on a rope? haha [16:33] haha.. halloween tradition, like bobbing for apples, but more sanitary [16:33] lol [16:34] our younger daughter, Zoë: https://scontent-a-iad.xx.fbcdn.net/hphotos-xap1/v/t1.0-9/10538654_10152847638537028_3233943882042977754_n.jpg?oh=8f5b74c34d729c2175c4a99e3895f8d7&oe=54EB41C1 [16:35] natefinch: that smile is infectious :D [16:36] the giggles even more so :) [16:36] you have some cute ladies :D [16:45] katco: thanks :) [16:52] natefinch, love the picture of Zoe! [16:52] Lily is also very cute, but Zoe's smile is awesome [17:11] natefinch, mgz, http://reviews.vapour.ws/r/320/ - should fix part of 1387936 and 1388073 [17:12] tasdomas: thanks! [17:22] * perrito666 is back [17:23] mgz, so is the last blocking the commit from Ian? [17:25] alexisb: I'm still not sure where it started breaking [17:26] ideally someone in the US can follow up and track down the point - it's confused by one of the failures being a variable segfault when running the tests [17:27] natefinch, I know you have tasks today, perrito666, katco, ericsnow, tasdomas, cmars ^^ can any of you help out [17:27] we need to get 1.21 released so this is top priority [17:27] fwereade, ^^ [17:28] alexisb: I'll take a look but no one should let that stop them from looking too :) [17:29] ericsnow: basically, test with -compiler gccgo in provider/openstack is fubar [17:29] I'm not sure exactly which change regressed, it's very messy [17:30] mgz: that means I would need to set things up to test against an openstack provider? [17:31] ericsnow: nope just the unit tests [17:31] go test -compiler gccgo [17:31] mgz: k [17:34] could somebody take another look at http://reviews.vapour.ws/r/320/ ? [17:37] natefinch: <3 ladybug girl. I read all of those to my duahgter. [17:37] mgz: so should I expect "go test -compiler gccgo" to have failed? (it didn't) [17:38] it fails in provider/openstack for me [17:38] well ubuntu upgrade plus power issues let my machine in a not so nice shape [17:39] mgz: I just did an "apt-get install gccgo" before running it (didn't have gccgo installed yet) [17:39] jrwren_: Yeah, we got one as a gift, loved it, and have been getting all the rest of them. [17:39] mgz: go test -compiler gccgo for provider/openstack or all of it? [17:40] mgz: never mind (was in the state directory) [17:41] perrito666: provider/openstack is the interesting bit [17:41] ericsnow: you have the failures/segfault now? [17:41] mgz: running [17:41] * perrito666 runs all of provider just in case [17:45] ls [17:45] heh [17:45] agent cloudinit dependencies.tsv etc mongo rpc upgrades [17:46] holy shit it worked [17:46] :-) [17:47] mgz: aborted ("munmap of stack space failed: errno 22") in apiserver [17:47] mgz: ok github.com/juju/juju/provider/openstack 51.623s [17:47] master [17:47] go test -compiler gccgo github.com/juju/juju/provider/... [17:48] mgz: (still running) [17:48] on master I can't even get the test to compile in that directory [17:48] /tmp/go-build382239753/github.com/juju/juju/provider/openstack/_test/_testmain.go:52:15: error: reference to undefined identifier ‘testing.MainStart’ [17:48] m := testing.MainStart(matchString, tests, benchmarks, examples) [17:48] ^ [17:48] FAIL github.com/juju/juju/provider/openstack [build failed] [17:49] hm... I wonder if I screwed myself by upgrading to go 1.4 yesterday [17:49] hm... [17:49] mgz: I seem to be immune to these issues [17:50] ahh, I think I know what it is... the new go tool running with the old gccgo ... funny [17:51] perrito666: gccgo version? [17:52] gccgo (Ubuntu 4.9.1-16ubuntu6) 4.9.1 [17:52] ubuntu 14.10 [17:53] mgz: segfault in github.com/juju/juju/environs/simplestreams and in github.com/juju/juju/provider/openstack [17:53] mgz: also, github.com/juju/juju/worker/uniter/jujuc failed [17:54] mgz: so more than just provider/openstack failed for me [17:54] the jujuc one tasdomas has branch up fixing [17:55] I think simplestreams/openstack is the same regression, but I have been failing to get it to pass by reverting the changes I think may be to blame [17:56] I need one of these: http://goo.gl/OPhW6a [17:56]