=== kadams54-away is now known as kadams54 [00:05] katco: ping === kadams54 is now known as kadams54-away [00:47] thumper: https://bugs.launchpad.net/juju-core/+bug/1456398 [00:47] Bug #1456398: cmd/jujud/agent: runtime panic during tests [00:47] this is me today [00:47] git bisect for the git gods [00:50] Bug #1456398 was opened: cmd/jujud/agent: runtime panic during tests [01:02] Bug #1456398 changed: cmd/jujud/agent: runtime panic during tests [01:08] Bug #1456398 was opened: cmd/jujud/agent: runtime panic during tests [01:24] davecheney: i'm a big fan of git bisect [01:43] axw: around? [01:47] thumper: hey [01:47] axw: oh hai [01:47] axw: just talking to wallyworld about the storage provider and multiple environments [01:47] axw: just bitten us [01:48] thumper: bugger. what's the problem? [01:48] thumper or wallyworld: txn pruning integration with Juju: http://reviews.vapour.ws/r/1720/ [01:48] axw: runner exited: watching volumes: permission denied [01:49] axw, wallyworld: likely to be a simple fix [01:49] just finding the right line to add [01:50] * thumper tries to run up to speed [01:50] axw: the storage provider is a machine agent worker? [01:50] thumper: the "storageprovisioner" worker is run twice: once in the env manager, once in each machine agent [01:51] thumper: kinda like how we have a env-level provisioner and one per machine for containers [01:51] axw: well... we have an env-level provisioner per environment [01:51] not just one on the state server machine [01:51] I'm wondering if this is the gotcha [01:52] thumper: quite possibly [01:52] axw: what is the worker called? [01:52] found it [01:54] thumper: if you get stuck, and can give me a repro, I can debug it [01:54] the environ-storageprovisioner is running per environment [01:54] so that should be fine [01:54] waigani: where was the error exactly? [01:57] axw: it was waigani that ran into the issue, you two can talk directly so I can go smash the cli some more [01:57] okey dokey [01:57] thumper: ERROR juju.worker runner.go:219 [01:58] waigani: hey, there is this cool function you can add... [01:58] * thumper goes to search for it [01:58] I wrote it for this type of situation... [01:58] I think [01:58] thumper: I created a hosted env, deployed wordpress, work-load status hangs on: message: Waiting for agent initialization to finish. Debug-log then is full of perm denied errors [01:59] axw: if it helps, I saw similar error in featuretests - i wasn't on multiple env tho [01:59] LoggedErrorStack [02:00] axw: i was runing one cli command and than followed by another in the same test [02:00] menn0: review done [02:00] so there is this line: return isEnvironManager && tag == st.EnvironTag() [02:00] see worker/provisioner/provisioner.go line 181 [02:00] axw: i was going to talk to u about it but got distracted :( === kadams54 is now known as kadams54-away [02:00] storageprovisioner.go:78 [02:01] if the error is not nil and the feature flag is set, it logs the error stack [02:01] very handy for debugging [02:01] when you really don't expect an error return there [02:01] waigani: st should be scoped to the env at that point? or not? [02:01] if it is okay to watch volumes in hosted envs, the fix should just be a matter of removing the isEnvironManager line [02:02] axw: yes, it should be [02:02] axw: I can double check though [02:02] wallyworld: thank you [02:02] np [02:02] waigani: don't understand, why would you want to remove that line? [02:03] anastasiamac: thanks for your review too [02:03] axw: I haven't tested, but it seems that would be what is stopping volumes from being watched in the hosted env [02:04] waigani: we only want state servers to be able to access environments [02:04] axw: as it will return false if it's hosted [02:04] axw: you've got this comment in the code: [02:04] / TODO(axw) allow watching volumes in alternative [02:04] // environments? Need to check with thumper. [02:04] menn0: always nice :D tyvm for PR [02:05] waigani: yeah, I don't think that's relevant. that was before all the JES stuff went in, now I see we have one worker per env [02:05] waigani: before I thought we might have one worker that spoke to multiple envs [02:05] axw: ah okay [02:09] waigani: where is the isEnvironManager line you are talking about? [02:09] thumper: line 79 [02:09] waigani: of which file [02:09] storageprovisioner.go [02:10] apiserver [02:11] no one will ever log in to an apiserver with an environ tag [02:11] that block should be removed [02:12] the only things that log in are machines, units or users [02:12] should we add cases for units and users? [02:13] as it currently doesn't have any [02:13] thumper: what do you mean nothing will log in with an environ tag? what bit of code leads you to think that's happening? [02:14] line 72 or apiserver storageprovisioner.go [02:14] thumper: the "case names.EnvironTag" is not the tag used to log in, it's a tag that the storage provisioner uses to scope things [02:14] uuurrrggghhhh.... [02:15] this seems overly complicated and ackward.... [02:15] thumper: no worker ever passes the login tag to canAccess [02:15] I think more logging around this access would show where the problem is [02:15] * thumper leaves it to waigani [02:15] * thumper runs away [02:15] haha [02:15] thanks :/ [02:16] waigani: the gist is: the worker asks to watch volumes. it passes a scope tag, and that is either a names.MachineTag or a names.EnvironTag [02:16] waigani: if it's an EnvironTag, the apiserver will watch volumes scoped to that environ [02:16] waigani: if it's a MachineTag, it'll watch volumes scoped to that machine [02:17] axw: right, that makes sense [02:17] waigani: so, if it's an EnvironTag we want to make sure the logged-in user is an environ manager [02:17] and also that the state's EnvironTag matches the scope [02:18] waigani: my understanding of JES is limited, so that last bit may be where things are broken [02:19] axw: when you say scope, you mean the state passed in to NewStorageProvisionerAPI ? [02:20] waigani: no, I mean the tag passed in the API request, in WatchVolumes/WatchFilesystems [02:21] waigani: those methods take params.Entities, each of which is a MachineTag or EnvironTag as described above [02:25] axw: yeah but "matching the scope" is matching the EnvrionTag passed in with that of the state obj used to make the api, right? [02:25] waigani: yes [02:27] axw: so the EnvironTag case is never going to pass in a hosted env [02:27] waigani: st should be scoped to the env at that point? or not? [02:27] axw: yes, it should be [02:27] ? [02:28] so the "state.State" is *not* scoped to the env? [02:28] (hosted env) [02:34] axw: yes, it should be scoped, but I'll check. I'm just looking to see which user is isEnvironManager() checking? [02:35] waigani: AuthEnvironManager [02:36] i.e. a machine agent with JobManageEnviron [02:37] axw: so in the hosted env scenario, which machine would that be? [02:38] waigani: you're asking me how JES works? :) [02:38] it should be a state server machine? [02:38] axw: haha, okay I'm going to dig into isEnvironManager [02:39] axw: if it's not the state server machine, it's going to fail. Also, as you point out if st isn't scoped, it's going to fail [02:39] waigani: yup [02:39] axw: so let me dump out some info and we can go from there [02:40] axw: shall I remove your comment? [02:41] waigani: yes please, if you're making changes [02:41] thanks [02:41] axw: I'll take it out now, in case this turns into a PR [02:41] np [02:58] axw: it does appear that azure really wants people to stop using affinity groups which is good for us. let me know if you have time for a quick chat as there's a couple of points i'd like to clarify. no rush [03:00] wallyworld: yeah, that sounds like the case. can chat now, tanzanite standup? [03:01] sure [04:22] got it [04:29] so st is scoped to hosted env, but the tag passed in is always the server env, so they never match. jujud's ReadConfig (cmd/jujd/agent/agent.go:106) takes a string tag and uses that to determine the location of the agent's config file to read. [04:31] the string tag is "machine-0" [04:31] (which can no longer be assumed to be the state server, as the first machine in a hosted env is also machine-0) [04:37] agent.ConfigPath returns the agent.conf of machine-0 the state server, not machine-0 the first machine in hosted env [04:37] .juju/local/agents/machine-0/agent.conf [04:40] waigani: well done :D [04:43] axw: thumper: ^^ [04:45] waigani: are you saying you can have two machines called machine-0? [04:45] axw: yes [04:45] sounds crazy, but ok [04:46] waigani: so... is that a jujud bug? that it's reading the agent.conf for the wrong agent? [04:46] axw: yep [04:46] axw: so will most likely have side effects elsewhere [04:47] axw: juju environment create env1; juju deploy wordress [04:47] axw: the juju status and checkout the machine name [04:50] waigani: I'm pleased you found it [04:50] waigani: hrm, I'm a little confused. I would think that machine-0 the state server would be running env workers per hosted env? [04:50] axw: machine-0 machines are in different environments [04:51] * thumper takes the dog out while it is still light [04:51] thumper: can we talk this one over tomorrow morning? [04:51] I need to run also === urulama is now known as urulama__ [06:33] * thumper heads off to make dinner [06:33] laters [06:45] wallyworld: looks like the whole azure model has changed. the little bit I've read makes it sound a lot more like other IaaS now, and Cloud Services are no more (with the new ARM API). portal is excruciatingly slow though, so can't actually tell for sure... [06:47] wallyworld: if that is the case, we might be able to migrate away from what the either-or HA/placement implementation [07:31] axw: yeah, they are changing terminology to use Region etc as well - so i think it's going to be goof [07:31] good [07:31] wallyworld: yes, looks good [07:32] will be nice to kill some of that old hacky code [07:32] well [07:32] relegate it anyway [07:52] anyone fancy a small review of a LoggingSuite change? https://github.com/juju/testing/pull/69 [08:16] dimitern: http://reviews.vapour.ws/r/1714/ [08:16] voidspace, looking [08:16] dimitern: plus http://reviews.vapour.ws/r/1715/ [08:17] voidspace, will do [08:17] dimitern: basically the same code as landed, with minor fixes for 1.24/master plus adding the upgrade step to steps124 too [08:17] dimitern: thanks [08:17] voidspace, have a look at our shiny new and empty board https://canonical.leankit.com/sapphire [08:17] dimitern: oooh... [08:17] dimitern: shiny :-) [08:18] voidspace, we'll start using it by friday I hope [08:18] dimitern: cool === ashipika1 is now known as ashipika [08:35] dimitern: but then we'll soon fill it and it won't be so shiny anymore *scnr* [08:36] dimitern: hehe, and opening the parking lot show's already some cards [08:41] TheMue, :) that's right, yeag [08:41] yeah even [08:42] I'll groom the parking lot cards a bit and see what we didn't manage to finish by thursday [08:42] dimitern: now it only has to be a visual frontend to our tasks in launchpad ;) [08:58] voidspace, both reviewed [08:58] dimitern: ta [08:58] dimitern: I'll have to go and fix those on 1.23 as well [09:00] voidspace, well, at least check if with 1.23 and string ids there the test still passes, other than that it's too late for 1.23 I think [09:00] dimitern: the tests pass [09:00] dimitern: the branch landed and I ran tests for 1.24 and master [09:01] voidspace, cool, I think it's fine then [09:01] on 1.24 (current version not my branch) featuretests time out [09:01] they pass on master [09:01] odd and annoying [09:01] hmmm [09:20] dimitern: where's the right place to submit bugs against juju-core - github or launchpad? [09:21] rogpeppe3, lp I guess [09:21] dimitern: "you guess"? :) === rogpeppe3 is now known as rogpeppe [09:21] dimitern: ok, will submit against lp [09:22] rogpeppe, I guess >> it was like that originally, but I know there are some GH issues filed now [09:22] dimitern: ok. [09:23] dimitern: i'll file on lp anyway [09:23] i guess :) [09:25] dimitern: https://bugs.launchpad.net/juju-core/+bug/1456519 [09:25] Bug #1456519: apiserver/client: clientSuite.TestProvisioningScript is flaky [09:26] i guess noone else is using go1.3 or later to run juju tests [09:28] dimitern: any chance of a review of https://github.com/juju/testing/pull/69 please? [09:28] dimitern: (it's very small) [09:29] or anyone else ( voidspace? fwereade?) [09:30] rogpeppe, ship it [09:30] fwereade: ta! [09:30] rogpeppe, I'll have a look after standup [09:30] rogpeppe, ah, ok then [09:31] fwereade: it would be nice to have your LGTM on it [09:31] rogpeppe, it does? http://reviews.vapour.ws/r/1724/ [09:31] fwereade: (we need two, and an explicit signoff from -core is always good) [09:32] fwereade: ha! i'm not used to reviews.vapour.ws! [09:32] fwereade: i was looking at github only [09:32] fwereade: ta! [09:33] Bug #1456519 was opened: apiserver/client: clientSuite.TestProvisioningScript is flaky === urulama__ is now known as urulama [10:43] reviews on http://reviews.vapour.ws/r/1725/ please - it fixes bug 1442257 [10:43] Bug #1442257: lxc network.mtu setting not set consistently across hosts

[10:45] morning [10:49] voidspace, TheMue, dooferlad, ^^ [10:49] dimitern: *click* [10:50] dooferlad, ta! [11:10] dimitern: back from lunch, looking too [11:20] TheMue, cheers [11:52] wallyworld: perrito666: we're still on for the relation status discussion, right? [11:52] yep [11:53] wwitzel3: we're also supposed to meet sometime, right? Do you want to meet before the retrospective tomorrow? [11:53] yep [12:53] jam: that sounds good to me [13:03] perrito666: on the concept of VR hangouts, I'm told that having the world move for you is the really-really big producer of nausea [13:09] jam: I only used a VR headset once and I got really confused about not seeing my hands [13:09] the movement was not very annoying when sitting [13:26] jam, I was just about to schedule a planning meeting in that slot :) [13:26] alexisb: for our scheduling? [13:27] alexisb: did you just cancel the interlock? [13:27] perrito666, yes I forgot to do it yesterday [13:27] * perrito666 notices google calendar loves a refresh after a couple of weeks up [13:27] cloudbase is at ODS this week [13:27] ah true [13:28] jam, yes [13:28] alexisb: sure. if you get it scheduled first you win :). I also sent a message to cmars about if there is another time we could use [13:29] yeah the hour after the leads call is definitely my preference [13:29] but I am going to grab that weds slot just in case [13:29] sure [14:10] natefinch: standup [14:10] voidspace: weren't you the one that moved us away from "anything" on our SSL cert code? [14:26] voidspace, is this bug Fix Committed in 1.24 https://bugs.launchpad.net/juju-core/+bug/1348663 [14:26] fwereade: hey want to chat about the container stuff a bit early? we're all in moonstone? [14:26] Bug #1348663: DHCP addresses for containers should be released on teardown

[14:26] voidspace, I think your last commit to 1.24 fixes bug 1441206 [14:29] katco, just making coffee, will join you in there though if that makes sense? [14:29] Bug #1441206: Container destruction doesn't mark IP addresses as Dead

[14:29] fwereade: yeah, we're all in ther efrom the standup [14:29] fwereade: https://plus.google.com/hangouts/_/canonical.com/moonstone?authuser=1&hceid=Y2Fub25pY2FsLmNvbV9pYTY3ZTFhN2hqbTFlNnMzcjJsaWQ5bmhzNEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t.7g85fo35lqi339e7ppmrdgeoi0 [14:32] ericsnow: no, that was wwitzel3 [14:32] sinzui: it does fix a bug [14:32] sinzui: let me confirm === kadams54 is now known as kadams54-away [14:50] sinzui: I've marked 1441206 as fix committed for 1.24, still trying to merge into master [14:51] voidspace, thank you [15:01] jw4: you around? [15:11] command line tool to open hangouts at your next meeting [15:11] https://t.co/gNRi41ELLk [15:28] natefinch: yeah - I'm OTP for the next 30 minutes or so, but I'm somewhat around [15:28] jw4: quick question - am I correct in my observation that action results are always converted to strings? [15:29] natefinch: hmm; that doesn't seem quite right to me [15:29] jw4: for example, I set a value of 16, but it gets converted into "16" [15:29] natefinch: I don't know for certain, but I *thought* result types would be preserved [15:29] natefinch: bodie_ worked on that part so he may remember more accurately [15:30] natefinch: I'll investigate too when I get a chance [15:30] jw4: ok no worries. Could well be my fault, too. [15:36] heh, interesting, from python's time formatting, about parsing seconds: "The range really is 0 to 61; this accounts for leap seconds and the (very rare) double leap seconds." [15:37] reviews on http://reviews.vapour.ws/r/1726/ are most welcome [15:37] dimitern: *click* [15:46] Bug #1454697 was opened: jujud leaking file handles

[15:49] katco, you around? [15:49] alexisb: yep [15:50] one thing I have to give it to python for, it is very googable for common questions [15:53] tvansteenburgh: Thanks for the merge. === kadams54 is now known as kadams54-away [16:14] natefinch: it looks like all the result values are converted to strings... [16:14] natefinch: the tests seem to explicitly expect that [16:15] boo! [16:15] why even use yaml if you can't use nice types? :/ [16:16] teh suk [16:17] well, at least I'm not crazy [16:18] natefinch: correction "at least this situation doesn't demonstrate whether you're crazy" - you may still be but not as shown by this situation :) [16:18] jw4: I was going to make that joke, but couldn't figure pout a way to word it. [16:18] natefinch: well, I didn't do too well either [16:18] :) [16:19] https://bugs.launchpad.net/juju-core/+bug/1456703 [16:19] Bug #1456703: action-set converts everything to strings

[16:19] boom [16:19] :) [16:19] at least it's recorded now [16:28] Bug #1456703 was opened: action-set converts everything to strings

[16:28] way to be on the ball there, mup [16:31] haha === kadams54-away is now known as kadams54 [16:34] * natefinch really needs pyformat [16:36] python peeps... is there an official or semi official pyformat? [16:37] I see pyformat, I see autopep8 ... what should I use and do I have to worry that it'll blow up my code? [16:38] natefinch: nothing official [16:38] natefinch: autopep8 is the closest [16:38] good enough for me [16:46] natefinch: run flake8 and that's the basics [16:47] natefinch: it's what we've been using + a few manual ideas copied from LP days for all our python stuff === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 [17:01] Bug #1456714 was opened: assignCleanSuite.TearDownTest fails

=== natefinch is now known as natefinch-afk === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 === urulama is now known as urulama__ === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 === kadams54 is now known as kadams54-away [21:55] wallyworld: hey got a few minutes to talk about a spec? [21:55] katco: yeah, give me a minute [21:55] wallyworld: ack [21:56] wallyworld: doh, nevermind the family just got home. i'll catch up with you later [21:56] ok, ping me , i ave a meeting in an hour but free after that [22:12] cmars: around?\ [22:12] thumper, hey [22:12] thumper, yeah, hang on [22:12] cmars: I'm in our hangout [22:13] wwitzel3: Hi! Just wondering if the reason you switched the status of bug 1453644 was related to the bug itself, or just simply you've got other priorities you need to switch to etc. [22:13] Bug #1453644: destroy-env leaves one lxc which remains pending on redeploy

[22:17] thumper: git bisect success [22:17] machine handed off to rsc [22:17] i'm off for a coffee [22:17] \o/ [22:17] and the offending diff looks like a plausible candidate this time [22:18] not a change to an unrelaed .h file [22:46] davecheney: \o/ [23:00] axw: i can't get the meeting link to work, can you? [23:00] wallyworld: nope [23:07] wallyworld, do you have a moment to review http://reviews.vapour.ws/r/1729/ [23:07] sure === noodles775 is now known as michaeln === michaeln is now known as mnelson === mnelson is now known as noodles775 [23:10] wallyworld, 386 unit tests might fail for 1.24. The failures for the first 2 tries are flaky, so I am declaring build 2664 releasable for beta4. [23:12] \o/ [23:15] yay [23:15] wallyworld: I may be slightly late to the 1.23 retrospective, but I will be there [23:15] ok === noodles775 is now known as miken [23:59] mwhudson: thanks for breaking the seal on the 'compiler is slow argument, let's no gc' [23:59] i was thinking of writing the same to russ privately [23:59] but public is better for everyone's karma