[00:05] menn0: ping, re: #1597601 this is in the process of a deploy, why can't we just catch that clear message and retry for a bit? [00:05] Bug #1597601: ERROR cannot deploy bundle: cannot deploy application: i/o timeout <2.0> [00:05] menn0: it's not clear why the user can retry when it's in our control from a bundle deployment (blocking operation atm) and know the issue? but they can? [00:12] thumper: k reviewing [01:02] thumper: I have a few comments [01:02] perrito666: ok [01:02] dont hate me [01:03] blame the checklist [01:03] k ppl, dinner, bbl [01:07] rick_h_: it goes deeper than that [01:07] rick_h_: this error can happen at any time... not just during deploy [01:07] rick_h_: when mongodb does a leader election, it drops all it's connections and all our workers die [01:08] rick_h_: if an api request comes in just as the mongodb conns go down the i/o timeout happens [01:08] rick_h_: the apiserver gets taken down so we can't retry internally [01:08] rick_h_: changing that requires some pretty fundamental changes to the way our workers are arranged and how state.State works [01:09] I don't think we can get that done before 2.0 [01:10] rick_h_: oh I see what you're saying... I guess I wasn't clear enough [01:10] rick_h_: the "please retry" error would come from the apiserver and be handled by the client. the end user wouldn't have to do anything. [01:11] updating the ticket [01:38] menn0: in case you were planning to change any other workers to depend on environ-tracker, I'm updating storage-provisioner to do that now [01:38] menn0: (related to my ongoing work) [01:39] axw: ok cool. I wasn't planning on doing that one but makes sense to do it while we're thinking about it [01:41] axw: that will mean that the minimal provider we inject in this other test will probably need to implement a little more, but I guess that won't be too ahrd [01:41] hard [01:42] menn0: as long as it has a "Config()" method, it'll be fine for the storage provisioner [01:42] axw: that's fine then, as long as the config it has to return isn't too onerous [01:42] menn0: AFAICR, it won't need anything at all [01:42] I mean, just a base config [01:43] I'll make sure of that before I land [01:43] axw: sounds great then. [01:43] axw: I'm working on the provisioner changes right now [01:44] * menn0 wants this done [01:44] :) [01:57] Wallyworld tx for the review [02:09] rick_h_: yea, I still can't seem to get the instance to deploy .... [02:10] rick_h_: or any instance for that matter [02:12] rick_h_: yea, the bootstrap instance deployed to the vpc I specified === natefinch-afk is now known as natefinch [02:24] Hey, just realized I hit my 3 year anniversary [02:25] (at canonical) [02:27] natefinch: congrats? :) [02:27] natefinch: sweet, congrats [02:27] by all going to sleep [02:28] perrito666: nite nite [02:28] see ya [02:28] thumper: permission to JFDI this? (test only change) https://github.com/juju/juju/pull/5908 [02:28] perrito666: good night [02:39] * thumper looks at menn0's patch [02:39] menn0: ack [02:40] thumper: thanks [02:48] menn0: http://reviews.vapour.ws/r/5349/ [02:48] menn0: and http://reviews.vapour.ws/r/5350/ [02:48] menn0: when those two have landed I have a juju branch [02:48] that needs these changes to make it work with the new loggo [02:49] 27 files changed in juju [02:49] thumper: ok, will look soon. just pushing my next thing. [02:50] menn0: np [02:51] ok... starting a stab at filesystem migration now [02:51] when I lose the will on that I may attack an intermittent failure [02:55] haha [02:59] axw: here's the provisioner change: http://reviews.vapour.ws/r/5351/ [03:00] menn0: thanks, will look shortly. if you have time, I've got the storage provisioner one up here: http://reviews.vapour.ws/r/5348/ [03:00] axw: will do [03:05] thumper: how much do you know about github.com/juju/schema? [03:07] (or anyone else for that matter). Mostly wonder if anyone will kick and scream if I unmangle the API very slightly. [03:09] schema and environschema suffer from "I wish this was python" syndrome [03:14] thumper: reviews done [03:14] menn0: ta [03:14] natefinch: a reasonable amount [03:14] natefinch: why? [03:15] axw: just an FYI but I'm pretty sure QA steps are supposed to be detailed step by step instructions [03:15] axw: alexis has pulled me up on that recently [03:16] axw: don't worry about it for this one [03:16] menn0: okey dokey, thanks [03:16] thumper: so, the Coerce method on the checker returned by FieldMap uses a lot of reflection to avoid just casting the interface to map[string]interface..... the end result really just being that it effectively works with any key that is a string or has a String() method [03:17] thumper: AFAIK, core code always passes in map[string]interface{} .... would it be kosher to strip out the reflection and just cast to map[string]interface{} inside fieldMap's coerce? [03:17] I don't know of other users [03:18] but it shouldn't hurt juju [03:18] but my question is why? [03:18] what are you doing to schema that got you looking there? [03:18] thumper: it would make the code a lot simpler [03:18] I'm just thinking priorities [03:18] that's all [03:19] thumper: I'm adding the concept of dependencies between attributes in a field map. So, like, A should only be set if B is set to value XXX. [03:19] ah... [03:19] thumper: this is to support add-cloud using the environschema to generate the add-cloud interactive prompts [03:19] is this mostly in environschema? [03:20] or the schema package itself? [03:20] but to go back to the original... [03:20] thumper: the two are pretty intertwined... but the dependency stuff needs to be in schema [03:20] I'm pretty sure that juju doesn't passing "Stringer"s into the schema checking [03:21] pretty sure, as you said, that it all goes through map[string]interface{} [03:21] thumper: it's not really important. I don't want to break anyone... just struck me as making the code obtuse for no reason. [03:23] axw: review done [03:24] menn0: thanks, yours too if you didn't see. I didn't get the name references sadly, so can't follow ;p [03:24] something about Cats I think? [03:24] axw: thanks for your review. your suggestion is spot on. I'll add a TODO. [03:24] brb [03:25] axw: I have NFI either. Just noted the high brow sounding names :) [03:25] hehe [03:25] axw: looks like cats :) [04:07] menn0: last one [04:07] http://reviews.vapour.ws/r/5353/ [04:08] * menn0 looks [04:26] Bug #1325968 changed: juju 1.18.3 - MAAS Node - cannot execute binary file: Exec format error [05:45] axw: here's the final PR for the TestHostedModelWorkers problem: http://reviews.vapour.ws/r/5354/ [05:46] menn0: cool, looking [05:52] menn0: LGTM, thanks [06:34] rick_h_: dimitern: before going, James put a fix for bug 1602054. PR is linked to the bug. Would be awesome to get into next release \o/ it was highlighted in sts call last week [06:34] Bug #1602054: juju deployed lxd containers are missing a default gateway when configured with multiple interfaces <2.0> [07:01] anastasiamac: ok, I'll have a look === frankban|afk is now known as frankban [07:24] Bug #1608818 opened: Default storage pools are not registered for hosted models [07:24] Bug #1608821 opened: "juju storage-pools --provider=..." does not work correctly [07:36] Bug #1608818 changed: Default storage pools are not registered for hosted models [07:36] Bug #1608821 changed: "juju storage-pools --provider=..." does not work correctly [07:39] Bug #1608818 opened: Default storage pools are not registered for hosted models [07:39] Bug #1608821 opened: "juju storage-pools --provider=..." does not work correctly [08:03] dimitern: tyvm \o/ === sinzui_ is now known as sinzui [08:16] reviewboard seems to be having issues - or it's just for me? [08:16] can anybody open http://reviews.vapour.ws/r/5333/ for example? [08:20] ah, it's back now [08:21] :/ gone again === cppforlife__ is now known as cppforlife_ [08:44] macgreagoir: hey, are you around? [08:57] should've checked the cal [09:12] axw: just FYI, following up from yesterday, i reported https://bugs.launchpad.net/juju-core/+bug/1608494 [09:12] Bug #1608494: state: not all public methods are tested [09:12] rogpeppe: ok, thanks [09:13] axw: there are others which seem to have test coverage only coincidentally (no dedicated tests), e.g. State.Cloud (tested only because NewModel calls it) [09:13] axw: i'm just fixing UpdateCloudCredentials now and writing a test for it [09:14] rogpeppe: great, thank you [09:15] axw: currently trying to work out how to create a named cloud [09:15] rogpeppe: to bootstrap with? [09:16] axw: so that i can have a valid cloud name to pass to UpdateCloudCredentials [09:16] axw: for the test [09:16] rogpeppe: for now you cannot add clouds, you can only refer to the cloud you bootstrapped with. in all the state tests (I think all?) that's "dummy" [09:17] axw: ah, ok, cool [10:00] axw: hmm, so the dummy provider has no attributes in its cloud credentials, and there seems to straightforward way to make more clouds [10:00] axw: so i think it's not going to be easy to test the UpdateCloudCredentials logic [10:01] s/to straightforward/no straightforward/ [10:01] axw: i guess i could write a new dummy provider and redo something like ConnSuite from scratch, but that seems... quite a bit of work [10:02] rogpeppe: I don't think you need to write a new provider, just initialize state with a different cloud definition [10:02] axw: ok, but ConnSuite has already initialized the state [10:02] rogpeppe: yeah, you would have to not use ConnSuite [10:03] rogpeppe: or... maybe change ConnSuite so that you can parameterise the cloud def [10:03] axw: yeah, not sure which way's best [10:03] rogpeppe: i.e. override before SetUpSuite [10:03] or SetUpTest or whatever [10:03] that's probably the least amount of work [10:04] axw: i guess i'd need a separate test suite for every cloud variation then [10:05] axw: or... i guess i could call SetUpTest inside each test directly [10:05] rogpeppe: ew, but yeah :) [10:06] axw: or provide some more modular functionality inside state/testing, i suppose [10:07] axw: or provide the capability to add more clouds [10:07] rogpeppe: it's probably fine to do that TBH, it's not going to harm anything [10:08] rogpeppe: and it's going to happen sooner or later [10:08] so if it's easy... [10:08] axw: :) [10:08] axw: this is all unbudgeted-for stuff [10:11] rogpeppe: gtg, thanks for fixing it up - I'm still neck high in config stuff [10:11] axw: np [10:11] axw: have fun! [10:17] reviews on http://reviews.vapour.ws/r/5356/ will be much appreciated! [10:50] dimitern: I'll trade you! ;) Take a look at these ones? http://reviews.vapour.ws/r/5312/ http://reviews.vapour.ws/r/5317/ [10:50] babbageclunk: sure ;) looking [10:54] anyone familiar with the new cloud stuff? named clouds and that sort of thing? [10:54] axw: i'm presuming you're done for the day... [11:09] anastasiamac, are you going to schedule something this week to talk about the bootstrap acceptance criteria? [11:13] dimitern: Reviewed, a couple of minor things. [11:14] babbageclunk: cheers! still looking at yours I'm afraid [11:17] oh balloons: I'd love to :D r u available? any pref days/times? [11:19] anastasiamac, it is easy for me to meet about this time. Does tomorrow work? [11:19] balloons: i'll add it to calendar! tyvm :) [11:20] balloons: bootsrap? there is add cloud.. not botstrap done :) [11:22] anastasiamac, is your github PR all up to date? [11:23] ahh right.. add-cloud: https://github.com/anastasiamac/juju/blob/acceptance-add-cloud/acceptance/cloud/add_cloud.feature [11:27] morning [11:42] babbageclunk: thanks for fixing the vsphere issue btw - LGTM [11:44] balloons: yes [11:45] balloons: at least for now an as far as I got with everyone sprinting... it needs more discussions but i believe it's at the next logical step and can be useful for u to kick things off on ur side :P [12:09] Bug #1571593 changed: lxd bootstrap fails with unhelpful 'invalid config: no addresses match' [12:17] rogpeppe: yes EOD. going away again :) [12:17] axw: np :) [12:18] axw: just wondering if clouds should be immutable [12:39] Bug #1443942 changed: SNAT for externally routed traffic should be only for EC2 and for subnets in the VPC [12:39] Bug #1540832 changed: Hard wired bridge prevents the use of the fan overlay network [12:39] Bug #1608723 changed: "juju test" command needs updating for juju 2.0 [12:39] Bug #1535891 opened: Feature request: Custom/user definable cloud-init user-data [13:03] babbageclunk: updated http://reviews.vapour.ws/r/5356/ can you have a second look please? [13:03] dimitern: yup, looking now [13:04] babbageclunk: ta! [13:06] babbageclunk: and if you can also have a look at https://github.com/juju/juju/pull/5871 at some point will be awesome! :) [13:15] Bug #1608723 opened: "juju test" command needs updating for juju 2.0 [13:15] Bug #1608938 opened: clientSuite.SetUpTest mongo could not create index [13:15] Bug #1608942 opened: UpgradeCharmSuccessSuite.SetUpTest cannot set admin password [13:19] dimitern: looking at 5871 now - why no RB for it? [13:19] babbageclunk: dunno - perhaps due to formatting of the description the RB bot didn't pick it up .. [13:30] Bug #1608947 opened: MSpan_Sweep: bad span state after sweep in github.com/juju/juju/api/provisioner === redelmann_ is now known as redelmann_wfh [13:39] Bug #1608947 changed: MSpan_Sweep: bad span state after sweep in github.com/juju/juju/api/provisioner [13:45] Bug #1608947 opened: MSpan_Sweep: bad span state after sweep in github.com/juju/juju/api/provisioner [13:46] Bug #1608952 opened: Deployer: KeyError: 'uuid' connecting to environment [13:46] Bug #1608956 opened: local charms can be deleted while still referenced [13:49] Bug #1608952 changed: Deployer: KeyError: 'uuid' connecting to environment [13:49] Bug #1608956 changed: local charms can be deleted while still referenced [13:49] morning, I am OCR so ill be reviewing stuff :) [13:55] Bug #1608952 opened: Deployer: KeyError: 'uuid' connecting to environment [13:55] Bug #1608956 opened: local charms can be deleted while still referenced === redelmann is now known as redelmann_brb [14:05] babbageclunk: review poke ? :) [14:06] dimitern: Still reading - sorry, was interrupted by call with alexisb. [14:06] nearly done [14:06] babbageclunk: no worries [14:11] dimitern: done! [14:11] babbageclunk: tyvm! [14:13] Bug #1608959 opened: deleted local charms not removed from caches [14:13] fwereade: are you up for cleaning the charm/cache problems 'the right way'? [14:14] fwereade: if you are cool with it would love to tackle those together with a central plan that removes the array of issues we're nipping at from different directions [14:14] rick_h_, I... really rather sort of am, yeah [14:14] fwereade: cool, let's break down the steps then for the kanban board and make sure to collect the array of bugs that we're targeting to make sure they fall away when the work is complete. [14:15] rick_h_, ack, will try to pull it all together [14:15] fwereade: <3 ty much [14:18] fwereade: hey, i'm not sure what to do based on your email (i.e. which "fix" is "the fix" :)). are we ok to go ahead with the fix we discussed yesterday? [14:25] katco, heh, I was sort of raising a big heap of related issues for general consideration [14:26] katco, yesterday we were saying "delete the charm data, keep the doc, see what happens"? [14:26] fwereade: yeah, it looks like our charm caching is in a bad way atm [14:26] fwereade: yeah, that's what i have proposed [14:27] katco, ah, hadn't spotted that -- I'll take a look [14:27] fwereade: http://reviews.vapour.ws/r/5344/ [14:27] fwereade: it already has a ship it, i just wanted to see where we're at with ian's input, and if you were raising additional points [14:28] katco, but, yeah, the worst consequence I can think of is the occasional failed download as a result of the related issues that are too big to drag in [14:28] fwereade: yep [14:28] katco, and that's mitigated by the evil caching anyway [14:28] katco, so, yeah, I think it's the best step forward [14:28] fwereade: ok, cool. i'll go ahead and land [14:28] fwereade: ta [14:29] katco, fix the docstring on deleteCharmarchive if you haven't triggered yet [14:29] fwereade: np i can do that [14:29] katco, ta [14:31] rick_h_: looks like master is blocked. what do i do? [14:31] katco: mark your card as blocked and put it in landing please [14:31] katco: and let's see what's up and what needs to happen to unblock master [14:32] rick_h_: apparently it's blocked on the other bug i am to be working on [14:32] katco: k, then leave the current card in landing/blocked and grab the other one please [14:32] rick_h_: so i am my own grandma and i am blocking myself [14:33] katco: :) [14:33] katco: more like someone didn't stack the bugs in order of actual todo priority [14:33] * rick_h_ hides real quick [14:33] rick_h_: oh... jujubot has accepted my merge request... odd [14:34] juju.fail is lying to me i suppose. i wish ci just had a dashboard [14:37] katco: you can checkout the python script check-blockers from juju-ci-tools and run it trivially btw ;) [14:38] dimitern: thanks, i might just start doing that. i dunno why ci can't provide that info on our dashboard though [14:39] that's a good question... not a priority I guess [14:46] dimitern: katco we'll take that into advisement as we work on the new workflows/processes [14:46] rick_h_: +1 [14:47] rick_h_: good luck. i raised that 2 years ago haha [14:53] k I am about to tackle the queue as it is, anyone needs an urgent review before that? [14:57] oh distro packages, you're so wacky. installing the golang package installs gcc and make and some perl crap... geez [14:58] Hi, I've asked a question about glance-charm in the OpenStack dev mailing list - http://lists.openstack.org/pipermail/openstack-dev/2016-August/100660.html is it a suitable place for the question or there is a better place present? [15:00] natefinch: odd, why would it install gcc? [15:01] natefinch: simply golang or golang-go ? the latter shouldn't depend on gcc [15:01] andrey-mp: hi there, I think #juju is a more suitable place but you are free to ask and if someone knows the answer will gladly tell it to you [15:01] dimitern: golang... I certainly don't know the difference between thw two [15:02] perrito666: thanks, will ask in #juju [15:02] perrito666: http://pastebin.ubuntu.com/21900049/ [15:02] natefinch: one is a metapackage, the other specifically using the gc toolchain [15:02] (IIRC) [15:02] natefinch: it most likely depends on build-essentials [15:02] ah, yes it does [15:03] natefinch: build essential is the bag of stuff that is required to build pretty much everything (in terms of tools) [15:03] dimitern: you notice rb failing earlier? [15:04] dimitern: I am getting a 500 [15:05] perrito666: evidently not actually essential ¯\_(ツ)_/¯ [15:06] natefinch: the name made sense when it was crafted [15:07] natefinch: ideally in any debian based distro you install build-essential and apt-get build-dep from whatever package you want and you can compile it by hand [15:09] mgz: sinzui abentley could any of you take a peek at the rb server? I am getting a consistent 500, I will put all my chips in lack of space [15:09] lol, but golang *doesn't* depend on bzr. Nice. [15:10] perrito666: what is rb? [15:10] reviewboard [15:10] perrito666: working for me [15:11] natefinch: you marked bug 1604474 as fixed but ci still cannot deploy windows charms. I can when we rest with juju from the first week of July [15:11] Bug #1604474: Juju 2.0-beta12 userdata execution fails on Windows [15:12] sinzui: uh, weird. Ok, I'll have to look at it again. [15:22] katco: strange, It seems to be a random failure [15:22] katco: would you try to enter this for me? http://reviews.vapour.ws/r/5355/ [15:23] perrito666: weird, that gives me a 500 [15:24] It is a django, so that must be logging errors somwehere [15:48] natefinch: heads up https://bugs.launchpad.net/juju-core/+bug/1604474 [15:48] Bug #1604474: Juju 2.0-beta12 userdata execution fails on Windows [15:49] natefinch: moved the card back to the high prio lane for after the current work/pr to see if we can understand why they're still seeing it [15:53] * rick_h_ goes out for lunchables today [16:25] Bug #1609041 opened: Race in github.com/juju/juju/cmd/modelcmd [17:07] bbl lunch [17:13] gah, gotta call an electrician... the outlet for our electric range is only feeding it 120, not 240, so I can't cook anything :/ === frankban is now known as frankban|afk [17:23] natefinch: rediscover fire [17:28] Natefinch most likely you have a breaker for one phase that you only use for that [17:36] here's a fix to UpdateCloudCredentials. a review would be much appreciated, thanks! https://github.com/juju/juju/pull/5917 [17:37] perrito666: there's a double breaker just for the stove, but toggling it doesn't help. Google says it's likely either a bad breaker or bad outlet. [17:39] Do you have a multimeter? [17:40] If so measure the input to the breaker [17:41] And make sure that you measure between two phases otherwise you will get 120 [17:43] I do have a multimeter, but I don't think I can get at the input to the breaker without taking the breaker box apart [17:43] and whether it's the outlet or the breaker - I'm not doing that myself. I could in theory do the outlet, but I doubt my wife would let me. [17:45] Geek up [19:15] * rick_h_ goes to do kid duty, back later. [20:25] so natefinch, everytime I have to bootstrap without the interactive stuff I get annoyed :) [20:25] liking the interactive bootstrap [21:01] any python enthusiasts want to help me debug installing ci tools? [21:04] katco: what's up? [21:05] * rick_h_ dusts off python stuff [21:06] rick_h_: getting an error after doing make install-deps. retrying after tweaking something... looks like it's potentially gotten past the error point [21:07] katco: k, let me know how it goes [21:08] rick_h_: yep, different error this time; timeout. looks like it's doing something with s3... [21:10] * katco is now debugging a python script to install dependencies for a CI system so she can debug an actual Go bug in Juju [21:11] * katco feels fantastic about this [21:14] i just tried re-running the test and it looks like it's doing something, so i'll just hope i don't need whatever this script is trying to download [21:17] katco: heh ok, welcome to the polyglot future :-) [21:18] rick_h_: been there for awhile :) [21:23] I either need to stop nodding on standups or buy a webcam [21:34] katco: ping [21:34] menn0: pong [21:34] katco: how's bug 1603221 going? [21:34] Bug #1603221: Charms utilizing storage fail on LXD [21:34] katco: is there a need for handoff before you finish up? [21:35] menn0: i just started to try and reproduce today; nothing to really hand off atm [21:35] katco: ok cool. [21:35] katco: is it really worthy of blocker status? [21:36] menn0: that i don't know. maybe not since it's limited to 1 feature on 1 substrate [21:36] katco: cool. I'll bring it up in the standup. [21:37] menn0: cool. i'm not sure who deems things blockers these days [21:37] katco: in this case it was alexisb [21:37] :) [21:38] menn0: ah. well maybe she can give some more info as to why [21:38] katco: that's what I'm thinking [21:40] menn0, we are on lock down [21:40] it gets decided in the release call [21:41] we are working towards a bless for this weeks release [21:42] alexisb: ok cool [23:06] I'm seeing this error repeated in my mongodb deploy log (for a ci test) Not sure what may cause it: ERROR juju.worker.dependency engine.go:539 "leadership-tracker" manifold worker returned unexpected error: leadership failure: lease manager stopped [23:33] sigh, might need another coffee. Just tried --veebers instead of --version :-\ [23:34] lol [23:34] new option \o/ [23:34] hah, I would hate to think what it did [23:36] functional test coverage? :) [23:36] heh :-) [23:44] veebers: that's not good. which version of Juju? [23:45] veebers: and do you have full logs? [23:47] menn0: I realised before it was a slightly older version (2.0-beta13-xenial-amd64) [23:48] menn0: I can put that log up somewhere if it's useful. I'm just about to try a run with beta14 if that'll be of more interest [23:51] veebers: it would be good to confirm if the problem still exists in a later version [23:51] veebers: I think fwereade_ has been doing some work to fix things in this area [23:52] menn0: ack. I'll put these current logs aside so they don't get destroyed and do a couple of runs with a newer juju binary [23:53] veebers: great, thanks [23:53] nw