[00:00] rick_h__: something weird is going on; the ec2 config schema for master is identical to 1.25, so control bucket support should also be the same. so a bug would be good [00:00] oh wait, you said rax? [00:01] how is control-bucket getting into rackspace config [00:01] huh [00:01] and there it is [00:02] looks like someone cut and paste some ec2 code into rackspace provider [00:02] rick_h__: ^^^^ so yeah, that's a bug sadly [00:04] wallyworld: ok, will file and go back to previous alpha for demo hopefully [00:05] wallyworld: is there anything I can pass it to make it less cranky in alpha2? [00:05] yeah, sorry, it's a one line fix by the looks, just a cut and paste error [00:05] rick_h__: sadly no, i don't think so [00:06] wallyworld: k, ty for checking for me [00:06] rick_h__: np, i'll follow up with QA to see how our tests can improve [00:07] rick_h__: if you need a build for demo, i can do that for you [00:20] wallyworld: [00:20] * wallyworld waits in anticipation [00:20] wallyworld: ty for the QA follow up. [00:21] haven't done it yet, but will do tomorrow when we meet :-) [00:21] wallyworld: guessing we're not trying to create-model on each provider [00:21] guess not :-( [00:21] wallyworld: yea, just got me thinking about how that slipped by [00:22] rick_h__: i have a feeling a lot of the QA scripts are pre-JES [00:59] anastasiamac: would you please review this small one? https://github.com/juju/juju/pull/4417 [01:00] axw: of course \o/ [01:03] axw: looks gr8! tyvm [01:03] anastasiamac: ta [01:20] axw: quartz could b multi-coloured too... and does not imply location (best opals are from Oz \o/) and does not start with "O" like some other teams already... [01:20] :D [01:23] anastasiamac: good points [01:28] wallyworld: axw: show-contorller cleaned up http://reviews.vapour.ws/r/3814/ [01:28] yay [01:30] anastasiamac: will look shortly [01:30] oh, I already did shipit [01:31] axw: u did but I've added some output that u might want to agree to... [01:31] anastasiamac: ok [01:31] axw: no rush, whenever u get a chance \o/ [01:33] anastasiamac: why change to using file-based store? [01:34] anastasiamac: in the tests [01:36] wallyworld: a thought just occurred to me, we could change the "restore-bootstrap -b" on its head, and add a "--restore" flag to bootstrap [01:36] wallyworld: not for now, maybe later [01:36] wallyworld: I think that would make intent a bit clearer [01:37] hmm, not a bad idea, i'm asking anastasiamac to do a one pager on this, so she can consider it [01:37] axw: easier to load in tests... [01:37] and actually clearer what m loading too... [01:38] anastasiamac: I'd rather we stop messing with the filesystem if possible. I suspect my change to jujuclienttesting (merging now) would make it clearer/easier for the in-mem case [01:39] anastasiamac: I won't block it for now, but next time could you please look at the new MemStore, see if it's easier with the new struct? [01:39] anastasiamac: ... and if not, try and make it easier, so we don't have to touch the filesystem [01:40] axw: k, m happy to wait for ur changes, rebase and updte the tests. [01:41] axw: one liner http://reviews.vapour.ws/r/3858/ [01:41] wallyworld: LGTM [01:41] ta [01:42] axw: altho, considering list-controller tests depend ont he same stuff.. i'd rather land this and then fix both command testsonce in memory store lands [01:42] anastasiamac: fine by me [01:42] axw: awesome \o/ [01:43] axw: so if u r happy with smaple output (in current tests probably esier to see), then m going to respect ur original ship-it and land [01:43] anastasiamac: just a moment please, still looking [01:44] axw: nps [01:46] anastasiamac: couple of small things, and one biggish one -- I think we shouldn't be showing passwords by default [01:46] wallyworld: ^^ [01:46] agree? [01:46] of course [01:47] axw: when u say "by default" do u mean not at all? [01:47] or do u mean have a flag? [01:47] cherylj: thanks for letting aaron know -- I've been emailing him about this. Aaron says it's a trivial change [01:47] axw: ah, ok [01:48] anastasiamac: I mean, clear out the password flag unless the user provides "--include-passwords" or something [01:48] also isnt this command only displays to users their own accounts?... we are not going out.. [01:48] anastasiamac: someone might want to get the addresses for their controller, and inadvertantly show a conference room their password [01:49] axw: ha. i'll add a flag [01:49] cherylj: pm-ed u [01:52] wallyworld: I've created a "backup/restore 2.0" card under Juju 2.0 [01:53] great ty [01:53] the board is getting out of hand :p [01:59] yes, too much work :-( [02:43] wallyworld: do you have time to review another one? https://github.com/juju/juju/pull/4419 -- ignoring the first commit, and the jenv.go stuff in the second one [02:43] (covered by other PRs) [02:43] axw: just heading out, will get to it as soon as i can [02:43] wallyworld: ok, nps [02:54] argggggghhhhhhhh. have the state tests gotten worse, or were they always this shit [02:54] axw: probably always [02:56] seems like every 2nd test run fails because "no reachable servers" or similar [02:57] :( [03:15] anyone know what generating /etc/rsyslog.d/25-juju.conf ? its got a dodgy IP in it that I need to remove, from virbr0 as well as all the juju state nodes. it only appears to be in the state nodes [03:26] thumper or waigani: this will be used to replace most of what the current statestarter worker does: http://reviews.vapour.ws/r/3860/ [03:27] * thumper leaves for waigani [03:32] Bug #1544846 changed: restore fails with could not exit restoring status

[03:34] menn0: sorry just saw your PR looking now [03:35] Bug #1544846 opened: restore fails with could not exit restoring status

[03:35] waigani: np [03:37] thumper: btw didn't spot anything weird in the diff. Though I release I merged in tip, not latest blessed master. But still, they're not seeing it at all in master. [03:38] *realize [03:38] waigani: weird [03:38] Bug #1544846 changed: restore fails with could not exit restoring status

[03:38] waigani: I have to head out shortly, so either check in with menn0 or we could go through it tomorrow [03:39] thumper: will do. cheers. [03:53] Bug #1545562 opened: Restore assumes systemd present on machines

[03:53] Bug #1545563 opened: Cannot deploy service after restoring

[03:53] Bug #1545568 opened: api/backups/restore.go has no unit tests

[03:59] Bug #1545562 changed: Restore assumes systemd present on machines

[03:59] Bug #1545563 changed: Cannot deploy service after restoring

[03:59] Bug #1545568 changed: api/backups/restore.go has no unit tests

[04:02] Bug #1545562 opened: Restore assumes systemd present on machines

[04:02] Bug #1545563 opened: Cannot deploy service after restoring

[04:02] Bug #1545568 opened: api/backups/restore.go has no unit tests

=== alexlist` is now known as alexlist [05:35] Bug #1545589 opened: Destroying a model leaves stale current-model file

[05:49] wallyworld, axw, could you guys take a look at http://reviews.vapour.ws/r/3861/ ? [05:58] cherylj: I don't understand this comment: This is necessary as [05:58] / the error type returned from a facade call is rpc.RequestError [05:58] / and we cannot use params.IsCodeUpgradeInProgress [05:58] cherylj: RequestError is an rpc.ErrorCoder, so should be usable in params.ErrCode [05:59] (which is what IsCode... functions use under the hood) [06:00] an the "code" is just a string [06:02] ah, I may have mixed up my comments. We can't use params.IsCodeUpgradeInProgress, because the code returned from the rpc call *isn't* apiserver.CodeUpgradeInProgress [06:03] cherylj: sure, looking [06:03] it's the inUpgradeError returned from the upgrading_root [06:03] it was hard to keep track of all the type mismatches [06:04] I'll update the comment [06:06] actually, maybe I'll go back to what it was before, and add in a new function into upgrading_root.go to check IsInUpgradeError [06:06] axw: how does that sound? ^^ [06:06] (and I mean, what I had it as before I committed) [06:06] cherylj: as in do the check on the servert, and translate to the code that the client was expecting? [06:06] server* [06:08] axw: that's not quite what I was thinking, although I could do that. [06:09] cherylj: can you explain more? sorry, slow. I'm not keen to have snowflakes that don't use the IsCode... functions [06:09] Yeah, I'm sorry, I'm super tired [06:09] I was going to add a func IsCodeInUpgrade().... [06:10] or I could have the existing in Upgrade check see if it matches CodeUpgradeInProgress or the upgrade_root.go inUpgradeErr [06:11] am I still not making sense? It is after midnight here. Maybe I should go to sleep and continue this tomorrow [06:12] cherylj: you may be, but I still don't understand :p [06:12] heh, ok, let me push up a proposed change [06:13] cherylj: sounds good, I comprehend code better than english [06:35] axw: sorry for delay, reviewed your earlier pr, see what you think of suggestions [06:35] axw: did you have any other concerns with the PR? [06:41] cherylj: nothing else jumped out [06:42] wallyworld: thanks, looking in a sec [06:42] ok, I'm just retesting with my change. This takes forever... [06:44] axw: later when you are free http://reviews.vapour.ws/r/3862/ [07:00] omfg why is it taking 20 minutes to do apt-get update on an AWS instance?! [07:10] wallyworld: PTAL. looking at yours now [07:14] axw: wallyworld: we don't write to models or accounts files yet, riite?.. [07:14] anastasiamac: I'm working on it [07:14] axw: \o/ [07:14] axw: so m not going crazy a little here.. :D [07:18] wallyworld: just one main concern, about the rpc change [07:18] ok [07:20] axw: with the remaining ClientStore() comment, i was really trying to look for a way not to have that method [07:20] the store is really internal to the command [07:21] if the method is merely used to see what's been set is nil, can we cater for that another way [07:24] wallyworld: each controller or model command will need to call ClientStore() [07:24] wallyworld: well maybe not all of them, but some of them do [07:24] ok [07:24] wallyworld: e.g. list-controllers will call ClientStore() to get the store and then call AllControllers() [07:25] fair enough lgtm [07:28] axw: the version != 0 thing. that appeared fundamentally broken. it totally rejected rpc calls where the version was not 0, and with all the old code removed, tests failed [07:28] i don't know a reason why we'd not allow rpc calls with version != 0 [07:32] wallyworld: version isn't used in that code below there, so I'm a bit concerned that something is passing in a version wrongly [07:32] axw: yeah, it was only used in that bogus (IMO) check [07:33] why would we return an error if version != 0 ? [07:33] I've tested the branch live with lxd, everythign seems fine [07:34] i'd have to check again, i think the test failure was on the Admin facade [07:44] wallyworld: sorry, it just doesn't look right. if an object doesn't support versioned methods, then the client shouldn't be requesting them. can you find the object & method that's being requested? [07:45] i'll check soon, just diggin into some python client stuff [07:48] Bug #1545126 changed: juju/maas do not create ptr reccords for bare metal servers with multiple networks [08:11] ugh, this upgrade in progress error is funky - the error string is put into the Message and not the Code [08:12] cherylj: omg, i hate being the clock but is it not 2am for u? [08:12] anastasiamac: it is [08:13] cherylj: wow [08:13] cherylj, that's because it's not a coded error - just like all generic errors it has no code and just message [08:14] dimitern: yeah, but I can't inspect it using errors.Cause() == upgradeErr because it's an rpc.ResultError so the types are different [08:14] so I need to use the check I have [08:14] bleh [08:15] cherylj, yeah, that's a result of not using concrete errors, unfortunately - and need to resort to strings.HasSuffix(err.Error(), ...) :. [08:15] :/ [08:15] just got a ship it from thumper, so fuck it. I'm merging and trying to get some sleep [08:16] cherylj, you should +1 [08:16] heh [08:16] thanks for taking a look at that bug, dimitern [08:16] no worries [08:17] cherylj, I hope some of my friday rant on that blocker bug helped a bit :) [08:17] cherylj: it appears that the IsCodeUpgradeInProgress only works if the original error is state.UpgradeInProgressError [08:18] cherylj: and not the one in apiserver [08:18] axw: yeah, I found that out the hard way yesterday [08:18] cherylj: ok. well, anyway, sleep :) [08:23] axw: that rpc FindMethod() call is being asked to find the "Login" method on the "Admin" facade with version = 3. That's when we fail the test. That code comes from api/state.go Login() which tried v2 then fell back to v1 and then v0.So the only reason it was all working is due to some obsolete fallback code in that client Login() method which I removed [08:24] wallyworld: so the other versions never worked? :/ [08:24] master always fell bacl to v0 [08:24] or so the code seems to suggest [08:25] there were 2 lots is params.IsCodeNotImplemented() [08:25] in that api state Login() [08:25] SERVERS Y U NO REACHABLE [08:25] i hate that error [08:25] wallyworld: sorry, I still need to convince myself [08:25] axw: open up api/state.go [08:26] see the Login() method at the top [08:26] i removed the fallbacks [08:26] wallyworld: I'm *fairly* sure I don't see three Login RPC calls each time I connect though... [08:27] need to check i guess, maybe it's the tests only affected. but the code path to the error makes sense. we are passing a version 3 and it complains when it has no right to [08:28] the method is there on the facade [08:29] wallyworld: I'm going to pull your branch and dig a bit. if I can convince myself it's ok, I'll merge it (unless you're still around) [08:29] i'll be here. i'll see what master does also [08:36] wallyworld: I think it must be tests only, because live test without that code removed works [08:36] wallyworld: at the top level we should have apiserver.anonRoot, which has its own FindMethod. I suspect some test is constructing a login facade and serving that directly [08:38] axw: but live, we should still be passing the login call to anonroot i would have thought [08:38] wallyworld: yes... testing live works. it goes to anonRoot, but anonRoot is not one of those rpcreflect things that you changed [08:39] ok, i'll dig into tests later fater dinner. but i still can see why we'd just blanket reject calls with non 0 version [08:39] can't [08:42] wallyworld, axw, fwiw apiserver/doLogin calls maintenanceInProgress which fakes up a login request with a user tag [08:42] might be related, as I discovered while looking into why upgrade/restore didn't work as expected [08:55] wallyworld: the problem is the errRoot [08:56] wallyworld: changing that to have a FindMethod which ignores the version, and using ServeFinder does the trick [08:57] wallyworld: the reason for not accepting a non-zero version is strictness: objects of that type don't understand versions, ergo the client shouldn't request versions. if they do, they're probably doing something wrong [08:57] safer to ignore versions on a case-by-case basis [09:00] jam: just rebooting to locate my headset... [09:01] frobware: k === ashipika1 is now known as ashipika === Odd_Blok1 is now known as Odd_Bloke [09:32] dimitern: frobware: dooferlad: if you get a chance http://reviews.vapour.ws/r/3848/ [09:33] voidspace, sure, looking [09:39] voidspace, reviewed [09:43] dimitern: you think the local machine will never attempt to login (your comment about that code path never being taken)? [09:43] dimitern: won't agents that use the api login as the local machine [09:44] voidspace, they'll try, but have a look in maintenanceInProgress and follow what happens when the validator return an unknown error (i.e. not UpgradeInProgress or one of the others) [09:45] voidspace, actually doLogin is the place [09:46] dimitern: hmmm... [09:47] dimitern: I see the code, however that returns a concrete error - and we see the "space discovery in progress" error [09:47] voidspace, that's why I suggested (not now, but soon) to add a concrete error for discovery in progress and check it early in doLogin [09:47] dimitern: which indicates to me that code path isn't being hit [09:48] dimitern: we hit the code path earlier which just fails - we bail out before maintenanceInProgress is called [09:48] the switch on concrete error types [09:48] dimitern: but those cases all return a restricted api - we either return full api or no api [09:49] mind you, not if the login succeeds [09:49] voidspace, yeah [09:49] ah, so for the local machine doLogin still fails [09:49] so why bother allowing local machine at all [09:49] voidspace, it fails for any machine [09:49] it will attempt to login, succeed, then double check to see if arbitrary logins work [09:49] when they don't it will bail [09:50] voidspace, if we get as far down as doCheckCreds and maintenanceInProgress [09:50] well, only if there's an error [09:50] if they logged in with local machine (e.g. the discover spaces worker which needs access to the api) [09:50] they should get that far and doCheckCreds should succeed [09:51] so it will never hit maintenanceInProgress [09:51] voidspace, nope, because of line 68 above [09:51] frobware: did I miss you somehow? [09:51] dimitern: if the login is from the local machine they will get "case nil" [09:51] dimitern: no error at all [09:51] jam: nope was still there, saw you come back and thought you disconnected [09:52] dimitern: and if the login isn't local machine it will fail completely on line 68 [09:52] dimitern: and that's all desired behaviour I think [09:52] frobware: yeah, I had to restart the connection, but now I don't seem to see you anymore [09:53] jam: I got disconnected [09:53] jam: live migration. :) [09:53] frobware: k. I'm back in the hangout [09:54] jam: trying.. [09:54] on error from checkCreds the s.validator is called again with a user tag.. what a mess :/ [09:55] dimitern: right, but checkCreds shouldn't fail should it? [09:56] axw: changes pushed [09:57] voidspace, I'd expect so [09:58] voidspace, but maybe it's deeper than that - e.g. something around empty uuid (for backwards compatibility), the MA trying v2 then v1 / v0 ? [09:58] dimitern, voidspace, frobware: may be a tiny bit late for standup - just off to get coffee. [09:58] dooferlad, np [09:59] wallyworld: reviewed [09:59] ta [09:59] gotta go take care of the kids for a while, bbl [10:06] fwereade: would you mind having a look at http://reviews.vapour.ws/r/3860/ pls? [10:06] fwereade: a bit of infrastructure for converting the state worker to work under the dependency engine [10:08] voidspace, standup? [10:09] dimitern: oops, sorry - omw [10:11] dimitern: frobware: uhm, I can't login [10:11] dimitern: frobware: I lost my phone over the weekend - I thought I had some saved codes for two factor auth in my gmail account [10:11] dimitern: frobware: but I don't - I'll have to contact IS to get them to turn off two factor temporarily [10:12] dimitern: frobware: sorry :-( [10:12] dimitern: frobware: dooferlad: my standup report - test fixes and discovery reliability landing now [10:13] voidspace, ok, np [10:13] dimitern: frobware: dooferlad: I've been working on having the maas provider (maasInstance) Addresses method add provider id [10:14] dimitern: frobware: dooferlad: that's complete but needs gomaasapi support for the spaces endpoint in the test server [10:14] dimitern: frobware: dooferlad: which is partly implemented but not usable (or tested) - so I've been working on that [10:14] when that is done I can test my branch for provider ids and land that [10:14] dimitern: frobware: dooferlad: caveat is that we need both the provider id *and* the space name [10:15] dimitern: frobware: dooferlad: so addresses out of the provider will have space name and id set - but this will be a *maas name* not a juju name for the space [10:15] dimitern: frobware: dooferlad: so we have to ensure it is converted before ever storing it in state or we will confuse ourselves a great deal [10:16] dimitern: frobware: dooferlad: *or* we add SpaceProviderName *as well*, which is annoying but might be better [10:17] or we just use provider id and don't set the provider space name, this might actually be better [10:17] but we have code that currently relies on having the name set I think [10:17] voidspace, let's sync up on this after standup [10:18] dimitern: ok [11:02] morning all [11:27] mm, odd for some operations lxd provider seems to be unregistered [12:03] Bug #1545686 opened: Multi-series not backwards compatible [12:06] Bug #1545686 changed: Multi-series not backwards compatible [12:12] Bug #1545686 opened: Multi-series not backwards compatible [12:18] Bug #1545686 changed: Multi-series not backwards compatible [12:24] Bug #1545686 opened: Multi-series not backwards compatible [12:51] wallyworld: there are assumptions in the code that the initial controller name == initial model name [12:51] wallyworld: so I've had to revert the model name back to controller name, from "admin" [12:51] damn, i thought we could work around those [12:51] wallyworld: possibly in a follow up, but it wasn't working right [12:52] sure np [12:52] i wasn't fully across how deeply ingrained the assumption was [12:53] :( [12:53] thanks for trying! [12:53] rick_h__: temporary setback, it will be fixed :-) [12:53] wallyworld: sorry about the rambling PR, the tendrils keep spreading [12:53] understood [12:53] wallyworld: is there anyone that can hand hold an outside commit? Looks like OCR is out today (domas) [12:54] wallyworld: https://github.com/juju/juju/pull/4426 is from an outsider through the mailing list trying to get the m4 types to work [12:54] wallyworld: so needs some kid gloves and "yay outside commits" love and such [12:54] rick_h__: perrito666 would love to :-) [12:54] ty :) [12:54] he's at SOD [12:54] k [12:55] rick_h__: longer term, we want to query those instance types using the new aws api [12:55] maybe for 2.1 [12:55] wallyworld: completely understand and agree [12:55] yep [12:56] marcoceppi: ^ fyi [12:56] wallyworld: at the very least, we should probably stop caring about the region and assume theyre the same relative costs across all [12:56] marcoceppi: faster to reply than me :P [12:57] axw: that is sensible, the costs are just approximate anyway [12:57] used in sorting [12:57] to select the cheapest by default [12:58] interesting. [12:59] I'm off, bonne nuit [13:06] Bug #1545704 opened: no-auto-upgrade not respected in environments.yaml

[13:08] marcoceppi: ? [13:08] marcoceppi: also fyi email on the way I need help on asap please so I can help the team out [13:08] rick_h___: I like the idea of moving aws queries out of the codebase [13:09] marcoceppi: oic, yea. I more meant that we have a reviewer nominated so we don't both bug folks on it. [13:09] marcoceppi: but yea, moving the instance types out is a long standing wishlist thing for the team heh [13:09] rick_h___: oh, cool. I just say I'm going to bug someone but never actually do it ;) [13:09] marcoceppi: :P [13:12] rick_h__: marcoceppi: until recently we haven't had the option for aws. was built into openstack from day one [13:12] Bug #1545712 opened: WARNING juju.network address.go:268 no hostPorts found in space "default"