[00:00] wallyworld: thanks [00:02] menn0: actually, i don't have credentials for jujubot, i'll ask the qa guys [00:05] wallyworld: I would have thought you could do it with your own account? === kadams54 is now known as kadams54-away [00:09] menn0: that's don [00:09] e [00:10] menn0: yeah, i could, i didn't realise i could :-) [00:14] wallyworld: cheers :) [00:14] np [00:19] menn0: fyi, i'm working on the status 2.0 spec based on notes from the sprint; still very much wip https://docs.google.com/a/canonical.com/document/d/19ljAmIe3wVC-jpDRFl8KfvJKQzx0HIC0yuPu3cCMh7Q [00:23] wallyworld: having a quick look === kadams54-away is now known as kadams54 [00:28] wallyworld: looking good so far [00:28] wallyworld: I made a few tiny typo corrections [00:29] menn0: oh ty, didn't mean to burden you just yet with corrections :-) [00:29] wallyworld: I couldn't help myself :) [00:29] :-) [01:11] wallyworld, Do you have minute to review http://reviews.vapour.ws/r/1497/ [01:12] sure [01:12] sinzui: you always give me the difficult ones [01:13] wallyworld, its an opportunity for you to say STOP, I have another issue that needs fixing [01:14] sinzui: i left off the :-) [01:14] :) [01:25] wallyworld, http://reports.vapour.ws/releases/2570 shows that 1.23.2 Passed every test, even against adverse dirty substrates and flakey vivid [01:26] yay [01:26] This is could be the highest score ever gotten [01:26] now all we need is for 1.24 to do the same [01:28] let's hope it's reproducable then [01:41] wallyworld: reviewed your branch [01:41] ty === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 [01:47] axw: with the client facade, william didn't want to bump that beyond 0. the only other alternative is to introduce a whole new facade and stick the method on that. I started going down that path but couldn't decide on a suitable facade so took the path of least resistence (the branch was already big enough and i was trying to get it done in time before branching). given we already have AddMachinesV2, any proper refactoring work would [01:47] have to sort out that stuff also, which is getting beyond the scope of removing a storage feature flag [01:48] wallyworld: why does he not want to bump the version? [01:50] gets messy - we would need to fire up api servers for both 0 and the new version, and we want to ultimately move stuff off client anyway [01:50] we support uniter v 0, 1, 2, but that's a much smaller facade [01:50] i was trying to avoid depending on a charm repo change with the patching but i guess that's unavoidable :-( [01:51] and even with the uniter stuff, the tests are incomplete as it's hard to write common tests to cover all facade versions [01:52] s/hard/tedious due to code cut and paste [01:55] wallyworld: can we at least hide this ugliness in api/client? so if the caller of AddMachines specifies storage, it calls AddMachinesV3 instead of V2 [02:25] wallyworld: not sure ify ou saw this before [02:25] wallyworld: can we at least hide this ugliness in api/client? so if the caller of AddMachines specifies storage, it calls AddMachinesV3 instead of V2 === kadams54 is now known as kadams54-away [03:10] axw: here's a patch to the charm repo to allow juju to patch the NewCharmStore from any package https://github.com/juju/charm/pull/123 [03:11] wallyworld: ?! why? [03:11] where is this needed? [03:12] axw: to allow core tests to run - stuff in the apiserver/service package calls into methods in apiserver/client and the apiserver/client code calls NewCharmRepo and this needs to be patched. the apiserver/client package patches NewCharmRepo but you objected to tha being exposed to apiserver/service package [03:13] wallyworld: I object to patching across packages in general, not that one case [03:14] * axw looks at code in question [03:14] so it's easier to just patch at the source of the charm repo creation instead of indirectly [03:16] axw: sadly we already patch charmrepo.CacheDir [03:16] so i'm at least lining up with what's done already [03:17] yeah, I want to stop the sadness :) I'm looking if there's a better alternative.. [03:17] and the client code patches across packages anyway to patch charmrepo.NewCharmRepo [03:17] longer term, agreed. but this is simply to remove a feature flag to get 1.24 out [03:18] it's not introducing new badness [03:18] wallyworld: service doesn't call into client, client calls into service [03:18] service calls s.APIState.Client().AddCharmWithAuthorization(curl, nil) [03:18] client previously *internally* patched that function, which isn't great but at least doesn't spread the hack around [03:18] to set up a test [03:18] * axw looks again [03:19] I see [03:19] yes it did, but any code can also patch cahrmrepo.CahedDir [03:20] this is a consequence of starting to move service apis off the client facade [03:20] and onto their own Service facade [03:20] there's a whole bunch of other service apis with todos to move [03:21] sadly, it we designed our code using inversion of control, this wouldn't be an issue, but we didn't :-( [03:24] wallyworld: this can be fixed by changing apiserver/service to operate on interfaces rather than *state.State. getting out of scope, so please ditch the juju/charm change and use the original patch change and add a TODO(wallyworld) [03:25] if we're adding hacks, let's at least confine them to juju/juju [03:27] axw: wallyworld: on a different note, here is the start of dynamic add - http://reviews.vapour.ws/r/1498/ [03:49] anastasiamac: reviewing now [03:49] axw: tyvm :D [03:55] anastasiamac: done [04:15] axw: so, just to confirm you happy to retain the x-package patching as per what i submitted so that we don't have to modify charm.v5? that was my original thought and rationale also [04:22] axw: FFS, disconnected again, may have missed your reply [04:48] wallyworld: sorry, was afk. yes. [04:49] wallyworld: provided it gets replaced later with an interface and the cross-package patching removed [04:51] axw: yeah, there's a bit of work to do there. won't happen for 1.24 is suspect, best we can do is 1.25. i've updated the PR on RB [04:51] looking [05:28] axw: i guess i should add a MachineManager facade and stuck the new AddMachine API on that. i was previously trying to avoid missing the 1.24 branching but given that's already happened, might be best not to introduce another lagacy API to have to support [05:29] wallyworld: that would be ideal [05:30] yeah, agreed. sigh. branch already soo bg [05:31] wallyworld: I already gave a shipit, you don't have to do it immediately [05:31] axw: ok, i'll land but then do an immediate followup [05:31] cool [05:39] wallyworld: is that a straight backport? do I need to review again? [05:43] Bug #1449367 was opened: remove storage feature flag

[05:49] Bug #1449367 changed: remove storage feature flag

[05:55] Bug #1449367 was opened: remove storage feature flag

=== rawang is now known as Guest64369 [07:07] Bug #1449390 was opened: storage: charms must wait for storage to be attached before running "install" hook

=== rogpeppe2 is now known as rogpeppe [09:02] voidspace: sorry for not being in hangout, have to help daughter [09:03] TheMue: no problem [09:03] TheMue: it's just you and me anyway :-) [09:03] TheMue: I'm working on forward porting the addressable container feature flag to master [09:03] TheMue: nearly done [09:03] TheMue: hard work - just about none of the patch applied cleanly [09:03] TheMue: only two more files to do though [09:03] TheMue: that's my report :-) [09:13] Bug #1449436 was opened: Environment variables are not propagated to jujud on vivid

[09:14] voidspace: so, "first aid" done, daughter has troubles with her car on the highway and asked what to do. and the other daughter is ill at home. :( [09:15] voidspace: I'm currently writing a little conference report about last week, the continue with the process changes with Katherine, and will then take a look in the list Dimiter sent [09:22] TheMue: ok [09:25] Bug #1449436 changed: Environment variables are not propagated to jujud on vivid

[09:31] Bug #1449436 was opened: Environment variables are not propagated to jujud on vivid

[10:19] jam: gotta go, fix up for review here FYI -- http://reviews.vapour.ws/r/1500/ [10:19] axw: have a good night [10:19] axw: I believe this should also go into 1.23, right? [10:19] Since that is what we're actually releasing on V [10:21] lads, when I bootstrap my env and then do 'juju ensure-availability', juju will fire up additional three units, set up replicaset for mongodb and all of that. Now when unit 0 dies (the initial bootstrap unit) I can't connect to my env any more. I need to manually change my environment.jenv file and remove references to machine 0 from there. Then juju works ok again. Now I want to remove machine 0 from juju, as it no longer exists, but juju won't let me [10:21] do so, saying: "ERROR no machines were destroyed: machine 0 is required by the environment". [10:21] Is this by design, or am I doing something wrong? [10:21] Using juju 1.22 currently, but observed same behaviour on 1.23 [10:22] hooooly shit. just went to write a fake object to fulfill the CloudConfig interface, and umm.. it has over 60 functions :/ [10:29] that's no interface .... mother of god! [10:29] Mmike: machine 0 is special, it cannot be removed [10:30] luckily the function I need to mock it for only uses like 6 of those functions, so I can replace the interface with a more narrow one, but still, jeezus. [10:31] davecheney, Mmike: you definitely should be able to kill machine 0 and have your environment still work fine. [10:31] (when in HA) [10:32] I don't know about actually tell juju to remove the phantom machine from its DB, but your environment should definitely still work. [10:33] natefinch: seems a bit oversized. Though you can do ttype [10:33] type MyFake struct { CloudConfig } [10:33] and then only override the ones you actually want to [10:33] (you'll get nil pointer dereference failures for anything that gets used that you didn't define) [10:33] Mmike: so I'd like to walk through it a bit with you. [10:33] After "juju ensure-availability" do you end up with 3 or 4 total machines? [10:33] (I would expect 3) [10:33] second, after those machines are up and running, you should be able to run "juju status", and connect to that env [10:33] and then it will notice that there are more machines [10:33] and record them as alternatives in your environments.jen [10:33] It won't actually record new machines until they are up and running and connected (AIUI) [10:33] but you shouldn't have to edit your JENV normally [10:33] just you have to connect to the environment once the extra machines are running. [10:33] Mmike: third, after machine-0 has died, you probably need to run "juju ensure-availability" again, where it should now notice that you have a dead machine, and try to take it out of official API server status [10:33] (this is a bit of a wart, and we're looking at splitting up ensure-ha into more precise commands) [10:33] and then you can "juju destroy-machine machine-0" (sp?) [10:33] once it has lost its voting status [10:34] jam: i see [10:34] jam: let me try that [10:35] jam: good idea embedding the interface in the struct === natefinch is now known as natefinch-afk [10:50] morning all [10:55] jam: where can I find the password jujud is using to connect to mongod? [11:03] Mmike: you have to look on machine-0 for it, it should be in /var/lib/juju/agents/machine-0/agent.conf I think [11:03] Mmike: sorry, looks like "/var/lib/juju/agents/machine-0.conf" [11:05] jam: ack, thnx [11:05] nope /var/lib/juju/agents/machine-0/agent.conf [11:05] jam: unlest it changed [11:05] perrito666: that's what I originally thought, but our backups test disagree [11:06] perrito666: ./state/backups/files_test.go:114: filepath.Join(s.root, "/var/lib/juju/agents/machine-0.conf"), [11:06] jam: actually on 1.22 it's in /var/lib/juju/agents/machine-0/agent.conf [11:06] just verified [11:06] jam: ill remember to ping eric on that one [11:07] perrito666: k. That sounds suspiciously like its a bad test. [11:08] perrito666: but "cmds/juju-restore" does use /var/lib/juju/agents/machine-0/agent.conf [11:08] jam: it is [11:08] jam: looking at the code that it is testing it does not really matter since that seems the result of faking a glob result [11:08] but is missleading [11:09] perrito666: yeah, I'm not saying the test doesn't pass, but we shouldn't have incorrect paths in a backup test :) [11:09] I have opened and restored that files enought times to know it is named agent.conf :p [11:09] * perrito666 makes a note for a less urgent time [11:10] perrito666: :) [11:10] yeah, I would guess you did [11:17] voidspace, ping [11:34] Mmike: did you get things to work? [11:34] jam: nope, not yet, my openstack deploy is a bit slow as I'm testing some other stuff too, so it takes time... [11:34] Mmike: np [11:35] just curious [11:35] jam: yup, have a case about that, so I need to know how it works [11:35] and for convenience too :) [11:37] fwereade, http://reviews.vapour.ws/r/1501/ [11:38] dimitern: is it actually allowed to start with - or _ [11:38] ? [11:38] jam, it is in fact [11:39] jam, rvba just confirmed that [11:40] dimitern: is there any reason not to be slightly more restrictive than them? [11:40] I think avoiding opening "-" is probably a good thing [11:40] jam, apart from another critical blocker from OIL [11:40] can't think of one :) [11:41] in case they now decide to have names like "-my_NET" [11:41] dimitern, jam: so the core of the problem is that we use provider ids as network names [11:42] dimitern, jam, which means we need to come up with a regexp that matches every possible valid provider network id [11:42] fwereade, jam, yeah, that's the crux of it; but it's much harder to fix it properly [11:42] dimitern, jam, but is still somehow useful [11:42] fwereade, jam, also, fwiw this only applies to maas anyway now [11:44] dimitern, jam: with that patch, can we still transform name<->tag safely? [11:44] dimitern, jam: and can we depend on correctly munging globallKeys? [11:44] fwereade, jam, yes - there are tests for that in fact, which didn't stop passing, and we don't allow / [11:44] or # right? [11:44] yeah [12:04] jam: so, here is what I did: juju bootstrap; juju ensure-availability. Then I did 'juju status' until I had all three machines with "state-server-member-status: has-vote". Then I deployed some service (percona-cluster, with 3 units). [12:05] Waited for that service to settle down, verified all is ok. [12:05] After that I killed the machine 0 (as this is openstack, I did nova delete $instance_id). Did juju status after that, waited for like 2-3 minutes, and then juju status returned status with machine0 being down. [12:05] Also, some percona-cluster units where shown as 'agent-state: down'. [12:05] Mmike: 2-3 minutes sounds odd to me [12:06] Mmike: one option is to use "juju status --verbose" (-v) which should have it report what its trying to do [12:06] is it still 2-3 min per status? [12:06] or is status fast now? [12:07] Then I did juju ensure-availability, juju instantiated another machine. Waited for that one to have 'state-server-member-status: has-vote'. Then did: "juju destroy-machine 0", but juju complained about 'machine 0 being needed for the env' [12:07] no, it is fast now [12:07] just the first run after I killed machine 0 was slow. [12:08] ok, now all of my percona-cluster units are up, and marked ok (no more 'agent-state: down'). [12:08] But I still can't remove machine 0. [12:09] Mmike: can you pastebin the "juju status" output ? [12:09] sure [12:10] jam: http://pastebin.ubuntu.com/10924694/ [12:10] jam: I can paste whole termlog from the first bootstrap, if needed [12:11] Mmike: so i *believe* that now that machine-0 has been removed from voting, you can do another "juju ensure-availability" and it will be removed as a state server entirely, and then you can "juju destroy-machine machine-0" [12:12] * Mmike tries [12:12] I think each transition needs another ensure-availability call. starts with has-vote, then goes to no-vote, then goes to no-longer-a-state-server [12:13] jam: ack, confirmed. [12:13] jam: excellent! :) [12:13] jam: thank you very much [12:13] just one more, though [12:13] so let's say I had 3 state machines. One of them died. Then another died. Now mongodb on remaining unit is in readonly state. [12:14] I don't have juju backup. [12:14] I'm thinking, then, connecting to the remaining machine, and 'reseting' mongodb - forcing it to become primary again. [12:14] Will that work? That is, will I be able to use juju afterwards? [12:14] (Havent tried, just curious) [12:15] Mmike: so if you manually poke the replicaset document in mongo [12:15] you should be able to do anything you want :) [12:15] just don't do it wrong. [12:15] obviously with db surgery you can make anything work. I'm not 100% sure how much Juju would let you get away with, because it will likely try to write the replicaset document to match its own state [12:16] well, obviously the 'right thing to do' is to have backups. [12:17] But, there being only two kinds of people in the world... sooner or later I'll have to deal with 'we WILL be doing backups' kind of people :) [12:17] jam: thnx for the inputs, will try breaking my env later again to verify if poking with mongo will bring it back to life [12:18] Mmike: have you verified that Juju becomes completely unhappy with 2 machines down? [12:18] It is *possible* that "juju ensure-availability" would rewrite things to try and get back into 3-man mode. [12:18] but I can entirely believe that we didn't get there. [12:18] And IIRC, we don't support "juju ensure-availability -n1" to get back out of HA mode [12:19] jam: nop, tbh. But I know from previous mongodb experience that if two nodes go down the last remaining node is in read-only, as it has no quorum. [12:19] Mmike: but being able to get a DB backup at that point seems *really useful* === rawang is now known as Guest89813 [13:05] * perrito666 uses deployer for the first time ever === rawang is now known as Guest7759 === natefinch-afk is now known as natefinch === kadams54 is now known as kadams54-away [14:02] natefinch: standup [14:03] natefinch: ping [14:04] aznashwan: sup? === kadams54-away is now known as kadams54 [14:13] natefinch: sorros [14:13] natefinch: got distracted [14:14] natefinch: could give me an idiot's guide to using that perl script for syscalls on windows? === urulama is now known as urulama__ [14:15] natefinch: (if you could exemplify on your npipe package what the result should be; that would be awesome) [14:50] * perrito666 drops a bucket of debug messages onto deployer [14:55] aznashwan: all I really did was look at the files in the syscall stdlib and copy what they did there. The zsyscall_* files give the command line to use, and show the input file, which contains the patterns that generate the syscalls in comments at the top of the file. [14:55] aznashwan: I meant to write a blog post about how to do it back when I wrote npipe, but never got around to it. I should brush up on it and do so, it would be pretty useful, I think. [14:59] dimitern: just FYI, I merged the tests as well [14:59] dimitern: that was easier [14:59] dimitern: I have a bunch of failing tests to fix now [15:00] dimitern: not all obvious, but I'll work through them [15:01] voidspace, sweet! [15:05] dimitern: hey, we have a test "TestNewCloudInitConfigNoFeatureFlag" [15:05] dimitern: testing maas config generation [15:05] https://github.com/juju/juju/compare/master...voidspace:addressable-featureflag [15:05] jam, are you around this week? [15:05] (the first s.SetFeatureFlags() in that branch is spurious I assume - and I've removed it locally) [15:06] dimitern: the intent of the test seems to be to test we get the same config with feature flag on and off [15:06] dimitern: but we don't [15:06] dimitern: we get the juju-br0 stuff when the flag is off [15:06] mattyw: I'm around [15:06] dimitern: so the test seems flawed (and is failing for this reason) [15:06] though its EOD for me now [15:07] dimitern: I failed to find you in person, but can I have a review please? :) [15:07] dimitern: https://github.com/go-goose/goose/pull/8/files [15:08] Bug #1449613 was opened: service/windows is missing unit tests.

[15:20] katco: can I wave bug 1447841 at you? [15:20] Bug #1447841: eu-central-1 AWS region V4 signing required and not supported

[15:24] mgz_: sure tal [15:25] mgz_: should be an easy fix [15:27] perrito666: review please? http://reviews.vapour.ws/r/1504/ [15:32] Bug #1449617 was opened: service.Service implementations are missing functional tests.

[15:33] natefinch: reviewed, just one issue [15:35] vo [15:35] mgz_: very small review for that bug: http://reviews.vapour.ws/r/1505/ [15:35] voidspace, hey [15:36] pants, we have in fact not yet switched to gopkg.in goose yet [15:37] voidspace, why is that happening can be gnarlly - check basically jujuconnsuite's setupsuite and setuptest - there might be something overriding the set FF [15:38] katco: change lgtm - no tests need updating? also, s3 auth works with the same auth object as ec2 okay? [15:38] I'm confident our testing will fail fast if anything is wrong [15:39] mgz_: yeah s3 defaults to v4, we deprecated v2 signing for s3 [15:39] mgz_: i haven't run tests, let me do that rq for just the ec2 provider package [15:40] mgz_: those tests pass. i'll try and land and see what happens [15:43] ericsnow, as always thank you! [15:44] Bug #1449633 was opened: Cannot terminate/remove broken state server after ensure-availability

[15:46] should we be sorting external import alphanum-wise? what can I use to do that? [15:47] hm, seems not, I have github.com and gopkg.in both ways round [15:49] mgz_: go fmt will alphabetize [15:50] mgz_: it sorts each section independently (where a section is delimited by blank lines) [15:55] perrito666: thanks [15:55] np [15:56] Bug #1449633 changed: Cannot terminate/remove broken state server after ensure-availability

[15:58] natefinch: yeah, I was wondering about sorting the blocks after that - which we seem not to do [16:01] mgz_: go fmt definitely sorts all the blocks... try it out here: http://play.golang.org/p/lb4Nc0Nz2B the blocks are intentionally out of order to start [16:02] Bug #1449633 was opened: Cannot terminate/remove broken state server after ensure-availability

[16:02] perrito666: hey do you need any help with 1441826? [16:03] katco: I think I just nailed it :) [16:04] perrito666: course you did... bc you're AWESOME ;p [16:04] perrito666: nice [16:04] I am writing the patch now to see if it works, but apparently its a shim missing in the multiwatcher status === kadams54 is now known as kadams54-away [16:16] ocr: can I request another review on http://reviews.vapour.ws/r/1473/ [16:17] I needed to bump goose version (and location) to add a test) [16:21] mm, that is a nasty thing to read on reviewboard [16:22] perrito666: blame Ian [16:22] ok [16:22] git blame ian [16:22] it was a nice clean change till I had to update all the imports [16:23] * perrito666 impatiently awaits for his stream to update [16:23] * perrito666 reads in github and gets a pill for the headache [16:24] perrito666: looking at each commit in turn may be more informative [16:25] mgz_: do you arrange your tests in lexicographic order? === kadams54-away is now known as kadams54 [16:29] perrito666: I arrange them in random walk order [16:31] mgz_: brb, will review upon return, I promise [16:32] perrito666: no problem [16:32] I'm off for now at least [16:34] natefinch: sorry; got carried away again... [16:34] aznashwan: did you see my prior responses? [16:35] natefinch: figured it out; it's quite handy [16:35] natefinch: yes I did [16:35] aznashwan: ahh cool. Yeah. it's nice. [16:35] natefinch: I used the go one in the syscall package [16:35] natefinch: they should really raise awareness for it [16:36] aznashwan: yeah, I remembered they added a go version finally... after I had to struggle with installing perl on windows ;) [16:36] natefinch: oh; it's yours? [16:36] natefinch: Windows salutes you :D [16:37] aznashwan: no no... I was sorely tempted to port it, but never found the time. Not sure who did, but not me :) [16:39] natefinch: works great; but there was a lot a lot of guesswork involved in figuring out how to use it for procs' that weren't in kernel32 === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 [19:10] perrito666: how is the bug coming? [19:11] katco: fixed It looks, fixing the tests now [19:11] perrito666: awesome :) [19:14] mmpf I am pretty sure network manager indicator has less functions than before :( I dont seem to be able to priorize my connections === kadams54 is now known as kadams54-away [19:59] talk about useful test failure errors [19:59] Error: entity mismatch; got len 1; want 1 [20:05] perrito666: heh wow [20:09] natefinch: hey is https://bugs.launchpad.net/juju-core/+bug/1446871 landed yet? [20:09] Bug #1446871: Unit hooks fail on windows if PATH is uppercase

[20:11] katco: I hadn't thought it was targetted at 1.24-alpha1, so I just have a PR up against master, which is blocked... but I could easily put it on 1.24 instead [20:11] katco: so, no :) [20:12] Nate looks like it's just targeted against 1.24 [20:12] natefinch: mgz was working on https://bugs.launchpad.net/juju-core/1.24/+bug/1441826 but it's past his EoD... we're blocked until that is complete [20:12] Bug #1441826: deployer and quickstart are broken in 1.24-alpha1

[20:13] natefinch: looks like you might have had some insight there, so if you are blocked, it would be a good thing to look into [20:13] katco: I can certainly look at that one, though I thought perrito666 had that one handled [20:14] natefinch: i thought he was looking at https://bugs.launchpad.net/juju-core/+bug/1441826? [20:14] Bug #1441826: deployer and quickstart are broken in 1.24-alpha1

[20:14] gah... stupid copy/paste mistake [20:14]