[00:20] wallyworld: I've got a bug fix to land. Shall I target 1.24 first and after shipit land it in master? [00:21] waigani: which bug? [00:21] wallyworld: https://bugs.launchpad.net/juju-core/1.24/+bug/1384549 [00:21] Bug #1384549: Running Juju ensure-availability twice in a row adds extra machines

[00:21] waigani: 1.24 please. master is blocked. we will look to merge 1.24 into master and get all the fixes in one go [00:22] wallyworld: okay, sounds like a plan - thanks [00:22] np, great that you have a fix for that bug [00:23] wallyworld: yeah, well see what you think... [00:23] ah, have to branch 1.24, I was working against master [00:35] thumper: HA bug fix: http://reviews.vapour.ws/r/1570 [00:59] waigani: did you ever get your virtual maas setup? [01:00] wwitzel3: yes I did! [01:00] wwitzel3: then I read a comment from nate that the bug I was working on didn't need maas to be reproduced :/ [01:01] wwitzel3: but I now have maas setup on my machine for future debugging [01:06] Bug #1451626 was opened: Erroneous Juju user data on Windows for Juju version 1.23 <1.23>

[01:37] wallyworld: hey, sorry I missed the standup. kids kept waking up last night, so I slept in [01:37] axw: np. hope everything's ok [01:38] i've been trying to fix my farking internet [01:38] wallyworld: yeah, just a bit cold and they keep rolling around and kicking their covers off... [01:38] got a new modem [01:38] oh, finally :) [01:38] well, i didn't know what the issue was , so i'm trying swapping out components :-) [01:39] axw: we'll be releasing 1.24 alpha 1 tomorrow, and then will look to merge 1.24 into master to bulk forward port the fixes [01:41] axw: o/ [01:41] anastasiamac: howdy [01:41] wallyworld: same question as waigani [01:41] I have the patch for the debug hooks issue [01:41] perrito666: great, merge into 1.24 first please [01:42] perrito666: but do it tomorrow as it's past your eod [01:42] axw: m good :D i have a branch for u to review when u settle :D [01:42] wallyworld: seems there's some issues with charm.v5? mgz proposed a revert of one of my branches which updates dependencies.tsv [01:42] axw: storage stuff is beta than coffee, u know :) [01:43] wallyworld: k just wondering if it was required before the cut [01:43] anastasiamac: sure, I'd like to try and prevent a revert of my work first though [01:43] nah, that bug has been there for a while [01:43] axw: sounds good [01:44] axw: reverts r always painfull and 1.24 takes priority over my branch (against master) anyway [01:44] axw: i looked at the revert and it looked like the commit u added for charm.v5 is more recent?.. [01:44] wallyworld: http://reviews.vapour.ws/r/1571/ [01:44] cheers people [01:45] perrito666: thanks, will look [01:45] anastasiamac: commits were added to v6-unstable, and dependencies.tsv apparently had that rev listed. so it says v5, but the commit wasn't on even in that branch [01:46] axw: what a mess :( [01:46] indeed [01:50] axw: perrito666's change looks ok but i'd like a 2nd opinion, so after your charm.v5 adventures, would be great if you could look. no rush as the work will be landed tomorrow anyway [01:53] menn0: with that upgrade bug, i really think juju should by default be changed to upgarde to latest stable. this whole only 1 version at a time thing is a relic of when we didn't do upgrades very well [01:54] wallyworld: I agree [01:55] i think there's a 1.24 bug related to that [01:55] or at least a similar issue [01:55] wallyworld: related to what? [01:55] wallyworld: https://github.com/juju/charm/pull/126 <- PTAL [01:56] an upgrade selected a surprising version [01:56] wallyworld: right [01:56] menn0: so doing upgrades right will sort of fix that implicitly [01:56] axw: looking [01:56] wallyworld: if we are going to support big upgrade jumps then we need to socialise that and have a CI test that upgrades from 1.18 to current stable [01:57] yes [01:57] wallyworld: I think i've found yet another upgrade bug going from 1.18 to 1.23 [01:57] wallyworld: just figuring it out now [01:57] best find them now before we change behaviour :-) [01:57] axw: how was the wrong rev added to dependencies.tsv? [01:57] wallyworld: fuck knows [01:58] someone was lazy [01:58] * axw shrugs [01:59] axw: i added $$merge$$, can't recall if there's a bot [01:59] will merge manually if needed [01:59] wallyworld: thanks, I think I can merge [01:59] tests all pass right? [01:59] ok [01:59] what, run tests? I suppose I should, but it's pretty trivial [02:00] all tests pass :) [02:02] \o/ [02:04] axw: so when you proposed the juju core hooks change and updated the charm.v5 revision, did you just grab the tip revision from master? [02:05] wallyworld: no. I branched off v5 and added my commit, then updated juju to use that. someone else had previously pulled a commit from v6-unstable [02:05] so when I made my change, the fixes on v6-unstable were lost [02:07] ah right [02:15] wallyworld: ok, definitely another upgrade problem [02:15] axw: looks like it was a cloud sigma change - they use a stale version of v5 [02:16] wallyworld: don't understand [02:17] they commited to dependencies.tsv and updated it with a stale charm.v5 revision ie one that was about a week old [02:17] wallyworld: I was trying to avoid laying blame, but the problem occurred on this commit https://github.com/juju/juju/commit/73b4f331085fabd5bef5188e5af193118ec573fb [02:18] wallyworld: note the change in dependencies.tsv [02:18] the commit changed to does not exist in v5 [02:18] not so much laying blame but understanding the cause so we can ensure it doesnt happen again [02:19] wallyworld: I replied to mgz's email, with a note on what to avoid doing [02:19] and explained what happened [02:19] rightio, need to check mail [02:31] waigani: can you please hold off on retrying that build for a minute [02:31] your change doesn't look quite right [02:31] axw: yeah for sure [02:31] axw: what's your concern? [02:32] waigani: I'm just reading over the code and thinking, pretty sure what was there was doing the right thing... need to go over it some more [02:33] axw: servers that were not available and wanted vote where getting demoted [02:33] axw: that's the crux of the bug [02:34] waigani: that's exactly the specified behaviour of ensure-availability [02:34] axw: with the fix servers that not available, have vote and want vote now get demoted [02:35] axw: "adding vote" == want vote + don't have vote [02:35] axw: we shouldn't be demoting servers that are getting a vote added to them [02:36] waigani: why not? [02:36] the machine is unavailable, so why would we not demote it? [02:36] that prevents another machine from becoming a state server [02:40] waigani: I suspect what's happened in the bug scenario is that the machine's pinger hasn't started yet [02:40] so it looks like the machine isn't available, but it kinda-sorta really is [02:40] axw: I've manually tested this on aws, I ran ensure-availability, turned off an instance, reran ensure-availability, dead server got demoted an new one got added [02:41] waigani: yes... that is what is meant to happen [02:41] axw: and it still does [02:44] waigani: how long did you wait? if you wait long enough, that state server should lose its "has vote" status [02:45] waigani: i.e. because the peer grouper has noticed that the mongo has stopped responding [02:45] waigani: and once that happens, the machine will be "maintained" forever [02:47] axw: I've just destroyed the environment, but let me do the same again and we can poke around [02:48] waigani: thanks. I think you should be able to see from the peergrouper logs which machines have/haven't got a vote [02:48] oh you already said it's in the status, never mind [02:49] axw: yep. this is from last run: http://pastebin.ubuntu.com/10987944/ [02:50] axw: I'm just bootstrapping again now, I'll run HA, turn off an instance, run HA again. Then I'll ping you. [02:50] waigani: thanks [03:00] wallyworld: can you please close https://github.com/juju/juju/pull/2209? [03:00] axw: yep, sure [03:01] wallyworld: ehm, hmm. on 1.24, I get "2015-05-05 03:01:02 WARNING juju.cmd.envcmd environmentcommand.go:253 invalid JUJU_CLI_VERSION value:" [03:01] I guess something is not checking for "" [03:01] I guess something is not checking for "" [03:01] oops [03:02] axw: hmmm, was supposed to check [03:02] i'll double check and fix [03:03] axw: ah, the warning is printed unnecessarily [03:03] ffs [03:07] axw: here is where the env is up to: http://pastebin.ubuntu.com/10987993/ [03:07] axw: stopped machine is being demoted, new machine is getting the vote [03:07] waigani: it'll do that as long as the machine "has-vote" [03:07] waigani: I think it should lose the vote after a long enough time of mongo not being running [03:10] axw: are you saying, if I stop an instance and don't run HA straight away, but wait, that instance will loose "has-vote" and then the logic is screwed? [03:10] waigani: right (assuming has-vote does get lost) [03:11] axw: how about l stop another instance and let it sit for the afternoon - checking in now and then [03:12] waigani: I'd appreciate it. I'm looking at the code now, but do need to get back to other things soon [03:12] axw: I'll also read through the code and see if I can spot anywhere that removes it [03:12] waigani: worker/peergrouper is the place to look [03:12] axw: ha, I was just about to ask - thanks [03:13] axw: on an interesting aside, I used destroy-environment --force on last env, but it didn't terminate the stopped instance - it's still there in aws [03:14] waigani: hrm :/ [03:15] axw: yeah. Other instances are marked as terminated. But that one is still there as "stopped". [03:15] I'm guessing that we're filtering for only running instances [03:18] axw: a reasonable guess. [03:22] waigani: okay, here's a scenario where it would (I think) occur: ensure-availability, then immediately stop jujud and mongod on the new state server [03:22] i.e. regardless of has-vote being lost [03:22] (I think has-vote will never be lost unless another eligible state server comes along) [03:24] axw: I currently have two stopped state server machines: one with vote, one without. How about I spin up the one without? [03:24] axw: then we'd have another eligible state server right? [03:24] waigani: sure. I'll try this in my own env too === beisner- is now known as beisner [03:36] axw: is this the scenario we need: http://pastebin.ubuntu.com/10988070/ [03:37] axw: machine 2 is started and has no vote. machine 3 is stopped and has vote [03:38] waigani: that's *a* scenario, yeah. what happens with ensure-availability once all of the agents are running/available? [03:40] axw: yikes, I just noticed that 0 and 1 are down - they shouldn't be? [03:41] axw: okay they are up now, I don't understand why they went down? [03:41] waigani: so anyway, I'm going to have to NOT LGTM this. with your change, if a state server machine never gets provisioned, then it'll never get replaced [03:42] waigani: there's a small window as machines and units start where they are marked as down - the default presence value is 0 [03:43] this is all messed up but that's how it was written [03:43] axw: yeah, but 0 and 1 were started, I just stopped and started 2. Anyway, focusing on the bug at hand [03:44] waigani: sorry, I don't know why those agents are going up and down [03:44] axw: so they are up now. the stopped server still has no vote. [03:44] waigani: I don't think there is a simple fix here. We need to improve agent presence, as wallyworld mentions [03:46] s/improve/burn in hell [03:46] it is horrid [03:46] hehe [03:46] wallyworld: not landing stuff on a blocked branch are you? [03:46] axw: how could a state server get the vote without being provisioned? [03:47] thumper: yes i am [03:47] thumper: i talked to curtis a few days ago and we agreed onecould be prmatic [03:47] and this ia a small fix to a bug in a previous landing [03:48] waigani: it doesn't ever need to have gotten a vote. [03:48] axw: sorry, got my logic backwards - it needs the vote to get demoted [03:48] right [03:48] axw: once you finish your conversation, do you have time for a quick chat about storage with me and anastasia? [03:49] axw: so we can't tell the difference between a state server that is un-provisioned because of an error and one that is in the process of being provisioned [03:49] waigani: or one where provisioning failed but can be retried, or a provisioned one where the agents's dead, or mongo's dead, or ... [03:50] wallyworld: sure, tanzanite hangout? [03:50] yep [03:50] anastasiamac: ^^^ [03:50] wallyworld: m here :D [03:52] wallyworld: can u hear us? [03:52] no, hangout is hanging [03:52] ironic [03:53] ffs [03:59] axw: what if we use a wait and retry policy? [03:59] axw: after n retries and x time, it gets demoted? [04:05] waigani: IMO that should be part of the "availability" [04:05] which is more general than HA [04:05] although HA availability *should* consider more than just agent availability [04:09] axw: but it's not (or the window is not big enough). because we are getting servers that are not available because they are in the process of being provisioned [04:09] I know it's not, I'm saying it should be [04:09] it's known that agent availability/presence is not great [05:11] wallyworld or axw: I am seeing a case of incorrect field ordering of docs in mongodb which is causing an assert to fail [05:12] oh? [05:12] wallyworld or axw: could be upgrade related. i see it when upgrading to 1.23 [05:12] example? [05:12] wallyworld or axw: it breaks address setting on machines [05:13] wallyworld: i see it happen fairly often on either the addresses and or machineaddresses fields [05:13] "machineaddresses" : [ [05:13] { [05:13] "addresstype" : "ipv4", [05:13] "networkscope" : "local-machine", [05:13] "value" : "127.0.0.1" [05:13] }, [05:13] { [05:13] "value" : "192.168.122.107", [05:13] "addresstype" : "ipv4", [05:13] "networkscope" : "local-cloud" [05:14] }, [05:14] { [05:14] "addresstype" : "ipv6", [05:14] "value" : "fe80::5054:ff:fe92:1645" [05:14] }, [05:14] { [05:14] "value" : "fe80::c8a9:ceff:fef4:18fb", [05:14] "addresstype" : "ipv6" [05:14] } [05:14] ], [05:14] note how the field ordering isn't consistent [05:14] mongo cares about field ordering when doing comparisons [05:14] so the assertion fails [05:14] there have been issues in the past where something would sort addresses when it shouldn't; order of addresses in state is meant to be maintained [05:14] axw: it's not the order of the docs that's the issue. that seems to be correct. [05:14] axw: it's the order of the fields inside the docs [05:15] hmmm [05:15] menn0: perhaps someone changed the struct definition [05:15] axw: in this case the 2nd and 4th docs matches the struct definition. the others don't [05:15] hrm :/ [05:15] wallyworld: i've checked that and it doesn't look like it [05:16] wh should mongo care about field ordering ffs? [05:16] what sort of retarted "db" it that [05:16] i'm wondering if it's related to the dict randomisation in Go 1.3 [05:16] i'm on vivid today [05:16] but mgo/bson seems to do everything right [05:16] i might stick some debugging stuff into the transaction layer and see what I can find [05:17] even with map odering, surely a map comparison doesn't care [05:17] wallyworld: mgo/txn actually makes a query to MongoDB to do the comparison [05:17] wallyworld: so it's being done at the MongoDB side, so field ordering does matter. [05:17] so are you saying if mgo gets a map{1:2, 3:4} that will be considered diferent to {3:4, 1:2} [05:18] if so, wtf [05:18] wallyworld: if you replace the word "map" with "document" then yes [05:18] why? [05:18] that just makes no sense to me [05:18] wallyworld: beats me but that's what it does [05:18] ffs [05:18] i hate mongo even more now [05:19] http://devblog.me/wtf-mongo [05:19] wallyworld: ^^^ first item [05:19] great link title [05:21] menn0: so the link offers a work around [05:21] alter the find syntax slightly [05:21] wallyworld: that doesn't really work for this code. [05:21] mwhudson: https://go-review.googlesource.com/#/c/9526 [05:21] wallyworld: it's asserting that a long list of network address structs matches [05:21] if you have time [05:21] a bit messy [05:22] but it gets the job done [05:22] wallyworld: I guess you *could* translate it [05:22] menn0: we *may* have to :-( [05:23] wallyworld: but first I'd like to understand what's going on because as far as I can see we only write using our structs which should preserve order (i've been trying to break mgo/bson and so far it's doing the right thing) [05:23] menn0: are you sure the struct field order is that same for 1.18 vs 1.23 etc? [05:24] wallyworld: fairly sure... I've been comparing both codes bases [05:24] wallyworld: i'll do some more digging [05:24] wallyworld: this is pretty frustrating [05:24] indeed :-( [05:41] wallyworld: well fuck, I see the problem and this is big [05:41] oh no [05:41] wallyworld: just about all the env UUID migrations use one helper function [05:42] wallyworld: and that helper loads the doc to be migration into a bson.M, modifies the _id, adds an env-uuid field and writes it back out [05:42] wallyworld: b/c it's loading it into a bson.M which is just a map (and in this case a map of maps) [05:42] wallyworld: and b/c it's Go 1.3 [05:43] wallyworld: the field orderings get messed up [05:43] oh dear [05:43] needs to be into a slice [05:43] wallyworld: this will be a problem where-ever we have a Go 1.3+ compiled Juju and nested docs or arrays of docs [05:44] luckily our releases are all 1.2 at present [05:44] wallyworld: I'm pretty sure that the releases for vivid are being compiled on 1.3 [05:44] wallyworld: at least I thought I saw sinzui say that [05:44] hmmm, could be [05:44] shit [05:45] I can fix the migration helper [05:45] luckily i feel most vivid usages of juju are to play with openstack kilo [05:45] new environments [05:46] but still [05:46] i'll create a new ticket [05:46] i think we are fine to ship aplha with this unfixed [05:47] but needs fixing for 1.24.0 [05:47] wallyworld: yeah, it's still alpha. if you upgrade an important environment to an alpha release then you deserve it. [05:47] yup [05:48] and we will add in big red letters to release notes [06:15] Bug #1451674 was opened: Broken DB field ordering when upgrading to Juju compiled with Go 1.3+

=== urulama is now known as urulama__ [07:42] fwereade, ping? === bradm_ is now known as bradm === liam_ is now known as Guest74585 [08:09] morning o/ === axw_ is now known as axw [08:14] wallyworld: when you're free, please take another look at the storage hook order review. it's changed a bit, to fix unit termination === ashipika1 is now known as ashipika [09:00] hangout, omw [09:01] voidspace: hangout! [09:01] dooferlad: omw [09:28] mgz: hiya [09:29] rogpeppe: hey [09:29] mgz: i've just landed a change to godeps that should make our life easier: https://codereview.appspot.com/230460044/ [09:29] mgz: (i wanted your review but you weren't in sight :-) ) [09:30] rogpeppe: I saw that one, thanks (I also added a hack in CI to delete stuff, your change is much nicer) [09:30] mgz: this should put paid to all those transitory dependency issues [09:30] rogpeppe: yeah, sorry, but saw it was getting looked at [09:31] mgz: if your change just deleted extraneous dependencies, then that was kinda flawed... :) [09:31] rogpeppe: indeed. relies on the build/test actually catching stuff it's missing but cares about. [09:32] mgz: well, the whole point of the test (and why the build was failing) was to catch places where we have a dependency that's not mentioned in dependencies.tsv, i think [09:32] yup [09:32] mgz: you'd've been better off just removing that check... [09:32] mgz: but anyway, now there is a Better Way :) [09:32] anyway, not using godeps is just much better [09:33] what's the alternative? [09:33] or did you mean "not using go get" ? [09:33] rogpeppe: we also needed an actually correct tarball for releasing, which means not including code we don't declare (and have done debian copyright checking on and such like) [09:34] rogpeppe: I did [09:34] using godeps, not using go get [09:36] mgz: actually deleting the extras was probably fine in fact, 'cos it would cause the build to fail if the dep was actually being used [09:36] that was the idea. [09:36] mgz: BTW I was using godeps -P 20 to fetch dependencies and it seemed to work reliably. i wonder if you might want to experiment with that to speed up turnaround time. [09:36] mgz: re https://github.com/juju/juju/pull/2209#issuecomment-99005800 -- the last windows unit test job passed, so I don't think 1.24 is blocked by the issue anymore? [09:37] axw, right, in process of opening both branches, one more forward port and some joyent junk to look at [09:38] ok then [09:41] oh, and functional-restricted-network is borked on 1.24 still... did that get a bug filed... [09:54] can I have a re-stamp on eric's fix, http://reviews.vapour.ws/r/1576 [09:56] mgz: stamped [09:56] ta [09:59] morning all [09:59] hey nate [10:01] sorry if I came off as pissy in the email, should not be tracking down bugs late at night [10:02] I thought it was fine. You're absolutely right that we shouldn't be landing branches that don't fix the blockers. To be fair, I did land a bugfix on 1.24 that was marked critical... but that one on master I had intended to let just sit there until master was unblocked. [10:15] davecheney: looks good to me [10:19] davecheney: thanks, btw [10:48] morning === lazyPower_ is now known as lazyPower [11:11] lazyPower: around? [11:12] axw: back from soccer, will have dinner and then look [11:13] wallyworld: your network seems to have settled, what did you do? [11:13] perrito666: replaced the @$*@&!^ modem [11:13] that happens [11:16] * perrito666 destroy one cheap AP per year and one modem every two [11:19] * perrito666 frowns at bugs that asume one knows things [11:31] perrito666: i am [11:35] lazyPower: https://bugs.launchpad.net/juju-core/+bug/1444861 [11:35] Bug #1444861: Juju 1.23-beta4 introduces ssh key bug when used w/ DHX

[11:36] I would appreciate a bit more detail on reproducing this, I have never used the plugin [11:36] so what provider where you using, version of the plugin, steps? [11:38] perrito666: I have a plan for that fix, I'll add to the bug [11:38] it's basically an api versioning/breakage issue [11:40] mgz: tx [11:41] mgz: do you know what time sinzui starts his day? I am waiting for his cut to merge a minor bug in 1.24 [11:41] perrito666: shortly [11:41] perrito666: I'm trying to open both master and 1.24 currently, master should be clear shortly [11:42] perrito666: launchpad is rejecting edits to the bug - is adding it as a comment fine? [11:43] lazyPower: it is [11:43] Updated as comment #2 [11:45] tx [11:58] axw: reviewed with a test suggestion [12:06] mgz: should I wait for your input on the bug or go on and try to fix it? [12:06] perrito666: enarly there [12:20] perrito666: commented, yell if anyhting doesn't make sense [12:21] * perrito666 yells because life is senseless [12:21] I think you can do a 1.23 fix that unbreaks the api, but refactors enough to be useful for a new api call with a better result struct [12:22] * perrito666 just read: make a dirty hack just don t make it look as a dirty hack [12:22] ... I think it's pretty elegant really... :) [12:23] you make runSSHKeyImport do what we want, the just adapt how it's mapped to ErrorResults in ImportKey different [12:23] I havent read the comment, I was talking about what you justsaid [12:24] :P [12:39] mgz: are there any unverified bug fixes on 1.24? can I JFDI a fix for storage? [12:40] axw: just gone through, functional-restricted-network is still unhappy but all blockers done [12:40] one sec and I'll unblock, no forcing needed [12:40] mgz: sweet, thanks [12:41] axw: go ahead and $$merge$$ [12:42] mgz: thanks [12:42] I'll file something about the network test, may not be juju breakage but something needs fixing [12:50] meh, I think 1.24 bootstrapping may actually be doing something wrong on limited networks [12:54] mgz: if I cancel the current github-merge-juju job on Jenkins, will it clean up the instance and so on? [12:54] missed a test fix while changing a signature [12:55] axw: possibnly not, but we'll catch and clean it up later anyway [12:55] axw: beware, that might be mine [12:55] mgz: ok, thanks [12:55] so perhaps you still have some room to fix :p [12:56] perrito666: it's not, promise [12:56] anh, axw won the race [12:56] axw: bummer [12:58] mgz: :( where'd that blocker come from [12:58] axw: is terminated [12:59] so yeah, our script does handle manual aborts fine [12:59] mgz: cool [13:00] menno filed it, probably doesn't need to block 1.24 or trunk at this point [13:01]