| wallyworld_ | davecheney: in a meeting, be with you soon | 00:00 |
|---|---|---|
| davecheney | wallyworld_: done | 00:06 |
| wallyworld_ | davecheney: tyvm, will look after my meeting | 00:06 |
| thumper | sinzui: how long does ci take to go through its tests? | 00:08 |
| thumper | sinzui: I'm wondering how long before we know if ericsnow's branch fixes the problem | 00:08 |
| davecheney | aboput 18 minutes atm | 00:09 |
| thumper | davecheney: that is just to land, I'm talking about the 'ci' tag being taken off the bug so it unblocks landings | 00:09 |
| perrito666 | davecheney: I could but I dont understand what that means | 00:14 |
| davecheney | perrito666: i don't either | 00:15 |
| davecheney | is something broken ? | 00:15 |
| davecheney | i'm completely lost as to where we left our discussion | 00:15 |
| perrito666 | lol, ok, you answered a review request for a wrapper around chmod with an email asking something | 00:15 |
| perrito666 | rings a bell? | 00:16 |
| davecheney | right, | 00:16 |
| davecheney | yes, why do we need this ? | 00:16 |
| perrito666 | davecheney: so I answered that mail but in summary, chmod, even though it "runs" on windows, does not work | 00:17 |
| davecheney | ok, but why do we need to chmod files ? | 00:17 |
| === kadams54 is now known as kadams54-away | ||
| perrito666 | davecheney: yes, we do sometime ago I did a check with the cloudbase guys to see what was not working on windows workloads and that one was one of the outstanding | 00:19 |
| davecheney | ok, my question is can we solve this problem by removing the cause ? | 00:19 |
| davecheney | chmod might be busted on windows | 00:19 |
| davecheney | but if we can remove the requirement to change permissions | 00:19 |
| davecheney | then that also solves the problem | 00:20 |
| perrito666 | well its used for charms and It seems necessary | 00:20 |
| davecheney | how is it used for charms ? | 00:21 |
| davecheney | come on man, work with me here | 00:21 |
| perrito666 | davecheney: sorry I am re-reading the code as we speak, hold | 00:21 |
| === kadams54-away is now known as kadams54 | ||
| perrito666 | davecheney: I dont fully see all implications but, except for one case where it it actually there for windows, it seems that we can remove/do only for linux the chmods | 00:25 |
| davecheney | to me that sounds like a better solution | 00:26 |
| perrito666 | davecheney: Ill have a talk with fwereade tomorrow and see if he remembers why we decided to do this instead of nuke all appearances of chmod, Ill be glad to se this go (and with windows tests in place it might never come back) | 00:29 |
| davecheney | perrito666: so this is a charm hook that needs to chmod a file ? | 00:30 |
| perrito666 | davecheney: the uses of Chmod=? | 00:33 |
| davecheney | yup | 00:34 |
| davecheney | i'm still digging for the why | 00:34 |
| perrito666 | there are around 26, for what I see almost half are tests simmulating lack of permissions or similar situation, the rest are giving more permissions to certain files, there is one case in environs/config.go that corrects a possible wrong permission on a file and the rest I would have to look | 00:38 |
| perrito666 | we use a lot chmod for my taste | 00:39 |
| perrito666 | so, looking at the existing chmod most if not all are of no consequence for windows, I worry a bit about future uses, now we actually need to care for windows workloads and if only os.Chmod would panic as file.Chmod does I would not fear that it might silently crawl under our radar | 00:42 |
| perrito666 | I might be just over engineering | 00:43 |
| sinzui | thumper trunk looks to be in bad shape. Lots of tests failed | 00:44 |
| davecheney | perrito666: i'm only interested in charm hooks that use chmod | 00:44 |
| davecheney | imo there shuld be none | 00:44 |
| sinzui | thumper, I am retesting those that looks like cloud failures | 00:44 |
| thumper | sinzui: ta | 00:44 |
| sinzui | thumper, but ha-backup-restore is not fixed, the bug has mutated. I add the new error message https://bugs.launchpad.net/juju-core/+bug/1398837 | 00:49 |
| mup | Bug #1398837: cannot extract configuration from backup file: "var/lib/juju/agents/machine-0/agent.conf <backup-restore> <ci> <regression> <juju-core:In Progress by ericsnowcurrently> <https://launchpad.net/bugs/1398837> | 00:49 |
| * thumper groans | 00:49 | |
| === kadams54 is now known as kadams54-away | ||
| ericsnow | thumper, sinzui: I already have http://reviews.vapour.ws/r/573/ that should address that EOF issue | 00:50 |
| sinzui | hurray | 00:50 |
| === kadams54-away is now known as kadams54 | ||
| ericsnow | it just needs a review | 00:51 |
| thumper | ericsnow: I'm looking at it | 00:56 |
| ericsnow | thumper: thanks | 00:56 |
| thumper | ericsnow: but I don't see how your change improves anything | 00:56 |
| thumper | ericsnow: can you explain? | 00:56 |
| ericsnow | thumper: if we notice a dropped connection we reconnect and try again | 00:57 |
| thumper | ericsnow: a minor change then land it | 00:58 |
| ericsnow | thumper: k | 00:58 |
| davecheney | ericsnow: http://reviews.vapour.ws/r/573/ not lgtm, yet | 00:59 |
| ericsnow | k | 00:59 |
| davecheney | 12noon ppl | 01:00 |
| davecheney | oh | 01:00 |
| davecheney | the meeting is actually at 2pm | 01:00 |
| davecheney | ignore me | 01:00 |
| davecheney | or one pm | 01:00 |
| davecheney | calendars are hard, let's go shopping | 01:01 |
| perrito666 | davecheney: I would really like, but most imports are closed to make people buy local for christmas | 01:01 |
| perrito666 | :p | 01:01 |
| ericsnow | davecheney: I replied to your comment | 01:15 |
| ericsnow | davecheney: basically, I'm pretty sure that code should stick around | 01:15 |
| wallyworld_ | axw: standup? | 01:18 |
| ericsnow | davecheney: but I'll do it cleaner | 01:18 |
| davecheney | ok | 01:18 |
| wallyworld_ | axw: oops sorry, you're not here | 01:18 |
| thumper | davecheney: http://reviews.vapour.ws/r/531/diff/# needs a full review | 01:22 |
| davecheney | thumper: on it | 01:22 |
| thumper | cheers | 01:22 |
| thumper | anastasiamac: any ETA on your blocking branch? I don't really want to put my machine branch up for review until I have that merged in | 01:24 |
| === _sEBAs_ is now known as sebas5384 | ||
| anastasiamac | thumper: today?.. | 01:32 |
| * anastasiamac fingers xrossed | 01:32 | |
| ericsnow | davecheney: I've updated http://reviews.vapour.ws/r/573/ | 01:34 |
| * thumper crosses fingers too | 01:34 | |
| davecheney | don't forget to add your things to the agenda | 01:35 |
| davecheney | https://docs.google.com/a/canonical.com/document/d/1eeHzbtyt_4dlKQMof-vRfplMWMrClBx32k6BFI-77MI/edit | 01:35 |
| davecheney | if you don't, then I get to talk for the whole time | 01:35 |
| davecheney | and you probably want to avoid that | 01:35 |
| davecheney | ericsnow: ok | 01:36 |
| davecheney | ericsnow: review done | 01:41 |
| davecheney | i'm not happy with the specialised logic inside getBackupTargetDatabases | 01:42 |
| davecheney | the DBSession interface should describe what it needs | 01:42 |
| davecheney | if it needs a .Copy method | 01:42 |
| davecheney | then the mock needs to implement that as well | 01:42 |
| ericsnow | davecheney: it has to match mgo's Session.Copy which returns *mgo.Session, making the interface method kind of goofy | 01:43 |
| ericsnow | davecheney: but I agree with you :) | 01:44 |
| davecheney | ericsnow: perhaps this is not the right place for a mock then | 01:44 |
| davecheney | however unpleasent that is | 01:45 |
| ericsnow | davecheney: you may be right :( | 01:45 |
| ericsnow | davecheney: in the meantime for the sake of opening the landing bot... | 01:46 |
| davecheney | ericsnow: please raise a bug against the next milestone | 01:46 |
| ericsnow | davecheney: k | 01:46 |
| davecheney | I need a second reviewer for http://reviews.vapour.ws/r/573/ | 01:56 |
| davecheney | thumper: wallyworld_ menn0 ? | 01:56 |
| wallyworld_ | sure | 01:56 |
| waigani | my cal says we have the team meeting now, but no one is here? | 02:04 |
| waigani | ah, just my crappy connection | 02:05 |
| === kadams54 is now known as kadams54-away | ||
| perrito666 | thumper: I finally found someone that would do an affogato en in argentina :) I only had to go 90Km away to get it | 02:27 |
| thumper | haha | 02:28 |
| thumper | davecheney: I replied to you comments | 02:52 |
| ericsnow | wallyworld_: I'm pretty sure the dropped session (the io.EOF) is due to the replicaset functionionality of HA | 02:52 |
| thumper | davecheney: one key change with putting the feature flags into juju/utils is that the flags themselves had to be agnostic of any particular environment variable | 02:53 |
| thumper | so it can't really return a map unless we pass the key in | 02:53 |
| thumper | which we could make a helper to do... | 02:53 |
| wallyworld_ | ericsnow: ok, np. i'm not across it enough. is only one retry enough? | 02:54 |
| ericsnow | wallyworld_: I just updated the patch to retry 10 times | 02:54 |
| ericsnow | wallyworld_: but still only for io.EOF (which mgo returns when the connection drops) | 02:54 |
| thumper | davecheney: I'm beginning to think that we should have the feature flag init in each of the main blocks | 02:55 |
| thumper | davecheney: then we wouldn't need the test for osenv | 02:56 |
| wallyworld_ | ericsnow: so why not wait until replicaset is up before doing backup? | 02:56 |
| wallyworld_ | don't we do that elsewhere? | 02:56 |
| ericsnow | wallyworld_: I don't know about elsewhere, but I wouldn't mind waiting until the replicaset is up | 02:57 |
| ericsnow | wallyworld_: wouldn't that but in the CI test script though? | 02:57 |
| ericsnow | s/but/be/ | 02:57 |
| wallyworld_ | our code needs to wait | 02:58 |
| wallyworld_ | if backup invoked, it needs to ensure replicaset is up before proceeding | 02:58 |
| ericsnow | wallyworld_: k | 02:58 |
| wallyworld_ | like i think state server might do when starting | 02:58 |
| wallyworld_ | that's IMHO | 02:58 |
| wallyworld_ | but seems like the right thing to do | 02:59 |
| ericsnow | wallyworld_: okay | 02:59 |
| === kadams54-away is now known as kadams54 | ||
| ericsnow | wallyworld_: I'm not well versed in the HA stuff but what you're saying makes sense | 03:00 |
| wallyworld_ | ericsnow: nate knows all about HA etc :-) | 03:00 |
| ericsnow | wallyworld_: lovely :P | 03:00 |
| ericsnow | wallyworld_: I'm pretty sure y'all don't want to wait until Nate is back online before I get this pushed :) | 03:01 |
| natefinch | I'm here :) | 03:01 |
| natefinch | why would I not be here? :) | 03:01 |
| ericsnow | magic | 03:01 |
| wallyworld_ | ericsnow: i'd rather not alnd a bad solution | 03:01 |
| ericsnow | lol | 03:01 |
| natefinch | haha | 03:01 |
| ericsnow | natefinch: wallyworld_ is suggesting that we wait for replicaset to finish coming up before running backup | 03:02 |
| ericsnow | natefinch: how do we test for that? | 03:02 |
| ericsnow | wallyworld_: agreed | 03:02 |
| natefinch | ericsnow: so, iirc, it's non-trivial | 03:04 |
| ericsnow | natefinch: not the right answer :P | 03:04 |
| axw | wallyworld_: I'm still going to be out for a while longer. I'll be working late tonight | 03:05 |
| wallyworld_ | axw: np at all | 03:05 |
| natefinch | ericsnow: michael did a bunch of work on that relatively recently, you should talk to him in the morning | 03:11 |
| ericsnow | natefinch: k | 03:12 |
| natefinch | ericsnow: I think there were updates to the replicaset code and/or tests in order to determine that. IIRC it was something like ping until we get something reasonable back. | 03:13 |
| ericsnow | natefinch: got it | 03:13 |
| ericsnow | natefinch: keep in mind that for backup we can do the check on the API server side, so we have access to state | 03:15 |
| natefinch | ericsnow: yeah, the replicaset tests assume DB access too. There's just no actual flag saying "replicasets are up" | 03:16 |
| === kadams54 is now known as kadams54-away | ||
| === kadams54-away is now known as kadams54 | ||
| === kadams54 is now known as kadams54-away | ||
| wallyworld_ | axw: ping | 04:37 |
| sebas5384 | https://gist.github.com/anonymous/de06b097d25d690b684f after seeing this log i'm pretty sure i'm not able to use kvm | 04:39 |
| sebas5384 | hehe | 04:39 |
| ericsnow | wallyworld_: I've updated http://reviews.vapour.ws/r/573/ | 05:04 |
| wallyworld_ | ok, looking, wow, must be late for you | 05:05 |
| ericsnow | wallyworld_: is that what you meant about checking if HA is ready? | 05:05 |
| ericsnow | wallyworld_: well, I hate leaving CI blocked | 05:05 |
| wallyworld_ | ericsnow: looks ok i think, but backup doesn't seem to call WaitUntilReady() and haEnabled() always returns true | 05:08 |
| ericsnow | wallyworld_: isn't HA always enabled (even if not utilized), i.e. the --replset option is always passed to mongod | 05:09 |
| ericsnow | wallyworld_: do you think WaitUntilReady would be more appropriate than IsReady? | 05:09 |
| wallyworld_ | true, so why the haEnabled() function? and what about older environments? or are they upgraded to ha ? | 05:10 |
| wallyworld_ | i think they are upgraded | 05:10 |
| ericsnow | wallyworld_: the new backups only applies to 1.22+ | 05:10 |
| ericsnow | wallyworld_: and haEnabled gets patched to return false in the tests (since we don't use HA there) | 05:11 |
| ericsnow | wallyworld_: is "haEnabled" the wrong name? (perhaps "replSetEnabled"? | 05:11 |
| wallyworld_ | so then that's a bit misleading, the block of code should be extracted from create | 05:11 |
| wallyworld_ | and put inside a func | 05:12 |
| wallyworld_ | and that func should be what's patched | 05:12 |
| ericsnow | wallyworld_: fair enough | 05:12 |
| wallyworld_ | but isn't that what WaitUntilReady is for? | 05:12 |
| wallyworld_ | if you just call WaitUntilReady, it should all be fine, just patch WaitUntilReady? | 05:13 |
| wallyworld_ | and then IsReady doesn't need to be exported | 05:13 |
| ericsnow | wallyworld_: it depends on it we want backup to fail immediately if HA isn't ready or if we make users wait | 05:15 |
| wallyworld_ | oh right i see | 05:15 |
| ericsnow | wallyworld_: currently it fails immediately, but it sounds like you would rather we take the waiting approach | 05:15 |
| wallyworld_ | for now just do what's needed to unblock so you csn goto bed | 05:15 |
| ericsnow | wallyworld_: I could drop the WaitUntilReady func | 05:16 |
| ericsnow | k | 05:16 |
| wallyworld_ | yes, drop that for now | 05:16 |
| wallyworld_ | jut the minimum, but then come back and fix if needed | 05:16 |
| wallyworld_ | i'd just like to see the code extracted | 05:16 |
| wallyworld_ | so the haEnabled() can be dropped | 05:16 |
| ericsnow | k | 05:16 |
| ericsnow | doing it now | 05:16 |
| wallyworld_ | ty | 05:16 |
| axw | wallyworld_: pong | 05:25 |
| wallyworld_ | axw: quick hangout? | 05:26 |
| axw | sure | 05:26 |
| wallyworld_ | 1:1 one | 05:26 |
| axw | ok | 05:26 |
| axw | wallyworld_: hypothetically the openstack provider could do bad things if your environment name contains regexp meta characters | 05:45 |
| wallyworld_ | axw: do we allow that? i thought env names were constrained | 05:46 |
| wallyworld_ | to valid chars | 05:46 |
| axw | maybe... trying to find where | 05:46 |
| axw | wallyworld_: seems we just check that it doesn't contain "/" | 05:48 |
| wallyworld_ | oh | 05:48 |
| wallyworld_ | in that case i need to do more in the azure one also | 05:48 |
| axw | I'll see what openstack allows for machine names... | 05:48 |
| wallyworld_ | azure is ok | 05:52 |
| wallyworld_ | "alphanumeric characters and underscores are valid in the name" | 05:52 |
| axw | hmm, can't find any info about it on openstack... | 05:55 |
| axw | possibly this? https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L61 | 05:57 |
| axw | which includes . | 05:58 |
| ericsnow | wallyworld_: updated http://reviews.vapour.ws/r/573/ (and added tests) | 05:59 |
| wallyworld_ | looking | 06:00 |
| wallyworld_ | ericsnow: +1, but i just realised, the CI script might still fail | 06:04 |
| ericsnow | FWIW, I plan on following up with mfoord in the morning | 06:04 |
| ericsnow | wallyworld_: how so? | 06:04 |
| wallyworld_ | as it will still see an error | 06:04 |
| wallyworld_ | it will try to backup and get a "not ready, try again later" error | 06:04 |
| ericsnow | wallyworld_: oh, duh | 06:04 |
| wallyworld_ | as opposed to a EOF error | 06:04 |
| ericsnow | wallyworld_: dang it | 06:04 |
| wallyworld_ | i think we can ask that the script be changed | 06:05 |
| wallyworld_ | maybe | 06:05 |
| wallyworld_ | or else we will need that retry loop | 06:05 |
| wallyworld_ | but you go to bed, i'll follow up | 06:05 |
| ericsnow | wallyworld_: and change it to a CI bug rather than a core bug? | 06:05 |
| wallyworld_ | maybe, i'll have to ask | 06:05 |
| wallyworld_ | i can see both sides of the argument | 06:06 |
| ericsnow | wallyworld_: k, I'll get the merge started | 06:06 |
| wallyworld_ | sure, tyvm | 06:06 |
| wallyworld_ | and then we can add to it if needed to put in the wait until ready | 06:06 |
| ericsnow | wallyworld_: that WaitUntilReady function is still in the commit history ;) | 06:06 |
| wallyworld_ | indeed :-) | 06:07 |
| ericsnow | wallyworld_: because I'm sneaky like that :) | 06:07 |
| wallyworld_ | lol | 06:07 |
| ericsnow | wallyworld_: okay, it's running the merge CI right now | 06:08 |
| wallyworld_ | ty :-) | 06:08 |
| ericsnow | wallyworld_: I'll leave it in your hands (thanks!) | 06:08 |
| wallyworld_ | night night | 06:08 |
| davecheney | wallyworld_: have a cheeky glass of red for me | 06:22 |
| wallyworld_ | wish i could | 06:22 |
| wallyworld_ | i'm still working | 06:22 |
| wallyworld_ | i was saying good bye to eri | 06:23 |
| wallyworld_ | c | 06:23 |
| wallyworld_ | axw: here's the gwacl branch https://code.launchpad.net/~wallyworld/gwacl/prefix-service-match/+merge/243620 | 06:23 |
| wallyworld_ | i need to modify juju to pass in a separator | 06:23 |
| axw | looking | 06:24 |
| wallyworld_ | axw: i might do as you suggest, i can just abandon the gwacl branch | 06:33 |
| axw | wallyworld_: I kinda wish those convenience functions in gwacl weren't there | 06:33 |
| wallyworld_ | yeah | 06:34 |
| axw | that one in particular is trouble, obviously | 06:34 |
| wallyworld_ | indeed | 06:34 |
| === fuzzy_ is now known as Fuzai | ||
| davecheney | anyone else need a review ? | 07:02 |
| wallyworld_ | axw: http://reviews.vapour.ws/r/580/ | 07:04 |
| axw | looking | 07:04 |
| wallyworld_ | davecheney: i do, but andrew can look as we've discussed befre hand | 07:04 |
| davecheney | wallyworld_: done, fwiw | 07:14 |
| wallyworld_ | thanks dave :-) | 07:14 |
| axw | wallyworld_: done also | 07:17 |
| wallyworld_ | thanks | 07:17 |
| wallyworld_ | i checked the azure doc, service names don't allow meta chars | 07:18 |
| wallyworld_ | but i'll add the quoting to be sure | 07:18 |
| axw | wallyworld_: they don't *now*, but who's stopping them from changing that? | 07:18 |
| wallyworld_ | sure | 07:18 |
| wallyworld_ | jam1: you ok for storage meeting a bit later? | 07:27 |
| jam1 | wallyworld_: yep | 07:27 |
| wallyworld_ | jam1: great | 07:27 |
| wallyworld_ | jam1: axw: i may be a minute or 2 later as i have to drive my wife to the city and am not sure of traffic | 07:28 |
| wallyworld_ | i should be back on time | 07:28 |
| axw | okey dokey | 07:28 |
| axw | wallyworld_: when you have a moment, can you see if this makes sense to you? https://docs.google.com/a/canonical.com/document/d/1-9ZPfdgpkj2R9mBG_tlSclGGyK3tRpMf2L4C37mzYD8/edit#bookmark=id.g0p2mahykmz | 08:00 |
| TheMue | morning | 08:44 |
| dimitern | morning all | 09:24 |
| voidspace | dimitern: morning | 09:27 |
| dimitern | morning voidspace, TheMue | 09:28 |
| TheMue | dimitern: voidspace: o/ | 09:28 |
| voidspace | TheMue: hiya | 09:39 |
| dimitern | jam1, voidspace, standup? | 10:01 |
| jam1 | dimitern: just working on a meeting, will be there soon | 10:02 |
| dimitern | ok | 10:02 |
| voidspace | dimitern: omw | 10:02 |
| * fwereade_ out for a bit | 10:05 | |
| perrito666 | morning | 10:21 |
| voidspace | perrito666: morning | 10:33 |
| TheMue | perrito666: heya | 10:33 |
| jam1 | wallyworld_: axw: are we back in https://plus.google.com/hangouts/_/canonical.com/juju-storage | 10:35 |
| jam1 | ? | 10:35 |
| axw | omw | 10:35 |
| * dimitern out for a 1h | 11:38 | |
| voidspace | perrito666: if you get a chance care to look at this one: http://reviews.vapour.ws/r/583/ | 13:02 |
| voidspace | perrito666: you're probably more familiar with this code than others, but it's a simple change | 13:02 |
| voidspace | perrito666: restores (and tests) some work by ericsnow | 13:02 |
| perrito666 | voidspace: looking | 13:03 |
| voidspace | perrito666: thanks | 13:03 |
| perrito666 | voidspace: only nits not worth mentioning, LGTM, but it is not worthy that I LGTM since you will need for david tomorrow to ship it :p so look for a better source of lgtmness | 13:09 |
| voidspace | perrito666: ok, cool - thanks | 13:10 |
| voidspace | anyone else who fancies an easy review | 13:12 |
| voidspace | http://reviews.vapour.ws/r/583/ | 13:12 |
| voidspace | I'm off to lunch | 13:12 |
| === jcw4 is now known as jw4 | ||
| natefinch | sinzui: where are we on those blockers? | 14:45 |
| sinzui | natefinch, still taking stock. I just reported https://bugs.launchpad.net/juju-core/+bug/1399229 | 14:45 |
| mup | Bug #1399229: win client cannot get status after bootstrap <ci> <regression> <status> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1399229> | 14:45 |
| perrito666 | sinzui: can we get a --debug run on that? | 14:53 |
| sinzui | perrito666, I am trying. I need to setup another win machine first | 14:53 |
| sinzui | natefinch, I suspect ha is brittle, but we have so many hot issue, I am going to get you better info for the current bugs than investigate ha | 14:55 |
| sinzui | natefinch, the ha issues are connect shutdown/refused talking to the api server, which is the current problem with the backup-restore test, so maybe we already have a bug tracking the problem | 14:58 |
| sinzui | perrito666, damn it. We cannot get a debug with a matching server because the testing streams are for 1.21. I will try a bootstrap anyway and hope for a reproduction | 15:02 |
| perrito666 | sinzui: hold, I have both windows and a fake stream | 15:02 |
| perrito666 | tell me how to try that | 15:02 |
| sinzui | perrito666, The test is just a bootstrap into aws, proof that it can bring up and talk to a state-server | 15:03 |
| perrito666 | sinzui: ok, that is trunk? | 15:03 |
| * perrito666 goes again | 15:03 | |
| sinzui | perrito666, yes | 15:03 |
| perrito666 | ok, firing up windows | 15:03 |
| perrito666 | can you believe it? compiling juju in your machine and inside a vm is a bad idea | 15:11 |
| sinzui | dimitern, do we really need to backport bug 1397376 to 1.20? the stakeholders are pointing to 1.20 as an example that works for them | 15:15 |
| mup | Bug #1397376: maas provider: 1.21b3 removes ip from api-endpoints <api> <cloud-installer> <fallout> <landscape> <maas-provider> <juju-core:In Progress by dimitern> <juju-core 1.20:In Progress by dimitern> <juju-core 1.21:In Progress by dimitern> <https://launchpad.net/bugs/1397376> | 15:15 |
| perrito666 | ok sinzui please dont hate me for the stupid thing I am about to ask | 15:22 |
| perrito666 | where should I store environments.yaml in windows? | 15:22 |
| dimitern | sinzui, I wasn't sure, let me check if the logic differs in 1.20 | 15:22 |
| sinzui | perrito666, /users/<you>/.juju/environments.yaml and you need a .ssh/id_rsa | 15:23 |
| katco | perrito666: there should be an environmental variable for /users/<you>.... forget what it is | 15:23 |
| katco | perrito666: perhaps %HOME% | 15:23 |
| dimitern | sinzui, confirmed - 1.20 is not affected by the same address ordering issue | 15:24 |
| sinzui | \o/ | 15:25 |
| perrito666 | asdkjhaskdjhna ksdhjask windows will not allow me to name something .juju from the gui | 15:26 |
| perrito666 | oh you have to name it .juju. | 15:27 |
| perrito666 | and windows removes the dot at the end | 15:27 |
| natefinch | perrito666: you need to turn off the "hide extensions" thingy, I think... I've never had a problem using wacky extensions in windows before. | 15:28 |
| perrito666 | natefinch: the issue was not with the extension is that the file began with . | 15:29 |
| perrito666 | so, no, windows is not looking for %HOME%/.juju | 15:29 |
| natefinch | oh yeah, you can't make a file in the UI that starts with a ..... sorry | 15:30 |
| natefinch | er with a . that is | 15:30 |
| jw4 | perrito666: %USERPROFILE%\.juju\environments.yaml ? | 15:30 |
| perrito666 | %UserProfile% | 15:30 |
| perrito666 | looking | 15:30 |
| natefinch | I thought we made juju home on windows just "Juju" | 15:30 |
| perrito666 | natefinch: let see | 15:31 |
| natefinch | perrito666: yeah it's %APPDATA%/Juju | 15:32 |
| alexisb | dimitern, you still around? | 15:33 |
| natefinch | alexisb: my chrome just froze, will be back on the call in a minute.... | 15:34 |
| perrito666 | bootstraping | 15:34 |
| perrito666 | sinzui: jw4 katco natefinch tx for your help | 15:34 |
| dimitern | alexisb, yes | 15:34 |
| katco | perrito666: anytime perrito666 | 15:34 |
| alexisb | on the cross team call, was curious on status on the api endpoints bugs | 15:35 |
| sinzui | natefinch, axw targeted https://bugs.launchpad.net/juju-core/+bug/1388860 to 1.20.14. It is a backport of a fix. He suggests it because there is another bug reporting that the issue does indeed affect the 1.20 series. Can you get someone to look into the backport now, and if it is ugly, then maybe we shouldn't...just go head with the release. | 15:45 |
| mup | Bug #1388860: ec2 says agent-state-info: 'cannot run instances: No default subnet for availability zone: ''us-east-1e''. (InvalidInput)' <deploy> <ec2-provider> | 15:45 |
| mup | <network> <juju-core:Fix Committed by axwalk> <juju-core 1.20:Triaged> <juju-core 1.21:Fix Released by axwalk> <https://launchpad.net/bugs/1388860> | 15:45 |
| dimitern | alexisb, i'm working on a fix now | 15:46 |
| dimitern | alexisb, it turned out to be more complicated to test than to fix :) - I should be ready later today or tomorrow morning | 15:46 |
| perrito666 | sinzui: natefinch http://pastebin.ubuntu.com/9368819/ | 15:47 |
| sinzui | perrito666, try again, that is aws's sucky mirrors | 15:47 |
| perrito666 | sinzui: I am about ot | 15:48 |
| perrito666 | to | 15:48 |
| sinzui | perrito666, oh.. | 15:48 |
| sinzui | perrito666, you can tell juju to not do updates and upgrades! | 15:48 |
| * perrito666 had to step down a moment to go order food, which requires me standing on a bench on the backyard, single place in the house with cell reception | 15:49 | |
| sinzui | perrito666, you can improve your chances if success: | 15:49 |
| sinzui | enable-os-upgrade: false | 15:49 |
| sinzui | enable-os-refresh-update: false | 15:49 |
| sinzui | ^ we do that with canonistack because its network is unreliable | 15:49 |
| perrito666 | sinzui: is the test running that way? | 15:50 |
| alexisb | dimitern, awesome, thank you for turning that around so quickly | 15:50 |
| natefinch | man, it really sucks we're still using launchpad for bugs... it makes everything so much more manual | 15:50 |
| perrito666 | I would like to stay as close as possible to the tests | 15:50 |
| natefinch | sinzui: do you happen to have a link to the PR that fixed https://bugs.launchpad.net/juju-core/+bug/1388860 for 1.21? | 15:50 |
| mup | Bug #1388860: ec2 says agent-state-info: 'cannot run instances: No default subnet for availability zone: ''us-east-1e''. (InvalidInput)' <deploy> <ec2-provider> | 15:50 |
| mup | <network> <juju-core:Fix Committed by axwalk> <juju-core 1.20:Triaged> <juju-core 1.21:Fix Released by axwalk> <https://launchpad.net/bugs/1388860> | 15:50 |
| dimitern | alexisb, no worries | 15:51 |
| sinzui | perrito666, no, but I think it is somewhat irrelevant because we have fresh images in aws, charms need updated and upgrades, but this failure is about talking to the state server | 15:51 |
| perrito666 | sinzui: running second bootstrap with your options added | 15:51 |
| sinzui | natefinch, I do not, I will look | 15:53 |
| voidspace | I have a fix for issue 1398837 | 15:53 |
| voidspace | waiting for a review | 15:53 |
| natefinch | sinzui: I found it | 15:53 |
| voidspace | bug 1398837 | 15:54 |
| mup | Bug #1398837: cannot extract configuration from backup file: "var/lib/juju/agents/machine-0/agent.conf <backup-restore> <ci> <regression> <juju-core:In Progress by ericsnowcurrently> <https://launchpad.net/bugs/1398837> | 15:54 |
| natefinch | sinzui: looks to be a trivial change | 15:54 |
| sinzui | natefinch, I see the change, but not the actual pull request | 15:55 |
| natefinch | https://github.com/juju/juju/search?q=1388860&type=Issues&utf8=%E2%9C%93 | 15:55 |
| natefinch | amazing, a search that actually works the way you'd expect :) | 15:56 |
| sinzui | natefinch, https://github.com/juju/juju/commit/9e1f40588eb6befcc543ae64e15cf7d8b11fd090 | 15:56 |
| sinzui | natefinch, I was slow, I looked at the date and read the commits | 15:56 |
| natefinch | sinzui: that's the one. Super simple. Want me to backport it? | 15:56 |
| sinzui | natefinch, please do. pretty please | 15:56 |
| perrito666 | odd, ECONREFFUSEd | 16:05 |
| ericsnow | sinzui: regarding 1398837, part of the problem is that the failing test does not wait long enough for HA to be ready before trying to run backup | 16:05 |
| sinzui | ericsnow, okay, what time should we set? | 16:05 |
| ericsnow | sinzui: I'm not sure how long it takes for HA to get ready, but I see that voidspace's patch has a timeout of 60 seconds | 16:06 |
| sinzui | what! | 16:07 |
| ericsnow | voidspace: thanks for taking that over, by the way | 16:07 |
| sinzui | ericsnow, I think we wait 10-20 minutes | 16:07 |
| perrito666 | sinzui: uff, /proc/self/fd/9: 9: exec: varlibjujutoolsmachine-0/jujud: not found | 16:07 |
| perrito666 | someone broke paths | 16:07 |
| voidspace | ericsnow: no problem, care to review it? | 16:07 |
| ericsnow | sinzui: that test has a total runtime of about 15 minutes | 16:07 |
| sinzui | perrito666, \o/, or old nemesis win paths | 16:08 |
| ericsnow | voidspace: sure, though it looks strangely familiar :) | 16:08 |
| perrito666 | sinzui: I wonder how in the universe does a windows client affects that path | 16:08 |
| sinzui | ericsnow, I am looking at the lib now and hoping for an easy timeout change | 16:09 |
| ericsnow | sinzui: k, thanks | 16:09 |
| voidspace | ericsnow: heh | 16:09 |
| voidspace | ericsnow: sinzui: in my investigations I could find *no* deterministic way to tell when a replicaset is ready | 16:10 |
| sinzui | perrito666, the win module is convoluted. it switched between native path separators and posix depending on the code *knowing* that it will be executed against the state server | 16:10 |
| sinzui | voidspace, :( | 16:10 |
| voidspace | ericsnow: sinzui: even connecting separately to all of them and waiting until the configuration from *all members* reports that they're all ready wasn't enough after a reconfigure | 16:10 |
| voidspace | ericsnow: sinzui: at which point I gave up | 16:10 |
| ericsnow | voidspace: I wanted to ask you about that (the IsReady function I added) | 16:10 |
| voidspace | ericsnow: sinzui: this fix will definitely help *sometimes* | 16:10 |
| perrito666 | nnnnnailed | 16:10 |
| perrito666 | sinzui: https://github.com/juju/juju/commit/ad420d9 | 16:10 |
| voidspace | ericsnow: I have been down this road and this will help sometimes | 16:11 |
| voidspace | ericsnow: but sometimes they report ready and the next operation can still fail | 16:11 |
| voidspace | ericsnow: although your problem is with initiation - my problem was with reconfigure | 16:11 |
| voidspace | ericsnow: so it's likely to be better | 16:11 |
| voidspace | ericsnow: the Initiate function could call WaitUntilReady | 16:11 |
| ericsnow | voidspace: ah, cool | 16:12 |
| voidspace | ericsnow: I didn't make that change as I was focussed on fixing the specific problem | 16:12 |
| perrito666 | sinzui: Ill try to fix it | 16:13 |
| ericsnow | voidspace: ack | 16:13 |
| ericsnow | natefinch, perrito666: standup? | 16:15 |
| natefinch | ericsnow: trying... google doesn't like me | 16:17 |
| voidspace | google is not your friend... | 16:17 |
| perrito666 | natefinch: use firefox | 16:17 |
| natefinch | it was trying to join as my gmail account | 16:18 |
| voidspace | ericsnow: why did you remove WaitUntilReady? | 16:18 |
| voidspace | ericsnow: a timeout in the script won't help without a retry loop | 16:19 |
| ericsnow | voidspace: we weren't using it so wallyworld_ asked me to remove it | 16:19 |
| ericsnow | voidspace: right, a timeout won't help but a sleep will :) | 16:19 |
| voidspace | ericsnow: ah, it was wallyworld_ who asked me to put it back in | 16:20 |
| voidspace | this morning | 16:20 |
| ericsnow | voidspace: yeah, I told him it was in the commit history still :) | 16:20 |
| voidspace | ericsnow: a retry loop is better than a sleep, surely? | 16:20 |
| voidspace | ericsnow: I'm agnostic on it - up to sinzui really | 16:20 |
| ericsnow | voidspace: sure, but that's up to sinzui | 16:21 |
| voidspace | whether we fix it in juju or they fix it in their test harness | 16:21 |
| voidspace | ericsnow: ok | 16:21 |
| voidspace | adding WaitUntilReady to Initiate slows the test suite down a lot | 16:23 |
| voidspace | I'll tell you by how much when it actually finishes! | 16:23 |
| ericsnow | voidspace: I noticed that each test I added for IsReady added about 12 seconds to the tests | 16:24 |
| voidspace | well, from 244 seconds to 365 - including one failure (probably needs a mock) | 16:24 |
| voidspace | ericsnow: heh, ouch | 16:24 |
| voidspace | ericsnow: my tests for WaitUntilReady are fast because they all mock IsReady... | 16:24 |
| ericsnow | voidspace: nice | 16:27 |
| voidspace | and use a timeout of 1 second... | 16:27 |
| sinzui | voidspace, the test sets the timeout for ha to 1200...could something else be timing out before then? | 16:28 |
| sinzui | well obviously something is | 16:28 |
| voidspace | sinzui: backup was set to hard fail if the replicaset was not ready | 16:29 |
| ericsnow | voidspace: IsReady may need a tweak (it will return an error for anything but io.EOF) | 16:29 |
| voidspace | sinzui: so it's not a timeout you need as much as a *retry* if it fails for that reason | 16:29 |
| voidspace | ericsnow: that sounds correct to me | 16:29 |
| voidspace | ericsnow: what other error would you expect? | 16:29 |
| sinzui | voidspace, retry of what, status, ha? | 16:29 |
| voidspace | sinzui: of the backup itself I think | 16:29 |
| voidspace | sinzui: that's where the failure was IIUC | 16:29 |
| sinzui | oh, backup, doh | 16:30 |
| ericsnow | voidspace: "dial tcp 172.31.3.149:17070: connection refused" | 16:30 |
| voidspace | ericsnow: ah | 16:30 |
| voidspace | ericsnow: what's the actual error type? | 16:30 |
| voidspace | ericsnow: or should I do a match for "connection refused"? | 16:30 |
| ericsnow | voidspace: not sure, sinzui noted it in bug 1398837 | 16:30 |
| mup | Bug #1398837: cannot extract configuration from backup file: "var/lib/juju/agents/machine-0/agent.conf <backup-restore> <ci> <regression> <juju-core:In Progress by ericsnowcurrently> <https://launchpad.net/bugs/1398837> | 16:30 |
| sinzui | ericsnow, the bug is really about the test failing. if we mark it fix released, we replace it with another bug that backup tests still fail | 16:31 |
| ericsnow | sinzui: +1 | 16:31 |
| voidspace | ericsnow: hmmm... that bug actually reports that restore fails | 16:32 |
| voidspace | ericsnow: are you sure that a WaitUntilReady in create will fix that? | 16:32 |
| ericsnow | voidspace: are you sure? "ERROR:root:Command '['juju', 'backup']' returned non-zero exit status 1" | 16:33 |
| voidspace | ericsnow: ah, that's further down the bug report | 16:33 |
| voidspace | "the bug has mutated" | 16:33 |
| ericsnow | voidspace: the original restore error was caused by backup though | 16:34 |
| voidspace | ericsnow: ok, fair enough | 16:35 |
| perrito666 | natefinch: you really need to work in your unmuting skills | 16:35 |
| ericsnow | voidspace: it's all because I stuck a "juju backups create" call in the backup plugin script a couple weeks back | 16:36 |
| ericsnow | voidspace: however, these are real issues that need addressing at some point so might as well be now | 16:36 |
| voidspace | ericsnow: so this would be my fix for the connection refused issue | 16:46 |
| voidspace | ericsnow: http://pastebin.ubuntu.com/9369537/ | 16:46 |
| voidspace | no, hang on | 16:46 |
| voidspace | if errors.Cause(err) == io.EOF || (err != nil && strings.Contains(err.Error(), "connection refused")) { | 16:47 |
| ericsnow | sinzui: so should we apply Michael's fix (http://reviews.vapour.ws/r/583/) or will you be able to add retries/sleep to the CI test script (the HA one) around the backup call? | 16:47 |
| sinzui | ericsnow, I am still looking at extending the timeout | 16:48 |
| ericsnow | voidspace: I so hate testing for strings in err.Error() :( | 16:48 |
| ericsnow | sinzui: I'm not sure a timeout will help | 16:48 |
| ericsnow | sinzui: it has to wait somehow for HA to be ready before running backup (or retry the backup if it fails) | 16:49 |
| ericsnow | sinzui: voidspace's fix would probably help, but I'd rather not do that *just* for the sake of the CI test if we can help it | 16:50 |
| sinzui | ericsnow, yep I see ensure-availability as the issue, I will report a new bug about this, close the backup bug and hope the patch works | 16:50 |
| ericsnow | voidspace: but you're probably right | 16:50 |
| voidspace | heh | 16:51 |
| ericsnow | voidspace: I was relying just on io.EOF for IsReady because of the precedent elsewhere in the replicaset code (I don't have a very good knowledge of the problem-space otherwise) | 16:51 |
| ericsnow | voidspace: davechaney had suggested checking for other kinds of failures but I didn't find any examples of that elsewhere in juju so I stuck with just io.EOF | 16:53 |
| ericsnow | voidspace: so I'm good with checking for "connection refused" (my dislike of checking err.Error() aside) | 16:53 |
| voidspace | right | 16:53 |
| voidspace | cool | 16:53 |
| voidspace | I've pushed it and we can let the PR lie until we get a definite decision | 16:54 |
| voidspace | I'm returning to IP address stuff | 16:54 |
| ericsnow | voidspace: k | 16:54 |
| ericsnow | sinzui: thanks! | 16:54 |
| ericsnow | sinzui: regardless, good came of the bug (though as the cost of CI being blocked for an extra day) | 16:55 |
| ericsnow | voidspace: I'll put up a separate patch just for the "connection refused" part of that so that it's not conflated with the WaitUntilReady part | 16:56 |
| voidspace | ericsnow: ah, I pushed that | 16:57 |
| voidspace | ericsnow: it doesn't do any harm is my thinking... | 16:57 |
| ericsnow | voidspace: that's okay | 16:57 |
| perrito666 | sinzui: natefinch found it, running tests now for proposal | 16:58 |
| sinzui | voidspace, ericsnow I reported https://bugs.launchpad.net/juju-core/+bug/1399277 about the ha issue, I add a line for beta4, because I /think/ this will help, but we can discuss it as out of scope | 17:03 |
| mup | Bug #1399277: ensure-availability is not reliable <ci> <ha> <regression> <juju-core:In Progress> <juju-core 1.21:Triaged> <https://launchpad.net/bugs/1399277> | 17:03 |
| ericsnow | voidspace: thanks! | 17:03 |
| ericsnow | sinzui: ^ | 17:04 |
| ericsnow | sinzui: it applies to 1.22 as well, right? (where we are seeing the CI backups failures) | 17:04 |
| sinzui | ericsnow, yess 1.22 is really hurting | 17:05 |
| ericsnow | sinzui: :( | 17:05 |
| arosales_ | rick_h_: to confirm where should we log bugs for jujucharms.com? | 17:15 |
| rick_h_ | https://github.com/CanonicalLtd/jujucharms.com/issues arosales_ | 17:15 |
| rick_h_ | arosales_: updated bug link is landed and will be in monday release | 17:15 |
| arosales_ | rick_h_: thanks | 17:18 |
| rick_h_ | arosales_: np, ty for the bug reports | 17:18 |
| arosales_ | np, thanks for responding to them :-) | 17:19 |
| perrito666 | uff this patch is awfully hard to revert | 17:44 |
| mgz_ | perrito666: ...subsequent changes? | 17:45 |
| perrito666 | mgz_: yes most likely | 17:46 |
| natefinch | Can someone review my backport? It's a very small change: http://reviews.vapour.ws/r/584/ | 17:50 |
| natefinch | ahh the old "on call reviewers are both done for the day by 8am" | 17:54 |
| natefinch | mgz_: can you look? ^ | 17:54 |
| mgz_ | natefinch: on it | 17:54 |
| mgz_ | natefinch: is the indent of that switch off or is that just reviewboard messing with me? | 17:55 |
| natefinch | mgz_: that's how gofmt likes it. | 17:57 |
| natefinch | just double checked | 17:57 |
| natefinch | personally, I prefer the cases to be indented, but it does then cause double indent for the stuff under the case, so.... yeah. *shrug* | 17:58 |
| ericsnow | natefinch: I reviewed that patch | 17:58 |
| mgz_ | I think I may just be misreading the html output... it looks like the returns are differently indented, but there are change-indent marks that implies otherwise | 17:58 |
| mgz_ | lgtm otherwise | 17:59 |
| natefinch | mgz_: oh I see what you mean | 17:59 |
| natefinch | mgz_: the left hand side code is only under a single "if", the right hand side is under an "if" and a switch, so it is indented more. | 18:00 |
| mgz_ | yup | 18:00 |
| natefinch | I honestly didn't even see the indent marks at first. It's nice that it doesn't mark the whole thing as just different when it's just the different indent. | 18:01 |
| ericsnow | natefinch: we should keep track of these (things people *like* about RB) | 18:04 |
| ericsnow | natefinch: all I ever hear is what annoys people (which is typical) :P | 18:04 |
| perrito666 | ericsnow: is easy, all that is not annoying we like | 18:05 |
| natefinch | the whole "make a ton of changes and then publish as one action" is pretty fantastic.... | 18:05 |
| ericsnow | natefinch: agreed | 18:05 |
| ericsnow | perrito666: heh | 18:06 |
| natefinch | ericsnow: being able to expand all files with a single click is also pretty awesome. I hate that I can't do that in github's diffs | 18:08 |
| voidspace | natefinch: what do you think of this as a conversion function - dotted quad (IP address) to decimal? | 18:29 |
| voidspace | natefinch: | 18:29 |
| voidspace | <natefinch> the whole "make a ton of changes and then publish as one | 18:29 |
| voidspace | oops... | 18:29 |
| voidspace | natefinch: http://pastebin.ubuntu.com/9370769/ | 18:30 |
| ericsnow | voidspace: FYI: http://reviews.vapour.ws/r/585/ (check other "conn dropped" conditions) | 18:30 |
| voidspace | natefinch: just wondering if it has any chance of passing review | 18:30 |
| voidspace | ericsnow: LGTM | 18:30 |
| voidspace | natefinch: the address is already validated, so no need to handle that potential error case | 18:31 |
| voidspace | although it should work fine anyway as ParseInt would fail | 18:31 |
| voidspace | and the *caller* will be constructing a sensible error message from that failure | 18:32 |
| voidspace | ooh, bug | 18:34 |
| voidspace | they need zero padding | 18:34 |
| voidspace | grr... | 18:34 |
| voidspace | right, g'night all | 18:43 |
| natefinch | voidspace: net.IP has some interesting stuff | 18:43 |
| voidspace | natefinch: ah, I'll check | 18:43 |
| voidspace | natefinch: I need int representations | 18:43 |
| voidspace | natefinch: it maybe that I don't need to write them myself | 18:43 |
| natefinch | voidspace: I always try to write as little as possible myself :) | 18:44 |
| voidspace | natefinch: it doesn't have a ToDecimal | 18:44 |
| voidspace | or equivalent | 18:44 |
| voidspace | nor the reverse | 18:45 |
| voidspace | I need both | 18:45 |
| natefinch | voidspace: I'll look around to see if something's already out there | 18:45 |
| natefinch | voidspace: don't want to keep you from EOD | 18:45 |
| voidspace | ok | 18:45 |
| voidspace | natefinch: thanks, if you see something an email would be awesome | 18:46 |
| natefinch | voidspace: will do | 18:46 |
| voidspace | natefinch: they're not hard to write functions - just converting via string functions is a little icky | 18:46 |
| voidspace | natefinch: decimal to dotted quad is less icky | 18:46 |
| voidspace | g'night | 18:46 |
| perrito666 | for the fantastic chance to unlock CI http://reviews.vapour.ws/r/586/ | 19:23 |
| perrito666 | it is extremely trivial | 19:23 |
| perrito666 | has been tested on windows and linux | 19:23 |
| natefinch | perrito666: will this work if deploying a unit to a windows machine? | 19:28 |
| perrito666 | natefinch: I dont know, I cannot deploy one of those | 19:29 |
| perrito666 | but if not, I am ok with it since it is a regression | 19:29 |
| perrito666 | and also https://bugs.launchpad.net/juju-core/+bug/1399322 | 19:29 |
| mup | Bug #1399322: ToolsDir should be series based <windows> <juju-core:New> <https://launchpad.net/bugs/1399322> | 19:29 |
| perrito666 | I opened a bug to have someone solve that doubt | 19:29 |
| natefinch | perrito666: let's ship it | 19:34 |
| perrito666 | aghh what was the _fixes thing? | 19:35 |
| ericsnow | perrito666: $$fixes-XXXXX$$ | 19:35 |
| perrito666 | tx ericsnow | 19:36 |
| perrito666 | sent | 19:36 |
| natefinch | I think the comments just need "fixes-123456" in them, and then $$whatever$$ works | 19:37 |
| perrito666 | ok, since the fix is hopefully being merged, I am going to step down for a moment | 19:45 |
| thumper | morning folks | 19:57 |
| rick_h_ | thumper: morning, shot you an email about rescheduling next week if you get time to peek | 19:58 |
| thumper | rick_h_: hey | 19:58 |
| thumper | rick_h_: can you do two days earlier and an hour later? | 19:59 |
| natefinch | man I hate mongo | 20:00 |
| thumper | sinzui: what's the status of CI? | 20:00 |
| rick_h_ | thumper: I can | 20:00 |
| rick_h_ | thumper: can you email that back and let urulama respond as well? | 20:00 |
| thumper | kk | 20:00 |
| rick_h_ | ty | 20:00 |
| sinzui | thumper, master is blocked by 2 regressions, both with fixes due soon | 20:00 |
| * thumper sighs | 20:02 | |
| === kadams54 is now known as kadams54-away | ||
| === kadams54-away is now known as kadams54 | ||
| === kadams54 is now known as kadams54-away | ||
| thumper | grr... | 20:50 |
| thumper | gc.Not is so fucked | 20:54 |
| anastasiamac | thumper: could u cast ur eyes over my changes for block commands | 20:54 |
| thumper | anastasiamac: sure | 20:54 |
| anastasiamac | thumper: the PR is so huge now, m thinking to break it into smaller pieces | 20:55 |
| anastasiamac | thumper: block functionality itself | 20:55 |
| anastasiamac | and than a PR for each command | 20:55 |
| thumper | as long as your first commands are add and remove machine I don't care :-) | 20:55 |
| anastasiamac | thumperbut if it's all good as it is now, I'd rather commit my monster without breaking it apart | 20:55 |
| anastasiamac | thumper: :) | 20:56 |
| anastasiamac | thumper: lets he how it reviews to u atm | 20:56 |
| thumper | I'm already going to have to do a mega-conflict merge | 20:56 |
| anastasiamac | /s/he/see | 20:56 |
| anastasiamac | thumper: :-( | 20:56 |
| thumper | that's fine | 20:56 |
| thumper | I'm used to it | 20:56 |
| anastasiamac | thumper: but beta u than me :) | 20:56 |
| thumper | sure | 20:56 |
| thumper | :) | 20:56 |
| anastasiamac | thumper: m off to sort kids and co... | 20:57 |
| thumper | kk | 21:00 |
| thumper | I think I'll fetch a coffee and my almond croisant before starting this review | 21:15 |
| thumper | it may take a while | 21:15 |
| menn0 | ericsnow: ping | 21:28 |
| ericsnow | menn0: o/ | 21:28 |
| menn0 | ericsnow: howdy. | 21:29 |
| menn0 | ericsnow: what's the state of play with the ensure-availability/backup issue? | 21:29 |
| menn0 | ericsnow: looks like you have a Ship It for PR 1271 | 21:29 |
| menn0 | ericsnow: is that PR not particularly important? | 21:30 |
| ericsnow | menn0: last I was aware, sinzui was going to see about tweaking the function-ha-backup-restore test so that HA is up before backup runs (or something along those lines) | 21:31 |
| menn0 | ericsnow: ok cool | 21:31 |
| ericsnow | menn0: that PR isn't important for unblocking CI | 21:31 |
| menn0 | ericsnow: there's also voidpspace's PR 1269 | 21:32 |
| menn0 | ericsnow: which seems relevant but hasn't been merged yet | 21:32 |
| ericsnow | menn0: it was just something we noticed would be good to do at some point (so it can wait until things are unblocked) | 21:32 |
| menn0 | ericsnow: ok so we're really just waiting on sinzui's change to get CI unblocked | 21:33 |
| ericsnow | menn0: yeah, Michael's patch (http://reviews.vapour.ws/r/583/) should help but I think the current behavior is correct for users (so I'd rather we not land that patch just for the sake of CI) | 21:34 |
| sinzui | menn0, not quite, because I don't know how to make ci know when ha is ready | 21:34 |
| ericsnow | menn0: yeah, and he had said he might change that bug to be a CI bug IIRC, which would unblock us | 21:34 |
| ericsnow | sinzui: can you just throw a sleep in there before calling backup or put that call to backup in a retry loop | 21:35 |
| ericsnow | ? | 21:35 |
| sinzui | ericsnow, sleeping for 5 minutes the running backup can still fail. | 21:36 |
| ericsnow | sinzui: yuck | 21:36 |
| sinzui | ericsnow, I need to poll something that means we are ready | 21:36 |
| ericsnow | sinzui: how long does it take for HA to be ready? | 21:36 |
| ericsnow | sinzui: I'm not the best resource for finding that polling solution but I'll give it some thought | 21:37 |
| ericsnow | natefinch: do you have any ideas off-hand on how sinzui might poll for HA-ready (from a script) | 21:38 |
| sinzui | ericsnow, it is variable. Our code calls a method named wait_for_ha() we don't start backup until | 21:39 |
| sinzui | ericsnow, We read status http://pastebin.ubuntu.com/9372831/ | 21:40 |
| sinzui | ericsnow, menn0 is there something else to read about juju *really* being ready | 21:40 |
| natefinch | sinzui: I wish I knew. I believe that can sometimes pass and mongo can still somehow not be ready-ready. | 21:41 |
| sinzui | could is "juju run" something on the state-server or the other voting machines? | 21:42 |
| sinzui | s/could is/could 1/ | 21:43 |
| * sinzui gives up | 21:43 | |
| sinzui | well, I think testing got lucky. I think the replica-set was ready this time | 21:44 |
| sinzui | natefinch, ericsnow I will add a 5m sleep if you think it will fix the issue 80% of the time | 21:45 |
| ericsnow | sinzui: If it's close enough that it just succeeded as-is, then I'd expect such a sleep (hacky as it is) would help a bunch | 21:46 |
| * sinzui adds hack | 21:46 | |
| ericsnow | sinzui: I would not expect a lot of variability in how long it takes for HA to come up | 21:46 |
| menn0 | sinzui, ericsnow, natefinch: wouldn't voidspaces change also greatly reduce the odds of the replicaset not being ready? | 21:48 |
| sinzui | menn0, I think so | 21:48 |
| ericsnow | sinzui: of course that doesn't solve the problem of accurately/reliably introspecting the HA status, but that the point of but 1399277, right? | 21:48 |
| ericsnow | menn0: if I were a user I would rather it fail when the replicaset isn't ready than have it wait | 21:49 |
| ericsnow | menn0: however, I'd expect the odds of this issue affecting actual users to be remote | 21:50 |
| ericsnow | menn0: I'd rather we didn't apply voidspace's patch just for the sake of CI (more -0 than -1) | 21:51 |
| natefinch | ericsnow: what, you don't think a lot of people will do "juju bootstrap && juju ensure-availability && juju backup"? :) | 21:52 |
| ericsnow | natefinch: :) | 21:52 |
| menn0 | ericsnow: ok, well if backup reports a clear error immediately about the replicaset not being ready can't the test detect that and retry a few times | 21:52 |
| menn0 | natefinch: you can't be too careful :) | 21:52 |
| ericsnow | natefinch: however, there's a chance this could bite someone if they did " juju ensure-availability && juju backup" on an existing env, no? | 21:52 |
| ericsnow | menn0: that was what voidspace suggested when we discussed it | 21:53 |
| ericsnow | sinzui, menn0: the error message is "HA not ready; try again later" | 21:54 |
| ericsnow | (apiserver/backups/create.go) | 21:54 |
| natefinch | ericsnow: I'm not too concerned if someone does "juju ensure-availability && juju backup" and the backup fails with "HA not ready" (or something similar). | 21:56 |
| ericsnow | natefinch: right | 21:56 |
| ericsnow | natefinch: that's the point of what I did yesterday | 21:56 |
| natefinch | ericsnow: well that's great. I'm fine with there being some times when backup can't be done. | 21:57 |
| natefinch | Gotta run | 22:00 |
| ericsnow | sinzui: would it be reasonable to drop the "ci" tag from bug 1399277? | 22:07 |
| mup | Bug #1399277: ensure-availability is not reliable <ci> <ha> <regression> <juju-core:In Progress> <juju-core 1.21:Triaged> <https://launchpad.net/bugs/1399277> | 22:07 |
| sinzui | no, the test has to consistent;y pass | 22:07 |
| ericsnow | sinzui: I meant since it's more of a CI bug but having a more reliable way to know when HA is ready is still something we want to get | 22:10 |
| sinzui | ericsnow, hell no. enterprises script this out like we do | 22:10 |
| ericsnow | sinzui: good point | 22:10 |
| sinzui | ericsnow, This issue is new, so I think something bad has happened. Regardless, I am adding a sleep | 22:11 |
| ericsnow | sinzui: what changed is I updated the "juju backup" plugin to call "juju backups create" | 22:12 |
| ericsnow | sinzui: but that was a few weeks ago so if this issue is new as of a matter of days then yeah | 22:13 |
| sinzui | ericsnow, yeah :( If I add a 5 minutes sleep, the test suite also sleeps. I need to do more work | 22:14 |
| ericsnow | sinzui: :( | 22:14 |
| ericsnow | sinzui: what about a loop around the backup call that checks for the "HA not ready; try again later" message? | 22:15 |
| sinzui | ericsnow, :/ doable but maybe award not all juju have this problem. We test 18 and 20 too | 22:16 |
| perrito666 | sinzui: can you not query status to determine if the ha servers are ready? or they are marked ready before replicaset is actually ready? | 22:17 |
| sinzui | perrito666, we do! status said has vote, so we started backup | 22:17 |
| ericsnow | perrito666: they are already doing that: http://pastebin.ubuntu.com/9372831/ | 22:17 |
| perrito666 | ok so status is lying | 22:18 |
| perrito666 | ok,EOD my brain is fried | 22:22 |
| perrito666 | sinzui: before I leave, where can I see https://bugs.launchpad.net/juju-core/+bug/1399229 job? does it run on the same CI? | 22:24 |
| mup | Bug #1399229: win client cannot get status after bootstrap <ci> <regression> <status> <windows> <juju-core:Fix Released by hduran-8> <https://launchpad.net/bugs/1399229> | 22:24 |
| perrito666 | being windows | 22:24 |
| sinzui | perrito666, this is the job, and your commit is the top result http://juju-ci.vapour.ws:8080/job/win-client-deploy/ | 22:25 |
| perrito666 | :D | 22:25 |
| perrito666 | I go in peace then, have a nice night everybody | 22:26 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!