[00:00] davecheney: in a meeting, be with you soon [00:06] wallyworld_: done [00:06] davecheney: tyvm, will look after my meeting [00:08] sinzui: how long does ci take to go through its tests? [00:08] sinzui: I'm wondering how long before we know if ericsnow's branch fixes the problem [00:09] aboput 18 minutes atm [00:09] davecheney: that is just to land, I'm talking about the 'ci' tag being taken off the bug so it unblocks landings [00:14] davecheney: I could but I dont understand what that means [00:15] perrito666: i don't either [00:15] is something broken ? [00:15] i'm completely lost as to where we left our discussion [00:15] lol, ok, you answered a review request for a wrapper around chmod with an email asking something [00:16] rings a bell? [00:16] right, [00:16] yes, why do we need this ? [00:17] davecheney: so I answered that mail but in summary, chmod, even though it "runs" on windows, does not work [00:17] ok, but why do we need to chmod files ? === kadams54 is now known as kadams54-away [00:19] davecheney: yes, we do sometime ago I did a check with the cloudbase guys to see what was not working on windows workloads and that one was one of the outstanding [00:19] ok, my question is can we solve this problem by removing the cause ? [00:19] chmod might be busted on windows [00:19] but if we can remove the requirement to change permissions [00:20] then that also solves the problem [00:20] well its used for charms and It seems necessary [00:21] how is it used for charms ? [00:21] come on man, work with me here [00:21] davecheney: sorry I am re-reading the code as we speak, hold === kadams54-away is now known as kadams54 [00:25] davecheney: I dont fully see all implications but, except for one case where it it actually there for windows, it seems that we can remove/do only for linux the chmods [00:26] to me that sounds like a better solution [00:29] davecheney: Ill have a talk with fwereade tomorrow and see if he remembers why we decided to do this instead of nuke all appearances of chmod, Ill be glad to se this go (and with windows tests in place it might never come back) [00:30] perrito666: so this is a charm hook that needs to chmod a file ? [00:33] davecheney: the uses of Chmod=? [00:34] yup [00:34] i'm still digging for the why [00:38] there are around 26, for what I see almost half are tests simmulating lack of permissions or similar situation, the rest are giving more permissions to certain files, there is one case in environs/config.go that corrects a possible wrong permission on a file and the rest I would have to look [00:39] we use a lot chmod for my taste [00:42] so, looking at the existing chmod most if not all are of no consequence for windows, I worry a bit about future uses, now we actually need to care for windows workloads and if only os.Chmod would panic as file.Chmod does I would not fear that it might silently crawl under our radar [00:43] I might be just over engineering [00:44] thumper trunk looks to be in bad shape. Lots of tests failed [00:44] perrito666: i'm only interested in charm hooks that use chmod [00:44] imo there shuld be none [00:44] thumper, I am retesting those that looks like cloud failures [00:44] sinzui: ta [00:49] thumper, but ha-backup-restore is not fixed, the bug has mutated. I add the new error message https://bugs.launchpad.net/juju-core/+bug/1398837 [00:49] Bug #1398837: cannot extract configuration from backup file: "var/lib/juju/agents/machine-0/agent.conf

[00:49] * thumper groans === kadams54 is now known as kadams54-away [00:50] thumper, sinzui: I already have http://reviews.vapour.ws/r/573/ that should address that EOF issue [00:50] hurray === kadams54-away is now known as kadams54 [00:51] it just needs a review [00:56] ericsnow: I'm looking at it [00:56] thumper: thanks [00:56] ericsnow: but I don't see how your change improves anything [00:56] ericsnow: can you explain? [00:57] thumper: if we notice a dropped connection we reconnect and try again [00:58] ericsnow: a minor change then land it [00:58] thumper: k [00:59] ericsnow: http://reviews.vapour.ws/r/573/ not lgtm, yet [00:59] k [01:00] 12noon ppl [01:00] oh [01:00] the meeting is actually at 2pm [01:00] ignore me [01:00] or one pm [01:01] calendars are hard, let's go shopping [01:01] davecheney: I would really like, but most imports are closed to make people buy local for christmas [01:01] :p [01:15] davecheney: I replied to your comment [01:15] davecheney: basically, I'm pretty sure that code should stick around [01:18] axw: standup? [01:18] davecheney: but I'll do it cleaner [01:18] ok [01:18] axw: oops sorry, you're not here [01:22] davecheney: http://reviews.vapour.ws/r/531/diff/# needs a full review [01:22] thumper: on it [01:22] cheers [01:24] anastasiamac: any ETA on your blocking branch? I don't really want to put my machine branch up for review until I have that merged in === _sEBAs_ is now known as sebas5384 [01:32] thumper: today?.. [01:32] * anastasiamac fingers xrossed [01:34] davecheney: I've updated http://reviews.vapour.ws/r/573/ [01:34] * thumper crosses fingers too [01:35] don't forget to add your things to the agenda [01:35] https://docs.google.com/a/canonical.com/document/d/1eeHzbtyt_4dlKQMof-vRfplMWMrClBx32k6BFI-77MI/edit [01:35] if you don't, then I get to talk for the whole time [01:35] and you probably want to avoid that [01:36] ericsnow: ok [01:41] ericsnow: review done [01:42] i'm not happy with the specialised logic inside getBackupTargetDatabases [01:42] the DBSession interface should describe what it needs [01:42] if it needs a .Copy method [01:42] then the mock needs to implement that as well [01:43] davecheney: it has to match mgo's Session.Copy which returns *mgo.Session, making the interface method kind of goofy [01:44] davecheney: but I agree with you :) [01:44] ericsnow: perhaps this is not the right place for a mock then [01:45] however unpleasent that is [01:45] davecheney: you may be right :( [01:46] davecheney: in the meantime for the sake of opening the landing bot... [01:46] ericsnow: please raise a bug against the next milestone [01:46] davecheney: k [01:56] I need a second reviewer for http://reviews.vapour.ws/r/573/ [01:56] thumper: wallyworld_ menn0 ? [01:56] sure [02:04] my cal says we have the team meeting now, but no one is here? [02:05] ah, just my crappy connection === kadams54 is now known as kadams54-away [02:27] thumper: I finally found someone that would do an affogato en in argentina :) I only had to go 90Km away to get it [02:28] haha [02:52] davecheney: I replied to you comments [02:52] wallyworld_: I'm pretty sure the dropped session (the io.EOF) is due to the replicaset functionionality of HA [02:53] davecheney: one key change with putting the feature flags into juju/utils is that the flags themselves had to be agnostic of any particular environment variable [02:53] so it can't really return a map unless we pass the key in [02:53] which we could make a helper to do... [02:54] ericsnow: ok, np. i'm not across it enough. is only one retry enough? [02:54] wallyworld_: I just updated the patch to retry 10 times [02:54] wallyworld_: but still only for io.EOF (which mgo returns when the connection drops) [02:55] davecheney: I'm beginning to think that we should have the feature flag init in each of the main blocks [02:56] davecheney: then we wouldn't need the test for osenv [02:56] ericsnow: so why not wait until replicaset is up before doing backup? [02:56] don't we do that elsewhere? [02:57] wallyworld_: I don't know about elsewhere, but I wouldn't mind waiting until the replicaset is up [02:57] wallyworld_: wouldn't that but in the CI test script though? [02:57] s/but/be/ [02:58] our code needs to wait [02:58] if backup invoked, it needs to ensure replicaset is up before proceeding [02:58] wallyworld_: k [02:58] like i think state server might do when starting [02:58] that's IMHO [02:59] but seems like the right thing to do [02:59] wallyworld_: okay === kadams54-away is now known as kadams54 [03:00] wallyworld_: I'm not well versed in the HA stuff but what you're saying makes sense [03:00] ericsnow: nate knows all about HA etc :-) [03:00] wallyworld_: lovely :P [03:01] wallyworld_: I'm pretty sure y'all don't want to wait until Nate is back online before I get this pushed :) [03:01] I'm here :) [03:01] why would I not be here? :) [03:01] magic [03:01] ericsnow: i'd rather not alnd a bad solution [03:01] lol [03:01] haha [03:02] natefinch: wallyworld_ is suggesting that we wait for replicaset to finish coming up before running backup [03:02] natefinch: how do we test for that? [03:02] wallyworld_: agreed [03:04] ericsnow: so, iirc, it's non-trivial [03:04] natefinch: not the right answer :P [03:05] wallyworld_: I'm still going to be out for a while longer. I'll be working late tonight [03:05] axw: np at all [03:11] ericsnow: michael did a bunch of work on that relatively recently, you should talk to him in the morning [03:12] natefinch: k [03:13] ericsnow: I think there were updates to the replicaset code and/or tests in order to determine that. IIRC it was something like ping until we get something reasonable back. [03:13] natefinch: got it [03:15] natefinch: keep in mind that for backup we can do the check on the API server side, so we have access to state [03:16] ericsnow: yeah, the replicaset tests assume DB access too. There's just no actual flag saying "replicasets are up" === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 === kadams54 is now known as kadams54-away [04:37] axw: ping [04:39] https://gist.github.com/anonymous/de06b097d25d690b684f after seeing this log i'm pretty sure i'm not able to use kvm [04:39] hehe [05:04] wallyworld_: I've updated http://reviews.vapour.ws/r/573/ [05:05] ok, looking, wow, must be late for you [05:05] wallyworld_: is that what you meant about checking if HA is ready? [05:05] wallyworld_: well, I hate leaving CI blocked [05:08] ericsnow: looks ok i think, but backup doesn't seem to call WaitUntilReady() and haEnabled() always returns true [05:09] wallyworld_: isn't HA always enabled (even if not utilized), i.e. the --replset option is always passed to mongod [05:09] wallyworld_: do you think WaitUntilReady would be more appropriate than IsReady? [05:10] true, so why the haEnabled() function? and what about older environments? or are they upgraded to ha ? [05:10] i think they are upgraded [05:10] wallyworld_: the new backups only applies to 1.22+ [05:11] wallyworld_: and haEnabled gets patched to return false in the tests (since we don't use HA there) [05:11] wallyworld_: is "haEnabled" the wrong name? (perhaps "replSetEnabled"? [05:11] so then that's a bit misleading, the block of code should be extracted from create [05:12] and put inside a func [05:12] and that func should be what's patched [05:12] wallyworld_: fair enough [05:12] but isn't that what WaitUntilReady is for? [05:13] if you just call WaitUntilReady, it should all be fine, just patch WaitUntilReady? [05:13] and then IsReady doesn't need to be exported [05:15] wallyworld_: it depends on it we want backup to fail immediately if HA isn't ready or if we make users wait [05:15] oh right i see [05:15] wallyworld_: currently it fails immediately, but it sounds like you would rather we take the waiting approach [05:15] for now just do what's needed to unblock so you csn goto bed [05:16] wallyworld_: I could drop the WaitUntilReady func [05:16] k [05:16] yes, drop that for now [05:16] jut the minimum, but then come back and fix if needed [05:16] i'd just like to see the code extracted [05:16] so the haEnabled() can be dropped [05:16] k [05:16] doing it now [05:16] ty [05:25] wallyworld_: pong [05:26] axw: quick hangout? [05:26] sure [05:26] 1:1 one [05:26] ok [05:45] wallyworld_: hypothetically the openstack provider could do bad things if your environment name contains regexp meta characters [05:46] axw: do we allow that? i thought env names were constrained [05:46] to valid chars [05:46] maybe... trying to find where [05:48] wallyworld_: seems we just check that it doesn't contain "/" [05:48] oh [05:48] in that case i need to do more in the azure one also [05:48] I'll see what openstack allows for machine names... [05:52] azure is ok [05:52] "alphanumeric characters and underscores are valid in the name" [05:55] hmm, can't find any info about it on openstack... [05:57] possibly this? https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L61 [05:58] which includes . [05:59] wallyworld_: updated http://reviews.vapour.ws/r/573/ (and added tests) [06:00] looking [06:04] ericsnow: +1, but i just realised, the CI script might still fail [06:04] FWIW, I plan on following up with mfoord in the morning [06:04] wallyworld_: how so? [06:04] as it will still see an error [06:04] it will try to backup and get a "not ready, try again later" error [06:04] wallyworld_: oh, duh [06:04] as opposed to a EOF error [06:04] wallyworld_: dang it [06:05] i think we can ask that the script be changed [06:05] maybe [06:05] or else we will need that retry loop [06:05] but you go to bed, i'll follow up [06:05] wallyworld_: and change it to a CI bug rather than a core bug? [06:05] maybe, i'll have to ask [06:06] i can see both sides of the argument [06:06] wallyworld_: k, I'll get the merge started [06:06] sure, tyvm [06:06] and then we can add to it if needed to put in the wait until ready [06:06] wallyworld_: that WaitUntilReady function is still in the commit history ;) [06:07] indeed :-) [06:07] wallyworld_: because I'm sneaky like that :) [06:07] lol [06:08] wallyworld_: okay, it's running the merge CI right now [06:08] ty :-) [06:08] wallyworld_: I'll leave it in your hands (thanks!) [06:08] night night [06:22] wallyworld_: have a cheeky glass of red for me [06:22] wish i could [06:22] i'm still working [06:23] i was saying good bye to eri [06:23] c [06:23] axw: here's the gwacl branch https://code.launchpad.net/~wallyworld/gwacl/prefix-service-match/+merge/243620 [06:23] i need to modify juju to pass in a separator [06:24] looking [06:33] axw: i might do as you suggest, i can just abandon the gwacl branch [06:33] wallyworld_: I kinda wish those convenience functions in gwacl weren't there [06:34] yeah [06:34] that one in particular is trouble, obviously [06:34] indeed === fuzzy_ is now known as Fuzai [07:02] anyone else need a review ? [07:04] axw: http://reviews.vapour.ws/r/580/ [07:04] looking [07:04] davecheney: i do, but andrew can look as we've discussed befre hand [07:14] wallyworld_: done, fwiw [07:14] thanks dave :-) [07:17] wallyworld_: done also [07:17] thanks [07:18] i checked the azure doc, service names don't allow meta chars [07:18] but i'll add the quoting to be sure [07:18] wallyworld_: they don't *now*, but who's stopping them from changing that? [07:18] sure [07:27] jam1: you ok for storage meeting a bit later? [07:27] wallyworld_: yep [07:27] jam1: great [07:28] jam1: axw: i may be a minute or 2 later as i have to drive my wife to the city and am not sure of traffic [07:28] i should be back on time [07:28] okey dokey [08:00] wallyworld_: when you have a moment, can you see if this makes sense to you? https://docs.google.com/a/canonical.com/document/d/1-9ZPfdgpkj2R9mBG_tlSclGGyK3tRpMf2L4C37mzYD8/edit#bookmark=id.g0p2mahykmz [08:44] morning [09:24] morning all [09:27] dimitern: morning [09:28] morning voidspace, TheMue [09:28] dimitern: voidspace: o/ [09:39] TheMue: hiya [10:01] jam1, voidspace, standup? [10:02] dimitern: just working on a meeting, will be there soon [10:02] ok [10:02] dimitern: omw [10:05] * fwereade_ out for a bit [10:21] morning [10:33] perrito666: morning [10:33] perrito666: heya [10:35] wallyworld_: axw: are we back in https://plus.google.com/hangouts/_/canonical.com/juju-storage [10:35] ? [10:35] omw [11:38] * dimitern out for a 1h [13:02] perrito666: if you get a chance care to look at this one: http://reviews.vapour.ws/r/583/ [13:02] perrito666: you're probably more familiar with this code than others, but it's a simple change [13:02] perrito666: restores (and tests) some work by ericsnow [13:03] voidspace: looking [13:03] perrito666: thanks [13:09] voidspace: only nits not worth mentioning, LGTM, but it is not worthy that I LGTM since you will need for david tomorrow to ship it :p so look for a better source of lgtmness [13:10] perrito666: ok, cool - thanks [13:12] anyone else who fancies an easy review [13:12] http://reviews.vapour.ws/r/583/ [13:12] I'm off to lunch === jcw4 is now known as jw4 [14:45] sinzui: where are we on those blockers? [14:45] natefinch, still taking stock. I just reported https://bugs.launchpad.net/juju-core/+bug/1399229 [14:45] Bug #1399229: win client cannot get status after bootstrap

[14:53] sinzui: can we get a --debug run on that? [14:53] perrito666, I am trying. I need to setup another win machine first [14:55] natefinch, I suspect ha is brittle, but we have so many hot issue, I am going to get you better info for the current bugs than investigate ha [14:58] natefinch, the ha issues are connect shutdown/refused talking to the api server, which is the current problem with the backup-restore test, so maybe we already have a bug tracking the problem [15:02] perrito666, damn it. We cannot get a debug with a matching server because the testing streams are for 1.21. I will try a bootstrap anyway and hope for a reproduction [15:02] sinzui: hold, I have both windows and a fake stream [15:02] tell me how to try that [15:03] perrito666, The test is just a bootstrap into aws, proof that it can bring up and talk to a state-server [15:03] sinzui: ok, that is trunk? [15:03] * perrito666 goes again [15:03] perrito666, yes [15:03] ok, firing up windows [15:11] can you believe it? compiling juju in your machine and inside a vm is a bad idea [15:15] dimitern, do we really need to backport bug 1397376 to 1.20? the stakeholders are pointing to 1.20 as an example that works for them [15:15] Bug #1397376: maas provider: 1.21b3 removes ip from api-endpoints

[15:22] ok sinzui please dont hate me for the stupid thing I am about to ask [15:22] where should I store environments.yaml in windows? [15:22] sinzui, I wasn't sure, let me check if the logic differs in 1.20 [15:23] perrito666, /users//.juju/environments.yaml and you need a .ssh/id_rsa [15:23] perrito666: there should be an environmental variable for /users/.... forget what it is [15:23] perrito666: perhaps %HOME% [15:24] sinzui, confirmed - 1.20 is not affected by the same address ordering issue [15:25] \o/ [15:26] asdkjhaskdjhna ksdhjask windows will not allow me to name something .juju from the gui [15:27] oh you have to name it .juju. [15:27] and windows removes the dot at the end [15:28] perrito666: you need to turn off the "hide extensions" thingy, I think... I've never had a problem using wacky extensions in windows before. [15:29] natefinch: the issue was not with the extension is that the file began with . [15:29] so, no, windows is not looking for %HOME%/.juju [15:30] oh yeah, you can't make a file in the UI that starts with a ..... sorry [15:30] er with a . that is [15:30] perrito666: %USERPROFILE%\.juju\environments.yaml ? [15:30] %UserProfile% [15:30] looking [15:30] I thought we made juju home on windows just "Juju" [15:31] natefinch: let see [15:32] perrito666: yeah it's %APPDATA%/Juju [15:33] dimitern, you still around? [15:34] alexisb: my chrome just froze, will be back on the call in a minute.... [15:34] bootstraping [15:34] sinzui: jw4 katco natefinch tx for your help [15:34] alexisb, yes [15:34] perrito666: anytime perrito666 [15:35] on the cross team call, was curious on status on the api endpoints bugs [15:45] natefinch, axw targeted https://bugs.launchpad.net/juju-core/+bug/1388860 to 1.20.14. It is a backport of a fix. He suggests it because there is another bug reporting that the issue does indeed affect the 1.20 series. Can you get someone to look into the backport now, and if it is ugly, then maybe we shouldn't...just go head with the release. [15:45] Bug #1388860: ec2 says agent-state-info: 'cannot run instances: No default subnet for availability zone: ''us-east-1e''. (InvalidInput)'

[15:45]

[15:46] alexisb, i'm working on a fix now [15:46] alexisb, it turned out to be more complicated to test than to fix :) - I should be ready later today or tomorrow morning [15:47] sinzui: natefinch http://pastebin.ubuntu.com/9368819/ [15:47] perrito666, try again, that is aws's sucky mirrors [15:48] sinzui: I am about ot [15:48] to [15:48] perrito666, oh.. [15:48] perrito666, you can tell juju to not do updates and upgrades! [15:49] * perrito666 had to step down a moment to go order food, which requires me standing on a bench on the backyard, single place in the house with cell reception [15:49] perrito666, you can improve your chances if success: [15:49] enable-os-upgrade: false [15:49] enable-os-refresh-update: false [15:49] ^ we do that with canonistack because its network is unreliable [15:50] sinzui: is the test running that way? [15:50] dimitern, awesome, thank you for turning that around so quickly [15:50] man, it really sucks we're still using launchpad for bugs... it makes everything so much more manual [15:50] I would like to stay as close as possible to the tests [15:50] sinzui: do you happen to have a link to the PR that fixed https://bugs.launchpad.net/juju-core/+bug/1388860 for 1.21? [15:50] Bug #1388860: ec2 says agent-state-info: 'cannot run instances: No default subnet for availability zone: ''us-east-1e''. (InvalidInput)'

[15:50]

[15:51] alexisb, no worries [15:51] perrito666, no, but I think it is somewhat irrelevant because we have fresh images in aws, charms need updated and upgrades, but this failure is about talking to the state server [15:51] sinzui: running second bootstrap with your options added [15:53] natefinch, I do not, I will look [15:53] I have a fix for issue 1398837 [15:53] waiting for a review [15:53] sinzui: I found it [15:54] bug 1398837 [15:54] Bug #1398837: cannot extract configuration from backup file: "var/lib/juju/agents/machine-0/agent.conf