[06:05] davecheney: mornin' boss [06:07] rog: ahoy! [06:08] davecheney: it's funny, i saw gustavo's response to 6304104 but not the original proposal email [06:08] davecheney: am looking at it now [06:12] rog: ta [06:18] davecheney: hmm [06:19] davecheney: i'm wondering if the interface shenanigans are worth it [06:19] davecheney: how about just using a different type for each operation, e.g. http://paste.ubuntu.com/1052155/ [06:20] rog: what is the type returned on the channel ? [06:20] interface{} ? [06:20] davecheney: we could put a private method on it [06:20] davecheney: then it could still be Operation [06:21] switch o := o.(type) { case ... } ? [06:21] davecheney: yeah, except we never need to do that AFAICS. i've only done a cursory look mind [06:21] davecheney: i think we always know what operation we're expecting [06:22] c.Assert(op, Equals, dummy.OpBootstrap{env, inst}) [06:22] davecheney: s/OpBootstrap/OpStartInstance/ [06:22] rog: sure [06:23] and that wouldn't work of course because equality isn't defined on Instances [06:23] and but you get the idea. [06:23] it means we can lost all the OperationKind stuff and the table. so we probably save lines overall. [06:23] s/lost/lose/ [06:27] rog: nice, I always wanted to switch on the type, rather than a synthetic kind field [06:28] davecheney: something like this perhaps, if we want an Operation type. http://paste.ubuntu.com/1052161/ [06:29] davecheney: cool [06:29] davecheney: oops, isOperation should be defined on OpEnv [06:30] davecheney: although... that won't print so nicely [06:30] davecheney: scratch that last suggestion [06:33] davecheney: could go with something like this: http://paste.ubuntu.com/1052167/; or just use interface{}. [06:34] rog: feels a bit forced [06:34] davecheney: the Operation type? [06:34] all those empty methods to satisfy the interface [06:35] davecheney: yeah. there is precedent (go/ast) but i think it's probably best to use interface{} [06:35] davecheney: it's not a public-facing API after all [06:55] moin. [06:56] Moin [06:57] * TheMue wanted to start later after doing some sports but the club is closed this morning. It seems the morning shift has overslept. [07:22] Aram: hiya [07:23] TheMue: hiya [07:26] rog: Hi, just had to restart after firmware update. [08:36] I like it when I'm reading an IRC conversation from some hours ago and thinking "I'd want to say *this*", then the conversation ends up at exactly that conclusion. [10:34] i'm off into town to fetch my bike and buy a replacement jacket for the one i left behind in Oakland (:-[). back in a couple of hours. [10:42] bokes are great [10:42] bikes [11:56] Aram, any thoughts on the loss of atomicity and ordering with mongodb (my reply on your thread on list)? [11:57] ie. given multiple collections to insert for observation/notification [11:58] there's no loss of ordering. [11:58] atomicity may be a problem. [11:58] but ordering should not be. [11:58] Aram, lacking atomicity, with concurrent ops, how can retain ordering [11:59] ie. given logical ops A and B and their respective inserts, they could go A1, B1, B2, A2 [12:00] there's an atomic sequencing mechanism that generates incrementing integers, and these integers can be used to query for data in a particular order. [12:02] hazmat: basically the sequencing mechanism is atomic because we make it so, it's a single document. [12:06] Aram, combined with locking? what's a sequence flow look like.. [12:06] of a logical op [12:08] get sequence value, lock observation collection, atomic update doc using sequence value as condition, insert into observation collection, unlock? [12:39] n reset by peer) [12:39] howdy, anyone seen mark [12:39] meh [12:40] dumb paste [12:40] [12:40] hazmat: there's no locking in mongodb. [12:40] hazmat: I can't answer your question because we haven't decided yet :-). [12:40] hazmat: I only made sure we can implement a mechanism if/when we need it, whatever that mechanism might be. [12:40] hazmat: for example we can implement full transactions using multi-phase commits, if we consider it necessary, but we hope we would not need such a heavy weight mechanism. [12:40] hazmat: I can tell you the technical details of various mechanisms we might use, but I can't tell you which one will use at this early stage. probably gustavo has some better insight. [12:41] Aram, right re locking i meant app locking not native primitives, and thanks for clarifying the current state [13:11] hey juju folks... for Juju on openstack, is swift required? [13:12] hrm... maybe wrong channel ;-) [13:14] i'll check in #juju [13:36] hi niemeyer [13:39] Yo! [13:47] niemeyer: hiya [13:49] * Aram is amused by people who censor profanity words on the internet. [13:52] Aram: Yeah, seriously [13:56] * TheMue moved on the veranda to continue work there. [13:56] niemeyer: Hi. [13:58] niemeyer: After a talk yesterday I started to read the mail thread and CL regarding format: 2. [13:59] niemeyer: A talk to fwereade [14:00] niemeyer: Shall I continue with that topic or with non-relation topics in state? Like e.g. hooks. [14:00] TheMue: Hmm [14:01] TheMue: Hooks are not really in state, and fwereade has already been moving towards having them working [14:01] TheMue: What else is missing from state? [14:02] niemeyer: Maybe I said it wrong. In Py in state there are hook, firewall, auth and security. [14:03] niemeyer: Those are still open. [14:04] niemeyer: I wanted to continue with relation but fwereade is doing there a lot and asked me to take another topic. [14:04] niemeyer: That's why he passed me infos about format 2. [14:05] TheMue: Yeah, handling the change to format 2 sounds like a good idea [14:05] TheMue: It's just out of state, AIUI [14:05] TheMue: Which is why I asked if you're leaving something else behind [14:09] niemeyer: Those four files above and relations, then state should have been done. [14:10] niemeyer: So far I compared it file by file and type by type. [14:10] TheMue: That's fantastic [14:10] niemeyer: But I could do a usage scan (Where is state used today?) and see what functionality is missing in the Go port. [14:11] TheMue: So it sounds like it's pretty much done indeed [14:11] TheMue: The last two are not going to be ported as we'll want to do something else [14:11] TheMue: Hooks, as I mentioned, is very close to the implementation of the unit agent, and shouldn't be done in isolation [14:11] niemeyer: Yes, you once said it when I started with auth. [14:11] TheMue: Firewall may be the more interesting bit [14:12] my T410 is overheating sometimes, had the same problem with my T400. [14:12] niemeyer: So I could do firewall first. [14:13] TheMue: I'm having a look at it [14:13] niemeyer: Just to get the state done. [14:13] Aram: I haven't been having it with T410, but I had it all the time on the T60 [14:14] TheMue: Hmmm [14:15] TheMue: We have to do firewall, but it shouldn't look like what is there right now [14:15] TheMue: It has a *provider* in the constructor, which makes no sense for a state package [14:17] niemeyer: Yes, just scanned the code quickly. It's using the state, but it is no state. [14:17] niemeyer: Interesting. [14:17] TheMue: It is state to some degree [14:18] TheMue: It's just mixed up [14:18] TheMue: Most of that file is really preparing for this: [14:18] for port, proto in to_open: [14:18] yield self.provider.open_port(machine, machine_id, port, proto) [14:18] for port, proto in to_close: [14:18] yield self.provider.close_port(machine, machine_id, port, proto) [14:19] TheMue: We have to invert the relationship as usual for watchers [14:20] TheMue: This is a type along the lines of PortChanges { Opened Closed } [14:20] TheMue: and Machine.WatchOpenPorts [14:20] niemeyer: IC [14:21] TheMue: Or really, Machine.WatchPorts [14:24] niemeyer: As today in Unit.WatchPorts [14:25] Hmm [14:26] Why do we watch ports for both units and machines [14:26] * niemeyer reads [14:27] niemeyer: Today it's only in Unit. [14:27] TheMue: I mean why the original code does that [14:28] niemeyer: Oh [14:28] TheMue: Ah, of course [14:28] TheMue: One is based on the other [14:32] niemeyer: do we want the machine agent to do the port opening and closing? it seems that it's currently the provisioning agent's job [14:32] niemeyer: hmm, i can see why we might not want that, as it currently requires ec2 creds [14:33] rog: We do want that, but this is a refactoring step we should do when we're done with the port [14:33] niemeyer: ok, so PA to start with [14:33] Yeah [14:40] * TheMue looks how firewall is used today. [14:50] TheMue, only in the provisioning agent [14:50] TheMue, it doesn't really belong in the state package [14:50] hazmat: Yes, seen it. [14:57] Fresh cherry cake by my wife, bad for my diet. *sigh* [15:12] niemeyer: ping [15:12] TheMue: i want some, please [15:12] rog: Heya [15:12] * TheMue dcc's rog some cake. [15:13] niemeyer: just looking at starting the PA [15:13] rog: Ok [15:13] niemeyer: there's a little awkwardness that i thought it might be worth discussing [15:13] rog: Cool. Can we cover this after lunch? [15:13] niemeyer: ok [15:13] It's food-o-time here [15:14] niemeyer: enjoy! ping me after lunch... [15:14] rog: Thanks, will do [16:18] rog: Yo, so what's up? [16:18] niemeyer: so... [16:18] niemeyer: we want the provisioning agent to know the machine address to give when starting new machines [16:19] niemeyer: so i thought we can easily use the metadata service to pass the address into the provisioning agent [16:19] niemeyer: same kind of thing we do with the machine agent [16:19] niemeyer: or initzk anyway [16:19] niemeyer: with instanceIdAccessor [16:20] niemeyer: i added hostNameAccessor [16:21] rog: How is it done today? [16:22] Aram: Just reviewed https://codereview.appspot.com/6304099 [16:22] thanks [16:22] niemeyer: good question. i should have looked. one mo [16:23] rog: We should look how it is currently done, because this problem has been solved in a way that works for several providers [16:23] rog: Maybe it's not optimal, but we should be sure we're improving on it if that are any changes to it [16:24] s/that/there [16:24] niemeyer: I did the change, can I just submit now? [16:24] Aram: Yeah [16:24] good [16:25] Thanks! Woohay first mstate branch going in! [16:26] * TheMue cheers [16:27] TheMue: I think we'll share a flight returning from the sprint. [16:29] Aram: Oh, good you mentioned it, I've got to change the wiki. BTS found a cheaper flight with KLM. So sorry, we do not share it. *sigh* [16:29] mramm, do you a percentage on travel arrangements ? ;-) [16:29] hmm, maybe I should ask for different flights, I know there are more flights than BTS told me. [16:30] hazmat: so far it's just pro-bono work [16:32] hazmat: Hehe, that question shocked him. ;) [16:32] niemeyer: ah! ok, problem solved. and dave's problem too. [16:33] niemeyer: i should've thought of it anyway [16:34] rog: So what's the deal? [16:35] niemeyer: the PA calls StateInfo [16:35] niemeyer: easy peasy [16:36] rog: Aha, ok [16:43] rog, Aram, TheMue: Folks, I think now is a good moment to do the move lp.net/juju-core/juju => lp.net/juju-core [16:43] Do you have any pending work that is about to be merged? [16:43] niemeyer: From my side +1. [16:43] * rog looks [16:43] +1. [16:44] niemeyer: no, that's fine [16:44] I'll give Dave a hand and see if any of his branches are mergeable and submit them myself now [16:44] fwereade will have to update, unfortunately [16:45] But it's almost as low a hit as we can manage I think [16:53] Running tests even for the dummy package is taking an awful lot of time. [16:54] s/package/provider/ [16:56] niemeyer: i have an idea about how we might be able to speed up the tests. the slowness is almost all caused by the zookeeper startup time [16:57] There seems to be something wrong.. [16:57] niemeyer: we could start one zk server on demand and use the zk chroot functionality so that it could be used across all tests in all packages. [16:57] START: /home/niemeyer/src/launchpad.net/juju-core/juju/environs/jujutest/test.go:96: LiveTests.SetUpTest [16:57] This is blocking [16:57] Sorry, no, it's not [16:59] * rog realises he's broken the build [17:01] rog: ? [17:01] niemeyer: my recent container "fix" [17:01] niemeyer: i could've sworn i ran the test, but evidently i didn't [17:04] rog: Hmm.. we'll probably need some kind of submission mechanism soon, hooked with lbox [17:04] rog: Btw, this line: [17:04] niemeyer: i think lbox should probably run all the test on the merged branch before actually submitting it [17:04] err := t.Env.Bootstrap(true) [17:04] rog: Is taking 36 seconds to execute [17:04] rog: With the *dummy* env [17:05] niemeyer: i suspect zookeeper, but i may be wrong. [17:06] rog: It may be zk.. we call state.Initialize there [17:06] * niemeyer checks [17:07] rog: PutTools [17:08] niemeyer: ah, it's taking ages to build the tools, i suppose [17:08] niemeyer: because nothing is installed [17:08] niemeyer: because you've just changed everything perhaps? [17:08] Could be [17:08] * niemeyer verifies [17:12] % time go install launchpad.net/juju-core/juju/cmd/... [17:12] go install launchpad.net/juju-core/juju/cmd/... 24.71s user 7.27s system 99% cpu 32.157 total [17:12] rog: ^ [17:12] rog: Now, the question is why is it taking such a long time *every time* [17:13] Clearly something is not ok [17:14] niemeyer: just using vanilla "go install" it always takes that long for you? [17:14] rog: What's vanilla go install? [17:14] niemeyer: just go install on the command line, no other stuff going on like GOBIN etc [17:15] niemeyer: for me it takes 1.12s the second time i run it [17:15] rog: Sorry, I still don't get it. The command above is go install on the command line. [17:15] niemeyer: right. and you ran it twice and got the same time each time. [17:15] niemeyer: i've no idea what's taking it so long there then [17:16] go install -x isn't even printing results [17:16] I'll rebuild my go tree [17:16] % time go install -x launchpad.net/juju-core/juju/cmd/... [17:16] WORK=/tmp/go-build238159301 [17:16] go install -x launchpad.net/juju-core/juju/cmd/... 24.93s user 7.07s system 99% cpu 32.182 total [17:16] WTF.. it's taking 32 seconds before it even does anything [17:17] niemeyer: if go install -x isn't printing results, it means it's doing nothing, yeah. [17:17] system 99% [17:17] try a syscall trace perhaps [17:18] I'll try rebuilding Go first, just to make sure I don't have something funky there [17:18] niemeyer: you get 32s before it even prints the WORK line? [17:18] rog: Yeah [17:18] rog: If I can reproduce with a rebuild I'll see where more exactly [17:19] niemeyer: at least this time, i think we can say the blame doesn't fall on the juju testing :-) [17:19] rog: Yeah! :) [17:20] niemeyer: can i propose/submit a quick fix for the container brokenness before you proceed? [17:20] rog: Hmm.. we could add a flag to lbox submit such as "--test-command" [17:21] rog: Yeah, definitely [17:21] niemeyer: that sounds like a good plan [17:21] rog: We can then have it in .lbox so we don't have to enter it every time [17:22] niemeyer: yeah. but it's important that the tests run on the merged version [17:22] rog: Of course [17:22] Still broken [17:23] Let's see where this time is being spent [17:23] Holy crap.. it's doing an awful lot of things [17:24] I suspect someone added some logic without much consideration for what else is under $GOPATH/src/, as it's walking the whole tree :-( [17:25] niemeyer: maybe it's not so clever about prefixes of ... patterns [17:26] niemeyer: but walking the tree doesn't take long, surely? [17:26] rog: Surely does [17:26] rog: If the tree is large [17:26] % time find $GOPATH/src > /dev/null [17:26] 0.01user 0.04system 0:00.05elapsed 98PU (0avgtext+0avgdata 4880maxresident)k [17:26] not for me :-) [17:27] niemeyer: presumably it's not reading all the files in the tree? [17:27] rog: Presumably it's spending 32 seconds doing that [17:27] niemeyer: how long does it take you to run that command? [17:28] [niemeyer@gopher ~/src]% time find > /dev/null [17:28] find > /dev/null 1.86s user 5.89s system 25% cpu 30.253 total [17:30] niemeyer: gosh. that is really really big. or you've got a slow disk/network drive. [17:30] rog: I have a lot of stuff in my src/ [17:30] rog: and an SSD [17:30] niemeyer: ah! you have GOPATH=$HOME ? [17:31] rog: Yep [17:31] * rog is glad he didn't do that [17:31] rog: Heh [17:31] niemeyer: the go tool should be more intelligent [17:31] rog: It should be less stupid, rather [17:32] Why on earth is it walking everything like that [17:32] niemeyer: because it's walking the directories then matching against the pattern [17:32] niemeyer: rather than walking the constant prefix of the pattern [17:33] niemeyer: at least that's my assumption [17:33] rog: OMG, I hope you're wrong [17:34] niemeyer: yeah, that's exactly what it's doing [17:34] It's like, "Oh you want everything under /home/joe? Okay! Let's start over in /var!" [17:34] filepath.Walk(src, func(path string, fi os.FileInfo, err error) error { [17:35] and src is everything [17:35] niemeyer: to be fair it was done quickly, and correctness is more important than quickness. [17:35] niemeyer: but it will be easy to fix. [17:35] rog: Yeah, I'll fix that now [17:36] niemeyer: i'd make matchPattern return the longest path prefix as well as the match function [17:36] rog: Doing this right is a no-brainer, to be honest.. it was just a brain fart from someone [17:38] rog: Actually, they've done in right elsewhere there [17:38] niemeyer: really? [17:38] rog: Yep.. see matchPackagesInFS [17:39] niemeyer: so it does. [18:01] niemeyer: i'm off for the evening. hope the switchover goes ok... [18:01] 'night al [18:01] l [18:01] rog: Thanks, have you seen the LGTM? [18:03] niemeyer: just making sure with a final test, then i'll submit [18:04] rog: Cheers [18:04] niemeyer: re: https://codereview.appspot.com/6304104/ i think we might have decided to go with a different, slightly simpler approach. [18:04] i get this sporadic failure: [18:04] FAIL: presence_test.go:328: PresenceSuite.TestWaitAlive [18:04] presence_test.go:348: [18:04] c.Assert(err, IsNil) [18:04] ... value *errors.errorString = &errors.errorString{s:"presence: still not alive after timeout"} ("presence: still not alive after timeout") [18:04] rog: The approach there looks pretty straightforward and clean to me [18:05] niemeyer: this was my suggestion to dave this morning: http://paste.ubuntu.com/1052155/ [18:06] rog: I mentioned it could stay simpler, but I also don't see an issue with doing it now, given Dave actually wrote it [18:06] niemeyer: which he seemed to quite like [18:06] niemeyer: in some similar form anyway [18:07] rog: That sounds fine too [18:07] niemeyer: it means we can get rid of operationKinds, the string table etc [18:09] niemeyer: https://codereview.appspot.com/6332044 submitted. build fixed. [18:10] rog: Thanks! [18:11] rog: Can you please add a comment to the review then? [18:11] * rog really is off now. see you tomorrow (my last day before i'm away for two weeks BTW) [18:14] Nevermind.. copied the IRC conversation on the proposal [18:20] niemeyer: i added a comment [18:21] niemeyer: ah, but replied to the wrong email! [18:21] darn [18:23] rog: np === imbrando1 is now known as imbrandon [20:23] So, fixed go problem.. can actually test code in a non-ridiculous timeframe now === robbiew1 is now known as robbiew [20:47] juju-core/juju => juju-core move is complete! [20:47] Please see juju-dev@ list for details. [20:48] * niemeyer steps out [21:46] niemeyer: :)