[06:05] <rog> davecheney: mornin' boss
[06:07] <davecheney> rog: ahoy!
[06:08] <rog> davecheney: it's funny, i saw gustavo's response to 6304104 but not the original proposal email
[06:08] <rog> davecheney: am looking at it now
[06:12] <davecheney> rog: ta
[06:18] <rog> davecheney: hmm
[06:19] <rog> davecheney: i'm wondering if the interface shenanigans are worth it
[06:19] <rog> davecheney: how about just using a different type for each operation, e.g. http://paste.ubuntu.com/1052155/
[06:20] <davecheney> rog: what is the type returned on the channel ?
[06:20] <davecheney> interface{} ?
[06:20] <rog> davecheney: we could put a private method on it
[06:20] <rog> davecheney: then it could still be Operation
[06:21] <davecheney> switch o := o.(type) { case ... } ?
[06:21] <rog> davecheney: yeah, except we never need to do that AFAICS. i've only done a cursory look mind
[06:21] <rog> davecheney: i think we always know what operation we're expecting
[06:22] <rog> c.Assert(op, Equals, dummy.OpBootstrap{env, inst})
[06:22] <rog> davecheney: s/OpBootstrap/OpStartInstance/
[06:22] <davecheney> rog: sure
[06:23] <rog> and that wouldn't work of course because equality isn't defined on Instances
[06:23] <rog> and but you get the idea.
[06:23] <rog> it means we can lost all the OperationKind stuff and the table. so we probably save lines overall.
[06:23] <rog> s/lost/lose/
[06:27] <davecheney> rog: nice, I always wanted to switch on the type, rather than a synthetic kind field
[06:28] <rog> davecheney: something like this perhaps, if we want an Operation type. http://paste.ubuntu.com/1052161/
[06:29] <rog> davecheney: cool
[06:29] <rog> davecheney: oops, isOperation should be defined on OpEnv
[06:30] <rog> davecheney: although... that won't print so nicely
[06:30] <rog> davecheney: scratch that last suggestion
[06:33] <rog> davecheney: could go with something like this: http://paste.ubuntu.com/1052167/; or just use interface{}.
[06:34] <davecheney> rog: feels a bit forced
[06:34] <rog> davecheney: the Operation type?
[06:34] <davecheney> all those empty methods to satisfy the interface
[06:35] <rog> davecheney: yeah. there is precedent (go/ast) but i think it's probably best to use interface{}
[06:35] <rog> davecheney: it's not a public-facing API after all
[06:55] <Aram> moin.
[06:56] <TheMue> Moin
[06:57]  * TheMue wanted to start later after doing some sports but the club is closed this morning. It seems the morning shift has overslept.
[07:22] <rog> Aram: hiya
[07:23] <rog> TheMue: hiya
[07:26] <TheMue> rog: Hi, just had to restart after firmware update.
[08:36] <rog> I like it when I'm reading an IRC conversation from some hours ago and thinking "I'd want to say *this*", then the conversation ends up at exactly that conclusion.
[10:34] <rog> i'm off into town to fetch my bike and buy a replacement jacket for the one i left behind in Oakland (:-[). back in a couple of hours.
[10:42] <Aram> bokes are great
[10:42] <Aram> bikes
[11:56] <hazmat> Aram, any thoughts on the loss of atomicity and ordering with mongodb (my reply on your thread on list)?
[11:57] <hazmat> ie. given multiple collections to insert for observation/notification
[11:58] <Aram> there's no loss of ordering.
[11:58] <Aram> atomicity may be a problem.
[11:58] <Aram> but ordering should not be.
[11:58] <hazmat> Aram, lacking atomicity, with concurrent ops, how can retain ordering
[11:59] <hazmat> ie. given logical ops A and B and their respective inserts, they could go A1, B1, B2, A2
[12:00] <Aram> there's an atomic sequencing mechanism that generates incrementing integers, and these integers can be used to query for data in a particular order.
[12:02] <Aram> hazmat: basically the sequencing mechanism is atomic because we make it so, it's a single document.
[12:06] <hazmat> Aram, combined with locking? what's a sequence flow look like..
[12:06] <hazmat> of a logical op
[12:08] <hazmat> get sequence value, lock observation collection, atomic update doc using sequence value as condition, insert into observation collection, unlock?
[12:39] <Aram> n reset by peer)
 howdy, anyone seen mark
[12:39] <Aram> meh
[12:40] <Aram> dumb paste
[12:40] <Aram>  
[12:40] <Aram> hazmat: there's no locking in mongodb.
[12:40] <Aram> hazmat: I can't answer your question because we haven't decided yet :-).
[12:40] <Aram> hazmat: I only made sure we can implement a mechanism if/when we need it, whatever that mechanism might be.
[12:40] <Aram> hazmat: for example we can implement full transactions using multi-phase commits, if we consider it necessary, but we hope we would not need such a heavy weight mechanism.
[12:40] <Aram> hazmat: I can tell you the technical details of various mechanisms we might use, but I can't tell you which one will use at this early stage. probably gustavo has some better insight.
[12:41] <hazmat> Aram, right re locking i meant app locking not native primitives, and thanks for clarifying the current state
[13:11] <marrusl> hey juju folks...  for Juju on openstack, is swift required?
[13:12] <marrusl> hrm... maybe wrong channel ;-)
[13:14] <marrusl> i'll check in #juju
[13:36] <Aram> hi niemeyer
[13:39] <niemeyer> Yo!
[13:47] <rog> niemeyer: hiya
[13:49]  * Aram is amused by people who censor profanity words on the internet.
[13:52] <niemeyer> Aram: Yeah, seriously
[13:56]  * TheMue moved on the veranda to continue work there.
[13:56] <TheMue> niemeyer: Hi.
[13:58] <TheMue> niemeyer: After a talk yesterday I started to read the mail thread and CL regarding format: 2.
[13:59] <TheMue> niemeyer: A talk to fwereade
[14:00] <TheMue> niemeyer: Shall I continue with that topic or with non-relation topics in state? Like e.g. hooks.
[14:00] <niemeyer> TheMue: Hmm
[14:01] <niemeyer> TheMue: Hooks are not really in state, and fwereade has already been moving towards having them working
[14:01] <niemeyer> TheMue: What else is missing from state?
[14:02] <TheMue> niemeyer: Maybe I said it wrong. In Py in state there are hook, firewall, auth and security.
[14:03] <TheMue> niemeyer: Those are still open.
[14:04] <TheMue> niemeyer: I wanted to continue with relation but fwereade is doing there a lot and asked me to take another topic.
[14:04] <TheMue> niemeyer: That's why he passed me infos about format 2.
[14:05] <niemeyer> TheMue: Yeah, handling the change to format 2 sounds like a good  idea
[14:05] <niemeyer> TheMue: It's just out of state, AIUI
[14:05] <niemeyer> TheMue: Which is why I asked if you're leaving something else behind
[14:09] <TheMue> niemeyer: Those four files above and relations, then state should have been done.
[14:10] <TheMue> niemeyer: So far I compared it file by file and type by type.
[14:10] <niemeyer> TheMue: That's fantastic
[14:10] <TheMue> niemeyer: But I could do a usage scan (Where is state used today?) and see what functionality is missing in the Go port.
[14:11] <niemeyer> TheMue: So it sounds like it's pretty much done indeed
[14:11] <niemeyer> TheMue: The last two are not going to be ported as we'll want to do something else
[14:11] <niemeyer> TheMue: Hooks, as I mentioned, is very close to the implementation of the unit agent, and shouldn't be done in isolation
[14:11] <TheMue> niemeyer: Yes, you once said it when I started with auth.
[14:11] <niemeyer> TheMue: Firewall may be the more interesting bit
[14:12] <Aram> my T410 is overheating sometimes, had the same problem with my T400.
[14:12] <TheMue> niemeyer: So I could do firewall first.
[14:13] <niemeyer> TheMue: I'm having a look at it
[14:13] <TheMue> niemeyer: Just to get the state done.
[14:13] <niemeyer> Aram: I haven't been having it with T410, but I had it all the time on the T60
[14:14] <niemeyer> TheMue: Hmmm
[14:15] <niemeyer> TheMue: We have to do firewall, but it shouldn't look like what is there right now
[14:15] <niemeyer> TheMue: It has a *provider* in the constructor, which makes no sense for a state package
[14:17] <TheMue> niemeyer: Yes, just scanned the code quickly. It's using the state, but it is no state.
[14:17] <TheMue> niemeyer: Interesting.
[14:17] <niemeyer> TheMue: It is state to some degree
[14:18] <niemeyer> TheMue: It's just mixed up
[14:18] <niemeyer> TheMue: Most of that file is really preparing for this:
[14:18] <niemeyer>             for port, proto in to_open:
[14:18] <niemeyer>                 yield self.provider.open_port(machine, machine_id, port, proto)
[14:18] <niemeyer>             for port, proto in to_close:
[14:18] <niemeyer>                 yield self.provider.close_port(machine, machine_id, port, proto)
[14:19] <niemeyer> TheMue: We have to invert the relationship as usual for watchers
[14:20] <niemeyer> TheMue: This is a type along the lines of PortChanges { Opened Closed }
[14:20] <niemeyer> TheMue: and Machine.WatchOpenPorts
[14:20] <TheMue> niemeyer: IC
[14:21] <niemeyer> TheMue: Or really, Machine.WatchPorts
[14:24] <TheMue> niemeyer: As today in Unit.WatchPorts
[14:25] <niemeyer> Hmm
[14:26] <niemeyer> Why do we watch ports for both units and machines
[14:26]  * niemeyer reads
[14:27] <TheMue> niemeyer: Today it's only in Unit.
[14:27] <niemeyer> TheMue: I mean why the original code does that
[14:28] <TheMue> niemeyer: Oh
[14:28] <niemeyer> TheMue: Ah, of course
[14:28] <niemeyer> TheMue: One is based on the other
[14:32] <rog> niemeyer: do we want the machine agent to do the port opening and closing? it seems that it's currently the provisioning agent's job
[14:32] <rog> niemeyer: hmm, i can see why we might not want that, as it currently requires ec2 creds
[14:33] <niemeyer> rog: We do want that, but this is a refactoring step we should do when we're done with the port
[14:33] <rog> niemeyer: ok, so PA to start with
[14:33] <niemeyer> Yeah
[14:40]  * TheMue looks how firewall is used today.
[14:50] <hazmat> TheMue, only in the provisioning agent
[14:50] <hazmat> TheMue, it doesn't really belong in the state package
[14:50] <TheMue> hazmat: Yes, seen it.
[14:57] <TheMue> Fresh cherry cake by my wife, bad for my diet. *sigh*
[15:12] <rog> niemeyer: ping
[15:12] <rog> TheMue: i want some, please
[15:12] <niemeyer> rog: Heya
[15:12]  * TheMue dcc's rog some cake.
[15:13] <rog> niemeyer: just looking at starting the PA
[15:13] <niemeyer> rog: Ok
[15:13] <rog> niemeyer: there's a little awkwardness that i thought it might be worth discussing
[15:13] <niemeyer> rog: Cool. Can we cover this after lunch?
[15:13] <rog> niemeyer: ok
[15:13] <niemeyer> It's food-o-time here
[15:14] <rog> niemeyer: enjoy! ping me after lunch...
[15:14] <niemeyer> rog: Thanks, will do
[16:18] <niemeyer> rog: Yo, so what's up?
[16:18] <rog> niemeyer: so...
[16:18] <rog> niemeyer: we want the provisioning agent to know the machine address to give when starting new machines
[16:19] <rog> niemeyer: so i thought we can easily use the metadata service to pass the address into the provisioning agent
[16:19] <rog> niemeyer: same kind of thing we do with the machine agent
[16:19] <rog> niemeyer: or initzk anyway
[16:19] <rog> niemeyer: with instanceIdAccessor
[16:20] <rog> niemeyer: i added hostNameAccessor
[16:21] <niemeyer> rog: How is it done today?
[16:22] <niemeyer> Aram: Just reviewed https://codereview.appspot.com/6304099
[16:22] <Aram> thanks
[16:22] <rog> niemeyer: good question. i should have looked. one mo
[16:23] <niemeyer> rog: We should look how it is currently done, because this problem has been solved in a way that works for several providers
[16:23] <niemeyer> rog: Maybe it's not optimal, but we should be sure we're improving on it if that are any changes to it
[16:24] <niemeyer> s/that/there
[16:24] <Aram> niemeyer: I did the change, can I just submit now?
[16:24] <niemeyer> Aram: Yeah
[16:24] <Aram> good
[16:25] <niemeyer> Thanks! Woohay first mstate branch going in!
[16:26]  * TheMue cheers
[16:27] <Aram> TheMue: I think we'll share a flight  returning from the sprint.
[16:29] <TheMue> Aram: Oh, good you mentioned it, I've got to change the wiki. BTS found a cheaper flight with KLM. So sorry, we do not share it. *sigh*
[16:29] <hazmat> mramm, do you a percentage on travel arrangements ? ;-)
[16:29] <Aram> hmm, maybe I should ask for different flights, I know there are more flights than BTS told me.
[16:30] <mramm> hazmat: so far it's just pro-bono work
[16:32] <TheMue> hazmat: Hehe, that question shocked him. ;)
[16:32] <rog> niemeyer: ah! ok, problem solved. and dave's problem too.
[16:33] <rog> niemeyer: i should've thought of it anyway
[16:34] <niemeyer> rog: So what's the deal?
[16:35] <rog> niemeyer: the PA calls StateInfo
[16:35] <rog> niemeyer: easy peasy
[16:36] <niemeyer> rog: Aha, ok
[16:43] <niemeyer> rog, Aram, TheMue: Folks, I think now is a good moment to do the move lp.net/juju-core/juju => lp.net/juju-core
[16:43] <niemeyer> Do you have any pending work that is about to be merged?
[16:43] <TheMue> niemeyer: From my side +1.
[16:43]  * rog looks
[16:43] <Aram> +1.
[16:44] <rog> niemeyer: no, that's fine
[16:44] <niemeyer> I'll give Dave a hand and see if any of his branches are mergeable and submit them myself now
[16:44] <niemeyer> fwereade will have to update, unfortunately
[16:45] <niemeyer> But it's almost as low a hit as we can manage I think
[16:53] <niemeyer> Running tests even for the dummy package is taking an awful lot of time.
[16:54] <niemeyer> s/package/provider/
[16:56] <rog> niemeyer: i have an idea about how we might be able to speed up the tests. the slowness is almost all caused by the zookeeper startup time
[16:57] <niemeyer> There seems to be something wrong..
[16:57] <rog> niemeyer: we could start one zk server on demand and use the zk chroot functionality so that it could be used across all tests in all packages.
[16:57] <niemeyer> START: /home/niemeyer/src/launchpad.net/juju-core/juju/environs/jujutest/test.go:96: LiveTests.SetUpTest
[16:57] <niemeyer> This is blocking
[16:57] <niemeyer> Sorry, no, it's not
[16:59]  * rog realises he's broken the build
[17:01] <niemeyer> rog: ?
[17:01] <rog> niemeyer: my recent container "fix"
[17:01] <rog> niemeyer: i could've sworn i ran the test, but evidently i didn't
[17:04] <niemeyer> rog: Hmm.. we'll probably need some kind of submission mechanism soon, hooked with lbox
[17:04] <niemeyer> rog: Btw, this line:
[17:04] <rog> niemeyer: i think lbox should probably run all the test on the merged branch before actually submitting it
[17:04] <niemeyer>         err := t.Env.Bootstrap(true)
[17:04] <niemeyer> rog: Is taking 36 seconds to execute
[17:04] <niemeyer> rog: With the *dummy* env
[17:05] <rog> niemeyer: i suspect zookeeper, but i may be wrong.
[17:06] <niemeyer> rog: It may be zk.. we call state.Initialize there
[17:06]  * niemeyer checks
[17:07] <niemeyer> rog: PutTools
[17:08] <rog> niemeyer: ah, it's taking ages to build the tools, i suppose
[17:08] <rog> niemeyer: because nothing is installed
[17:08] <rog> niemeyer: because you've just changed everything perhaps?
[17:08] <niemeyer> Could be
[17:08]  * niemeyer verifies
[17:12] <niemeyer> % time go install launchpad.net/juju-core/juju/cmd/...
[17:12] <niemeyer> go install launchpad.net/juju-core/juju/cmd/...  24.71s user 7.27s system 99% cpu 32.157 total
[17:12] <niemeyer> rog: ^
[17:12] <niemeyer> rog: Now, the question is why is it taking such a long time *every time*
[17:13] <niemeyer> Clearly something is not ok
[17:14] <rog> niemeyer: just using vanilla "go install" it always takes that long for you?
[17:14] <niemeyer> rog: What's vanilla go install?
[17:14] <rog> niemeyer: just go install on the command line, no other stuff going on like GOBIN etc
[17:15] <rog> niemeyer: for me it takes 1.12s the second time i run it
[17:15] <niemeyer> rog: Sorry, I still don't get it. The command above is go install on the command line.
[17:15] <rog> niemeyer: right. and you ran it twice and got the same time each time.
[17:15] <rog> niemeyer: i've no idea what's taking it so long there then
[17:16] <niemeyer> go install -x isn't even printing results
[17:16] <niemeyer> I'll rebuild my go tree
[17:16] <niemeyer> % time go install -x launchpad.net/juju-core/juju/cmd/...
[17:16] <niemeyer> WORK=/tmp/go-build238159301
[17:16] <niemeyer> go install -x launchpad.net/juju-core/juju/cmd/...  24.93s user 7.07s system 99% cpu 32.182 total
[17:16] <niemeyer> WTF.. it's taking 32 seconds before it even does anything
[17:17] <rog> niemeyer: if go install -x isn't printing results, it means it's doing nothing, yeah.
[17:17] <rog> system 99%
[17:17] <rog> try a syscall trace perhaps
[17:18] <niemeyer> I'll try rebuilding Go first, just to make sure I don't have something funky there
[17:18] <rog> niemeyer: you get 32s before it even prints the WORK line?
[17:18] <niemeyer> rog: Yeah
[17:18] <niemeyer> rog: If I can reproduce with a rebuild I'll see where more exactly
[17:19] <rog> niemeyer: at least this time, i think we can say the blame doesn't fall on the juju testing :-)
[17:19] <niemeyer> rog: Yeah! :)
[17:20] <rog> niemeyer: can i propose/submit a quick fix for the container brokenness before you proceed?
[17:20] <niemeyer> rog: Hmm.. we could add a flag to lbox submit such as "--test-command"
[17:21] <niemeyer> rog: Yeah, definitely
[17:21] <rog> niemeyer: that sounds like a good plan
[17:21] <niemeyer> rog: We can then have it in .lbox so we don't have to enter it every time
[17:22] <rog> niemeyer: yeah. but it's important that the tests run on the merged version
[17:22] <niemeyer> rog: Of course
[17:22] <niemeyer> Still broken
[17:23] <niemeyer> Let's see where this time is being spent
[17:23] <niemeyer> Holy crap.. it's doing an awful lot of things
[17:24] <niemeyer> I suspect someone added some logic without much consideration for what else is under $GOPATH/src/, as it's walking the whole tree :-(
[17:25] <rog> niemeyer: maybe it's not so clever about prefixes of ... patterns
[17:26] <rog> niemeyer: but walking the tree doesn't take long, surely?
[17:26] <niemeyer> rog: Surely does
[17:26] <niemeyer> rog: If the tree is large
[17:26] <rog> % time find $GOPATH/src > /dev/null
[17:26] <rog> 0.01user 0.04system 0:00.05elapsed 98PU (0avgtext+0avgdata 4880maxresident)k
[17:26] <rog> not for me :-)
[17:27] <rog> niemeyer: presumably it's not reading all the files in the tree?
[17:27] <niemeyer> rog: Presumably it's spending 32 seconds doing that
[17:27] <rog> niemeyer: how long does it take you to run that command?
[17:28] <niemeyer> [niemeyer@gopher ~/src]% time find > /dev/null
[17:28] <niemeyer> find > /dev/null  1.86s user 5.89s system 25% cpu 30.253 total
[17:30] <rog> niemeyer: gosh. that is really really big. or you've got a slow disk/network drive.
[17:30] <niemeyer> rog: I have a lot of stuff in my src/
[17:30] <niemeyer> rog: and an SSD
[17:30] <rog> niemeyer: ah! you have GOPATH=$HOME ?
[17:31] <niemeyer> rog: Yep
[17:31]  * rog is glad he didn't do that
[17:31] <niemeyer> rog: Heh
[17:31] <rog> niemeyer: the go tool should be more intelligent
[17:31] <niemeyer> rog: It should be less stupid, rather
[17:32] <niemeyer> Why on earth is it walking everything like that
[17:32] <rog> niemeyer: because it's walking the directories then matching against the pattern
[17:32] <rog> niemeyer: rather than walking the constant prefix of the pattern
[17:33] <rog> niemeyer: at least that's my assumption
[17:33] <niemeyer> rog: OMG, I hope you're wrong
[17:34] <rog> niemeyer: yeah, that's exactly what it's doing
[17:34] <niemeyer> It's like, "Oh you want everything under /home/joe? Okay! Let's start over in /var!"
[17:34] <rog> 		filepath.Walk(src, func(path string, fi os.FileInfo, err error) error {
[17:35] <rog> and src is everything
[17:35] <rog> niemeyer: to be fair it was done quickly, and correctness is more important than quickness.
[17:35] <rog> niemeyer: but it will be easy to fix.
[17:35] <niemeyer> rog: Yeah, I'll fix that now
[17:36] <rog> niemeyer: i'd make matchPattern return the longest path prefix as well as the match function
[17:36] <niemeyer> rog: Doing this right is a no-brainer, to be honest.. it was just a brain fart from someone
[17:38] <niemeyer> rog: Actually, they've done in right elsewhere there
[17:38] <rog> niemeyer: really?
[17:38] <niemeyer> rog: Yep.. see matchPackagesInFS
[17:39] <rog> niemeyer: so it does.
[18:01] <rog> niemeyer: i'm off for the evening. hope the switchover goes ok...
[18:01] <rog> 'night al
[18:01] <rog> l
[18:01] <niemeyer> rog: Thanks, have you seen the LGTM?
[18:03] <rog> niemeyer: just making sure with a final test, then i'll submit
[18:04] <niemeyer> rog: Cheers
[18:04] <rog> niemeyer: re: https://codereview.appspot.com/6304104/ i think we might have decided to go with a different, slightly simpler approach.
[18:04] <rog> i get this sporadic failure:
[18:04] <rog> FAIL: presence_test.go:328: PresenceSuite.TestWaitAlive
[18:04] <rog> presence_test.go:348:
[18:04] <rog>     c.Assert(err, IsNil)
[18:04] <rog> ... value *errors.errorString = &errors.errorString{s:"presence: still not alive after timeout"} ("presence: still not alive after timeout")
[18:04] <niemeyer> rog: The approach there looks pretty straightforward and clean to me
[18:05] <rog> niemeyer: this was my suggestion to dave this morning: http://paste.ubuntu.com/1052155/
[18:06] <niemeyer> rog: I mentioned it could stay simpler, but I also don't see an issue with doing it now, given Dave actually wrote it
[18:06] <rog> niemeyer: which he seemed to quite like
[18:06] <rog> niemeyer: in some similar form anyway
[18:07] <niemeyer> rog: That sounds fine too
[18:07] <rog> niemeyer: it means we can get rid of operationKinds, the string table etc
[18:09] <rog> niemeyer: https://codereview.appspot.com/6332044 submitted. build fixed.
[18:10] <niemeyer> rog: Thanks!
[18:11] <niemeyer> rog: Can you please add a comment to the review then?
[18:11]  * rog really is off now. see you tomorrow (my last day before i'm away for two weeks BTW)
[18:14] <niemeyer> Nevermind.. copied the IRC conversation on the proposal
[18:20] <rog> niemeyer: i added a comment
[18:21] <rog> niemeyer: ah, but replied to the wrong email!
[18:21] <rog> darn
[18:23] <niemeyer> rog: np
[20:23] <niemeyer> So, fixed go problem.. can actually test code in a non-ridiculous timeframe now
[20:47] <niemeyer> juju-core/juju => juju-core move is complete!
[20:47] <niemeyer> Please see juju-dev@ list for details.
[20:48]  * niemeyer steps out
[21:46] <andrewsmedina> niemeyer: :)