/srv/irclogs.ubuntu.com/2012/06/21/#juju-dev.txt

rogdavecheney: mornin' boss06:05
davecheneyrog: ahoy!06:07
rogdavecheney: it's funny, i saw gustavo's response to 6304104 but not the original proposal email06:08
rogdavecheney: am looking at it now06:08
davecheneyrog: ta06:12
rogdavecheney: hmm06:18
rogdavecheney: i'm wondering if the interface shenanigans are worth it06:19
rogdavecheney: how about just using a different type for each operation, e.g. http://paste.ubuntu.com/1052155/06:19
davecheneyrog: what is the type returned on the channel ?06:20
davecheneyinterface{} ?06:20
rogdavecheney: we could put a private method on it06:20
rogdavecheney: then it could still be Operation06:20
davecheneyswitch o := o.(type) { case ... } ?06:21
rogdavecheney: yeah, except we never need to do that AFAICS. i've only done a cursory look mind06:21
rogdavecheney: i think we always know what operation we're expecting06:21
rogc.Assert(op, Equals, dummy.OpBootstrap{env, inst})06:22
rogdavecheney: s/OpBootstrap/OpStartInstance/06:22
davecheneyrog: sure06:22
rogand that wouldn't work of course because equality isn't defined on Instances06:23
rogand but you get the idea.06:23
rogit means we can lost all the OperationKind stuff and the table. so we probably save lines overall.06:23
rogs/lost/lose/06:23
davecheneyrog: nice, I always wanted to switch on the type, rather than a synthetic kind field06:27
rogdavecheney: something like this perhaps, if we want an Operation type. http://paste.ubuntu.com/1052161/06:28
rogdavecheney: cool06:29
rogdavecheney: oops, isOperation should be defined on OpEnv06:29
rogdavecheney: although... that won't print so nicely06:30
rogdavecheney: scratch that last suggestion06:30
rogdavecheney: could go with something like this: http://paste.ubuntu.com/1052167/; or just use interface{}.06:33
davecheneyrog: feels a bit forced06:34
rogdavecheney: the Operation type?06:34
davecheneyall those empty methods to satisfy the interface06:34
rogdavecheney: yeah. there is precedent (go/ast) but i think it's probably best to use interface{}06:35
rogdavecheney: it's not a public-facing API after all06:35
Arammoin.06:55
TheMueMoin06:56
* TheMue wanted to start later after doing some sports but the club is closed this morning. It seems the morning shift has overslept.06:57
rogAram: hiya07:22
rogTheMue: hiya07:23
TheMuerog: Hi, just had to restart after firmware update.07:26
rogI like it when I'm reading an IRC conversation from some hours ago and thinking "I'd want to say *this*", then the conversation ends up at exactly that conclusion.08:36
rogi'm off into town to fetch my bike and buy a replacement jacket for the one i left behind in Oakland (:-[). back in a couple of hours.10:34
Arambokes are great10:42
Arambikes10:42
hazmatAram, any thoughts on the loss of atomicity and ordering with mongodb (my reply on your thread on list)?11:56
hazmatie. given multiple collections to insert for observation/notification11:57
Aramthere's no loss of ordering.11:58
Aramatomicity may be a problem.11:58
Arambut ordering should not be.11:58
hazmatAram, lacking atomicity, with concurrent ops, how can retain ordering11:58
hazmatie. given logical ops A and B and their respective inserts, they could go A1, B1, B2, A211:59
Aramthere's an atomic sequencing mechanism that generates incrementing integers, and these integers can be used to query for data in a particular order.12:00
Aramhazmat: basically the sequencing mechanism is atomic because we make it so, it's a single document.12:02
hazmatAram, combined with locking? what's a sequence flow look like..12:06
hazmatof a logical op12:06
hazmatget sequence value, lock observation collection, atomic update doc using sequence value as condition, insert into observation collection, unlock?12:08
Aramn reset by peer)12:39
Aram<davecheney> howdy, anyone seen mark12:39
Arammeh12:39
Aramdumb paste12:40
Aram 12:40
Aramhazmat: there's no locking in mongodb.12:40
Aramhazmat: I can't answer your question because we haven't decided yet :-).12:40
Aramhazmat: I only made sure we can implement a mechanism if/when we need it, whatever that mechanism might be.12:40
Aramhazmat: for example we can implement full transactions using multi-phase commits, if we consider it necessary, but we hope we would not need such a heavy weight mechanism.12:40
Aramhazmat: I can tell you the technical details of various mechanisms we might use, but I can't tell you which one will use at this early stage. probably gustavo has some better insight.12:40
hazmatAram, right re locking i meant app locking not native primitives, and thanks for clarifying the current state12:41
marruslhey juju folks...  for Juju on openstack, is swift required?13:11
marruslhrm... maybe wrong channel ;-)13:12
marrusli'll check in #juju13:14
Aramhi niemeyer13:36
niemeyerYo!13:39
rogniemeyer: hiya13:47
* Aram is amused by people who censor profanity words on the internet.13:49
niemeyerAram: Yeah, seriously13:52
* TheMue moved on the veranda to continue work there.13:56
TheMueniemeyer: Hi.13:56
TheMueniemeyer: After a talk yesterday I started to read the mail thread and CL regarding format: 2.13:58
TheMueniemeyer: A talk to fwereade13:59
TheMueniemeyer: Shall I continue with that topic or with non-relation topics in state? Like e.g. hooks.14:00
niemeyerTheMue: Hmm14:00
niemeyerTheMue: Hooks are not really in state, and fwereade has already been moving towards having them working14:01
niemeyerTheMue: What else is missing from state?14:01
TheMueniemeyer: Maybe I said it wrong. In Py in state there are hook, firewall, auth and security.14:02
TheMueniemeyer: Those are still open.14:03
TheMueniemeyer: I wanted to continue with relation but fwereade is doing there a lot and asked me to take another topic.14:04
TheMueniemeyer: That's why he passed me infos about format 2.14:04
niemeyerTheMue: Yeah, handling the change to format 2 sounds like a good  idea14:05
niemeyerTheMue: It's just out of state, AIUI14:05
niemeyerTheMue: Which is why I asked if you're leaving something else behind14:05
TheMueniemeyer: Those four files above and relations, then state should have been done.14:09
TheMueniemeyer: So far I compared it file by file and type by type.14:10
niemeyerTheMue: That's fantastic14:10
TheMueniemeyer: But I could do a usage scan (Where is state used today?) and see what functionality is missing in the Go port.14:10
niemeyerTheMue: So it sounds like it's pretty much done indeed14:11
niemeyerTheMue: The last two are not going to be ported as we'll want to do something else14:11
niemeyerTheMue: Hooks, as I mentioned, is very close to the implementation of the unit agent, and shouldn't be done in isolation14:11
TheMueniemeyer: Yes, you once said it when I started with auth.14:11
niemeyerTheMue: Firewall may be the more interesting bit14:11
Arammy T410 is overheating sometimes, had the same problem with my T400.14:12
TheMueniemeyer: So I could do firewall first.14:12
niemeyerTheMue: I'm having a look at it14:13
TheMueniemeyer: Just to get the state done.14:13
niemeyerAram: I haven't been having it with T410, but I had it all the time on the T6014:13
niemeyerTheMue: Hmmm14:14
niemeyerTheMue: We have to do firewall, but it shouldn't look like what is there right now14:15
niemeyerTheMue: It has a *provider* in the constructor, which makes no sense for a state package14:15
TheMueniemeyer: Yes, just scanned the code quickly. It's using the state, but it is no state.14:17
TheMueniemeyer: Interesting.14:17
niemeyerTheMue: It is state to some degree14:17
niemeyerTheMue: It's just mixed up14:18
niemeyerTheMue: Most of that file is really preparing for this:14:18
niemeyer            for port, proto in to_open:14:18
niemeyer                yield self.provider.open_port(machine, machine_id, port, proto)14:18
niemeyer            for port, proto in to_close:14:18
niemeyer                yield self.provider.close_port(machine, machine_id, port, proto)14:18
niemeyerTheMue: We have to invert the relationship as usual for watchers14:19
niemeyerTheMue: This is a type along the lines of PortChanges { Opened Closed }14:20
niemeyerTheMue: and Machine.WatchOpenPorts14:20
TheMueniemeyer: IC14:20
niemeyerTheMue: Or really, Machine.WatchPorts14:21
TheMueniemeyer: As today in Unit.WatchPorts14:24
niemeyerHmm14:25
niemeyerWhy do we watch ports for both units and machines14:26
* niemeyer reads14:26
TheMueniemeyer: Today it's only in Unit.14:27
niemeyerTheMue: I mean why the original code does that14:27
TheMueniemeyer: Oh14:28
niemeyerTheMue: Ah, of course14:28
niemeyerTheMue: One is based on the other14:28
rogniemeyer: do we want the machine agent to do the port opening and closing? it seems that it's currently the provisioning agent's job14:32
rogniemeyer: hmm, i can see why we might not want that, as it currently requires ec2 creds14:32
niemeyerrog: We do want that, but this is a refactoring step we should do when we're done with the port14:33
rogniemeyer: ok, so PA to start with14:33
niemeyerYeah14:33
* TheMue looks how firewall is used today.14:40
hazmatTheMue, only in the provisioning agent14:50
hazmatTheMue, it doesn't really belong in the state package14:50
TheMuehazmat: Yes, seen it.14:50
TheMueFresh cherry cake by my wife, bad for my diet. *sigh*14:57
rogniemeyer: ping15:12
rogTheMue: i want some, please15:12
niemeyerrog: Heya15:12
* TheMue dcc's rog some cake.15:12
rogniemeyer: just looking at starting the PA15:13
niemeyerrog: Ok15:13
rogniemeyer: there's a little awkwardness that i thought it might be worth discussing15:13
niemeyerrog: Cool. Can we cover this after lunch?15:13
rogniemeyer: ok15:13
niemeyerIt's food-o-time here15:13
rogniemeyer: enjoy! ping me after lunch...15:14
niemeyerrog: Thanks, will do15:14
niemeyerrog: Yo, so what's up?16:18
rogniemeyer: so...16:18
rogniemeyer: we want the provisioning agent to know the machine address to give when starting new machines16:18
rogniemeyer: so i thought we can easily use the metadata service to pass the address into the provisioning agent16:19
rogniemeyer: same kind of thing we do with the machine agent16:19
rogniemeyer: or initzk anyway16:19
rogniemeyer: with instanceIdAccessor16:19
rogniemeyer: i added hostNameAccessor16:20
niemeyerrog: How is it done today?16:21
niemeyerAram: Just reviewed https://codereview.appspot.com/630409916:22
Aramthanks16:22
rogniemeyer: good question. i should have looked. one mo16:22
niemeyerrog: We should look how it is currently done, because this problem has been solved in a way that works for several providers16:23
niemeyerrog: Maybe it's not optimal, but we should be sure we're improving on it if that are any changes to it16:23
niemeyers/that/there16:24
Aramniemeyer: I did the change, can I just submit now?16:24
niemeyerAram: Yeah16:24
Aramgood16:24
niemeyerThanks! Woohay first mstate branch going in!16:25
* TheMue cheers16:26
AramTheMue: I think we'll share a flight  returning from the sprint.16:27
TheMueAram: Oh, good you mentioned it, I've got to change the wiki. BTS found a cheaper flight with KLM. So sorry, we do not share it. *sigh*16:29
hazmatmramm, do you a percentage on travel arrangements ? ;-)16:29
Aramhmm, maybe I should ask for different flights, I know there are more flights than BTS told me.16:29
mrammhazmat: so far it's just pro-bono work16:30
TheMuehazmat: Hehe, that question shocked him. ;)16:32
rogniemeyer: ah! ok, problem solved. and dave's problem too.16:32
rogniemeyer: i should've thought of it anyway16:33
niemeyerrog: So what's the deal?16:34
rogniemeyer: the PA calls StateInfo16:35
rogniemeyer: easy peasy16:35
niemeyerrog: Aha, ok16:36
niemeyerrog, Aram, TheMue: Folks, I think now is a good moment to do the move lp.net/juju-core/juju => lp.net/juju-core16:43
niemeyerDo you have any pending work that is about to be merged?16:43
TheMueniemeyer: From my side +1.16:43
* rog looks16:43
Aram+1.16:43
rogniemeyer: no, that's fine16:44
niemeyerI'll give Dave a hand and see if any of his branches are mergeable and submit them myself now16:44
niemeyerfwereade will have to update, unfortunately16:44
niemeyerBut it's almost as low a hit as we can manage I think16:45
niemeyerRunning tests even for the dummy package is taking an awful lot of time.16:53
niemeyers/package/provider/16:54
rogniemeyer: i have an idea about how we might be able to speed up the tests. the slowness is almost all caused by the zookeeper startup time16:56
niemeyerThere seems to be something wrong..16:57
rogniemeyer: we could start one zk server on demand and use the zk chroot functionality so that it could be used across all tests in all packages.16:57
niemeyerSTART: /home/niemeyer/src/launchpad.net/juju-core/juju/environs/jujutest/test.go:96: LiveTests.SetUpTest16:57
niemeyerThis is blocking16:57
niemeyerSorry, no, it's not16:57
* rog realises he's broken the build16:59
niemeyerrog: ?17:01
rogniemeyer: my recent container "fix"17:01
rogniemeyer: i could've sworn i ran the test, but evidently i didn't17:01
niemeyerrog: Hmm.. we'll probably need some kind of submission mechanism soon, hooked with lbox17:04
niemeyerrog: Btw, this line:17:04
rogniemeyer: i think lbox should probably run all the test on the merged branch before actually submitting it17:04
niemeyer        err := t.Env.Bootstrap(true)17:04
niemeyerrog: Is taking 36 seconds to execute17:04
niemeyerrog: With the *dummy* env17:04
rogniemeyer: i suspect zookeeper, but i may be wrong.17:05
niemeyerrog: It may be zk.. we call state.Initialize there17:06
* niemeyer checks17:06
niemeyerrog: PutTools17:07
rogniemeyer: ah, it's taking ages to build the tools, i suppose17:08
rogniemeyer: because nothing is installed17:08
rogniemeyer: because you've just changed everything perhaps?17:08
niemeyerCould be17:08
* niemeyer verifies17:08
niemeyer% time go install launchpad.net/juju-core/juju/cmd/...17:12
niemeyergo install launchpad.net/juju-core/juju/cmd/...  24.71s user 7.27s system 99% cpu 32.157 total17:12
niemeyerrog: ^17:12
niemeyerrog: Now, the question is why is it taking such a long time *every time*17:12
niemeyerClearly something is not ok17:13
rogniemeyer: just using vanilla "go install" it always takes that long for you?17:14
niemeyerrog: What's vanilla go install?17:14
rogniemeyer: just go install on the command line, no other stuff going on like GOBIN etc17:14
rogniemeyer: for me it takes 1.12s the second time i run it17:15
niemeyerrog: Sorry, I still don't get it. The command above is go install on the command line.17:15
rogniemeyer: right. and you ran it twice and got the same time each time.17:15
rogniemeyer: i've no idea what's taking it so long there then17:15
niemeyergo install -x isn't even printing results17:16
niemeyerI'll rebuild my go tree17:16
niemeyer% time go install -x launchpad.net/juju-core/juju/cmd/...17:16
niemeyerWORK=/tmp/go-build23815930117:16
niemeyergo install -x launchpad.net/juju-core/juju/cmd/...  24.93s user 7.07s system 99% cpu 32.182 total17:16
niemeyerWTF.. it's taking 32 seconds before it even does anything17:16
rogniemeyer: if go install -x isn't printing results, it means it's doing nothing, yeah.17:17
rogsystem 99%17:17
rogtry a syscall trace perhaps17:17
niemeyerI'll try rebuilding Go first, just to make sure I don't have something funky there17:18
rogniemeyer: you get 32s before it even prints the WORK line?17:18
niemeyerrog: Yeah17:18
niemeyerrog: If I can reproduce with a rebuild I'll see where more exactly17:18
rogniemeyer: at least this time, i think we can say the blame doesn't fall on the juju testing :-)17:19
niemeyerrog: Yeah! :)17:19
rogniemeyer: can i propose/submit a quick fix for the container brokenness before you proceed?17:20
niemeyerrog: Hmm.. we could add a flag to lbox submit such as "--test-command"17:20
niemeyerrog: Yeah, definitely17:21
rogniemeyer: that sounds like a good plan17:21
niemeyerrog: We can then have it in .lbox so we don't have to enter it every time17:21
rogniemeyer: yeah. but it's important that the tests run on the merged version17:22
niemeyerrog: Of course17:22
niemeyerStill broken17:22
niemeyerLet's see where this time is being spent17:23
niemeyerHoly crap.. it's doing an awful lot of things17:23
niemeyerI suspect someone added some logic without much consideration for what else is under $GOPATH/src/, as it's walking the whole tree :-(17:24
rogniemeyer: maybe it's not so clever about prefixes of ... patterns17:25
rogniemeyer: but walking the tree doesn't take long, surely?17:26
niemeyerrog: Surely does17:26
niemeyerrog: If the tree is large17:26
rog% time find $GOPATH/src > /dev/null17:26
rog0.01user 0.04system 0:00.05elapsed 98PU (0avgtext+0avgdata 4880maxresident)k17:26
rognot for me :-)17:26
rogniemeyer: presumably it's not reading all the files in the tree?17:27
niemeyerrog: Presumably it's spending 32 seconds doing that17:27
rogniemeyer: how long does it take you to run that command?17:27
niemeyer[niemeyer@gopher ~/src]% time find > /dev/null17:28
niemeyerfind > /dev/null  1.86s user 5.89s system 25% cpu 30.253 total17:28
rogniemeyer: gosh. that is really really big. or you've got a slow disk/network drive.17:30
niemeyerrog: I have a lot of stuff in my src/17:30
niemeyerrog: and an SSD17:30
rogniemeyer: ah! you have GOPATH=$HOME ?17:30
niemeyerrog: Yep17:31
* rog is glad he didn't do that17:31
niemeyerrog: Heh17:31
rogniemeyer: the go tool should be more intelligent17:31
niemeyerrog: It should be less stupid, rather17:31
niemeyerWhy on earth is it walking everything like that17:32
rogniemeyer: because it's walking the directories then matching against the pattern17:32
rogniemeyer: rather than walking the constant prefix of the pattern17:32
rogniemeyer: at least that's my assumption17:33
niemeyerrog: OMG, I hope you're wrong17:33
rogniemeyer: yeah, that's exactly what it's doing17:34
niemeyerIt's like, "Oh you want everything under /home/joe? Okay! Let's start over in /var!"17:34
rogfilepath.Walk(src, func(path string, fi os.FileInfo, err error) error {17:34
rogand src is everything17:35
rogniemeyer: to be fair it was done quickly, and correctness is more important than quickness.17:35
rogniemeyer: but it will be easy to fix.17:35
niemeyerrog: Yeah, I'll fix that now17:35
rogniemeyer: i'd make matchPattern return the longest path prefix as well as the match function17:36
niemeyerrog: Doing this right is a no-brainer, to be honest.. it was just a brain fart from someone17:36
niemeyerrog: Actually, they've done in right elsewhere there17:38
rogniemeyer: really?17:38
niemeyerrog: Yep.. see matchPackagesInFS17:38
rogniemeyer: so it does.17:39
rogniemeyer: i'm off for the evening. hope the switchover goes ok...18:01
rog'night al18:01
rogl18:01
niemeyerrog: Thanks, have you seen the LGTM?18:01
rogniemeyer: just making sure with a final test, then i'll submit18:03
niemeyerrog: Cheers18:04
rogniemeyer: re: https://codereview.appspot.com/6304104/ i think we might have decided to go with a different, slightly simpler approach.18:04
rogi get this sporadic failure:18:04
rogFAIL: presence_test.go:328: PresenceSuite.TestWaitAlive18:04
rogpresence_test.go:348:18:04
rog    c.Assert(err, IsNil)18:04
rog... value *errors.errorString = &errors.errorString{s:"presence: still not alive after timeout"} ("presence: still not alive after timeout")18:04
niemeyerrog: The approach there looks pretty straightforward and clean to me18:04
rogniemeyer: this was my suggestion to dave this morning: http://paste.ubuntu.com/1052155/18:05
niemeyerrog: I mentioned it could stay simpler, but I also don't see an issue with doing it now, given Dave actually wrote it18:06
rogniemeyer: which he seemed to quite like18:06
rogniemeyer: in some similar form anyway18:06
niemeyerrog: That sounds fine too18:07
rogniemeyer: it means we can get rid of operationKinds, the string table etc18:07
rogniemeyer: https://codereview.appspot.com/6332044 submitted. build fixed.18:09
niemeyerrog: Thanks!18:10
niemeyerrog: Can you please add a comment to the review then?18:11
* rog really is off now. see you tomorrow (my last day before i'm away for two weeks BTW)18:11
niemeyerNevermind.. copied the IRC conversation on the proposal18:14
rogniemeyer: i added a comment18:20
rogniemeyer: ah, but replied to the wrong email!18:21
rogdarn18:21
niemeyerrog: np18:23
=== imbrando1 is now known as imbrandon
niemeyerSo, fixed go problem.. can actually test code in a non-ridiculous timeframe now20:23
=== robbiew1 is now known as robbiew
niemeyerjuju-core/juju => juju-core move is complete!20:47
niemeyerPlease see juju-dev@ list for details.20:47
* niemeyer steps out20:48
andrewsmedinaniemeyer: :)21:46

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!