/srv/irclogs.ubuntu.com/2012/09/06/#juju-dev.txt

davecheney./conn_test.go:117: undefined: juju.NewConnFromAttrs02:54
davecheneygrrr, thanks rog02:54
TheMueGood morning06:23
fwereade_TheMue, heyhey06:31
davecheneyhello06:31
fwereade_davecheney, heyhey06:32
TheMuefwereade_: Had a nice party yesterday?06:32
fwereade_TheMue, ha, just a sleep :)06:32
TheMuefwereade_: Thought you would be too upset to fell asleep. (Do you say so?)06:33
fwereade_TheMue, excited maybe more than upset06:33
davecheneywhat did I miss ?06:34
fwereade_TheMue, upset usually carries unhappy connotations :)06:34
fwereade_davecheney, I ran something on juju :)06:34
davecheneyfwereade_: indeed you did06:34
davecheneya cause for celebration, not for sadness by my account06:34
fwereade_davecheney, definitely, it's just that I celebrated by going to sleep06:34
davecheneysounds like a valid choice to me06:35
TheMuefwereade_: Ah, excited is the missing word. Thx.06:36
TheMuefwereade_: Already felt that upset is wrong.06:36
davecheneypossibly elated06:36
TheMueBtw, my vacation starts today. But I'll be in later today again as I have two LGTMs.06:39
davecheneyok06:39
TheMueWhile I'm away you can reach me by mail, thx to those little portable computers which also can phone. ;)06:40
davecheneyTheMue: i have a really bad camera that also has email on it06:46
davecheneybtw, http://codereview.appspot.com/6497070/06:46
davecheneythis might be the solutoin to the strange EOF problems we have talking to LP and AWS06:46
davecheney*might*06:46
davecheneyor it coudl just be the intertubes in australia06:46
fwereade_TheMue, btw, if you's still around, why do you favour the switch over the if/else in loose-hook-info-members?06:55
fwereade_TheMue, it's clearly a perfectly legitimate thing to do but I have a strong bias toward switching on just one thing... is this just because I'm used to "broken" switch statements?06:56
rogpeppedavecheney, fwereade_: morning07:09
davecheneymorning07:10
rogpeppedavecheney: the change to NewConnFromAttrs was niemeyer's suggestion BTW07:10
rogpeppedavecheney: sorry about that07:10
davecheneyrogpeppe: so'k07:10
davecheneywasn't hard to figure out07:10
rogpeppedavecheney: it made for better alignment with the entry points in environs07:10
davecheneyyeah, 'tis and improvement07:10
TheMuefwereade_: Just optical reasons. I like the switch more than if-else-chains (even if they are short). But don't care, just personal preferences. :)07:12
fwereade_TheMue, changed it, still somewhat ambivalent, we'll see07:14
rogpeppefwereade_: does this look better to you than the version as test fixture?  https://codereview.appspot.com/6495086/07:14
fwereade_rogpeppe, heyhey07:14
* fwereade_ looks07:14
rogpeppefwereade_: i generally prefer a switch if there are more than two branches07:15
TheMuefwereade_: Thx. So, off for the moment.07:15
fwereade_rogpeppe, there are only two here07:15
rogpeppefwereade_: in which case i'm probably on the fence, although i haven't seen the code in question07:16
fwereade_rogpeppe, it may just be that I had a couple of reviews requesting switch->if-else and have grown wary of them :)07:16
fwereade_rogpeppe, that looks much nicer to me07:19
rogpeppefwereade_: cool. i'm not entirely sure about the "series" helper methods (explicit better than implicit?) but i'm happier with it as a normal type not a fixture.07:19
fwereade_rogpeppe, definitely07:19
rogpeppedavecheney: good work on tls fix - let's hope it helps07:20
rogpeppefwereade_: the only other thing i considered was embedding LocalRepository or something like that.07:20
rogpeppefwereade_: 'cos they're really quite similar07:21
rogpeppefwereade_: but most uses don't care, and it's easy to make up a LocalRepository when needed07:21
fwereade_rogpeppe, IIRC I wanted to do that when I was first writing it but couldn't get them quite similar enough :)07:21
davecheneyrogpeppe: i'm not really sure it was the problem, but TSAN picked it up so it needed to be fixed07:22
rogpeppefwereade_: i think it's ok - they both represent a repository, but do different things with it. and all the state is in the directory, so it doesn't matter.07:22
rogpeppedavecheney: ah. i'm looking forward to that being included as standard. i'm interested to see what it makes of our code...07:23
davecheneyrogpeppe: there is a reason why I don't run GOMAXPROCS > 107:26
davecheneythat will make gozk EXPLODE!!07:26
rogpeppedavecheney: have you tried it?07:26
davecheneyonce07:26
davecheneygot a segfault in gozk07:26
rogpeppedavecheney: ha ha. i thought it *should* be safe.07:26
davecheneyrogpeppe: did you log that issue to have the signal details logged07:26
rogpeppedavecheney: ah, no. i'll do that.07:26
rogpeppedavecheney: it would be quite a hassle to fix sadly07:33
rogpeppedavecheney: because in that context we can't call any functions07:34
davecheneyisn't there a panicf07:35
davecheneyor a printf that doesn't allocate ?07:35
rogpeppedavecheney: hmm, maybe. i'm not sure of what invariants we need at that level.07:36
rogpeppedavecheney: of course, the real fix is to allow callbacks from non-Go-created threads.07:38
davecheneyrogpeppe: that fix might be saying 'the real fix is to relax the laws of reletivity to allow high paying packets to travel at their chosen speed'07:41
rogpeppedavecheney: yeah07:41
davecheneym and g are not available, but as long as the GC doesn't need to run, it should be safe to use runtime.print07:42
rogpeppedavecheney: do you know what linux signal behaviour is w.r.t. threads? is the signal delivered to one thread only? or all threads?07:45
rogpeppedavecheney: hmm, one random thread, it looks like. hurray.07:46
davecheneyrogpeppe: yup, but all threads wouldn't help either07:47
rogpeppedavecheney: if it was all threads we could block the signals in all but the one we care about and ignore if we find no m, no?07:47
davecheneyrogpeppe: good point, but i'm sure that was tried and found wanting07:48
rogpeppedavecheney: better would be if a thread had to explicitly enable signals rather than receiving them by default07:50
davecheney/home/dfc/src/launchpad.net/juju-core/testing/mgo.go:119: c.Fatal("Test left sockets in a dirty state")07:53
davecheneywhat does this mean ?07:53
fwereade_davecheney, sorry, no idea... failed to delete them maybe?08:04
davecheneyfwereade_: i was leaking a mgo.Session08:13
davecheneymy fault08:13
fwereade_np08:14
rogpeppedavecheney: do you know a way of reliably reproducing the signal problem. my naive attempt doesn't fail: http://paste.ubuntu.com/1188513/08:14
davecheneyrogpeppe: not really08:14
davecheneyit is quite rare08:14
davecheneydeinfitely run with high GOMAXPROCS08:14
rogpeppedavecheney: if the issue is what we think it is (the signal is randomly delivered to some thread), i can't see why the above code doesn't trigger the issue08:16
davecheneyrogpeppe: try firing the signal from the C side08:17
rogpeppedavecheney: i tried firing the signal from the command line. no difference.08:21
davecheneyrogpeppe: i'm not surprised, it's not supposed to happen08:22
rogpeppedavecheney: well... how can they be stopping it from happening?08:22
davecheney'supposed to happen' == no panic08:22
rogpeppedavecheney: i've seen a couple more failures of the TestSSHConnect test BTW. (one just now and another one yesterday).08:30
rogpeppedavecheney: it concerns me08:31
davecheneyrogpeppe: my suggestion to that would be to switch to the ssh crypto package08:31
rogpeppedavecheney: i'd love to. we started off trying that direction, but it wasn't mature enough.08:32
davecheneyrogpeppe: all it needs to support for loading keys off disk (in the same order that ssh does it)08:33
davecheneyand talking to the ssh agent (which we have now)08:33
davecheneyi'll put it on my list for this weekend08:33
davecheneymust fly08:33
Arammoin.08:42
fwereade_Aram, heyhey08:58
fwereade_rogpeppe, trivial: https://codereview.appspot.com/649008609:16
rogpeppefwereade_: LGTM09:17
fwereade_rogpeppe, cheers09:17
rogpeppefwereade_: pity it had never been tested live before...09:17
fwereade_rogpeppe, indeed09:17
* Aram has troubles finding flights to lisbon.09:34
rogpeppefwereade_: need trivial LGTM please: https://codereview.appspot.com/649409010:12
* fwereade_ looks10:12
fwereade_LGTM10:13
fwereade_rogpeppe, ^10:13
rogpeppefwereade_: ta!10:13
* rogpeppe is ashamed to have broken the build :-(10:13
rogpeppefwereade_: FWIW, i think a string argument to AgentToolsDir is still ok. each agent can decide on its own name, no?10:23
rogpeppefwereade_: (i just saw your discussion with gustavo last night)10:24
fwereade_rogpeppe, it happens ;p10:24
fwereade_rogpeppe, kinda... but, atm, everybody has to prepend unit-, and deslash, whenever they want to specify a unit name10:25
fwereade_rogpeppe, this feels kinda icky10:25
fwereade_rogpeppe, although I am also not 100% sold on KeyPath10:26
rogpeppefwereade_: i'm not sure what you mean. unit names aren't prepended with "unit-" AFAICS10:26
rogpeppefwereade_: the *agent name* for a unit is though10:27
fwereade_rogpeppe, isn't that what AgentToolsDir expects?10:27
rogpeppefwereade_: yes, but how often do you need to call AgentToolsDir?10:27
fwereade_rogpeppe, a few places that may eventually coalesce into one place10:28
fwereade_rogpeppe, the hook env needs it, EnsureJujucSymlinks needs it10:28
fwereade_rogpeppe, container needs it10:29
rogpeppefwereade_: why does the hook env need it?10:29
fwereade_rogpeppe, so we can put it on the PATH10:29
rogpeppefwereade_: why not just do that in the uniter main?10:30
fwereade_rogpeppe, and, later, add it as an explicit env var for use when setting up juju-run commands10:30
fwereade_rogpeppe, because it's only needed by the hooks?10:30
rogpeppefwereade_: fair enough. i'd probably do something like uniter.AgentName(unit *state.Unit) string10:31
rogpeppefwereade_: or even uniter.ToolsDir(unit *state.Unit) string10:31
fwereade_rogpeppe, not unreasonable for container to know about uniter, I guess... and environs doesn't need to know about uniters, because that's always done by machiner... yeah, probably good10:33
rogpeppefwereade_: yeah, i hadn't thought about container needing to know about uniter, but as you say, sounds reasonable, as it's starting the worker (indirectly)10:34
fwereade_rogpeppe, yeah, exactly10:34
fwereade_rogpeppe, a further thought10:36
fwereade_rogpeppe, if we do have a consistent "kind-name" badge for the various agents, which sounds sensible, we really should be using it in more places than just AgentToolsDir10:37
fwereade_rogpeppe, it should be in logfile names, for example10:37
rogpeppefwereade_: that seems like a good plan10:38
rogpeppedavecheney: did you have any further discussion with niemeyer about your CL https://codereview.appspot.com/649907111:12
rogpeppe?11:12
rogpeppedavecheney: i'm not sure i understand his objections11:12
rogpeppedavecheney: but perhaps his objections are *only* to the CL description11:14
davecheneyrogpeppe: no, i will try again tomorrow11:14
rogpeppedavecheney: and if it was changed to "most uses just want the *Machine" it would be ok11:14
davecheneyrogpeppe: couldn't hurt11:14
rogpeppedavecheney: although he's right - there's actually *no* real code that wants the Machine rather than the id11:16
davecheneyrogpeppe: it's probably not worth the discussion11:20
davecheneywhat I have works11:20
davecheneyand there are so many more important things do to11:20
rogpeppetrue 'nufff11:20
* rogpeppe goes for some lunch12:47
rogpeppeniemeyer: morning!12:58
niemeyerrogpeppe: Morning!13:35
niemeyerHello all!13:35
rogpeppeniemeyer: thanks for the review!13:35
niemeyerrogpeppe: np, thanks for all the niceties13:35
rogpeppeniemeyer: no problem at all13:35
rogpeppeniemeyer: i don't think there are two round trips to get machine id and machine. in state i think that's true anyway. or are you thinking of mstate?13:36
niemeyerrogpeppe: Sorry, your logic escapes me.. you mean that state.Machine(id) doesn't have a roundtrip to the state server?13:37
rogpeppeniemeyer: no, i'm saying that MachindId needs the round trip anyway13:37
rogpeppeMachineId13:38
rogpeppeAssignedMachineId even13:38
niemeyerrogpeppe: The logic still escapes me.. the CL makes it call Machine, always13:38
niemeyerrogpeppe: Machine has a roundtrip to the state server, both in state and in mstate13:38
rogpeppeniemeyer: ah yes, you're right. that's wrong. but we can construct the machine from the topology without needing an extra round trip.13:40
niemeyerrogpeppe: No, we can't, because we use cached state in mstate13:41
rogpeppeniemeyer: ah, so Unit.AssignedMachineId doesn't go to the server?13:41
niemeyerrogpeppe: Yes, and Machine has to go to the server to get the machine state13:42
rogpeppeniemeyer: right, gotcha. presumably we *could* fetch the machine too when we fetch the unit.13:42
niemeyerrogpeppe: Dude..13:42
rogpeppelol13:43
rogpeppeok ok13:43
fwereade_huh, are we not actually using --juju-directory for anything?14:02
mrammHi all.   I was on swap yesterday, but am back now.14:40
mrammJust got off a call about doing a Juju marketing video14:40
Aramhi mramm14:40
fwereade_mramm, heyhey14:40
fwereade_mramm, I deployed something yesterday, would you believe?14:40
mrammI'm also working on the info for the website, a juju webinar slide deck, and talking to jorge about marketing juju to developers and using it as a DIY PaaS.14:41
mrammfwereade_: That is AWESOME!14:41
mrammfwereade_: Everybody's hard work is coming together!14:41
fwereade_mramm, yep :D14:42
mrammI am excited!14:42
fwereade_mramm, now ofc I have to hammer it into proposable shape14:42
mrammand using too many exclamation points!!!!!!!!!!!!!14:42
fwereade_mramm, but that shouldn't be too far off14:42
mrammfwereade_: great!14:42
mrammI am excited about the juju video14:43
mrammthe people we've got working on it seem competent14:43
rogpeppefwereade: you'll be pleased to know that authorisation of internal traffic now works14:43
fwereaderogpeppe, sweet14:44
fwereademramm, (that was the only manual step)14:44
rogpeppefwereade: just waiting on a live test that checks that both machine agents upgrade14:44
rogpeppeYAY!14:44
rogpeppeit works14:44
* fwereade cheers14:44
mrammI also had a talk with Clint about packaging juju for Ubuntu on Tuesday night14:44
mrammrogpeppe: AWESOME!14:44
mrammso very much awesome going on right now!14:45
rogpeppefwereade: 3.87s to upgrade both agents FYI14:45
rogpeppefwereade: and that's using the proper deploy infrastructure, juju.AddService, AddUnits, etc14:46
fwereaderogpeppe, sweeet14:46
rogpeppefwereade: yeah, it's nice to have some functional tests that are a bit higher level14:47
rogpeppefwereade: it's just a pity that pushing the tools takes so long. sometimes as long as 5 minutes.14:49
rogpeppefwereade: it would have been done earlier if amazon's docs weren't so crap :-)14:51
fwereaderogpeppe, ha, I know your pain14:52
mrammrogpeppe: haha14:53
mrammSo, I have a product review meeting tomorrow14:53
mrammif you have things you think I should give management heads up on, let me know14:53
mrammI've got the unit agent, updater, mongo, and marketing update basics figured out14:54
mrammso if you have any additional detail you think is important on those things, or any other stuff that people need to know about, shoot it on over.14:55
fwereaderogpeppe, niemeyer: sanity check on http://paste.ubuntu.com/1189087/ if yu have a mo14:57
fwereadeniemeyer, KeyPath on state entities was not so nice in the end -- we don't always have the state entities available14:58
fwereadeniemeyer, but I *think* this ends up quite neat14:58
fwereadeniemeyer, rogpeppe: I would almost certainly want to pair it with a PathSuite or something that swaps out LibDir, LogDir, InitDir15:00
fwereadeniemeyer, (and possibly $HOME, why not, eh?)15:00
niemeyerjuju/testing/conn.go:// It also sets up $HOME and environs.VarDir to15:01
niemeyerfwereade: ^15:01
fwereadeniemeyer, yeah; and a lot of other tests swap one or both of those out here or there15:01
fwereadeniemeyer, it was moving environs.VarDir that made me think "ouch, too much duplication"15:02
niemeyerfwereade: My only concern is that right now we already have an Agent with an interface15:03
fwereadeniemeyer, but if I could find a sensible name, it would probably be ok?15:03
fwereadeniemeyer, and actually I'm not sure agent.Spec is all that much subject to confusion15:04
fwereadeniemeyer, wait a mo, there is no Agent interface that I can find15:05
fwereadeniemeyer, now I want to just call that an Agent :)15:05
niemeyerfwereade: cd cmd/jujud; grep Agent *15:06
fwereadeniemeyer, I see AgentConf and AgentState15:08
niemeyerfwereade: Uh?15:08
niemeyerfwereade: UnitAgent, MachineAgent, ProvisioningAgent, AgentState, AgentConf, etc etc15:08
fwereadeniemeyer, ok... you said we had an Agent with an interface, maybe I mistook what you meant15:09
niemeyerfwereade: AgentConf being JujuDir + StateInfo..15:09
fwereadeniemeyer, ah yeah, mean to mention that, JujuDir is not used15:09
niemeyerfwereade: I mean that the concepts you're playing with already exist15:10
niemeyerfwereade: We can't just add another package without sorting their situation out15:10
fwereadeniemeyer, ok, I see now15:10
fwereadeniemeyer, well, JujuDir doesn't actually do anything15:10
niemeyerfwereade: That won't change if we just add another package :)15:10
fwereadeniemeyer, but I'm pretty sure what it *should* do is overwrite environs.VarDir (which would be agent.LibDir)15:11
rogpeppeniemeyer: https://codereview.appspot.com/650008915:20
niemeyerrogpeppe: Cheers.. I think I'll focus a bit on the watcher today, though, otherwise we'll never have it15:21
rogpeppeniemeyer: no problem15:21
rogpeppeniemeyer: you'll be glad to know that this enables the uniter stuff to work live though15:21
niemeyerrogpeppe: Wow, sweet!15:21
rogpeppeniemeyer: and that it tests two machine agents upgrading at the same time15:21
rogpeppeniemeyer: through the juju.Conn deploy machinery15:22
niemeyerrogpeppe: Oh man15:22
niemeyerrogpeppe: It's coming together15:22
rogpeppeniemeyer: if nothing else, just have a brief glance at https://codereview.appspot.com/6500089/diff/2001/environs/jujutest/livetests.go15:22
rogpeppeniemeyer: (i'm really happy that worked)15:23
niemeyerrogpeppe: Impressive indeed.. even more impressive to have that as a stock test15:25
rogpeppeniemeyer: yeah, i'm happy about that15:27
niemeyerfwereade: wb15:30
rogpeppefwereade: https://codereview.appspot.com/650008915:30
niemeyerfwereade: I sent you and -dev a mail15:30
niemeyerfwereade: With the feedback, before I forget and so that I could move on15:30
fwereadeniemeyer, cheers, I will take a look15:30
fwereadeniemeyer, yeah, consider my mind already churning on these thoughts15:32
niemeyerfwereade: +1, cheers15:33
rogpeppefwereade: shouldn't JujuDir set environs.VarDir?15:38
fwereaderogpeppe, I intimated as much above15:38
rogpeppef.StringVar(&environs.VarDir, "juju-directory", environs.VarDir, "juju working directory")15:39
rogpeppefwereade: oh yes, i didn't notice that15:39
fwereaderogpeppe, although I am starting to feel dissatisfied with its globalness, and am wondering if we really need it15:39
rogpeppefwereade: yeah, i'd be happy for it to avoid globalness15:40
rogpeppefwereade: i think most of environs/tools.go could happily fit in a new package15:41
rogpeppefwereade: with no globals15:41
rogpeppefwereade: maybe just a package called "tools"15:42
rogpeppeaw15:42
rogpeppefwereade: last thing you saw?15:43
rogpeppefwereade: last thing from you i saw was15:43
rogpeppe[16:39:26] <fwereade> rogpeppe, although I am starting to feel dissatisfied with its globalness, and am wondering if we really need it15:43
fwereaderog, I saw  fwereade: i think most of environs/tools.go could happily fit in a new package15:48
rogpeppe16:41:37] <rogpeppe> fwereade: with no globals15:49
rogpeppe[16:42:19] <rogpeppe> fwereade: maybe just a package called "tools"15:49
fwereaderogpeppe, I was just pondering exactly that while having a ciggie :)15:49
rogpeppefwereade: because that's what this is all about, i *think*15:49
fwereaderogpeppe, yeah, there's a sideline in "bits of container look awfully like bits of cloudinit" -- which originally started me off on this jaunt -- but you are absolutely right15:50
niemeyerLunch break15:52
rogpeppefwereade: which bits were you thinking of?15:54
fwereaderogpeppe, just the bits that create Confs are awfully similar and mildly tediously inconsistent15:54
fwereaderogpeppe, not a big deal at all really, but it was nagging at me15:54
rogpeppefwereade: Conf?15:55
fwereaderogpeppe, upstart.Conf15:55
rogpeppefwereade: yeah, i think that code would fit quite happily in a tools package15:58
* rogpeppe is quickly hacking together a sketch16:03
rogpeppefwereade: how about something like this? http://paste.ubuntu.com/1189197/16:09
rogpeppefwereade: a few bits tidied up:16:10
rogpeppepackage tools16:10
rogpeppe// SearchFlags gives options when searching  for tools.16:10
rogpeppetype SearchFlags int16:10
rogpeppeconst (16:10
rogpeppe// HighestVersion indicates that versions above the version being16:10
rogpeppe// searched for may be included in the search. The default behavior16:10
rogpeppe// is to search for versions <= the one provided.16:10
rogpeppeHighestVersion SearchFlags = 1 << iota16:10
rogpeppe// DevVersion includes development versions in the search, even16:10
rogpeppe// when the version to match against isn't a development version.16:10
rogpeppeDevVersion16:10
rogpeppe// CompatVersion specifies that the major version number16:10
rogpeppe// must be the same as specified. At the moment this flag is required.16:10
rogpeppeCompatVersion16:10
rogpeppe)16:10
rogpeppe// List holds a list of available tools.  Private tools take16:10
rogpeppe// precedence over public tools, even if they have a lower16:10
rogpeppe// version number.16:10
rogpeppetype List struct {16:10
rogpeppePrivate []*state.Tools16:10
rogpeppePublic  []*state.Tools16:10
rogpeppe}16:11
rogpeppe// ListAll returns a List holding all the tools16:11
rogpeppe// available in the given environment that have the16:11
rogpeppe// given major version.16:11
rogpeppefunc ListAll(env Environ, majorVersion int) (*List, error)16:11
rogpeppe// Put builds the current version of the juju tools, uploads them16:11
rogpeppe// to the given storage, and returns a Tools instance describing them.16:11
rogpeppe// If vers is non-nil it will override the current version in the uploaded16:11
rogpeppe// tools.16:11
rogpeppeoops16:11
rogpeppesorry everyone16:11
rogpeppei meant this: http://paste.ubuntu.com/1189199/16:11
rogpeppefwereade: http://paste.ubuntu.com/1189199/16:13
rogpeppehmm, i wonder if i've been muted16:17
fwereaderogpeppe, sorry, I can hear you16:18
rogpeppenp16:18
fwereaderogpeppe, it looks sane, I think, I'm not sure exactly how it will square with the other wild ideas I am chasing16:18
rogpeppefwereade: what kind of thing are you thinking of?16:19
fwereaderogpeppe, agent.Agent, and what exactly it should have attached to it, if it should even exist16:19
rogpeppefwereade: i'm not convinced it should exist16:20
rogpeppefwereade: an agent is its own beast16:20
fwereaderogpeppe, there are an *awful* lot of commonalities across agents, I think there is a useful common structure waiting to emerge16:21
rogpeppefwereade: i go along with niemeyer's comments in this respect16:21
rogpeppefwereade: when it emerges, we can factor it out16:21
rogpeppefwereade: as particular chunks of functionality16:21
fwereaderogpeppe, I think it's worth an overnight ponder at least :)16:22
rogpeppefwereade: saying "*this* is an agent" is a bit like the inheritance way of thinking, i think16:22
fwereaderogpeppe, I guess it is somewhat predicated on the assumption that an agent should correspond to a single state entity16:22
rogpeppefwereade: indeed16:22
rogpeppefwereade: which may very well not be the case in the future16:22
rogpeppefwereade: well... i guess we will probably always have at least one item in the state for an agent16:23
fwereaderogpeppe, hmm -- so it's workers that should correspond to state entities?16:23
rogpeppefwereade: no16:23
rogpeppefwereade: it's running processes16:24
rogpeppefwereade: i.e. things that can upgrade themselves16:24
fwereaderogpeppe, ok (is there now a Provisioner state entity that I haven't noticed?)16:24
rogpeppefwereade: but i may very well be seeing through very upgrading-centric eyes currently :-)16:24
rogpeppefwereade: no, but there will be16:24
rogpeppefwereade: of some kind16:24
rogpeppefwereade: otherwise we can't tell when a PA manages to upgrade itself16:25
fwereaderogpeppe, cool, thought so, and this just makes all the agents look even more similar to me, but indeed the stars might not yet be aligned16:26
rogpeppefwereade: there are definitely similarities16:26
rogpeppefwereade: but we can abstract those out when we need some common functionaliy16:26
rogpeppety16:26
rogpeppefwereade: what kinds of operations do you envisage on agent.Agent?16:27
rogpeppefwereade: would it be a concrete type or an interface?16:28
SpamapSHello golangian friends. I am wondering, are there any examples of Go applications already packaged (via PPA, or even in the Ubuntu archive) ?16:48
SpamapSand on a related note, I am looking at writing a charm in Go as an exercise, and wondering how best to do it.. #!/usr/bin/gorun is great for prototyping, but at some point I think I'll want to just compile it16:49
niemeyerSpamapS: lbox and cobzr might be good examples to start with17:00
niemeyerSpamapS: They're both in PPAs17:00
SpamapSniemeyer: perfect17:00
niemeyerSpamapS: and auto-building17:00
* SpamapS already has the PPA's for those.. apt-get source to the rescue17:00
SpamapSniemeyer: any reason you had to be so explicit on binary-arch here: http://bazaar.launchpad.net/~niemeyer/lbox/package/view/head:/debian/rules17:02
SpamapSniemeyer: seems like an override_dh_install: might have handled that.17:03
TheMuehiho17:07
TheMueniemeyer: ping17:07
rogpeppeSpamapS: i had some ideas about writing charms in Go, but haven't had time to do anything about them yet17:07
SpamapSrogpeppe: it seems a bit heavy handed for most charm duties.17:07
rogpeppeSpamapS: yeah, a shell script is often a good fit17:08
SpamapSwhich boil down to "parse this file. put that file over there. run this command" ..17:08
niemeyerTheMue: Pong17:08
niemeyerSpamapS: Hmm17:08
SpamapSrogpeppe: I'm actually finding puppet's DSL quite good and succinct for charm duties. ;)17:08
TheMueniemeyer: just wanted to inform you that the CLs are in17:08
niemeyerTheMue: Beautiful, thanks17:09
* rogpeppe hasn't actually looked at puppet yet17:09
TheMueniemeyer: You had some notes on the first and not on the second and vice versa. I applied them on both.17:09
SpamapSrogpeppe: the biggest drawback is that you have to bust out to ruby to do anything interesting17:09
niemeyerTheMue: Well, actually you didn't, as I mentioned in the review, but that's fine17:10
TheMueniemeyer: Like the tabs, the panic message and the comment.17:10
niemeyerTheMue: We can sort out these details when you're back17:10
niemeyerTheMue: Ah, yeah, that was great,thanks17:10
TheMueniemeyer: What exactly you are referring?17:10
rogpeppeSpamapS: interesting. ruby's Yet Another Language I Have Never Used17:10
rogpeppeSpamapS: i always figured it was too close to python to be worth learning unless i needed to.17:10
niemeyerTheMue: The three points in the review17:10
TheMueniemeyer: WaitAgentAlive() on Unit?17:11
niemeyerTheMue: Where I suggested they might be done in a different CL17:11
SpamapSrogpeppe: more like perl meets javascript17:11
niemeyerTheMue: There's a bug opened as well with them now17:11
TheMueniemeyer: Yeah, the Dying and the Alive error, but not the removal of the loop.17:11
niemeyerTheMue: Don't worry, what's going in is a great start, and we can easily sort the points out in a follow up when you're back17:11
TheMueTheMue: You made the comment not on the last checkin.17:12
niemeyerSpamapS: I don't really know.. I bet there are better ways to handle it.. I'm just not a well versed packager17:12
rogpeppeSpamapS: anyway, Go might be good for charms that do a lot of orchestration dancing17:12
SpamapSniemeyer: its really only a future-proofing thing anyway.. the way its written now is 100% correct17:12
SpamapSrogpeppe: right, I was thinking of centralized things that have to handle all coming/going .. like monitoring or logging17:13
rogpeppeSpamapS: i reckon you could write it in good Go style, as a goroutine that receives "charm-hook-execute" events in a loop, then acts on the hook and replies.17:13
TheMueniemeyer: Please take a look at https://codereview.appspot.com/6494073/diff/2002/mstate/unit.go at line 230ff17:14
rogpeppeSpamapS: rather than our traditional "you get called when a hook happens" model17:14
niemeyerTheMue: Ok?17:14
TheMueniemeyer: It's now w/o the loop, like for Machine.17:15
rogpeppeSpamapS: but these thoughts are only fleeting and unformed...17:15
niemeyerTheMue: Ok, that's great17:15
SpamapSrogpeppe: thats actually what I was thinking too. But I think such things will only be necessary in the extreme cases where something has many thousands of related service units.17:17
SpamapSrogpeppe: for 99% of the cases.. python/ruby/bash will be up to the task of running a hook every few seconds.17:18
rogpeppeSpamapS: it's not necessarily about large scale - it could make it easier to make logic simpler17:18
SpamapSrogpeppe: I doubt that. :)17:18
TheMueFine, now I can leave relaxed. I will look into mail via phone to stay informed during my vacation. It's an exciting time for juju.17:18
rogpeppeSpamapS: you may well be right :-)17:19
SpamapSrogpeppe: simpler to a go developer maybe.. but not to a charmer. :)17:19
SpamapSthe most insanely complex charms are still pretty straight forward17:19
rogpeppeSpamapS: well, i certainly intend to write some more charms at some point :-)17:19
rogpeppeSpamapS: also, i have a feeling we'll get more golangers coming along in the python/ruby space, though i may be wrong in that17:20
SpamapShttp://www.cloudifysource.org/17:20
SpamapSanother juju-like project17:20
niemeyerTheMue: Yeah, have a great time there17:30
TheMueniemeyer: Thx, the weather forecast looks good, so we'll have some fine days (and evenings with a bottle of wine) at the water. It's only already too cold for swimming.17:31
rogpeppei'm off for too. have fun all. see you tomorrow!17:36
rogpeppeTheMue: enjoy your break!17:36
TheMuegn rogpeppe17:36
TheMueand thank you17:36
fwereadeniemeyer, ping21:06
niemeyerfwereade: Pongus21:06
fwereadeniemeyer, I think trunk is broken, I'm getting a panic from somewhere inside presence in the mstate tests... just verifying it *actually* happens on trunk, not just my merged version21:07
fwereadeniemeyer, if it is, is it still ok to submit over the top?21:07
niemeyerfwereade: Can you please paste the panic?21:07
fwereadeniemeyer, https://bugs.launchpad.net/juju-core/+bug/104705121:07
niemeyerfwereade: Yeah, if that's the only failure, it'd be fine21:07
fwereadeniemeyer, cool, thanks21:07
fwereadeniemeyer, yeah, trunk too21:07
niemeyerfwereade: Ah, ok.. I can tell what it is21:08
niemeyerfwereade: I've missed that when reviewing Frank's branch21:08
fwereadeniemeyer, ++psychic debugging :)21:08
niemeyerfwereade: We're creating a watcher and not stopping21:08
fwereadeniemeyer, jolly good, could be worse ;)21:08
niemeyerfwereade: Hehe :)21:08
niemeyerfwereade: I'll fix it as soon as I stop the current line of thinking here21:09
fwereadeniemeyer, lovely, tyvm21:09
niemeyerfwereade: Sorry for the trouble21:09
fwereadeniemeyer, no trouble :)21:09
niemeyerfwereade: ping22:11
fwereadeniemeyer, pong22:11
niemeyerfwereade: Yo22:11
fwereadeniemeyer, how's it going?22:11
niemeyerfwereade: Up for a quick review fixing that issue?22:11
fwereadeniemeyer, absolutely22:11
fwereadeniemeyer, then sleepytime ;)22:11
niemeyerfwereade: Cool, pushing22:11
niemeyerfwereade: I bet! 8)22:13
niemeyerfwereade: https://codereview.appspot.com/651004322:15
niemeyerI've screwed up the description.. fixing meanwhile22:15
fwereadeniemeyer, LGTM22:16
niemeyerDone22:16
niemeyerfwereade: Cheers!22:16
fwereadeniemeyer, btw, I think I have convinced myself that the reset --soft is pure superstition; and that in any cases which may exist in which deleting the lock file is insufficient, we really can't do anything about this22:30
fwereadeniemeyer, *but* that I also shouldn't even try to recover on unit agent startup -- the user could very easily be logged in and messing with the dir while the uniter is in an error state, and we can't just charge in and start doing stuff22:31
niemeyerfwereade: Ah, cool, thanks for the note.. I really wasn't sure22:31
fwereadeniemeyer, so I now want to just speculatively delete the lock file before every command that hits the index22:32
niemeyerfwereade: Sounds good.. your atomicity scheme kind of makes it unnecessary as long as the automated process is concerned, so sounds sane22:32
niemeyerfwereade: Uh oh22:32
fwereadeniemeyer, hm, is that bad?22:33
niemeyerfwereade: Sounds like the two extremes..22:33
niemeyerfwereade: We're not reseting because the user might be messing around, but we're happy to kill the lock all the time?22:33
fwereadeniemeyer, the most authoritative-sounding source I could find was a git list thread from 2009, in which they discussed making the "just delete the lock file" message carry more of that intent22:33
fwereadeniemeyer, the only times we have any reason to kill the lock are precisely when we're doing things with the index22:34
fwereadeniemeyer, this implies that we are upgrading, or running hooks, and if the user is messing around in the charm dir while we are working that is on hi own head ;)22:34
niemeyerfwereade: I see, so hitting the index as in changing it22:35
niemeyerfwereade: Not just reading, right?22:35
fwereadeniemeyer, I'm not sure it'll even read the index when something else has a lock22:35
fwereadeniemeyer, but this was in fact done by a process of pure research22:36
niemeyerfwereade: I'd rather start more conservative, if feasible22:36
niemeyerfwereade: The lock is there precisely to avoid processes from stepping on each other22:36
fwereadeniemeyer, of all the commands we use: add, commit, pull, reset all require that the lock file not exist22:37
niemeyerfwereade: Sure, because if it exists someone/something else may be acting on the directory22:37
niemeyerfwereade: But, as I understand all the changes are done in independent directories22:37
niemeyerfwereade: With your atomicity mechanisms22:37
niemeyerfwereade: Which means that if we ever see a lock, something awkward is happening22:37
niemeyerfwereade: Breaking when something awkward happens seems legit22:38
fwereadeniemeyer, not actually true, but it could be made to be so22:38
niemeyerfwereade: Ah, indeed, the final pull I guess22:38
fwereadeniemeyer, yeah, the charm dir itself is not a swappable symlink22:38
fwereadeniemeyer, although maybe it should be22:38
niemeyerfwereade: COol, we can leave that22:38
niemeyerfwereade: Even then, I'd be fine with asking the user to go there and fix it for the moment, before we're sure that stuff works fine22:39
niemeyerfwereade: Seems useful to learn about how frequently we'll observe lock files left around and whatnot22:39
fwereadeniemeyer, hm, ok, let me think a sec22:39
niemeyerfwereade: We can always change our mind and be less conservative22:40
fwereadeniemeyer, in that case I suspect I favour just explicitly blowing up when we see a lock file that wold impede the impending operation22:41
fwereadeniemeyer, sane granularity?22:41
niemeyerfwereade: Yeah, that's what I mean22:41
niemeyerfwereade: Ah, explicitly22:41
niemeyerfwereade: You mean checking it's there?22:41
fwereadeniemeyer, we shouldn't be operating on the charm dir except at times when the user really shouldn't be operating on the charm dir22:41
fwereadeniemeyer, hmm, just let it fail even :)22:42
niemeyerfwereade: Right.. there's no point in checking, since that's exactly the race that the lock file is meant to protect against22:42
fwereadeniemeyer, true :)22:42
fwereadeniemeyer, I'm just thinking it through... this is a potential new error state22:43
niemeyerfwereade: Really? How is this any different from any other error related to the charm deployment?22:44
niemeyerfwereade: Can't create dir, can't download, etc22:44
fwereadeniemeyer, ah, ok, it's a totally-screwed-agent-down-uniter-log-filled-with-errors situation, not a polite-error-message-log-in-nicely-and-resolve-it one?22:45
fwereadeniemeyer, I guess all of those are22:45
niemeyerfwereade: Right22:45
fwereadeniemeyer, yep, ok, I'm just more scared of git because I'm more aware of my ignorance22:46
niemeyerfwereade: Well.. some of those errors surely are resolvable too, right?22:46
niemeyerfwereade: I'm not sure if you're saying these are conditions we can't recover from or not?22:46
fwereadeniemeyer, I'm just saying that currently we don't22:47
niemeyerfwereade: Okay, I'm happy to move on and polish later, but in principle those conditions should also be recoverable22:47
niemeyerfwereade: E.g. a download error is a pretty unexceptional error22:48
niemeyerfwereade: As Amazon would gladly tell you22:48
fwereadeniemeyer, so *that* will work just fine assuming the resource is actuallythere... eventually it'll succeed22:48
niemeyerfwereade: Perhaps not gladly, no :-)22:48
fwereadeniemeyer, and therefore shouldn't be reported, at least not immediately, because we expect it to be transient22:49
niemeyerfwereade: Right, ok.. so that's what I mean.. I see all of those cases as "wedged, needs action"22:49
niemeyerfwereade: but "juju resolved" in general should mean "Alrightly, will try again and see if that works now"22:49
fwereadeniemeyer, I'm not sure I agree that a dropped connection during a download is reason enough to stop doing things until the user comes to help us22:51
niemeyerfwereade: Sorry, yes, you're right.. that's not a good example22:52
fwereadeniemeyer, that's the trouble22:52
niemeyerfwereade: I mean a case like "can't create directory"22:52
fwereadeniemeyer, some uniter errors are resolvable by time, and others justify a stop-the-world22:52
fwereadeniemeyer, we do not as yet have a way of classifying them22:53
niemeyerfwereade: I see, ok22:53
fwereadeniemeyer, it would be a nice thing to be able to do so but it also feels like it might get hairy22:53
niemeyerfwereade: I think we might do with some heuristics22:54
niemeyerfwereade: without having to put them in separate buckets22:54
fwereadeniemeyer, so, given that, I would like to just drop the concept of recovery entirely and let the process fail as it pleases22:55
niemeyerfwereade: E.g. if it's something we want to retry automatically, don't yet tell it's an error22:55
niemeyerfwereade: But if we retry so many times, or for so long, and it's not working, stop trying and get into an error state22:55
fwereadeniemeyer, yeah, reasonable -- feels like a different change to just dont-ever-recover though22:55
niemeyerfwereade: This is even a conservative approach to prevent a large system from self-destructing22:55
niemeyerfwereade: Agreed22:56
fwereadeniemeyer, but, yeah, absolutely, sounds like what we should be doing to me22:56
niemeyerfwereade: To make things simpler for us to move forward, it's probably fine to start with don't-ever-recover, and move towards auto-retry22:56
fwereadeniemeyer, well, auto-retry will happen on everything for free via upstart22:57
niemeyerfwereade: That's not very nice22:58
niemeyerfwereade: We're managing retrying internally in other cases already22:58
niemeyerfwereade: E.g. connections to state22:58
niemeyerfwereade: Well, all errors really22:59
niemeyerfwereade: In other agents we manage retry-after-error internally22:59
fwereadeniemeyer, oh, er, yeah, sorry, I *did* copy that bit in from the machine agent23:00
fwereadeniemeyer, ok, I see where you're coming from now23:00
fwereadeniemeyer, so managing the heuristics becomes a little easier, and it's still a good idea to add some sort of heuristics23:01
fwereadeniemeyer, er, sorry nonsense23:01
fwereadeniemeyer, it feels to me like "try again" should always be reasonable behaviour if we've written the workers right23:02
fwereadeniemeyer, the spinning and logspam are unsightly, it is true23:03
niemeyerfwereade: It is always true, but in some cases we may need authorization to do so, due to the non-idempotency of some operation23:04
fwereadeniemeyer, and that should be handled internally by the uniter, surely?23:05
fwereadeniemeyer, the uniter knows about these things; the unit agent just needs to keep a uniter running at all times, except when it's upgrading23:05
fwereadeniemeyer, ah ok I think I see a source of confusion23:06
niemeyerfwereade: That, what?23:06
fwereadeniemeyer, when I say always-retry is the default I am specifically referring to errors that cause the uniter to fail23:06
fwereadeniemeyer, hook errors do not cause the uniter to fail -- when state indicates a hook error, the uniter runs just fine, in a mode in which it waits for resolution23:07
fwereadeniemeyer, the uniter will not retry an upgrade or a hook unless requested23:07
fwereadeniemeyer, and so it should always be safe for the unit agent to launch a new one after the first has failed23:08
niemeyerfwereade: If a conflict happens on a merge.. how is that handled?23:08
fwereadeniemeyer, the GitDir returns ErrConflict, magic happens, the uniter runs ModeConflicted23:09
niemeyerfwereade: and then?23:10
fwereadeniemeyer, eventually the user resolves the error, and we pull again, commit, and continue in ModeStarted23:10
niemeyerfwereade: Okay, it is an error that doesn23:11
niemeyer't cause the uniter to bail out, then?23:11
fwereadeniemeyer, exactly23:11
niemeyerfwereade: How's a Conflicted error different from a "directory has a lock" one, from our perspective?23:11
fwereadeniemeyer, no different, essentially; this is a point of trouble :/23:12
niemeyerfwereade: How's it trouble? It seems like a solution.. if it's the same, let's unify the handling23:12
fwereadeniemeyer, ha, yes indeed23:13
fwereadeniemeyer, there's no good reason for any of the git operations to fail, none of them touch the network23:13
fwereadeniemeyer, if they *ever* do we can go straight to help-fix-git-please mode23:13
niemeyerfwereade: Right23:14
niemeyerfwereade: help-fix-deployment even23:14
niemeyerfwereade: heck-were-is-my-data alternatively23:14
niemeyer:-D23:14
niemeyerwhere23:15
fwereadeniemeyer, indeed, it's currently called ModeConflicted and assumes that it's because of an upgrade, but I don;t think that's fundamental, just conincidental23:15
fwereadehaha23:15
niemeyerfwereade: Yeah, ModeDeploymentError or something23:16
niemeyerfwereade: status-info ftw23:16
fwereadeniemeyer, yep, ok, I think it's all percolating nicely through my brain; time to sleep for now, I hope I'll have time to sort out smarter error handling before too long23:22
fwereadegn23:22
niemeyerfwereade: Have a good night23:22

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!