/srv/irclogs.ubuntu.com/2012/08/31/#juju-dev.txt

davechen1yniemeyer: you make a good point01:17
niemeyerFor context,01:17
niemeyer<davechen1y> http://codereview.appspot.com/6497057/ is my proposal01:17
niemeyer<niemeyer> I suspect it's a bit more involved than that01:17
niemeyer<niemeyer> As the comment suggests, we should only push secrets when they are not yet set, and should push *only* secrets01:17
niemeyer<davechen1y> should we take this to the chhanel ?01:17
niemeyer<niemeyer> Sure01:17
davechen1yniemeyer: what happenes currently is AllAttrs - SecretAttrs are pushed on bootstrap01:17
davechen1yso, with 649705701:17
davechen1ythe set of attrs not pushed would only be the secrets01:18
davechen1ybut this is only by coincidence, not design01:18
niemeyerdavechen1y: Hmm.. I'm not sure I understand what that means01:18
niemeyerdavechen1y: That's what have been aiming at01:18
* davechen1y finds code01:18
niemeyerdavechen1y: Also, it's not true that we currently push AllAttrs-SecretAttrs.. it will be so once wrtp's branch goes in with the fix, though01:19
davechen1yniemeyer: hmm, there are unit tests that check this01:19
davechen1ycertainly up to the point that they are yaml encoded and passed to cloudinit.Config01:20
niemeyerdavechen1y: If we have unittests for this, they're broken or not being run01:20
niemeyerdavechen1y: https://codereview.appspot.com/6500052/diff/2001/environs/ec2/ec2.go01:20
niemeyerdavechen1y: Line 132 of the left-hand side01:20
davechen1yniemeyer: http://codereview.appspot.com/6458161/patch/17001/1500701:24
niemeyerdavechen1y: Sure, it works because updateSecrets is broken, right?01:30
niemeyerdavechen1y: The configuration being tested there is after updateSecrets has set everything over agian01:30
niemeyersent01:30
davechen1yniemeyer: yes, you are correct01:31
davechen1yniemeyer: http://codereview.appspot.com/6458161/diff/17001/state/state_test.go?column_width=80, rhs, like ~ 8801:32
davechen1ythat doesn't go through the conn01:32
davechen1yso the problem must be the handoff to jujud bootstrap-state via cloudinit01:32
niemeyerdavechen1y: I'm not entirely sure about what you're trying to show me.. the test contains the full configuration..01:32
niemeyerdavechen1y: The problem is in the exact line I've shown you above, and wrtp's branch fixes it01:33
davechen1ycool, i'll wait for it to lang01:34
davechen1yland01:34
niemeyerdavechen1y: You can also use it as a pre-req if you'd like to build on it01:36
niemeyerdavechen1y: It's already approved01:36
davechen1ysweet, will do01:36
niemeyerOK: 1 passed02:11
niemeyerPASS02:11
niemeyerok      launchpad.net/juju-core/mstate/presence 1.111s02:11
niemeyer!!!!!02:11
niemeyerOn that note, I'm heading to bed as tomorrow will be an early day02:11
* davecheney opens the champaign02:11
niemeyerdavecheney: I wish I could enjoy it with you :)02:11
davecheneyTO LISBON!02:12
niemeyerdavecheney: Oh yeah!02:12
davecheneyMOAR PORT WINE02:12
niemeyerSO TRUE02:12
niemeyer8)02:12
niemeyer'davecheney: Have a great working day, and will see you in a bit!02:12
davecheneywill do02:12
* niemeyer takes off02:12
davecheneycommon LP, you can do it03:02
davecheney% lbox propose03:02
davecheneyerror: Failed to load data for project "juju-core": Get https://api.launchpad.net/devel/juju-core: unexpected EOF03:02
wrtpdavecheney: mornin'04:59
davecheneywrtp: hody!05:15
davecheneycan you commit your bootstrap branch05:15
davecheneyso I can unfuck my cross branch that requiresit05:15
* davecheney hates prereqs so much05:15
wrtpdavecheney: sorry for stepping on your toes with the provisioner fix - i'd done the fix before i checked the bug and found it was InProgress05:16
wrtpdavecheney: am just about to submit05:16
davecheneywrtp: no appology needed05:16
davecheneydon't care, it's fixed :)05:16
wrtp:-)05:16
davecheneyi have a secrets only branch to follow05:16
wrtpdavecheney: it was great that you'd added the last "happens in the wild" comment to the bug report05:16
wrtpdavecheney: 'cos otherwise i wouldn't have found the other firewaller bug05:17
davecheneywrtp: also, nice use of the watcher inside the provisioner test05:17
wrtpdavecheney: it was a little more involved than perhaps it should be, but i hate fuckin' timeouts05:18
wrtpdavecheney: we have far too many tests that rely on waiting for arbitrary time periods05:19
davecheneywrtp: 110% agree, see my whinge about https://bugs.launchpad.net/juju-core/+bug/103742105:20
wrtpdavecheney: yeah.05:20
wrtpdavecheney: i wonder if we should standardise on using debug log messages to determine "actively doing nothing"05:20
wrtpdavecheney: because that's usually what we're testing when we timeout05:21
davecheneywrtp: not sure, like it says in the ticket, i haven't looked into _why_ this happens at all05:21
wrtpdavecheney: presence may be different05:21
wrtpdavecheney: i looked at the timing of the presence tests BTW, and it's not that there's one test that's slow - there are many slow tests there.05:22
davecheneywrtp: 0% cpu usage while it is waiting, i smell timeout05:22
wrtpdavecheney: definitely05:22
wrtpdavecheney: just doing final test after merging trunk, then will submit05:27
davecheneykk05:27
wrtpdavecheney: the other thing to make tests faster: it would be great if all the tests could use the same state server.05:27
davecheneywrtp: i cant' see how we could do that in zk, but with mongo probably05:29
wrtpdavecheney: apparently it is possible in zk (you can do the equivalent of chroot) but mongo is more important now.05:30
wrtpdavecheney: submitted05:30
davecheneywrtp: i guess with mgo it would be some sort of unique prefix on the table names05:31
wrtpdavecheney: i guess.05:32
wrtpdavecheney: BTW do you think that https://codereview.appspot.com/6499056/ is superficial enough to submit without niemeyer's LGTM?05:32
davecheneywrtp: you have my permission to throw me under the bus if niemeyer objects05:33
davecheneyi've been submitting build fixes without waiting05:33
wrtpdavecheney: i didn't see any buses in lisbon05:34
davecheneyyou never say LONGO BUS ?05:34
wrtpdavecheney: oh yeah, one thing: there was a test in firewaller_test that looked like a debugging remnant. see line 81 of https://codereview.appspot.com/6499056/diff/5001/worker/firewaller/firewaller_test.go05:34
wrtpdavecheney: i left it in, but was it there for a particular reason?05:35
davecheneywrtp: dunno, i didn't write it (i think)05:35
wrtpdavecheney: actually i came from the airport on a bus. you may be in danger.05:35
wrtpdavecheney: frank thought you did actually05:35
davecheney s/say/saw05:35
wrtpdavecheney: that bit of it anyway05:35
davecheneywrtp: fuck it, it's called refactoring05:36
wrtpdavecheney: yeah. i didn't want to leave out tests that actually checked functionality.05:36
wrtpdavecheney: but that just tests that the instance has been started, which we test elsewhere05:36
davecheney# launchpad.net/juju-core/juju05:38
davecheney./conn.go:95: env.Get undefined (type *config.Config has no field or method Get)05:38
davecheney./conn.go:96: env.Set undefined (type *config.Config has no field or method Set)05:38
davecheney./conn.go:99: env.Write undefined (type *config.Config has no field or method Write)05:38
davecheneysadface05:38
davecheneyi was using those05:38
wrtpdavecheney: we could add Get.05:44
wrtpdavecheney: i don't think the other two are appropriate though05:45
davecheneywrtp: what CL changed this05:45
davecheneyi'll get my learn on05:45
wrtpdavecheney: in fact, i wonder if the lack of Write is a problem05:45
wrtpdavecheney: it's been going to happen for ages05:45
wrtpdavecheney: frank did it at niemeyer's request05:45
wrtpdavecheney: right, you're officially at risk05:49
davecheneywrtp: jolly good05:52
davecheneywrtp: any idea why this happens ?06:12
davecheneyubuntu@server-15347:~/go/src/pkg/net/http$ godoc launchpad.net/juju-core/environs/ec206:12
davecheneyPACKAGE06:12
davecheneypackage ec2 import "launchpad.net/juju-core/environs/ec2"06:12
wrtpinteresting06:12
davecheneytested from a clean checkout06:12
wrtpdavecheney: yes, happens for me too06:13
wrtpdavecheney: but works fine on dummy, for example06:14
davecheneyhow odd06:14
* wrtp refuses to get distracted :-)06:15
davecheneywrtp: i'll lot a bug to figure out WTF that is happening06:17
davecheneycan I test the waters with this CL, http://codereview.appspot.com/6492065/06:28
davecheneyI was playing on canonistack today06:29
davecheneyit is very possible to do a three line install of juju from source from scratch06:29
davecheneyhas anyone else run into this issue btw, https://bugs.launchpad.net/juju-core/+bug/104416406:34
wrtpdavecheney: i think that's a good idea06:43
wrtpdavecheney: i think it's a pity that go get doesn't have a way of asking for testing deps too06:43
davecheneywrtp: it's kind of a hack06:44
davecheneybut it works06:44
davecheneyand is reasonably self contained06:44
wrtpdavecheney: given the current limitations of go get, i think it's reasonable.06:45
wrtpdavecheney: i've fixed the issue you had above BTW06:45
wrtpdavecheney: (it happens all the time for me)06:45
davecheneygodoc ?06:45
davecheneyor AWS06:45
wrtpdavecheney: sig mismatch06:45
davecheneydo I need to update goamz ?06:46
wrtpdavecheney: i can't remember if i submitted or not06:46
wrtpdavecheney: hmm, maybe i never even proposed it06:47
wrtpdavecheney: ha, i never even committed the change!06:48
wrtpdavecheney: short story: goamz is totally broken currently06:48
wrtpdavecheney: fix: add / to the urls06:48
* davecheney facepalm06:48
davecheneyah yes06:49
davecheneybucket.s3.amazon.com is a host06:49
davecheneybucket.s3.amazon.com/ is a url06:49
davecheneyor something like that06:49
wrtpdavecheney: it signs the path, which should be "/" but is ""06:50
davecheneyright06:50
davecheneythe root of a URL is always tricky06:50
davecheneywell it's not tricky06:50
davecheneyit's very simple06:50
davecheneybut somehow always catches things out06:50
wrtpdavecheney: URLs are fundamentally broken06:51
davecheneywrtp: no argument there06:51
wrtpdavecheney: we just hobble around the brokenness06:51
wrtpdavecheney: any sequence of slashes should be a separator06:51
wrtpdavecheney: like in unix paths06:51
wrtpdavecheney: mind you, even in unix it's a bit broken: i think cat /etc/passwd/ should work06:52
davecheneywrtp: not sure I can follow you there06:52
davecheneyis a collection06:52
davecheney /index is a resource06:53
davecheney / is a collection06:53
wrtpdavecheney: i don't understand that06:53
davecheneyput it this way06:53
davecheney / is a dir06:54
davecheneyactually never mind06:54
davecheneyiw as talking cack06:54
wrtpdavecheney: the model becomes really simple if trailing slash (or several) make no semantic difference.06:55
wrtpdavecheney: with root being a special case of course, so that we get relative paths.06:55
davecheneywrtp: what does cat /etc/ do ?06:57
wrtpdavecheney: depends on the system06:58
davecheney:)06:58
davecheneyyou've used solaris06:58
wrtpdavecheney: once upon a time it gave you the contents of the dir in binary format06:58
wrtpdavecheney: (it still does in plan 9, and i think that's ok)06:58
davecheneywrtp: i kinda agree06:58
wrtpdavecheney: but the point is {operation /etc/} is the same as {operation /etc}06:58
davecheneyi'm not sure if I want /etc and /etc/ to *be* the same thing06:59
wrtpdavecheney: they are06:59
davecheneywell currently /etc/ isn't, it's an error06:59
wrtpdavecheney: no it's not - it's just fine06:59
wrtpdavecheney: ls /etc/06:59
wrtp% ls /etc/ | md5sum06:59
wrtp910833b8b809292bd85de24a9536df74  -07:00
wrtp% ls /etc | md5sum07:00
wrtp910833b8b809292bd85de24a9536df74  -07:00
davecheneybut then we get to the issue of ls /etc/passwd/07:00
wrtpdavecheney: /etc/passwd/ is just an error because something in the kernel is trying to be clever07:00
davecheneyi try not to be clever, to many things have gone wrong07:00
wrtpdavecheney: or probably because of the way the namec works internally07:00
wrtps/the/that/07:00
davecheneyhurry up ec207:01
wrtpif it's still called namec any more07:01
davecheneyi wanna propose and get to my friends leaving drinks07:01
davecheneynamec is the function ?07:01
wrtpin plan 9 you could do cat /etc/passwd/ if /etc/passwd existed :-)07:01
wrtpdavecheney: yeah, it's the kernel name lookup function, or used to be07:02
wrtpdavecheney: probably not under linux07:02
davecheneydamn open sores07:02
wrtpi may well be remembering very badly :-)07:02
wrtpdavecheney: my memory is indeed faulty. it's namei. http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/sys/sys/nami.c07:08
davecheneylater lads07:09
wrtpdavecheney: cheerio07:09
TheMuemorning08:32
Arammoin.08:45
TheMueAram: hi09:09
TheMueAram: how are the watchers doing?09:31
AramI ma fighting bugs.09:31
TheMueAram: if i can help you let me know.09:33
TheMueAram: i'm interested in how they are done now. so any paste is welcome.09:33
Aramok, let me bzr push09:33
wrtpAram, TheMue, fwereade__: morning09:34
Aramyo.09:34
fwereade__wrtp, hey09:34
TheMuewrtp: good morning, too09:34
fwereade__hey everyone else too :)09:34
TheMuefwereade__: and also you ;)09:34
wrtpi've actually been around for over 4 hours this morning, just forgot to say hello earlier!09:34
=== wrtp is now known as rogpeppe
AramTheMue: lp:~aramh/juju-core/62-mstate-watchers-mux09:35
Aramlook in watchserver.go09:35
Aramthere are bugs, but that's the general idea09:35
TheMueAram: great, thx09:35
Aramsome types will change, e.g. will use a chan []genericDoc instead of chan genericDoc, some loops will change from m x n to n x m etc.09:39
Arambut now I am debugging the thing first.09:39
Aramah, there will be e generic unmarshalling function using reflection09:40
Aramrogpeppe: I forgot how can I run a single test.09:42
rogpeppeAram: -gocheck.f pattern09:42
Aramrogpeppe: thanks, how do I find this information if I need it again?09:42
rogpeppeAram: run the test binary with -help09:43
Aramah yes, I was running go test itself with -help09:43
TheMuefor standard tests my editor scans the file and provides a shortcut to a menu where all or individual tests can be run. that's nice.09:44
rogpeppeTheMue: do you remember talking about the possibility of using a single watcher for the EnvironConfig for both provisioner and firewaller?09:49
rogpeppeTheMue: it looks like it should be ok to me, but i'm not entirely sure.09:49
Arambtw, in the new watcher implementation, there is a single real watcher. a watch server demultiplexer, so to speak.09:49
Aramit's only one of those per session.09:49
TheMuerogpeppe: yeah, we once talked about it. i don't remember anymore why we chosen watchers for each.09:50
rogpeppeAram: that sounds like a good move09:50
Aramit gets the data and distributes it to interested parties09:50
rogpeppeAram: i was hoping for something like that09:50
rogpeppeAram: in this case though, it's more about saving code than watchers per se09:50
TheMuerogpeppe: they should reference the same environ, so that one SetConfig() should be enought.09:50
rogpeppeTheMue: exactly09:50
TheMuerogpeppe: but there are TWO "should" in that sentence ;)09:50
rogpeppeTheMue: i think i'll do it. shouldn't be hard. and it removes some moving parts (and hence tests) from the firewaller and the provisioner.09:51
TheMuerogpeppe: +109:52
rogpeppefunc NewProvisioner(environ environs.Environ, st *state.State) *Provisioner09:52
TheMuerogpeppe: it simply sounds more logical09:52
rogpeppefwereade__, Aram: does that sound reasonable to you?09:52
* fwereade__ reads back for context09:53
rogpeppefwereade__, Aram: using a single shared environs.Environ for the various workers, and having a separate process watching it and updating it when necessary.09:53
fwereade__rogpeppe, +109:53
Aramlgtm09:53
TheMueAram: i'm no friend of the reflection based unmarshal idea. ok, it may save a few lines, but we only talk about four internal docs, so your actual way seems fine to me.10:00
Aramyes! the fix I dreamt last night was correct!10:29
Aramdreaming fixes has happened before, but not in a long time.10:29
AramI tried it in the morning, and thought it didn't work, but my fix was wrong because the zero value of interface{} is nil, not 0.10:30
Arambut what I have dreamt was correct :).10:30
fwereade__Aram, nice :)10:31
Aram"panic: interface conversion: interface is int, not mstate.Life"10:33
AramTheMue: ^ ^ this is the reason I want to use reflection, not because it produces less code.10:33
Arambut because the code is necessarily correct and doesn't break when you change the data model.10:34
fwereade__conversely, I just spent all frickin' morning delving deeeeep into the guts of the charm manager, trying to figure out what's going on, and I *just* realised it was a poorly merged test :(10:34
rogpeppefwereade__: my fault?10:38
fwereade__rogpeppe, nope, 100% mine10:39
rogpeppefwereade__: my commiserations10:39
rogpeppefwereade__: i hate it when that happens10:39
AramYES! everything works.10:39
rogpeppefwereade__: the bright side is you usually come out with a better understanding of everything goingon10:39
Aramfixed the damn thing.10:39
rogpeppefwereade__: and probably made some fixes while you were doing it10:39
fwereade__rogpeppe, sadly in this case all I have done is come up with a series of ever-more-batshit theories about how git works, and I fear they may pollute my brain for a while ;)10:40
rogpeppefwereade__: oh well10:40
AramTheMue: you might want to pull again, to see working watchers.10:40
Aramtests pass now10:40
* fwereade__ cheers at Aram10:41
Aramit's by far the most complex piece of machinery in mstate, sadly.10:41
Arambut I am happy with it from many perspectives.10:42
TheMueAram: great10:52
TheMueAram: and yes, the server is a little monster ;)10:52
rogpeppeAram: which piece is this?10:52
Aramrogpeppe: lp:~aramh/juju-core/62-mstate-watchers-mux10:52
rogpeppeAram: i will have a look :-)10:53
Aramwatchserver.go is the real thing, watcher.go is just a quick hack to be able to validate the design.10:53
rogpeppeenviron watchers ripped out of firewaller and provisioner. all tests pass (some new tests needed though). am happy.10:53
TheMueAram: btw, I'm just doing the EnvironConfig change for mstate10:54
AramTheMue: what change?10:54
TheMuerogpeppe: great., i hope https://codereview.appspot.com/6488057/ will then still run too10:54
TheMueAram: state.EnvironConfig now returns an environs/config.Config and there's an additional SetEnvironConfig10:55
Aramok.10:55
TheMueAram: test is already working here10:55
rogpeppehave we got a meeting now?11:01
davecheneywaiting on gustavo and mramm11:02
rogpeppedavecheney: just a quick query11:02
mrammI'm on my way11:02
niemeyerMorning all11:02
niemeyerDo we have invites sent yet?11:02
mrammnot yet11:02
mrammthat i see11:02
davecheneyrogpeppe:mmm11:02
mramminviting everybody now11:02
rogpeppedavecheney: i decided that it would be a good thing if the provisioner didn't listen for environ config changes itself11:03
rogpeppedavecheney: so something else listens and changes the environ; everyone uses that environ11:03
rogpeppedavecheney: does that sound reasonable to you?11:03
davecheneyrogpeppe: why the reason for the change ?11:03
rogpeppedavecheney: i was going to have to have exactly the same code in three places11:04
davecheneyit sounds like my beloved proxy environ idea :)11:04
rogpeppedavecheney: no need for a proxy i think11:04
mrammhttps://plus.google.com/hangouts/_/4ca5fa037ece34849caa79686808bc9e75cfae1911:04
Aramah, meeting today?11:04
rogpeppeAram: yup11:05
mrammhttps://launchpad.net/juju-core/+milestone/1.311:06
rogpeppefwereade__: meeting?11:06
fwereade__rogpeppe, whoops, ty11:06
=== Aram2 is now known as Aram
davecheneyAram: https://wiki.canonical.com/UbuntuEngineering/Sprints/JujuCoreIntegration11:48
* Aram goes to lunch11:58
TheMuelunchtime12:00
niemeyer                case change, ok := <-fw.environWatcher.Changes():12:06
niemeyer                        err := fw.environ.SetConfig(change)12:06
rogpeppeniemeyer: http://paste.ubuntu.com/1177699/12:07
=== ChanServ changed the topic of #juju-dev to: Milestone goals: 1) mstate in, state out 2) uniter working with simple charms; 3) upgrade-juju working
davecheneynight guys12:22
niemeyerrogpeppe: http://pastebin.ubuntu.com/1177736/12:35
niemeyerrogpeppe: fw.environ = environ12:46
niemeyerrogpeppe: worker.SetEnvironConfig(fw.environ, config)12:46
niemeyerrogpeppe: https://codereview.appspot.com/6488057/diff/1/worker/provisioner/provisioner_test.go12:49
niemeyerrogpeppe: case fw.environTest <- fw.environ: // do nothing12:52
niemeyerrogpeppe: someworker.SetEnvironWatcher12:53
niemeyerrogpeppe: someworker.SetEnvironObserver12:54
niemeyerrogpeppe: someworker.SetEnvironObserver(ch)12:54
niemeyerrogpeppe: case fw.environObserver <- fw.environ: // do nothing12:54
rogpeppeniemeyer: http://paste.ubuntu.com/1177767/13:05
rogpeppeniemeyer: perhaps you could mention to TheMue the thoughts you had on the worker tests, re: the above provisioner_test.go CL.13:11
niemeyerrogpeppe: Will do13:11
niemeyerWill get some breakfast first, though!13:14
rogpeppeniemeyer: are we concerned about what might happen if an upgrade is done concurrently with an environment setting change?13:49
niemeyerrogpeppe: In which sense?13:49
rogpeppeniemeyer: would it be acceptable for the upgrade request to be lost, for example?13:49
rogpeppe(in that case)13:49
niemeyerrogpeppe: Are you alluding to the race in SetEnvironConfig?13:51
rogpeppeniemeyer: yeah.13:51
niemeyerrogpeppe: Not a problem we have to worry for a while, IMO13:51
rogpeppeniemeyer: well, the race that happens when you must do: EnvironConfig followed by SetEnvironConfig.13:51
niemeyerrogpeppe: yeah13:51
rogpeppeniemeyer: ok, i thought so, but just wanted to check explicitly13:51
rogpeppeniemeyer: thanks13:52
niemeyerrogpeppe: We should fix that at some point, probably by having a method that enables applying deltas safely, but this is a nice problem for us to have when things actually work13:52
niemeyerrogpeppe: In practice, fiddling with the env is rare enough that it won't be an issue anytime soon13:52
rogpeppeniemeyer: definite agreement there. and fiddling with the env *concurrently* is even rarer.13:53
niemeyerrogpeppe: Yeha13:53
fwereade__niemeyer, before I polish this properly, can I get another pre-review sanity check on the charm upgrades please? https://codereview.appspot.com/648806214:00
niemeyerfwereade__: Sure14:02
niemeyerfwereade__: I'll just finish something and will be with you14:02
fwereade__niemeyer, cheers14:02
niemeyerFlights booked14:21
niemeyerfwereade__: Haven't reviewed it entirely, but sent some pre-review comments14:29
fwereade__niemeyer, thanks, that's what I was after -- and sorry to keep bugging you with this one14:29
niemeyerfwereade__: It still looks like you're fighting with the awkwardness of having the charm manager handling responsibilities which ought to be elsewhere14:30
fwereade__niemeyer, it seems excessive to me to have to keep *another* state file around to allow the uniter to keep track of what operation it's performing14:31
fwereade__niemeyer, it feels to me that being able to determine restart state purely from charm state and hook state is a good thing14:32
fwereade__niemeyer, charm state + hook state + uniter state feels wrong -- especially since IMO charm state and hook state are useful simplifying features14:32
fwereade__niemeyer, but once we're dealing with uniter state as well they actually become redundant and make life more complex14:32
fwereade__niemeyer, or have I misunderstod something?14:33
fwereade__niemeyer, whereas just making sure that the charm op and the hook op overlap STM like the clearest way to ensure that the appropriate ops are followed by appropriate hooks14:34
niemeyer/ charm directory, it returns ErrConflict. cbSuccess will be called before14:34
niemeyer/ status is set from st back to Deployed, to give clients the opportunity14:34
niemeyer/ to complete dependent actions.14:34
niemeyerfwereade__: This is a hack14:34
niemeyerfwereade__: This is a charm.Manager..14:35
niemeyerfwereade__: It has deployed the charm14:35
niemeyerfwereade__: There are no "dependent actions"14:35
niemeyerfwereade__: It has done it14:35
niemeyerfwereade__: No conflicts, no problems.. it's finished its job14:35
niemeyerfwereade__: It doesn't depend on anything else to say it has finished its job either14:35
niemeyerfwereade__: We can talk about different ways in which we can solve that, but first thing is to understand what we're doing and why14:36
fwereade__niemeyer, ok -- what I am trying to do is ensure that a charm operation is always followed by the apropriate hook14:37
fwereade__niemeyer, ISTM that the two ways to do that are to overlap the operations, so the charm op ends after the hook op has begun; or to add *another* layer of state that's specific to the uniter, which feels redundant14:38
niemeyerfwereade__: So what you mean is that you'd rather do the whole upgrade again than acknowledge the fact we can tell the merging has been done?14:39
fwereade__niemeyer, er, I don't think that's what I'm saying at all14:40
niemeyerfwereade__: That's what's in the code14:40
niemeyerfwereade__: If cbSuccess fails, we never set the status to deployed, despite the fact it *is deployed*14:40
fwereade__niemeyer, so what? same thing happens if writing "Deployed" fails14:41
niemeyer<niemeyer> fwereade__: So what you mean is that you'd rather do the whole upgrade again than acknowledge the fact we can tell the merging has been done?14:41
fwereade__niemeyer, are yu telling me I should be parsing git logs or something?14:42
niemeyerfwereade__: No.. I'm telling that we shouldn't have a cbSuccess that prevents us from acknowledging the merge has been done, for purposes that are completely external to the concerns of the Manager14:42
niemeyerfwereade__: Divide and conquer.. the Manager is doing a lot14:43
niemeyerfwereade__: We should have a pleasant API to work with, that makes sense by itself14:43
niemeyerfwereade__: I don't buy that we should be breaking abstractions and injecting that kind of weirdness because we can save a file from being written to disk elsewhere14:43
fwereade__niemeyer, I am trying, but I don't see the middle ground between what I suggest and dropping independent charm and hook state entirely14:45
fwereade__niemeyer, if I've got a super-state that tells me how to interpret charm and hook state, doesn't it end up noticeably simpler to simply write every operation out in just one place? ...but then we lose many of the benefits of separating out /charm and /hook14:47
fwereade__niemeyer, or...... hum, I think I see14:47
fwereade__niemeyer, the uniter state is basically independent of hook state and exists purely to manage the transition from charm op to associated hook?14:48
niemeyerfwereade__: Possibly. I'm not even entirely sure (yet) we need an extra state, to be honest14:49
fwereade__niemeyer, ok -- the problem I am trying to avoid is: write-charm; set-deployed; whoops-we-crashed-before-run-hook14:50
niemeyerfwereade__: If the uniter decides it needs to upgrade, what happens within the uniter itself?14:50
niemeyerfwereade__: Yeah, I kind of see it.. I just don't have as good a view on the details as you do14:50
niemeyerfwereade__: The basic semantics I imagine for the manager, in isolation is: write-charm-and-url, set-deployed14:51
niemeyerfwereade__: If there is a crash, the uniter should have a mechanism that tells it was going to over a charm upgrade.. it will try to write the charm again, and the manager will go "Ah, don't worry, that's what I have"14:52
niemeyerfwereade__: What is that mechanism?14:53
fwereade__niemeyer, the mechanism ATM is charm.ReadState14:53
niemeyerfwereade__: Okay, maybe we can still keep it like that14:53
niemeyerfwereade__: If there's really nothing else we need in terms of uniter information to make that decision14:54
fwereade__niemeyer, well... ok -- but the fact that the charm is deployed is not in itself enough for the uniter to know that it has completed all relevant operations (ie running a ollowup hook)14:55
niemeyerfwereade__: Yeah, there's a missing state14:55
fwereade__niemeyer, so it's uniter state? or are you thinking of something else?14:55
fwereade__niemeyer, or a missing state within an existing set of states?14:56
niemeyerfwereade__: No, it may be fine to do that within the manager if that makes it simple and doesn't disturb the manager itself14:56
niemeyerfwereade__: E.g.14:56
niemeyerfwereade__: Installing, Installed, InstalledAck, Upgrading, Upgraded, UpgradedAck14:57
fwereade__niemeyer, ah, ok, so the Manager is responsible or writing most o those states but the Uniter writes the Ack states afterwards?14:58
niemeyerfwereade__: yeah14:58
niemeyerfwereade__: Or even, Ack()s :-)14:58
fwereade__niemeyer, honestly that feels like a step back from what I originally proposed -- it smears the charm-state-writing responsibility into 2 places instead of one, and that eels to me like it's harder to ollow?14:59
fwereade__hm, my F key appears to be flaky, sorry14:59
niemeyerfwereade__: I'd be glad to understand why you think that's the case14:59
fwereade__niemeyer, well, beforehand I had a single func (changeCharm) that did all the writing of charm state15:01
niemeyerfwereade__: :-)15:01
niemeyerfwereade__: I hope that's not what you're proposing now :)15:01
fwereade__niemeyer, feels analogous to hook state, which while it's in 2 funcs is at least all in the same package15:01
niemeyerfwereade__: (as a single function of 300 lines might look bad)15:01
fwereade__niemeyer, I'm talking specifically about the charm state file15:01
niemeyerfwereade__: I feel like part of the reason we're having so much trouble is precisely that file15:02
niemeyerfwereade__: I'd like to kill the idea that we ReadState and WriteState15:02
niemeyerfwereade__: We have a Manager, that manipulates a charm directory15:02
niemeyerfwereade__: Anyone that is not the Manager should talk to the Manager to change its state15:02
fwereade__niemeyer, ah! if we can move all that state-writing into uniter itself, I'd be entirely happy15:03
niemeyerfwereade__: LOL15:03
niemeyerfwereade__: I was thinking the opposite :-)15:03
niemeyerfwereade__: But, tell me about that idea15:03
fwereade__niemeyer, fuzzily, I think I'm thinking that the charm state is actually uniter operation state15:04
niemeyerfwereade__: Ok15:04
niemeyerfwereade__: Makes sense15:04
niemeyerfwereade__: What would the Manager look like in such a world?15:05
fwereade__niemeyer, I think the manager would be more of a Deployer15:05
fwereade__niemeyer, ...except maybe not quite...15:05
fwereade__niemeyer, aside: when I originally moved the charm dir out of the manager, it really did feel like a step forward, but I ended up not going anywhere very good from there15:07
niemeyerfwereade__: Yeah ,the problem is drawing the line at the right place.. you probably were onto a good direction in some sense, but the line got blurred somehow15:07
niemeyerfwereade__: The idea of a deployer makes sense to me..15:08
niemeyerfwereade__: The only state we need within the deployer is probably the URL that is actually on disk (what we have as readURL)15:09
fwereade__niemeyer, yeah, exactly15:09
fwereade__niemeyer, ok, so assuming we were able to separate out a charm dir, a deployer, and a charm-operation-state file15:09
niemeyerfwereade__: In the uniter state, I suspect we'll still need extra states, though15:10
niemeyerfwereade__: Which some of the causation of the discussion above15:10
niemeyerfwereade__: Since we need to track "need to deploy => deployed and need to run hook => run hook"15:10
fwereade__niemeyer, I *think* that works OK though -- once we're deployed, yay, we're deployed -- but the state is still the uniter's responsibility to recover on unexpected abort15:11
niemeyerfwereade__: Yep15:11
niemeyerfwereade__: So State, ReadState and WriteState are all gone..?15:11
fwereade__niemeyer, well, I think they are divorced from Manager anyway15:12
niemeyerfwereade__: Yep15:12
fwereade__niemeyer, they still feel pretty "charm"y15:12
niemeyerfwereade__: Not to me15:12
niemeyerfwereade__: Anymore than the unit is charmy, anyway15:12
niemeyerfwereade__: "I need to run hook", as an example15:12
fwereade__niemeyer, I can see it both ways15:12
niemeyerfwereade__: Is pretty unity :)15:13
fwereade__niemeyer, yeah, indeed :)15:13
fwereade__niemeyer, ok, so, as a rough sketch: charm.Deployer + charm.GitDir; then some unexported uniter state manipulation in changeCharm around Deployer operations?15:14
fwereade__niemeyer, the readURL thing starts to feel pretty redundant now15:14
niemeyerfwereade__: +1 except for the last sentence15:16
niemeyerfwereade__: The URL has to be put as part of the deployed content if we want to be able to easily tell what's in the directory15:17
niemeyerfwereade__: The uniter state can't track what's *actually* there (after facing a revert, or merge, or whatever)15:17
fwereade__niemeyer, hmm, I'll think about it a bit more... ah, ok, we could quite easily have a *missing* operation state file meaning "deployed, check the charm dir"15:17
fwereade__niemeyer, ok, I think maybe I have direction again15:18
fwereade__niemeyer, cheers15:18
niemeyerfwereade__: I don't see that as something entirely neat.. if we have state, we can easily name it..15:18
niemeyerfwereade__: Having the same URL both in the uniter state and in the charm is a great sign of things being right :)15:19
fwereade__niemeyer, ok, I think my resistance may come from not being able to figure out how I should react if they don't match :)15:19
niemeyerfwereade__: They serve different purposes, anyway15:19
niemeyerfwereade__: Example scenario:15:20
niemeyerfwereade__: We're running an upgrade from revision 10 to 1115:20
niemeyerfwereade__: uniter state says we're in revision 10, trying to install 1115:21
niemeyerfwereade__: we ask the deployer to upgrade, and get a conflict15:21
niemeyerfwereade__: unit goes into error mode15:21
niemeyerfwereade__: user logs into the machine.. fiddles around, reverts15:22
niemeyerfwereade__: user calls resolved15:22
niemeyerfwereade__: uniter catches up, learns that directory has revision 10.. doesn't call upgrade hook, goes back into working mode15:22
niemeyerfwereade__: End of story point.15:22
fwereade__niemeyer, feels like the next stepis actually "goto10"15:23
niemeyerfwereade__: ?15:24
fwereade__niemeyer, uniter returns to working mode; detects an upgrade, and...15:24
niemeyerfwereade__: That's another story15:24
niemeyerfwereade__: At the end of the story above, it gets back onto the main loop, and can do anything it usually would15:25
niemeyerfwereade__: Including upgrading again, sure.. perhaps not even to the same version.15:25
rogpeppeniemeyer: this might be something like we discussed: https://codereview.appspot.com/650107215:27
fwereade__niemeyer, I don't see the value of that story, because what *will* happen is another upgrade, and we can easily and automatically handle a revert-and-upgrade-to-new-version ourselves, even from an error state15:27
fwereade__niemeyer, without a separate actually-don't-upgrade-this mechanism, I don't see why a user would ever need or want to revert15:28
niemeyerfwereade__: The value of that story is that the only way to make it possible is by knowing what's actually inside the directory15:28
niemeyerfwereade__: Ah, maybe it's that's the disagreement.. I totally see why people (or I :-) would like to resolve a conflict, and then revert back to the previous state15:29
niemeyerfwereade__: Because what was done was wrong15:29
niemeyerfwereade__: Even then, we're diving into too much detail here15:29
fwereade__niemeyer, are we talking about one specific unit, or a whole service?15:29
niemeyerfwereade__: The core of my argument is that there's value in knowing what a given revision of the charm directory actually contains15:29
niemeyerfwereade__: I was hoping that this wouldn't be very controversial15:30
fwereade__niemeyer, neither keep-the-url-in-the-charm-dir nor keep-the-url-in-a-state-file are individually controversial in my mind15:30
niemeyerrogpeppe: Cheers, will look15:30
fwereade__niemeyer, keeping *both* of them still makes me nervous15:30
fwereade__niemeyer, because I'm not sure that we can do anything useful with disagreements15:31
niemeyerfwereade__: Okay, I'm happy to have it just within the charm directory while we don't figure a reason to have it outside15:31
fwereade__niemeyer, hence my contention that what we're after is *operation* state, not charm state, and we shouldn't have a URL in operation state when we're not operating15:31
niemeyerfwereade__: I don't know how you'll manage to tell what version you're supposed to upgrade to before the upgrade actually happens, but that's part of the "while we don't figure a reason" part15:32
niemeyerfwereade__: Okay, that makes sense15:32
fwereade__niemeyer, if I go one step further and say we shouldn't have *any* operation state when we're not operating, are you still comfortable?15:33
fwereade__niemeyer, blast, sorry, brb15:35
niemeyerfwereade__: Yeah, I'd definitely not complain if you made that work nicely15:35
fwereade__niemeyer, cool, I'll have another crack at it15:40
fwereade__niemeyer, sorry this bit's taking so long15:40
rogpeppeflight to lisbon booked15:52
niemeyerfwereade__: No problem at all, I appreciated that conversation15:57
niemeyerrogpeppe: Reviewed, good stuff15:57
rogpeppeniemeyer: thanks15:57
fwereade__niemeyer, a pleasure, as always :)15:58
rogpeppeniemeyer: i thought about Errer, but couldn't quite bring myself - it just looks like a spelling mistake. still, if you think it's ok...15:58
niemeyerrogpeppe: It crossed my mind, but it still feels better than Errorer (which I'm probably unable to pronounce :-)15:59
rogpeppefwereade__: o fellow english speaker: what do you think: Errer or Errorer ? (as a name for interface{Err() error})16:02
rogpeppefwereade__: or some alternative?16:02
fwereade__rogpeppe, Errer is an ugly word, but also seems like a legitimate "that which errs", so I think I'd go with that16:03
Aramrename Err to Error and use the builtin error interface?16:03
rogpeppeAram: that would be kinda cool but... do we want all watchers to be errors ?16:04
Aramrogpeppe: ^16:04
rogpeppeAram: it would also be a far reaching change16:04
rogpeppeAram: tomb, for instance, is an external package that uses Err()16:05
niemeyerrogpeppe, Aram: Nope, Err is the proper name for that method, and it's actually a standard convention16:05
niemeyerrogpeppe, Aram: Err is seen in things that report an "error" (the interface); they are not errors themselves16:06
rogpeppeniemeyer: +1 having looked in the go source16:06
Aramwhite:pkg$ pwd16:06
Aram/home/aram/go/src/pkg16:06
Aramwhite:pkg$ csh '\.Err\(' | wc -l16:06
Aram2116:06
Aramit's only used in database and exp/html16:06
niemeyerAram: Yeah, but it's an agreed-to convention, and I actually pushed the patch prior to Go 1 to make it consistent16:07
rogpeppeAram: yeah - it wouldn't be right anyway because Error (error's method) returns a string not an error16:07
rogpeppeAram: which we definitely don't want to do16:07
Aramactually the very few .Err() in the go tree return error, not string.16:09
Aramso while I agree that changing to error is not a good idea, calling three usages of an Err in 400,000 LOCs hardly counts as convention, especially since our Err has a different signature :)16:09
niemeyerrogpeppe: https://codereview.appspot.com/6501069/ reviewed too16:12
rogpeppeniemeyer: ta!16:12
niemeyerrogpeppe: Thank you!16:13
rogpeppeniemeyer: i put the numbering in because it was so painful counting through 13 tests to find which one had failed.16:15
rogpeppeniemeyer: the alternative is to put the comment in the struct16:15
rogpeppeniemeyer: which is probably better, come to think of it16:15
niemeyerrogpeppe: Yes, there are multiple ways to solve that.. putting the comment as a summary in the struct is a generally a great one16:15
rogpeppeniemeyer: i'll go with that then16:15
niemeyerrogpeppe: Thanks!16:15
niemeyerStepping out for lunch16:16
Aramyay, I can use chan []map[string]interface{} as a map key16:18
* Aram wished Put wasn't near Undo in acme.16:25
rogpeppeAram: i don't get bitten by that very much, though i wish Put didn't appear under the mouse when you get to the end of a load of Redos...16:35
Aramyeah, that too16:35
rogpeppeAram: i have never thought of a good way to make it better though16:35
rogpeppeyay, ec2 is fixed again!16:35
niemeyerAram: Gosh, do we need that as a map key? :)17:20
Aramyeah...17:20
niemeyerAram: Why?17:20
Aramit's not that bad, it's only chan []genericDoc, genericDoc is map[string]interface{} because it represents any mongodb document (like bson.M). we need it because this is how client watchers register with the watch server.17:22
Aramthe code is actually quite clear, even if the key is complex.17:23
Aramthe key is what is is because it is the way the server sends event to clients.17:23
Aramclients expect slices of documents.17:23
niemeyerAram: Hmm.. that sounds a bit suspect17:23
niemeyerAram: Why do we have a genericDoc in lower level watcher?17:24
niemeyerAram: That wasn't in the design we talked about17:24
Aramwhat is a lower level watcher?17:24
niemeyerAram: The design we talked about, which I thought you've been working on17:24
AramI have.17:25
niemeyerAram: Well, that wasn't there17:25
niemeyerAram: We don't work with genericDocs in mstate so far17:25
niemeyerAram: The fact we can deal with actual values instead is quite convenient17:25
niemeyerAram: I'm concerned that what we talked about was simpler than what seems to be taking to get it in place17:26
Aramthere is a single entity doing all the queries for all the watchers. watchers have different documents, it has to use a map to be able to get data before converting it to a document.17:26
niemeyerAram: That's not what is in the design17:27
niemeyerhttps://gist.github.com/e1453bb97426081a12e517:27
niemeyerAram: This is the design17:27
Aram"A Watcher can watch any number of documents and collection" that implies arbitrary documents, hence maps instead of structs.17:28
niemeyerAram: You don't have to agree with it, and it's fine to go in a different direction, but when I go over the trouble of doing that after we stumbled upon several issues, I do expect feedback before we change gears17:28
niemeyerAram: Where's genericDoc in this design?17:28
Aramthe fact that you didn't mention genericDoc doesn't mean I can't use it, you didn't mention error or string or int either. genericDoc is just a name I gave for a map, I can change it to bson.M if you want.17:30
Arambecause it is essentially bson.M, not only in representation, but in concept as well.17:30
niemeyerAram: I don't care about the name.. I care about the fact I can't see a "generic map" or however you wanna call it in this design17:30
niemeyerAram: We simply do not *load* the document17:30
Aramsurely someone loads documents as the events delivered by watchers are full entities that have the full document inside them.17:32
niemeyerAram: I'm talking about the design in this page, not about whatever you're implementing on top of it17:32
niemeyerAram: This is a building block that we need, and agreed upon17:32
niemeyerAram: You seem to be talking about stuff that is not in this design, despite the fact we don't have this integrated or up for review anywhere'17:33
niemeyerAram: We seem stuck despite all my efforts, and I'd like to understand why17:33
* rogpeppe loves all it when 90 lines of new test code all passes after one minor initial glitch.17:39
rogpeppesomehow the initial glitch makes it even better, because you have positive verification that the test is actually running.17:39
niemeyerrogpeppe: True17:40
niemeyerrogpeppe: When I get a straight pass I'll generally tweak things a bit to make sure it's *really* passing :)17:40
rogpeppeniemeyer: i often leave a "this cannot pass" piece of code in the test just to make sure17:41
rogpeppeniemeyer: easier when you've got an ErrorMatches test17:41
niemeyerrogpeppe: +1, done that before as well17:42
rogpeppe5 minutes and 21 seconds to upload the tools that time.18:04
rogpeppe(running live upgrading test BTW)18:05
rogpeppeniemeyer: friday's last gasp: https://codereview.appspot.com/6490067/18:15
rogpeppeniemeyer: it passes live tests!18:15
rogpeppeniemeyer: and that should get us roughly back to where we were...18:16
rogpeppeand with that, i'm off to practice drinking port in preparation for Lisbon18:16
rogpeppefwereade__, mramm, niemeyer: have a great weekend!18:16
niemeyerrogpeppe: Superb!18:17
niemeyerrogpeppe: Thanks so much18:18
niemeyerrogpeppe: and have a brilliant weekend too18:18
mrammrogpeppe: have a fantastic weekend, and get good and practiced up on the port drinking!18:27
rogpeppemramm: darn, it's running out. need more supplies!18:32
mrammheading to a late lunch, back in a bit18:48
niemeyerPASS: presence_test.go:148: PresenceSuite.TestScale     1.978s21:56
niemeyer1000 pingers starting, pinging, half being killed, all of them observed.. 2 seconds. Woohay.21:56

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!