/srv/irclogs.ubuntu.com/2013/06/04/#juju-dev.txt

=== bigjools_ is now known as bigjools
=== wedgwood is now known as wedgwood_away
davecheneyhttps://bugs.launchpad.net/bugs/118706201:19
_mup_Bug #1187062: 1.11.0-1~1240~quantal1 cannot find Precise Image on HP Cloud <juju-core:New> <https://launchpad.net/bugs/1187062>01:19
davecheneydoes anyone have the creds to run sync tools ?01:19
wallyworlddavecheney: it's not sync-tools - there's no quantal image metadata uploaded to hp cloud yet, just precise01:33
wallyworldi can add a quantal image entry to the metadata01:33
wallyworldwhat image id do you want?01:34
davecheneywallyworld: thanks for replying01:43
davecheneyi'm wondering if I should tell antonio not to use quantal01:44
davecheneywe're only pushing the LTS01:44
wallyworldthat's what i thought when i uploaded the metadata - that people would just be using precise01:44
davecheneyi think that is a reasonable response01:44
davecheneyand is consistent with the company message01:44
wallyworldi thought that the charms would only be quaranteed to work with the LTS01:44
davecheneyalso, there are almost 0 charms for quantal01:45
wallyworldyeah01:45
wallyworldso i didn;t want to provide a footgun01:45
davecheneywallyworld: the charms also have a series, and there aren't really any for quantal, almost none for raring and ziltch for suacy01:45
davecheneylol @ footgun01:45
wallyworldi know charms have a series - just assumed there were only a number for precise and not many for anything else like you say01:46
wallyworldfoorgun amuses me too01:46
wallyworldfootgun even01:46
wallyworldfor demos, like i think he wants this for, precise is the way to go01:47
davecheneywallyworld: thanks, i'll write some stuff to antonio01:48
wallyworldi had a chat this morning - there was also an issue not reading the release notes i *think*01:49
wallyworlddavecheney: there's a meeting tomorrow where all this will be cleared up fwiw, so don't feel like you have to write too much today01:49
davecheneyok02:17
davecheneyyeah, simple streams02:17
thumperhi jam, you around yet?02:39
thumperdavecheney: I have a question for you if you are around03:26
thumperdavecheney: wondering about lockless reading of an int03:30
thumperdavecheney: when it is possible that another goroutine may be writing to it03:31
thumperany ideas on guarantees?03:31
thumperdavecheney: nm, I'll use a defined size and sync.atomic03:32
jamthumper: hey, what's up ?03:48
thumperjam: hi, I've just tweaked your branch a little03:48
thumperand about to merge it03:48
jamthumper: for loggo?03:48
thumperyeah03:48
thumperjam: I changed the globalMinLevel to be a uint32 explicitly03:48
thumper(and changed the type from int to uint32 for Level)03:49
thumperso I could use sync.atomic for the reads and writes03:49
thumperotherwise we'd be reading an int without a lock03:49
thumperpotential problem03:49
thumperso using sync.atomic for both reads and writes fixes this03:49
jamthumper: because it is a word size, you will either get the existing or previous value (I'm pretty sure), which you are racing on when the int gets set with when you are logging03:49
jamwhich is already a race.03:49
jamwell 'race'03:50
thumperwhile the writes were all within the mutex, the reads weren't03:50
jamthumper: sure, but it doesn't seem like an int that you have to be strict about03:50
thumperwhile most likely, it isn't guaranteed (for the reads)03:50
jambecause if you got the wrong value, you could have also gotten the 'wrong' value because the async call to change it was 1ms later.03:50
thumperyeah, but I'm mildly pedantic about shit like that03:50
jamthumper: for word-size, you won't ever read part of the value.03:50
jamthumper: for the lowest part of logging, it would be nice to avoid mutexes.03:51
jamor sync level calls.03:51
jamdid you check it with the benchmarks?03:51
thumperI seem to recall an exact example where it would03:51
jamthumper: for multi-word stuff, you can get craziness.03:51
thumperlet me poke it with and without the sync calls03:51
jamfor example 'interface' is 2 words03:52
jam(a pointer to type, and a pointer to value, which you can abuse with GOMAXPROCS>1)03:52
jamYou may still want uint32 instead of 'int' for that same reason.03:52
jamthumper: anyway, I have the feeling the specific overhead doesn't matter much, but it is a multi-cpu sychronization point to log something that you are then throwing away (sometimes)03:53
jamthumper: btw, you don't have a good way to change the log level of writers. To change the default logging you have to RemoveWriter() RegisterWriter(), because ReplaceDefaultWriter doesn't let you change the level.03:54
thumperjam: ah, good point...03:54
thumperI suppose we should add that at some stage03:54
jamthumper: I was a little surprised that Logger objects track their log level, but Writer objects have it tracked in a separate location.03:55
thumperseems to be about 5% slower on the fastest case to use sync03:55
jamthumper: tbh the fastest case probably doesn't trigger all that often, since getEffectiveLogLevel is probably going to be the primary DEBUG/TRACE filter?03:56
thumperaye03:56
jamthumper: the goal is to just make it cheap enough that you don't have to think about perf when adding logging, because it will be cheaply filtered out.03:56
thumperso I guess the question is with a read of uint32, will we always get a "whole" value?03:57
jamthumper: note, layering the calls has a measurable performance impact, though in the "NS" range.03:57
thumperwhat do you mean layering?03:57
jamthumper: Debug calling Log03:57
jamit is also measurable if you pass an extra parameter.03:57
jambut again, 13ns per call vs 19ns per call.03:58
jamsort of range.03:58
thumperthat is because I special case the one param03:58
thumperto avoid Sprintf03:58
jamthumper: no, actually, it doesn't get to Sprintf03:58
jamSprintf costs 600ns03:58
jamor so03:58
jamthis is just parameter passing03:58
thumperoh, so jsut the param?03:58
thumperweird03:58
jamif you change the fastest benchmark test03:58
jamthumper: yeah, but again, nanoseconds03:58
thumperand you really shouldn't be logging 1000s of things per second03:59
thumperhopefully03:59
jamthumper: well, millions of things, really03:59
thumperalthough I did log a metric fuck-ton of stuff in unity for testing cleanup03:59
jamI could see TRACE getting really verbose.303:59
thumperjam: exactly, which is why we want better module separation04:00
jambut if you are actually logging stuff, the Sprintfs start to add up.04:00
thumperso you can set trace on say, the provisioner04:00
thumperbut nothing else04:00
thumperyeah, I can imagine that it does add up, but the cost/benefit of the logging is worth it I think04:00
jamthumper: so on my machine, if you look at BenchmarkLoggingDiskWriterNoMessagesLogLevel04:01
jamwith s.logger.Debug(msg, i)04:02
jamit is 25.5ns/op04:02
jamif I change it to04:02
jams.logger.Debug(msg)04:02
jamit is 13.8ns/op04:02
thumperheh04:02
jamif I change it to s.logger.Log(trace.DEBUG, msg)04:02
jamIt is 9.2ns/op04:03
jamso, interesting that all that is measurable04:03
jamI imagine the overhead of varargs is because it has to allocate the slice and the backing array04:03
jambut you *are* talking 15ns absolute time.04:03
jamso if you call this 1000s of times per second, it has a net overhead of 15 microseconds/second (15ppm)04:04
jamI don't think that will explicitly ever show up in a pprof :)04:04
thumper:)04:04
jamcompared to TestWriters which is 1574ns04:05
jamso once you are actually formatting and writing stuff to a string04:05
thumperso, reading around the atomicity of 32 bit reads and writes...04:05
jamyou are ~100x slower.04:05
thumperseems fine on intel, and amd6404:05
thumperbut not sure about arm04:05
jamthumper: I *believe* that all platforms give you an atomic-word-aligned load because often pointers are used to implement atomic operations.04:06
thumperI'll take that 10ns hit for a nicer method signature04:06
thumperhmm...04:06
jamthumper: http://stackoverflow.com/questions/9399026/arm-is-writing-reading-from-int-atomic04:06
thumperjam: so a question is "does go word align integers?"04:07
thumperI think the answer should be "I damn well hope so"04:08
jamthumper: I don't know of anything that *doesn't* align objects unless you do crazy shit04:08
thumperjam: like std::vector<bool>?04:08
jamlike use an integer pointer into a byte array to manually extract stuff (which I've done, but I've also gotten BUS errors on Mac doing it :)04:08
jamthumper: does the stdlib treat it as wide pointers in memory?04:09
thumperI don't recall04:09
jamthere is a fair amount of C code (especially string searching) that knows that on certain platforms04:09
jamit is safe to do unaligned loads04:09
thumperthe general answer was don't do it04:09
jam(intel amd is fine, PPC is not)04:09
thumperew04:10
jamI think bzrlib itself has some, but also with platform checks.04:10
thumperok, so back to a global uint3204:10
thumpershould be word aligned?04:10
thumperhow would we know?04:10
thumperapart from taking the address of it04:10
jamif you want to *know* then you have to take the address and compare mod 4, but there is reflect.Type.Align as well.04:11
thumperyeah, you know what?04:13
thumperI'm just going to use sync.atomic04:13
thumperit isn't enough of a difference to care04:13
jamthumper: did you check what the over head is? If it is 10ns I agree, if it is 100 or 1000 then I might quibble04:14
thumperjam: it is about 4ns04:14
thumperon my machine04:14
davecheneythumper: back now (lunch)04:14
davecheneylooks like you already got an answer04:15
thumperdavecheney: that's ok, me and jam have just been talking about atomic stuff04:15
davecheneymaybe you've already covered it04:15
davecheneybut there is a difference between an atomic write, and a write that is safely published04:15
thumperwhat do you mean by safely published?04:16
davecheneyvisible to another thread04:16
thumperI don't care about delays04:16
thumperjust valid reads and writes04:16
davecheneyyou'll probaby be ok, but that is playing fast and loose with the memory model04:17
davecheneyie, the delay could be infinite04:17
* thumper pulls a face04:17
thumperso how does one safely publish something?04:17
davecheneysync/atomic or use a lock04:17
davecheneyie, you need a memory barrier04:17
davecheneyor send it through a channel04:17
thumperah, yes, I decided to use sync/atomic04:17
davecheneyall good then04:18
thumperok, cool04:18
* thumper considers something else04:18
thumperso, theoretically I could have one go routine set a logging level on a logger, and not have it visible to another go routine?04:20
thumperif not protected by locks?04:20
davecheneycorrect04:20
thumperor atomic reads/writes04:20
thumperhmm...04:20
davecheneyyes04:20
thumperpoos04:20
thumperdavecheney: so, I have "type Level uint32"04:21
thumperbut I can't have a Level variable and use atomic.LoadUint3204:21
thumperbecause it complains about the casts04:22
thumperso perhaps just better not to have the Level type?04:22
thumperalthough I kinda like the String method on it04:22
thumperor just have the Level at the public interface04:22
thumperand use uint32 internally?04:22
davecheneythumper: where is the code again04:22
thumperlaunchpad.net/loggo04:22
* davecheney looks04:23
davecheneytwo secs04:23
thumperparticularly considering the func (logger Logger) SetLogLevel method04:23
* davecheney twiddles fingers04:23
thumperdavecheney: also, FYI, I'm giving a talk on Go in about an hour04:24
davecheneyat the NZ meetup04:24
davecheneysweet04:24
davecheney>04:24
davecheney?04:24
davecheneythumper: let me check one thing04:25
davecheneyi think when you use atomic.SetUint32, you also need to read using atomic.ReadUint3204:26
davecheneylet me check in the channel04:26
thumperdavecheney: yes, was also using both04:27
thumperread and write for sync04:27
davecheneyok, cool04:27
thumperatomic.StoreUint32 and LoadUint3204:27
thumperhowever04:27
davecheneyindeed04:27
davecheneylogically that would appear to be the correct usage04:27
thumpercurrently for the logging levels, this isn't done04:27
thumperjust assigning to a Level variable04:28
thumperwhich is a uint3204:28
thumperbut the typing is just getting in the way of using the atomic functions to store and load04:28
thumperwhich is frustrating04:28
davecheneythumper: got the line(s)04:28
thumperdavecheney: I'm currently poking jam's branch04:28
thumperwhich isn't currently pushed04:28
davecheneyewww04:28
thumperewww what?04:29
davecheneyjust paste the few lines into paste.ubuntu.com04:29
thumperreturn level >= Level(atomic.LoadUint32(&globalMinLevel))04:36
thumpersorry, someone at the door04:36
thumperthat is what I have by saving globalMinLevel as a uint3204:36
thumperbut I can't seem to have globalMinLevel as a Level, and still use atomic reads and writes04:37
thumperdavecheney: http://play.golang.org/p/pB65DtZrXr so this is what I want to do but can't04:39
davecheneythumper: http://play.golang.org/p/0_ei1fpH9q04:39
thumperdavecheney: gah... ok, ta04:40
thumperI'm pleased it is possible04:40
thumperI was getting very frustrated04:40
davecheneyyou need the parens to disambiguate the * deref04:40
davecheneyplease hold, arguing about StoreUint32 in #go-nuts04:41
thumperdavecheney: as in arguing the need for it?04:43
davecheneynot really04:44
davecheneymore wandering a blind confusion at this point04:44
thumperinteresting04:55
thumperchanging the access to the module.level to use atomics made 0.1ns of difference04:56
thumperrather than the 4ns of difference changing the globalMinLevel for the writer04:56
davecheneythumper: i think i missed that part of the timeing discussion04:58
* davecheney looks for globalMinLevel04:58
davecheneythumper: honestly, if you're in the ns territory, it doesn't matter04:59
davecheneyeven in the us territory04:59
thumperdavecheney: it is in the branch jam proposed05:01
thumpernot in trunk05:01
davecheneycan't see the code05:02
davecheneybut there are probably a few things going oin there05:02
davecheney1. this is uncontented05:02
davecheneyno other CPU is stealing the cache line from the active one05:02
davecheneyso you're benchmarking the round trip to your L1/L2 cache05:02
davecheneywhich is fair05:02
davecheneymost synchronisation is uncontended05:03
davecheneyi'm assuming because globalMinLevel has the word global in it05:04
davecheneyyou're walking the logger tree to the root05:04
davecheneycorrect ?05:04
thumperI think so, but I'm off now05:05
thumperperhaps we could chat tomorrow?05:05
davecheneywho's used debug-hooks recently06:00
davecheney?06:00
=== bigjools_ is now known as bigjools
fwereade_davecheney, I haven't but I might be able to talk about it?06:20
davecheneyfwereade_: so I have a version of debug-hooks that replicates the ssh into tmux behavior06:21
davecheney(stole the script straight out of py juju)06:21
davecheneybut it doesn't really look right06:21
davecheneyie, shouldn't HOOK_CONTEXT be exported, blah blah ?06:21
fwereade_davecheney, yeah, I think so, HOOK_CONTEXT_ID is needed to make the hook tools run06:22
davecheneyand probably putting them in the path06:22
davecheneybut I think pyjuju cheats on that bit06:22
fwereade_davecheney, yeah -- it ought to be as close a copy of the normal hook environment as possible06:23
davecheneyfwereade_: did such a thing contextually exist in pyjuju ?06:23
fwereade_davecheney, there must have been *something* that communicated back with the main agent process... let me see if I can find it06:24
davecheneyfwereade_: i'm looking at control/debug_hooks.py06:24
fwereade_davecheney, you've seen the other half of it in hooks/executor.py?06:26
davecheneyfwereade_: nope06:26
davecheneythere was a concerning something.setdebug(true)06:27
davecheneythat i chose to ignore06:27
davecheneyfwereade_: even if there was a signal to the unit agent06:28
davecheneyi cant find any hook(sic) in the command that does anything but wait til it reached that point06:28
davecheneythen ssh in06:28
fwereade_davecheney, I'm afraid I don't recall the experience of actually using debug-hooks06:30
davecheneyfwereade_: i saw marco use it once06:31
davecheneyhttps://codereview.appspot.com/999604306:31
davecheneyit works, but doesn't really do much more than juju ssh atm06:31
fwereade_davecheney, (but yeah, I think you can forget the set debug log bit for now -- it'd be worth syncing up with thumper on what he has planned in that direction though)06:31
davecheneyfwereade_: i'm trying to figure out if my faxilily is poor, or if the debug hooks world is more complicated in our juju06:32
fwereade_davecheney, there's definitely some crazy magic that needs to happen at the other end, so that we do actually somehow inject the appropriate hook context while a hook is being run06:33
* davecheney scrobbles in the code06:34
davecheneylucky(~/devel/juju/juju/hooks) % ack tmux06:35
davecheneyexecutor.py06:35
davecheney34:# The beauty below is a workaround for a bug in tmux (1.5 in Oneiric) or06:35
davecheney37:tmux new-session -d -s $JUJU_UNIT_NAME 2>&1 | cat > /dev/null || true06:35
davecheney38:tmux new-window -t $JUJU_UNIT_NAME -n {hook_name} "$JUJU_DEBUG/hook.sh"06:35
davecheneyoh ffs06:35
davecheneythat is what set debug does06:35
fwereade_davecheney, ah, got you, I misunderstood what you said before06:36
fwereade_davecheney, you will indeed need to set some state flag so the unit can know to do that06:37
davecheneyfwereade_: ok, i'll talk to le thump tomorrow06:37
fwereade_davecheney, although... we will also need to make sure we unset it, however the connection ends, surely06:38
* davecheney noted that python didn't appear to unset whatever it sets06:38
fwereade_davecheney, oh, ffs, it uses a ZK ephemeral node that goes away when the cli client goes away06:41
* fwereade_ sighs deeply06:41
fwereade_davecheney, ...we could maybe use a presence node with a session guid? cleaning up feels likely to be racy, but perhaps it's actually equivalent06:42
* davecheney throws up his hands06:43
fwereade_davecheney, ZK ephemerals don't go away immediately either06:43
davecheneythis is over my head06:43
davecheneyso much for debug-hooks being 'simple' to implement06:43
fwereade_davecheney, yeah, I didn't have any recollection of that being the case myself06:43
davecheneythere was a suggestion of such on the ML06:43
fwereade_davecheney, roughly speaking, the presence stuff should enable the same stuff that ZK ephemerals do, so once you're operating at a certain level of rarefied abstraction I can imagine how it could seem simple :/06:45
davecheneysure, it's just programming, right ?06:45
fwereade_davecheney, it's just 1s and 0s, type fastr06:45
fwereade_davecheney, and, hmm, we can't get the ownership guarantees with a pinger that we could with a ZK ephemeral node06:49
* davecheney goes to find whisky06:49
* fwereade_ looks at the clock, and regretfully does not join davecheney06:50
davecheneyfwereade_: it's always 5pm somewhere in the world06:51
fwereade_davecheney, ok, we *could* fake up everything we need, I think -- I just need to get up properly before I can discuss this sanely06:51
fwereade_davecheney, I'll be back in a few minutes; will you be free for a quick hangout then? I think I can give you a rough sketch of what we need and you can decide whether it's doable without bloodshed06:52
davecheneyfwereade_: i don't think it is worth it06:53
davecheneythis is over my head06:53
davecheneyi'll throw this card back into the pool and leave this command for someone else to use06:53
davecheneythis is a much bigger job than I thought06:54
davecheneyand this work is not scheduled for a good reason06:54
dimiternmorning06:55
fwereade_dimitern, heyhey07:10
fwereade_TheMue, heyhey07:10
rogpeppemornin' all!07:14
dimiternfwereade_: hey07:14
dimiternrogpeppe: morning07:14
TheMuefwereade_, dimitern: heya07:14
rogpeppedimitern, TheMue: yo!07:14
dimiternfwereade_: about to propose the machiner facade stuff07:14
TheMuerogpeppe: oh, yes, welcome back too ;)07:14
fwereade_rogpeppe, heyhey07:14
fwereade_dimitern, cool07:14
rogpeppefwereade_: hiya!07:14
dimiternrogpeppe: you might be surprised by my next CL :)07:14
rogpeppedimitern: how did you get on with that set of branches?07:14
dimiternrogpeppe: didn't manage due to too much other stuff going on07:15
rogpeppedimitern: i haven't seen any emails go by, but that's i think because i never see any CLs i'm not directly involved in07:15
rogpeppedimitern: what's the "machiner facade" then?07:16
dimiternrogpeppe: I did 3 separate refactoring proposals last week, all of them changing the API in a different way07:16
dimiternrogpeppe: finally we agreed on how to move forward07:16
dimiternrogpeppe: so basically we decided to get rid of srvMachine, srvUnit, api.Machine, api.Unit and related stuff07:16
dimiternrogpeppe: and instead have srvMachiner, which implements a subset of the API only used by the machiner07:17
rogpeppedimitern: okay... what's the plan then?07:17
rogpeppedimitern: so you're putting the machiner in the API?07:17
dimiternrogpeppe: and have lightweight "MachinerMachine" objects proxying calls through the facade07:17
rogpeppedimitern: i'm not sure i understand what you mean by that07:18
dimiternrogpeppe: yeah, as an attempt to be have SOA oriented approach, rather than replicate the state api directly07:18
* rogpeppe doesn't really know what "service-oriented architecture" actually means07:19
dimiternrogpeppe: you'll see shortly - the code will explain it better07:19
rogpeppedimitern: could you point me towards a place where you were discussing this, so i can get some background?07:19
dimiternrogpeppe: take a look at the juju-dev ML messages and this document: https://docs.google.com/a/canonical.com/document/d/1Yd2Nil43AemnBq8qv003OkWLptiPzuhWGESBcwlI7Nc/edit#heading=h.fdyojyfogyn107:20
dimiternrogpeppe: it's not up-to-date now, but the proposal implements what was agreed upon07:20
rogpeppedimitern: so, by "bulk operation" you mean that you have a set of objects and you perform the same operation on all of them at once? kinda like vector math but for objects?07:26
TheMuerogpeppe: it's not in the typical sense of soa. but the style is service-oriented. instead m := getMachine(4711); m.DoThis(...); it's a machineService.DoThis(4711, ...).07:26
dimiternrogpeppe: yes07:26
fwereade_rogpeppe, yeah, the consensus is essentially that domain objects always suck07:27
fwereade_rogpeppe, client-side, we'll still be faking them up, so the refectoring doesn't kill us07:27
dimiternrogpeppe: instead of having Machine.SetStatus() -> Machiner(args []{Id, status, info}) -> []errorresults07:27
rogpeppefwereade_: i guess i don't quite see the issue.07:27
dimiternrogpeppe: s/Machiner/Machiner.SetStatus/07:27
rogpeppefwereade_: how do domain objects always suck?07:28
fwereade_rogpeppe, well, the API mimics state at the moment, and thereby preserves all the mistakes we made with state -- but is much harder to change07:28
fwereade_rogpeppe, consider the provisioner07:29
fwereade_rogpeppe, grabbing N machine objects individually is plainly insane07:29
rogpeppefwereade_: it calls AllMachines, no?07:30
fwereade_rogpeppe, yes, and AllMachines is total unjustifiable crack07:30
rogpeppefwereade_: interesting p.o.v.07:30
rogpeppefwereade_: why so?07:30
fwereade_rogpeppe, it's a load of unnecessary data that'll go stale, and which doesn't allow for useful bulk ops -- if we wanted to do the provisioner right, we'd be asking for 1000 life statuses, then getting the instance ids of the alive 800 of those, then getting whatever the next subset of information is07:32
fwereade_rogpeppe, talking about individual machines renders this approach unworkable to the point you don't even consider it07:32
rogpeppefwereade_: we can get bulk ops easily by issuing multiple concurrent requests - but do you consider that an unworkable approach?07:33
fwereade_rogpeppe, yes, it's insane07:33
rogpeppefwereade_: that's a strong statement :-)07:33
fwereade_rogpeppe, makes it impossible to leverage the db's support for bulk ops07:33
rogpeppefwereade_: ah, that's a good point.07:34
fwereade_rogpeppe, I understand *why* it happened like this -- we swapped backends without considering what the state package should look like in the new context07:34
rogpeppefwereade_: and mongodb has such support? (other than bulk query ops)?07:35
fwereade_rogpeppe, bulk query ops are exactly what we need in general07:35
fwereade_rogpeppe, and in the cases where we need bulk change ops (currently few, I suspect will become more and more important as the project matures)07:35
rogpeppefwereade_: i thought query ops were very fast, and not likely to be a bottleneck for us07:35
fwereade_rogpeppe, we still want to be able to express them in a compact way even if we end up needing to write intent to a queue and handle the txn changes in smaller batches07:36
rogpeppefwereade_: is there a risk that we're making things harder for ourselves by making a significant architecture shift here that might actually be premature optimisation?07:36
fwereade_rogpeppe, I accept responsibility for that risk07:36
rogpeppefwereade_: have you made some measurements to support these decisions?07:37
fwereade_rogpeppe, no, because we can't scale far enough to tell in the first place07:39
dimiternthere it is -> https://codereview.appspot.com/9896046/07:43
* dimitern ducks for cover with a steak :)07:43
fwereade_rogpeppe, please read through the lists for better context than IRC07:44
rogpeppefwereade_: which lists?07:44
fwereade_rogpeppe, juju-dev07:44
fwereade_rogpeppe, on a related note, btw, what is the deal with Pinger?07:44
rogpeppefwereade_: i haven't seen anything on juju-dev since dimiter's response to my handoff email07:45
fwereade_rogpeppe, have you switched to gmail yet?07:45
rogpeppefwereade_: ahhh07:45
rogpeppefwereade_: that will be the issue07:45
rogpeppefwereade_: i use gmail anyway - i need to point it to my canonical gmail i guess07:45
rogpeppefwereade_: it's a good thing actually that i didn't see any of this while i was away on holiday :-)07:46
fwereade_rogpeppe, ha, yeah07:46
TheMuerogpeppe: ;)07:47
rogpeppefwereade_: what about Pinger?07:47
fwereade_rogpeppe, I was very clear that I did not want pinger in the API07:47
rogpeppefwereade_: ah, what would you like to call it?07:48
fwereade_rogpeppe, and that I wanted it out of machiner so we could move forward without it complicating the issue, and give ourselves some breathing room to make a final decision without immediate pressure07:48
fwereade_rogpeppe, I don't think it's justifiable at all in the API, myself, as you know07:49
=== tasdomas_afk is now known as tasdomas
rogpeppefwereade_: ah, so we don't want the agents telling the state they're alive at all?07:49
fwereade_rogpeppe, we certainly do not want the machiner pretending it's the machine agent07:49
rogpeppefwereade_: what *is* the machine agent?07:50
fwereade_rogpeppe, the bit that runs the various workers07:50
fwereade_rogpeppe, not the workers themselves07:50
rogpeppefwereade_: presumably *some* worker has got to do it, no?07:50
fwereade_rogpeppe, I'm not sure why that would be the case at all07:51
fwereade_rogpeppe, why not keep it purely server-side?07:51
rogpeppefwereade_: how does the server side know a client is around?07:51
rogpeppes/client/agent/07:51
fwereade_rogpeppe, I *hope* it can tell, otherwise it'll be running a lot of watchers for a client that's disconnected07:52
fwereade_rogpeppe, anyway, the reason I wanted it out of the API was so that we could have this discussion independently of the critical path07:53
rogpeppefwereade_: so you're suggesting that the API see a connection from a given agent and run a pinger on its behalf as a result of that connection being made?07:54
fwereade_rogpeppe, that is indeed a possibility, as we discussed in detail before I went away on holiday07:54
rogpeppefwereade_: i remember a few discussions in that area. i hadn't realised you didn't want the pinger in the API at all though, sorry.07:55
rogpeppefwereade_: istr the suggestion to put it in the Agent.Entity call, which didn't work out07:56
fwereade_rogpeppe, I wanted it out of machiner so that we could discuss the answer to this question separately, without blocking the machiner work07:56
rogpeppefwereade_: ok, so let's just remove it then07:56
dimiternrogpeppe, fwereade_: https://codereview.appspot.com/9896046/ ?08:08
fwereade_dimitern, I'm reading it right now08:08
dimitern(it's big, but there are mostly removals)08:09
=== TheRealMue is now known as TheMue
fwereade_dimitern, reviewed with a few thoughts08:37
dimiternfwereade_: cheers08:37
fwereade_dimitern, some of them are a bit vague08:37
fwereade_dimitern, Machines in particular is maybe really just Exists()?08:38
fwereade_dimitern, but I'm ambivalent about the Refresh()/Life() thing in particular08:39
dimiternfwereade_: i think it's a good idea for Refresh calling Life and caching08:39
fwereade_dimitern, so long as we implement the API sanely, I think it's sensible to keep the *Machine interface as close as possible08:39
fwereade_dimitern, cool08:39
fwereade_dimitern, so long as we all know that's now how it "should" be long-term08:40
dimiternfwereade_: yeah08:40
dimiternfwereade_: not sure I get the point of converting NotFound into Dead?08:40
dimiternfwereade_: that surely the same as state works08:41
fwereade_dimitern, yeah, but all it means is that all the client code has to go around specifically handling NotFound and handling it as Dead ;p08:42
fwereade_dimitern, we kinda BDUFed the lifecycle stuff and that's one of the ickiness points08:42
fwereade_dimitern, there may or may not in general be a distinction between Dead and NotFound, and it's situational08:43
fwereade_dimitern, I suspect that if we're asking for Life explicitly it should probably be reported as Dead -- although this then makes auth errors interesting08:44
* fwereade_ grumbles at the world08:44
dimiternfwereade_: well, how about a true "not found" case?08:44
dimiternfwereade_: i mean you asked for a machine that was never there08:44
fwereade_dimitern, ok, lets start with Machines() -- what's the use case there on the server side? to get a domain object, we need to discover its Life, but that's the only call we need -- we already know the ID08:46
fwereade_dimitern, not-auth is actually a reasonable response to a Life query I guess08:46
fwereade_dimitern, (in general, anyway...) but from the POV of the machiner, I think that reporting notfound as Dead is actually quite useful... am I making sense here, or is this all too dependent on my internal context?08:48
fwereade_dimitern, it's like calling EnsureDead on a machine that doesn't exist -- it succeeds08:48
dimiternfwereade_: do you mean that only for life?08:48
dimiternfwereade_: i.e. Machines will still return not found as usual08:48
fwereade_dimitern, I'm trying to figure out what the use case is for Machines()08:49
dimiternfwereade_: what? no - ensuredead fails on a non-existing machine through the API08:49
dimiternfwereade_: because we need to get the machine first08:49
dimiternfwereade_: but i guess in state it operates on cached life and might succeed08:49
fwereade_dimitern, that's an interesting behaviour change08:49
fwereade_dimitern, g+ a mo?08:50
dimiternfwereade_: ok, just a sec08:51
dimiternfwereade_: https://plus.google.com/hangouts/_/971de9659aecd256626b1d52513288375093bf72?authuser=0&hl=en08:52
dimiternrogpeppe: can you take a look as well please? https://codereview.appspot.com/9896046/09:23
rogpeppedimitern: i'm currently going through all the backlog09:24
dimiternrogpeppe: ok09:25
=== rogpeppe2 is now known as rogpeppe
dimiternfwereade_: if i remove the authEnvironManager from the allowed perm checks10:25
dimiternfwereade_: then it kinda defeats the point of having bulk operations - any method on the machiner will only ever succeed for the machine this machiner is responsible for10:26
dimiternfwereade_: do you know what I mean?10:34
dimiternfwereade_: anyway, it's updated now https://codereview.appspot.com/9896046/11:06
wallyworldTheMue: hey frank, thanks for the review. as written, i think the provisioner tweak is safe to land now so i'd like to do that and then your changes can come along later?11:09
jamfwereade_: so, we currently have a test that is broken with latest mgo. And it will prevent us from using the juju-core go-bot. Care to give direction about how to fix it?11:13
jamTestOpenDoesNotDelayOnHandShakeFailure11:14
jamit was written by Dave, because he implemented the logic to have juju-core delay if it gets a connection failure, but *not* delay if it gets a TLS handshake failure.11:15
jamThe test now fails because mgo unconditionally delays 500ms on *any* failure.11:15
TheMuewallyworld: it's ok for me. the information ContainerType() is returning matches exactly my needs.11:17
wallyworldgreat :-)11:17
TheMuewallyworld: just code a little outline where it better can be seen how it will be used.11:17
TheMuewallyworld: eh, not "just code", but "i'm just coding" ;)11:17
wallyworldTheMue: ok. so are you wanting to land your work before mine? wouldn't mine need to land before yours?11:18
wallyworldso you can use the new ContainerType() method?11:19
fwereade_dimitern, the remaining point of bulk operations is in habit and consistency; and in that we *can't* really predict what a machiner will ultimately be responsible for, and we get some future-proofing by allowing ourselves to express multiple ops if we ever need them11:19
fwereade_dimitern, I do agree that the machiner is not in itself a compelling use case for bulk ops11:19
dimiternfwereade_: sure, np11:19
wallyworldfwereade_: i think you'll be happy with this now, hopefully https://codereview.appspot.com/9820043/11:20
dimiternfwereade_: wanna take a look now?11:20
fwereade_dimitern, wallyworld, thanks both, I'll take a look11:20
wallyworldthanks11:20
fwereade_jam, I would provisionally be ok dropping that test and that behaviour -- it seems like it's been taken out of our hands with the mgo change11:21
jamfwereade_: well *a* fix is to get something upstreamed11:21
jambut an *easier* fix is to just drop it :)11:21
fwereade_jam, but it would be good to have a word with davecheney for a bit more context11:21
jamfwereade_: AIUI the initial issue was that mgo always retried without any delay, which punished things a bit11:22
jamso we put in a delay, and now so has mgo directly.11:22
fwereade_wallyworld, what were your thoughts re unit-dirty vs container-dirty?11:22
fwereade_jam, in that case I would be happy dropping it, assuming davecheney's approval11:23
fwereade_just in case i missed something11:23
TheMuewallyworld: yours imho can already land, yes11:24
wallyworldfwereade_: hmmm. i must confess i forgot that if it was raised as a question. sorry. the flag as implemented is a unit-dirty flag i guess. but that's all we need now i think11:24
wallyworldTheMue: thanks. i promise to fix that method comment if you +1 it :-)11:25
fwereade_wallyworld, fair enough -- I think there will be some interestingness in future though -- possibly I raised it in a different CL11:25
TheMuewallyworld: great11:26
wallyworldfwereade_: the idea now is that a unit can be deployed if unit-clean=true, since it a machine has containers, it doesn;t really matter for that case11:26
wallyworldor so i understand11:26
fwereade_wallyworld, I'm just a bit antsy about what we really want to express -- "clean" and "unused" both feel like rational and distinct requests11:26
fwereade_wallyworld, I'm willing to call this progress, though -- nothing's using "unused" at the moment, right?11:27
fwereade_wallyworld, I *will* be nervous about reactivating unused while it's not able to take constraints into account, though11:27
wallyworldfwereade_: for now, afaiui, we only need care about clean rather than unused  (as far as containerisation goes)11:27
wallyworldfwereade_: i think the current AssignUnused realy should be AssignClean11:28
wallyworldsince that's the semantic we are really aiming for right now11:28
wallyworldafaiui11:28
fwereade_wallyworld, I'd be +1 on a rename there, but you don't need it in this CL11:28
danilosmumble trouble :/11:28
mgzyou're hoping a lot ::)11:29
wallyworldfwereade_: yes, i agree with the rename. as you say, that's part of the evolution of this work and not for this mp11:29
wallyworldthat also matches my thinking of the issue11:29
jamdanilos: we've put you in another room for now, hopefully you can get it working again.11:30
danilosjam: I am trying11:30
danilosjam: works fine then kicks me out 5 seconds later :/11:31
jamyou could try restarting completely...11:31
fwereade_wallyworld, you have an LGTM, sorry I let that one linger11:31
fwereade_dimitern, I'm on yours now11:31
wallyworldnp, thanks11:31
dimiternfwereade_: cheers!11:32
danilosjam: I can, if you mean rebooting? (I've killed mumble and tried again, fwiw)11:32
jamthat is what I meant, though we could also switch to a hangout11:32
jamwallyworld, danilos, mgz: https://plus.google.com/hangouts/_/8868e66b07fa02bdc903be4601200d470dae9ee311:33
jamdimitern: ^^11:33
dimiternfwereade_: blast, I realized I have to add MachinerMachine.EnsureDead() and also add client-side tests for the machiner11:53
fwereade_dimitern, you *could* very happily just strip out the client-side code from this CL, and repropose those with tests in a new one11:54
dimiternfwereade_: good idea, will do11:57
dimiternfwereade_: so the pipeline will be: 2) client-side + tests, 3) split suites, 4) split (apiserver|api)/machiner into a separate subpackage, 5) implement Machiner.Watch,11:58
dimiternfwereade_: sounds good?11:58
fwereade_dimitern, SGTM11:59
fwereade_dimitern, and LGTM11:59
dimiternfwereade_: tyvm11:59
fwereade_dimitern, (although lt me know your thoughts on the comments, and if you decide they're candidates for this CL so much the better)11:59
fwereade_dimitern, I'm very happy with splitting Authorizer out12:00
dimiternTheMue: since you're the OCR today, can you have a look as well? https://codereview.appspot.com/9896046/ (disregard the state/api/machiner.go stuff - will split it in a follow-up)12:00
TheMuedimitern: yep12:00
dimiternfwereade_: i'm looking at your review and will ask if something is unclear12:01
danilosTheMue, hi, I wonder if you can take a look at https://codereview.appspot.com/9876043/?12:32
TheMuedanilos: *click*12:33
danilosTheMue, thanks :)12:33
TheMuedanilos: Done.12:43
danilosTheMue, thanks12:44
fwereade_wallyworld, ping13:13
wallyworldhi13:13
fwereade_wallyworld, free for a chat about containers?13:13
wallyworldsure13:13
fwereade_wallyworld, I'll start one13:13
mgzcan I sit in?13:14
wallyworldoh, alright13:14
fwereade_mg, sure, I'll invite, just a sec13:14
dimiternfwereade_, TheMue: next one - https://codereview.appspot.com/9686047/13:34
=== tasdomas is now known as tasdomas_afk
TheMuedimitern: *click*13:43
fwereade_wallyworld, mgz, ffs, sorry13:45
mgzno problem13:45
fwereade_waiting for plus.google.com...13:46
* fwereade_ sighs13:47
fwereade_wallyworld, mgz, it really doesn't want to talk to me today :/13:48
wallyworldfwereade_: try mumble?13:49
fwereade_wallyworld, I really ought to have set that up at some point since I started here, shouldn't I :/13:49
wallyworldlol13:49
=== wedgwood_away is now known as wedgwood
fwereade_I'll bounce my router on general principles, bbiab13:50
wallyworldkk13:50
dimiternTheMue: thanks13:50
frankbanrogpeppe, dimitern,anyone else: I need another review for https://codereview.appspot.com/9641044/ . could you please take a look?13:50
TheMuedimitern: yw13:52
dimiternfrankban: looking13:54
frankbandimitern: thanks13:54
dimiternfwereade_, TheMue: and another small one: https://codereview.appspot.com/10003044/13:54
fwereade__TheMue, mramm, kanban?14:02
dimiternfrankban: LGTM14:03
frankbandimitern: great, thank you. The test with a float is already present as part of the schema tests.14:06
dimiternfrankban: wasn't sure, but ok14:06
TheMuesh..., sorry, just a technician arrived *grmpf*14:07
fwereade__grar14:13
fwereade__google dislikes me today14:13
fwereade__and it really is just google14:17
dimiternfwereade__: :/14:18
fwereade__launchpad is positively sprightly by comparison14:18
* fwereade__ sighs deeply14:23
fwereade__dimitern, rogpeppe1, mramm, I think I'm going to give up on this and go reboot ALL THE THINGS14:25
=== BradCrittenden is now known as bac
dimiternTheMue: https://codereview.appspot.com/10003044/14:58
dimiternalso second reviewer needed on https://codereview.appspot.com/9686047/14:58
TheMuedimitern: *click*14:59
dimiternTheMue: cheers14:59
dimiternTheMue: tyvm15:23
dimiternneed to relax a bit, i'm off for now; might be back later15:24
=== BradCrittenden is now known as bac
=== deryck is now known as deryck[lunch]
* rogpeppe2 is done for the day. g'night all.17:07
=== deryck[lunch] is now known as deryck
=== Daviey_ is now known as Daviey
=== _mup___ is now known as _mup_
=== benji_ is now known as benji
=== _mup___ is now known as _mup_
thumperfwereade_: around?21:36
* thumper sighs21:46
thumpercan anyone else confirm a build failure with trunk?21:46
* thumper wonders where our tarmac committer is21:46
thumpergrr21:57
thumperdimitern_: you broke trunk, naughty naughty21:58
thumperr124721:58
thumperjujud tests fail to build21:58
wallyworldthumper: tarmac is almost ready - was waiting on a failing mongo test to be fixed21:59
thumperwallyworld: hi there21:59
wallyworldbut that test will be deleted21:59
wallyworldhi21:59
wallyworldhow's the dog?22:00
thumperwallyworld: how am I supposed to know that? trunk build fails, this is bad22:03
thumperdog is fine, was sleeping22:03
wallyworldthumper: no, separate issue22:03
thumpernow is staring at me22:03
wallyworldtest failure (resulting from upstream mongo changes) was preventing tarmac bot bring deployed22:04
wallyworldand don't you love how we just pull upstream from tip so we are not isolated from breaking changes22:04
thumperyeah, it's awesome22:04
wallyworldah, who needs dependency management22:04
thumperdidn't kapil have a dep management thing to add in?22:05
thumperhazmat: where is that?22:05
wallyworldyeah, but now someone else has proposed yet another Go solution22:05
thumperwallyworld: do you know what needs to be fixed in the failing test?22:05
thumperwallyworld: I'm not sure what the test is trying to test22:05
wallyworldthumper: gopm or something. but given how the last person who proposed something was shot down in flames, i'm not optimistic22:06
thumperwho proposed a go thing?22:06
wallyworldthumper: no idea. i just looked at irc and saw your comments. i know nothing of the build failure yet22:06
wallyworldthe failing test is being deleted22:06
thumperyou said someone proposed a go solution...22:06
thumperi was asking about that22:06
wallyworldyeah, i was told, let me try and find something22:06
wallyworldthumper: https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/k8pmk8FQC8w22:07
wallyworldhttps://github.com/GPMGo/gopm-api/22:08
wallyworldso just a proposal really it seems at this stage22:08
wallyworldthumper: if you add your +1 to this i can land it https://codereview.appspot.com/9820043/22:11
fwereade_thumper, hey dude22:13
fwereade_thumper, sorry I haven't been around much at sociable hours the last few days22:14
thumperwallyworld: sorry, doggy break needed22:21
thumperfwereade_: hey22:21
thumperwallyworld: we should land anything new until trunk is fixed22:21
wallyworldthumper: sure, i just want to get it unblocked, not going to land immediately22:22
thumperfwereade_: so trunk is broken due to r124722:22
thumperfwereade_: just wondering what to do with the now failing test22:22
thumperwallyworld: ack, I'll look shortly22:22
wallyworldno hurry22:22
* thumper sighs22:28
thumperit isn't obvious how to fix this test22:28
* thumper comments out the whole test22:28
thumperwallyworld: can I get a +1 trivial on this? Rietveld: https://codereview.appspot.com/1002204322:44
thumperI could just merge it in, but someone else agreeing helps22:44
* wallyworld looks22:44
wallyworldthumper: done. i had a quick look too but it wasn't immediately obvious what the replacement api to call was22:46
thumperyeah22:46
wallyworldwe could have found it i guess, but other things to do22:46
fwereade__thumper, sorry, just getting up to date22:49
fwereade__thumper, oh, hell, what's broken?22:49
thumperfwereade__: just a test, see review just above22:50
fwereade__thumper, ah, ok, the issue is that API no longer has a .Machine()?22:51
fwereade__thumper, I'd just drop that bit22:51
fwereade__thumper, being able to log in should be evidence enough that something's serving the API22:51
thumperfwereade__: well, actually submitted already22:51
thumper:)22:51
fwereade__thumper, ok, just mail dimitern_ with some light bitching about running *all* the tests then ;p22:52
* thumper nods22:52
thumperfwereade__: do you have time for a quick catch up?22:52
thumperlike a hangout?22:52
fwereade__thumper, sure, woudl you start one please? with you in 222:53
thumperfwereade__: ok22:53
hazmatthumper, its kinda of lame.. ie works for ci use case only.  its at lp:goreq22:53
thumperfwereade__: https://plus.google.com/hangouts/_/69cccc01076c5b15bb3afbf54ba00501977e7b80?hl=en22:54
hazmatthere's better vcs management in go juju's deployer impl lp:juju-deployer/darwin22:54
hazmatdefinitely a few go build tools popping up22:55
thumpernot surprising22:55
thumperthe problem hits everyone22:55
hazmatbesides the ones on the golang list.. there's also mozilla's heka-build tool which  supports compile time plugins.22:55
hazmatas well as the frozen/repeatable vcs version sets22:55
hazmatthe later of which is all goreq does, update a tree/gopath to a known set of versions22:56
fwereade__mramm, ping23:03
thumpermramm: are you alive?23:03
mrammthumper: fwereade__: yep I'm here23:03
mrammand alive23:04
thumpermramm: can you join us in a hangout?23:04
mrammsure23:04
thumpermramm:  https://plus.google.com/hangouts/_/69cccc01076c5b15bb3afbf54ba00501977e7b80?hl=en23:04
=== wedgwood is now known as wedgwood_away
thumperwallyworld: got a few minutes?23:34
wallyworldthumper: on a csll23:34
wallyworldcall23:34
thumperwallyworld: ack23:34
thumperwallyworld: I'm going to go to the gym in about 20 minutes, so perhaps we'll chat when I'm back?23:35
wallyworldsure23:35

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!