[00:00] <SpamapS> haha
[00:00] <jimbaker> i'm pretty much convinced that Twisted is the best choice for this in the python frameworks
[00:00] <SpamapS> It drives me *batty*
[00:02] <jimbaker> SpamapS, oh sure, it definitely does do that
[00:03] <jimbaker> i personally think that we could write equally robust and deterministic tests using threaded code, along with some conventions. the advantage over twisted would being able to use any arbitrary library. but we would have to come up with some good conventions that at the end of the day do look like go
[00:04] <SpamapS> Go is a nice idea.. but at what point do you stop forward progress to rewrite in it?
[00:05] <jimbaker> SpamapS, progress is so slow right now because testing is currently too hard
[00:05] <jimbaker> so i think that answers it for me
[00:05] <SpamapS> I found it rather easy to do tests for LXC.. but.. thats because I made LXC block.
[00:06] <jimbaker> SpamapS, correct. blocking is much easier to work with. that's why i mentioned using threads. (i was thinking of the paper you shared)
[00:08] <SpamapS> Yeah it pretty much blows up all of the arguments I've heard for event based programming
[00:11] <jimbaker> the inline callbacks model and how it supports coroutines through a trampoline basically gives us what we want there, except that constantly need to remember our calling conventions. so stuff randomly blows up if we forget. and we lose the stack.
[00:12] <jimbaker> in comparison goroutines are very nice
[00:13] <SpamapS> Whats the obsession w/ concurrency in a single process? I honestly can't see more than 1000 things happening at any one time in any one of the agents or clients... thats well within the capabilities of multi-processing.
[00:16] <jimbaker> well, i'm quite certain that multiprocessing would be the wrong way to architect it
[00:16] <jimbaker> we have i/o concurrency here
[00:16] <jimbaker> so not compute bound
[00:20] <kirkland> m_3: around?
[00:24] <_mup_> ensemble/robust-hook-exit r291 committed by jim.baker@canonical.com
[00:24] <_mup_> Doc strings and comments about exit vs end
[00:25] <_mup_> ensemble/robust-hook-exit r292 committed by jim.baker@canonical.com
[00:25] <_mup_> Merged trunk
[00:39] <_mup_> ensemble/robust-hook-exit r293 committed by jim.baker@canonical.com
[00:39] <_mup_> PEP8
[00:43] <_mup_> ensemble/trunk r286 committed by jim.baker@canonical.com
[00:43] <_mup_> merge robust-hook-exit [r=niemeyer,hazmat][f=810808]
[00:43] <_mup_> Distinguishes between a process exits and the process has closed its
[00:43] <_mup_> file descriptors, so that hook output is captured properly without
[00:43] <_mup_> unnecessarily waiting on misbehaving hooks with child processes that
[00:43] <_mup_> haven't closed their FDs.
[00:46] <jimbaker> negronjl, fix released into trunk
[00:47] <negronjl> jimbaker:  Thanks man
[00:47] <jimbaker> negronjl, cool, hopefully this works well as we expect for cloudfoundry
[01:40] <_mup_> ensemble/states-with-principals r299 committed by kapil.thangavelu@canonical.com
[01:40] <_mup_> integrate principal creation/group management into relevant domain states (service, unit, machine)
[01:42] <hazmat> jimbaker, i think the greenlet approach might be a bit nicer
[02:06] <jimbaker> hazmat, the problem i see with that is you cannot switch across threads with greenlets. but it's required to use different threads when working with ZK (or at least the current c api)
[02:07] <jimbaker> obviously it can be made to work, but so much of the greenlet advantage is seen when using switch
[02:28] <m_3> kirkland: hey man... ssup?
[02:49] <hazmat> jimbaker, you can definitely switch across threads with greenlets
[02:49] <hazmat> jimbaker, you can wait on another greenlet to finish and capture its return value and you can yield control of a thread
[03:16] <sidnei> SpamapS, do you happen to be around?
[03:44] <jimbaker> hazmat: http://packages.python.org/greenlet/#greenlets-and-python-threads - "Greenlets can be combined with Python threads; in this case, each thread contains an independent "main" greenlet with a tree of sub-greenlets. It is not possible to mix or switch between greenlets belonging to different threads."
[03:50] <SpamapS> sidnei: here now, sup?
[03:51] <sidnei> SpamapS, oh, was checking #736216, then i realized i enabled proposed but never actually installed the package from proposed.
[03:51] <_mup_> Bug #736216: bzr crashed with error in _curl_perform(): (28, 'SSL connection timeout at 298225') connecting to Launchpad <amd64> <apport-crash> <error-reporting> <https> <natty> <running-unity> <verification-needed> <Bazaar:Invalid> <Launchpad itself:Invalid> <bzr (Ubuntu):Invalid> <curl (Ubuntu):Fix Released> <bzr (Ubuntu Maverick):Invalid> <curl (Ubuntu Maverick):In Progress> <bzr (Ubuntu Natty):Invalid> <curl (Ubuntu Natty):Fix Committed>
[03:52] <sidnei> incidentally seems like niemeyer was hitting it today with bzr
[03:54] <SpamapS> sidnei: Cool. bzr has a micro-release exception, so not all the bugs have to be verified. The package just has to sit in -proposed for 7 days and pass the test suites.
[03:55] <sidnei> SpamapS, yes, but the bug is in curl, and bzr is affected due to python-pycurl linking against libcurl3-gnutls
[03:55] <SpamapS> OH
[03:57] <sidnei> anyway, i enabled proposed but forgot to install the upgrade from proposed until today. seems like the updated package fixes the problem, but i will leave landscape-client running overnight and check the logs in the morning to be real sure.
[13:51]  * hazmat wonders if his diy nas will boot
[14:01] <jcastro> SpamapS: good luck today on your talk, knock em dead!
[14:02] <kooolhead11> hi all
[14:04] <kim0> hazmat: I'm interested to know more about ur nas
[14:07] <hazmat> kim0, nutshell. recycling old diy desktop (core2duo + 4g ram), into a cooler master centurion 590 case, a pair of 5x3 drive cages, and 8 port sata controller, i'm pretty set on a zfs setup at this point (freenas), although i'm going to see if i can test btrfs before i do. sadly my first problem just cropped, need an add on video card, no mb graphics.
[14:09] <kim0> hazmat: was just checking out btrfs since I converted my /home to that (getting too many bad sectors lately). so btrfs can't raid5/6 yet, no working fsck, and no raid conversion yet. Otherwise, seems functional :)
[14:10] <niemeyer> Good mornings!
[14:11] <kim0> Morning 
[14:11] <fwereade> niemeyer: heyhey
[14:16] <hazmat> kim0, i have hopes for btrfs, and keeping check in on the dev, but it still seems a bit experimental for trustworthy storage.. i'm planning on using  raidz2 (3 drive fail to loose data) mode of zfs across eventually 14 drives (starting with 6 atm), in two vdevs of 5-7 each
[14:17] <kim0> yeah although fedora is switching to it, I think it's too early to trust it with family photos :)
[14:19] <SpamapS> jcastro: thanks.. I may still change the pics if I have some time.. but for now.. its lolcats ftw
[14:19] <jcastro> SpamapS: heh, when in doubt, go with what you know
[14:19] <kim0> hehe
[14:50] <niemeyer> hazmat: Physical drivers?
[14:50] <niemeyer> hazmat: Physical drives?
[14:51] <hazmat> niemeyer, 6 atm (5 2tb, 128 gb ssd from old laptop).. total capacity is 14 with the 8-port sata card (not raid) and 6 mb sata slots
[14:53] <hazmat> as far as picking a sata controller with good foss driver support, i found this blog post very helpful.. http://blog.zorinaq.com/?e=10
[14:53] <fwereade> hey all, I'm confused about something
[14:54] <hazmat> niemeyer, re security discussion from yesterday, let me know when you've got time to pick it back up.. 
[14:54] <hazmat> fwereade, what's up?
[14:54] <fwereade> you know we get authorized keys and send them to instances... what do we actually use them for?
[14:54] <fwereade> I thought we were meant to be doing all communication via ZK?
[14:54] <hazmat> fwereade, ssh into the machine
[14:54] <niemeyer> hazmat: Ok.. I have time right now, but I think I'd like to spend some more time with it if you don't mind, so that I can be more helpful.
[14:54] <fwereade> is it only a sort of emergency damn-I've-screwed-up channel
[14:55] <niemeyer> hazmat: I've thought further about the problem yesterday, and have some ideas, but these ideas have holes still
[14:55] <hazmat> fwereade, we use that for the debug hooks command and ssh, and in future scp
[14:55] <hazmat> niemeyer, i've got a nice solution
[14:55] <niemeyer> hazmat: I need to stand next to a white board for a moment
[14:55] <fwereade> hazmat: ah, cool
[14:55] <niemeyer> hazmat: Oh, what's that?
[14:55] <fwereade> hazmat: thanks :)
[14:55] <hazmat> fwereade, np
[14:56] <hazmat> niemeyer, well first.. i think its useful to think of the topology as an identity db, not particularly relevant to the solution.. but most sequence nodes don't have identity prior to their existance in the topology
[14:57] <hazmat> for example look at a failure mode for creating a machine, it leaves an unused sequence node
[14:57] <niemeyer> hazmat: Ok
[14:57] <hazmat> niemeyer, so in particular, i just backed off doing acls for these nodes at creation time, and instead update the acl using policy rules after they've been added
[14:58] <hazmat> to the topology
[14:58] <niemeyer> hazmat: Yeah, that was the direction of my thinking as well
[14:58] <niemeyer> hazmat: Have you figured what's the problem with this too?
[14:59] <hazmat> niemeyer, haven't run into anything yet, do you foresee an issue?
[14:59] <niemeyer> hazmat: Yeah.. what's the reason why we create the entry before putting it in the topology?
[14:59] <hazmat> niemeyer, ah.. yes there is a gap i did notice that
[15:00] <hazmat> before we apply acls
[15:00] <niemeyer> hazmat: Right, we're going back to the original problme
[15:00] <hazmat> niemeyer, not really
[15:00] <niemeyer> hazmat: Why not?
[15:01] <hazmat> niemeyer, the one acl that the standard policy can put forth is an owner policy, which free us from worrying about an acl security gap, so we can modify it post topology add.
[15:02] <niemeyer> hazmat: THe reason why we create the entry before putting it in the topology is because once the topology is touched agents will notice that and will act accordingly
[15:02] <hazmat> niemeyer, as to your question of why create before topology, so we can have an atomic update of the topology which drives  identity/rel watches and that's its reflective of what's present
[15:02] <niemeyer> hazmat: So to make it "atomic" we change the topology once the system is ready to take it
[15:02] <niemeyer> hazmat: If we change in the suggested way, we'll put in the topology something that is not usable
[15:02] <hazmat> niemeyer, yeah.. the gap
[15:02] <hazmat> hmm
[15:05] <hazmat> niemeyer, so that brings up back to nodes should have identity upon creation via contents, but that's complicated as well for the named principal which is going to be reference the sequence value
[15:06] <niemeyer> hazmat: Yeah, I feel like this is an ugly hack we shouldn't dive into
[15:06] <hazmat> niemeyer, alternatively we could write support for the new zk multi-node api into the libzk and zkpython
[15:06] <hazmat> niemeyer, yeah.. i'd prefer to avoid content sniffing as well
[15:07] <hazmat> bummer
[15:07] <hazmat> yeah.. a whiteboard sounds good
[15:07] <hazmat> and some coffee
[15:08] <niemeyer> hazmat: I think your suggestion is going into the right direction.. let's just ponder for a second on the new problem that arrives from that.  I have a vague idea, just need to see if it's reasonable or not.  Give me a few minutes to brainstorm on this
[15:10] <hazmat> niemeyer, so the failure modes on these from a topo conflict are pretty much non-existant
[15:10] <hazmat> its just a retry
[15:11] <hazmat> the sequence nodes are guaranteed unique
[15:12] <hazmat> we could have a local topology for the policy to utilize before saving it to the final topology, effectively making the acl update part of the topo change function, that finishes prior to return contents
[15:12] <hazmat> s/finishes/modifies the acl prior to returning the topology contents
[15:14] <hazmat> hmm.. it also needs to do the otp there, as the rules will likely reference
[15:15] <niemeyer> hazmat: Why?
[15:15] <niemeyer> hazmat: The OTP doesn't have to change
[15:15] <niemeyer> hazmat: on every iteration of the retry, that is
[15:15] <jcastro> jamespage: hey do you think it's feasable to replace pad.ubuntu.com with -lite deployed via ensemble? That would be a nice dogfood for next UDS.
[15:16] <jamespage> jcastro: hmm - maybe; its quite bleeding edge ATM and requires stuff which is not in the archive
[15:16]  * jcastro nods
[15:16] <jamespage> jcastro: will probably hit the 'is-it-packaged' challenge 
[15:17] <jcastro> yeah but if we have an ensemble formula we should stop caring about that right?
[15:17] <jamespage> jcastro: :-)
[15:17] <jamespage> jcastro: well the formula pull in and builds quite a few bits and pieces not from Ubuntu
[15:18] <jcastro> I thought that was a feature, but yeah, I can see that.
[15:18] <jamespage> jcastro: just pushed an initial version of the formula BTW
[15:18] <jcastro> jamespage: I saw, that's what prompted me to ask.
[15:19] <jcastro> well, we can certainly dogfood it during ensemble team events. :)
[15:19] <jamespage> don't try it with haproxy yet......
[15:19] <jamespage> but works OK with a mysql backend
[15:19] <jcastro> ok
[15:19] <jcastro> yeah this is one of those that just will demo great and let people try it immediately.
[15:21] <jamespage> agreed
[15:39] <hazmat> niemeyer, true the otp identity is stable and doesn't ref the topology.
[15:47] <_mup_> ensemble/debug-log-relation-settings-changes r275 committed by jim.baker@canonical.com
[15:47] <_mup_> Merged trunk and resolved conflicts
[16:12] <robbiew> m_3: hey...did you ever hear back from the MediaWiki folks?
[16:17] <m_3> robbiew: nopw
[16:17] <m_3> I'll bug Ryan again
[16:18] <robbiew> m_3: cool...I think SpamapS was planning on stopping by the offices...but not sure when
[16:18] <m_3> yeah, #wikimedia-tech's quiet at the moment... no sign of ryan
[16:19] <robbiew> probably OSCon related
[16:19] <m_3> hopefully
[16:19] <m_3> it's still early
[16:27] <niemeyer> Lunch time
[17:45] <niemeyer> fwereade: ping
[17:59] <hazmat> bcsaller, jimbaker, fwereade, niemeyer weekly meeting?
[18:00] <bcsaller>  trying g+ again?
[18:02] <niemeyer> +1
[18:05] <niemeyer> jimbaker: ping?
[18:24] <jimbaker> niemeyer, hi
[18:30] <jimbaker> niemeyer, so is this g+ meeting happening?
[18:32] <niemeyer> jimbaker: Yeah, it's rolling
[18:32] <niemeyer> jimbaker: We'll invite you
[18:32] <jimbaker> niemeyer, ok
[18:32] <niemeyer> jimbaker: What's your gmail address
[18:52] <m_3> SpamapS: break a leg!
[18:53] <SpamapS> :)
[18:54] <SpamapS> t minus 3 hours 16 minutes
[18:54] <SpamapS> err
[18:54] <SpamapS> 4 hours
[18:54] <m_3> time to enjoy the conf
[18:55] <SpamapS> s/enjoy/document/ ;)
[18:55] <m_3> niemeyer: please add me to the right lp group to edit the ensemble.ubuntu.com wiki when you get a chance
[18:55] <niemeyer> m_3: Done
[18:55] <m_3> thnsk
[19:18] <jimbaker> fyi, james.edward.baker@gmail.com
[19:45] <hazmat> niemeyer, one problem i have with current review queue stackand security.. the fixes for security-rules depend depend on stuff latter in the stack.
[19:45] <hazmat> the rules are standalone as is, so i think addressing some of the comments, and then moving forward with the rest of it, till a latter point in the branch stack probably makes the most sense
[19:46] <niemeyer> hazmat: How do you mena?
[19:47] <niemeyer> hazmat: When there is a dependency chain like this, the natural thing to do is to review them in order, apply comments in order, and merge in order
[19:47] <niemeyer> hazmat: Jumping back and forth will make things more complex
[19:49] <hazmat> niemeyer, right.. security rules are an integration piece though, i'm doing integration work and branches right now, but the rest of the branches in the stack that's part of review, including those that follow security-rules are implementation focused. some of them may contain fixes that will be needed to finish up security-rules. currently its standalone, effectively unused (though tested), but unfortunately merged into the rest of the implementati
[19:49] <hazmat> on stack of branches
[19:50] <hazmat> i guess its doable as is, it might just be a while till everything gets merged, which means i'll have a lot of pending branches
[19:50] <niemeyer> hazmat: Yeah, that's why I say that when the dependency has been introduced and they are in a chain, the natural thing to do is to apply reviews and move forward
[19:50] <niemeyer> hazmat: My suggestion is to get the bottom of the stack, fix it, and merge it
[19:50] <niemeyer> hazmat: and so on
[19:50] <hazmat> niemeyer, yeah.. its an artificial dependency at this point which is going to hold up the merge of the other 4 branches in review
[19:50] <niemeyer> hazmat: and then move forward with everything in trunk
[19:51] <niemeyer> hazmat: It is, but it's there..
[19:51] <niemeyer> hazmat: The problem you have is that you're trying to push things forward while there are reviews pending
[19:52] <niemeyer> hazmat: If you take care of the branches that are in review and get them merged, the problem disappear
[19:52] <niemeyer> s
[19:54] <hazmat> niemeyer, perhaps.. the real problem is that its an unrelated branch, which introduced before the implementation of other features, ie. an artifical dependency... i probably shouldn't have pushed it to review.. but i wanted to get feedback on the approach... not a big deal, it can be redressed in the current process.
[20:26] <jcastro> any idea what could cause this? http://pastebin.ubuntu.com/653982/
[20:27] <jcastro> oh nevermind
[20:27] <jcastro> it's that minute delay thing
[20:40] <fwereade> gents, I'm very sorry I missed you
[20:41] <fwereade> in my defence, it did start almost 2 hours after the theoretical end of my day 
[20:42] <niemeyer> fwereade: Not a problem.. we were entirely aware of that
[20:43] <niemeyer> fwereade: We decided to go forward just because we've been missing it again and again..
[20:43] <niemeyer> fwereade: We should do another one tomorrow, or very soon anyway, with you
[20:43] <fwereade> niemeyer: anything I should know from it?
[20:43] <niemeyer> fwereade: Nothing critical to what you've been doing.. hazmat is sending notes to the list
[20:43] <niemeyer> fwereade: Btw,
[20:43] <fwereade> niemeyer: cool
[20:43] <niemeyer> fwereade: Just as a note for the start of your day tomorrow
[20:44] <niemeyer> fwereade: It looks like some of the branches you've been pushing for review are missing part of the workflow
[20:44] <niemeyer> fwereade: They're not showing in the Kanban
[20:44] <niemeyer> fwereade: No rush, but worth checking early tomorrow so they get into the review flow
[20:44] <fwereade> niemeyer: hmm, lack of bugs perhaps?
[20:45] <niemeyer> fwereade: Quite possibly
[20:45] <niemeyer> fwereade: Or not assigned to the right milestone
[20:45] <niemeyer> fwereade: Or not in progress.. or.. :)
[20:45] <fwereade> niemeyer: hmm, probably all 3 for most of them
[20:45] <niemeyer> fwereade: Again, no big deal.. just worth checking
[20:45]  * fwereade looks shifty
[20:45]  * hazmat lol
[20:45] <fwereade> niemeyer: cheers :)
[20:45] <niemeyer> fwereade: Sorry for missing you on the meeting
[20:46] <fwereade> niemeyer: no worries :)
[20:46] <niemeyer> fwereade: I hope we automate that at some point.. I'm conscious this procedure is too heavy
[20:47] <fwereade> niemeyer: it's sweetness and light compared to what I'm used to :)
[20:48] <fwereade> niemeyer: however, I think what I'm used to gave me a mild phobia of kanban boards
[20:48] <niemeyer> LOL
[20:48] <fwereade> niemeyer: you'd almost think the software was written by anti-agile activists, it seemed to make a point of putting you off
[20:48] <niemeyer> fwereade: "Wait.. I see you've pushed too branches in a row.. why are you doing things so fast!?"
[20:49] <fwereade> niemeyer: haha
[20:54] <_mup_> Bug #817732 was filed: can't communicate with an orchestra server running cobbler <Ensemble:New> < https://launchpad.net/bugs/817732 >
[20:55] <niemeyer> bcsaller: re. the order of things on LXC, some vague ideas:
 1) Make multiple units work on a single machine across the board (no LXC)
 2) Make local deployments work with one or multiple units (no LXC)
 3) Make LXC work to deploy units 
[20:55] <niemeyer> bcsaller: Of course, each of these can be split further
[20:55] <bcsaller> yeah
[20:55] <niemeyer> bcsaller: But these high-level steps feel like making things more manageable
[20:56] <bcsaller> sounds like a good place to start
[20:57] <_mup_> Bug #817735 was filed: cobbler system names can change <Ensemble:In Progress by fwereade> < https://launchpad.net/bugs/817735 >
[21:00] <hazmat> bcsaller, i'm writing up meeting notes, is your next post-config item spec for containers or co-location?
[21:00] <bcsaller> yes
[21:00] <hazmat> "or" }
[21:00] <hazmat> ?
[21:00] <hazmat> bcsaller, both?
[21:00] <bcsaller> of, well, specs for both, but co-location is first
[21:00] <hazmat> bcsaller, cool, thanks
[21:01] <_mup_> Bug #817736 was filed: FileStorage interface is inconsistent <Ensemble:In Progress by fwereade> < https://launchpad.net/bugs/817736 >
[21:03] <_mup_> Bug #817738 was filed: duplicated code in ec2 LoadState and SaveState operations <Ensemble:In Progress by fwereade> < https://launchpad.net/bugs/817738 >
[21:05] <_mup_> Bug #817739 was filed: no FileStorage implementation for orchestra <Ensemble:In Progress by fwereade> < https://launchpad.net/bugs/817739 >
[21:08] <_mup_> Bug #817740 was filed: orchestra provider can't discover running zookeepers  <Ensemble:In Progress by fwereade> < https://launchpad.net/bugs/817740 >
[21:10] <_mup_> Bug #817743 was filed: orchestra provider can't launch an instance that does anything useful <Ensemble:In Progress by fwereade> < https://launchpad.net/bugs/817743 >
[21:16] <fwereade> niemeyer: btw, re: orchestra development strategy
[21:17] <fwereade> niemeyer: you're absolutely right about the shadow-trunk, that's a nicer model for future work
[21:18] <fwereade> niemeyer: but I think I'm missing something: how does that help us help us transition the code already in adres' branch?
[21:18] <fwereade> niemeyer: well, I guess it doesn't have to
[21:18] <fwereade> niemeyer: it's just that it works, and I'd prefer to transition the working ocde smoothly if I can
[21:19] <fwereade> niemeyer: ...and as soon as I say it in public, it seems much dumber than it did in my head
[21:20] <fwereade> with one more branch (in progress), I think we'll have something that's a viable base for RoAkSoAx to branch from
[21:23] <niemeyer> fwereade: There's no way to transition his work into something compatible with trunk in a mergeable way besides doing it piece by piece
[21:25] <fwereade> niemeyer: yep, I think you're right, it just seems like a shame :)
[21:26] <niemeyer> fwereade: Spikes are spikes.. we have to recognize them by the benefit they provide, but transitioning from a spike into a stable implementation takes known pain
[21:27] <fwereade> niemeyer: and, to be fair, less total pain than "it works? SHIP IT!"
[21:27] <niemeyer> fwereade: Exactly
[21:30]  * fwereade has an uncomfortable feeling he may have quoted directly from someone at his first job
[21:31] <robbiew> lol
[22:00] <SpamapS> crazy idea
[22:00] <SpamapS> we should have a "sneakernet" provider
[22:00] <SpamapS> all it does is create nodes in zookeeper...
[22:01] <SpamapS> and feed you back admin credentials to spawn machine agents
[22:01] <SpamapS> that way you can join existing machines to ensemble
[22:07] <niemeyer> SpamapS: As a first step, defining a formula for an external service would probably work
[23:05] <robbiew> SpamapS:  done your talk yet?
[23:09] <adam_g> robbiew: starts in 2 min
[23:10] <robbiew> adam_g: ah...how's the attendance?
[23:11] <robbiew> I noticed it was still in the OS track b/c of the previous upstart topic
[23:11] <robbiew> not cloud :/
[23:40] <_mup_> ensemble/debug-log-relation-settings-changes r276 committed by jim.baker@canonical.com
[23:40] <_mup_> Fixed faulty merge