#ubuntu-ensemble 2011-07-18
<fwereade> good mornings
<_mup_> Bug #812343 was filed: formula exec env should include DEBIAN_FRONTEND=noninteractive <Ensemble:New> < https://launchpad.net/bugs/812343 >
<adam_g> m_3: pong
<m_3> adam_g: was just responding with the rabbit stuff
<m_3> adam_g: I attached it to the bug report
<m_3> adam_g: was able to reproduce your problem, then got it 'started'
<m_3> adam_g: misbehaving package dep
<m_3> adam_g: lemme know if that fixes it for you pls
<SpamapS> rabbit is a strange beast
<SpamapS> causes all kinds of problems for LP devs because of its insistance that its hostname never change.
<m_3> SpamapS: I think it was just a startup script in a dependent package not playing nicely
<SpamapS> m_3: what package?
<m_3> don't know which one...
<m_3> apt-get install -y rabbitmq-server didn't work
<m_3> adding noninteractive and --no-install-recommends fixed it for me
<adam_g> m_3: i actually was using the non-interactive frontend for a while, but took it out to see if it made any difference
<adam_g> ill give no-install-recommoends a shot, thanks
<SpamapS> erlang-base | erlang-base-hipe, erlang-asn1, erlang-corba, erlang-crypto, erlang-docbuilder, erlang-edoc, erlang-eunit, erlang-ic, erlang-inets, erlang-inviso, erlang-mnesia, erlang-odbc, erlang-os-mon, erlang-parsetools, erlang-percept, erlang-public-key, erlang-runtime-tools, erlang-snmp, erlang-ssh, erlang-ssl, erlang-syntax-tools, erlang-tools, erlang-webtool, erlang-xmerl
<m_3> I can narrow it down to a particular package if this gets you past the hurdle
<SpamapS> lots of dependencies from erlang-nox
<m_3> yup
<adam_g> m_3: still no luck with that install hook and those apt-get options.
<adam_g> m_3: were you able to deploy to a completely fresh system?
<fwereade> hey guys, didn't notice you come in :)
<fwereade> it's the end of my day now but I'll be back shortly
<fwereade> my mind in in a maze of twisty little tests, all alike, and I'm not going to be able to resisit another look :p
<m_3> adam_g: yeah, from totally shutdown to rabbitmq 'started'
<m_3> adam_g: are you specifying the ami or anything in your environment?
<adam_g> m_3: yeah, an oneiric daily from last week. which are you using?
<m_3> not specifying ami... defaults to natty
<m_3> ami-06ad526f
<m_3> which ensemble rev?
<adam_g> sec
<m_3> fwereade: hey man
<m_3> adam_g: yeah, just tested again and it comes up fine... only bzr diff with your branch is http://pastebin.com/76H4dsBF
<m_3> adam_g: my env (less keys) http://pastebin.com/fL0fSn3r
<fwereade> m_3: heyhey :)
<fwereade> oh, dammit
<fwereade> I just spent half an hour reexamining my apparently-flawed understanding of mocker and twisted
<fwereade> turned out to be a lower-case 'r' in some_mock.callRemote
<jimbaker> fwereade, they are tricky to combine
<fwereade> jimbaker: yeah, I assumed I'd encountered some subtlety that would teach me something
<fwereade> all it's actually taught me is lrn2type
<RoAkSoAx> fwereade: \o/
<fwereade> RoAkSoAx: hey, how's it going
<SpamapS> we've all been humbled by twisted a bit. :)
<RoAkSoAx> fwereade: good good
<RoAkSoAx> you?
<fwereade> RoAkSoAx: yeah, recovering, not used to this jetlag lark
<RoAkSoAx> fwereade: hehehe yeah!! fun times !! :)
<jimbaker> fwereade, in zed shaw's learning python the hard way, learning to type (and the resulting debugging) is a key piece of the learning strategy
<fwereade> jimbaker: still haven't looked into that... I usually enjoy zed's stuff
<fwereade> jimbaker: first encountered him ranting about statistics in the ZSFA days :)
<jimbaker> fwereade, some interesting insight. i was perusing it for teaching my daughter python
<fwereade> jimbaker: cool :) how old is she?
<jimbaker> fwereade, she's 10. she has already programmed in C (albeit through the use of a visual editor) and squeak
<jimbaker> she learned squeak on her own. C was for a robotics camp
<fwereade> jimbaker: cool, I have that to look forward to, Laura's only 2 :)
<m_3> adam_g: they're coming up, but rabbit's not fully installed... still trying to figure out where it's going
<fwereade> jimbaker: I have some blurred photos of her waving a cuddly snake, and that's as far as I've got
<SpamapS> jimbaker: My 8 year old found Scratch this last weekend. He had quite a bit of fun with it. :)
<jimbaker> fwereade, so zed shaw's premise is that those of who learned how to program by typing in code from a magazine/book is fundamentally different than the latter practice of doing cut & paste
<SpamapS> Still its really hard to keep him focused on it.. its an unbelievable leap for him from that to what the xbox does. ;)
<fwereade> jimbaker: hmmm, yes
<jimbaker> i don't know if he uses this analogy, but looking at code is like an expert chess player looking at a board. we can use pattern recognition to really grok what's going on - especially what's wrong
<SpamapS> crap.. I suck at chess
<m_3> _love_ scratch
<fwereade> jimbaker: nice
<jimbaker> SpamapS, yeah, my daughter was playing with scratch over at a friend's house. what kids do when you're monitoring them, it's dangerous ;)
<jimbaker> you're *not* monitoring ...
<fwereade> jimbaker: it'll be like wargames all over again
<jimbaker> fwereade, indeed. i think there will be a tv movie about it
<fwereade> RoAkSoAx: hey, cool, glad you're coming in august
<RoAkSoAx> fwereade: mee too.. this week I'll be working on getting everything Clint'
<RoAkSoAx> did onto the ensemble bootstrap
<RoAkSoAx> and I guess in the sprint we'll figure final details
<fwereade> RoAkSoAx: awesome
<fwereade> RoAkSoAx: I have the final(!?) state of that branch in review again, and I'm starting to integrate stuff from our shared branch on top of it
<fwereade> RoAkSoAx: it crossed my mind that from what Gustavo said I shouldn't be pulling from or pushing to that branch directly
<fwereade> RoAkSoAx: so I guess it doesn't need to be shared
<fwereade> RoAkSoAx: but no harm done :)
<RoAkSoAx> fwereade: right, but well I guess that as long as nothing changes significantely and breaks the stuff i'm working on, there wouldn't be much difference on how e handle that
<RoAkSoAx> fwereade: but still I'd like to work with a branch that's in sync with your work
<RoAkSoAx> so I can be sure nothing will break significantly
<fwereade> RoAkSoAx: indeed not -- I just thought you should know
<RoAkSoAx> fwereade: :)
<RoAkSoAx> ;)
<fwereade> RoAkSoAx: the major source of confusion I think is that I'm having to use the twisted xmlrpc client, and test it, and so the code is coming out the other end a ...little less recognisable
<fwereade> RoAkSoAx: but given I have an environment, I think I can be pretty confident in it if it has the same results as yours
<fwereade> RoAkSoAx: that said it might be sensible to get you to review my (your) changes once they've passed through my hands ;)
<RoAkSoAx> fwereade: hehe yeah I was planning to send my changes for your review, as really, the main concern I have is just use all the goodness of your refactoring
<RoAkSoAx> rather than me putting all the code in one single function
<RoAkSoAx> :)
<fwereade> RoAkSoAx: as long as it's moderately spread out I'm not bothered :p
<fwereade> RoAkSoAx: just shout if you find you're ending up duplicating code
<fwereade> RoAkSoAx: which would imply a deficiency in itc current shape
<fwereade> RoAkSoAx: actually I had a question
<fwereade> netboot_enabled
<fwereade> acquire_system will only grab systems with it enabled
<fwereade> start_system always tries to set it to True
<fwereade> I presume we only actually need one of those, and I was wondering which you thought would be better
<RoAkSoAx> fwereade: let me check
<RoAkSoAx> fwereade: no I believe they should be kept
<RoAkSoAx> fwereade: acquire_system will obtain an available system
<RoAkSoAx> fwereade: while start system will tell it to start by issueing the commands, like containing the system through IPMI or stuff like that
<fwereade> RoAkSoAx: not sure I follow
<fwereade> if a system is already netboot_enabled, why would we set that to True again on start?
<RoAkSoAx> fwereade: acquire_system browses for a system that is under foo-available management class, and grabs one 
<fwereade> similarly, if we can set it to True on launch, why exclude systems that don't have it set?
<fwereade> RoAkSoAx: but acquire_system also skips those without netboot_enabled
<fwereade> RoAkSoAx: ...unless I'm looking at the wrong branch :/
<RoAkSoAx> fwereade: then start_system, tells cobbler to contact any configured device to actually power on the machine. Setting it to netboot_enable when acquiring the system makes sure to acquire one that has netboot enabled
<RoAkSoAx> fwereade: but when starting the system, it is making sure that netboot enabled is correctly set, otherwise the system willfail to netboot
<RoAkSoAx> fwereade: it is just a way of making sure
<RoAkSoAx> fwereade: cause let's say for whatever reason we acquired a system, and suddently someone changes the system profile on cobbler, then when ensemble tells it to boot the machine up
<RoAkSoAx> fwereade: the machine wont be able to PXE
<fwereade> RoAkSoAx: if we also _set_ it when we acquire the system, I'm happy, but as it is we just skip the system if it isn't already set the way we want
<fwereade> RoAkSoAx: it makes sense to me to set it immediately before launch, even if it seemed to be set ok before
<fwereade> RoAkSoAx: but if we _can_ set it, why would we avoid acquiring a system _without_ netboot_enabled?
<RoAkSoAx> fwereade: maybe it would be a system that we would like to not use for a moment but still keep under foo-available
<RoAkSoAx> fwereade: I know it does not seem unnecessary, but for me I think it is a necessar evil :)
<fwereade> RoAkSoAx: I'd be inclined to make in-foo-available the single point of truth for that
<_mup_> ensemble/expose-provision-machines-reexpose r292 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<fwereade> RoAkSoAx: but the critical thing is that it is actually intended
<fwereade> RoAkSoAx: and I don't need to worry that I'm totally misunderstanding ;)
<RoAkSoAx> fwereade: yeha, but AFAIK I think we should not care that much right now about that, as it is super easy to override
<RoAkSoAx> fwereade: cause as I mention, i think we need better ways to ensure the interaction with the management classes
<RoAkSoAx> as I explained last week, we could set the management class from foo-available to foo-acquired and then the physicl machine might fail to start
<RoAkSoAx> while ensemble status still shows that machine as available
<RoAkSoAx> and doing the check, we can be assuming that "there's a reason by it has been set to disable. but let's just make sure we enable it before starting the machine"
<RoAkSoAx> s/by/why
<fwereade> RoAkSoAx: yep, makes sense
<fwereade> RoAkSoAx: thanks :)
<fwereade> anyway, I think that really is it for me now
<fwereade> gn all, see you soon
<_mup_> Bug #812441 was filed: ensemble ssh should passthrough args to ssh <Ensemble:New> < https://launchpad.net/bugs/812441 >
<_mup_> ensemble/expose-provision-machines-reexpose r293 committed by jim.baker@canonical.com
<_mup_> Support reexposing a service
<niemeyer> Hallo!
<niemeyer> m_3: Nice blog post1
<niemeyer> !
<jcastro> m_3: SpamapS: *hand wave* who wants to go over the first cut of my ensemble slides?
<SpamapS> jcastro: fire away
<SpamapS> m_3: feedback on the NFS formula.. I don't think we should keep repeating the paradigm of using the remote service name for stuff.
<SpamapS> m_3: I'd rather see us implement a mode where the consuming side suggests a name... that way two services can share the same NFS export.
<SpamapS> m_3: we also need to put together some interface docs .. :)
<jcastro> m_3: nice blog post!
<jcastro> I wonder if it makes sense to do a debconf prompt when you install ensemble for the EC2 keys instead of punting the user to editing a dotfile right off the bat
<m_3> niemeyer: thanks!
<m_3> jcastro: are your latest on the U1 share?
<m_3> SpamapS: yup... do you have any interfaces written up so far?
<jcastro> m_3: they are
<m_3> seems like it's naturally part of the formula doc... we then have an 'interfaces' page on the wiki that scrapes formula docs from the repo
<m_3> jcastro: used 22G on U1 already... gotta get it off the web
<jcastro> heh, nice!
<m_3> jcastro: which talk you wanna go over first?
<jcastro> m_3: 5 minute is the same you saw, 10 minute is just the exanded version of that, it just needs a once over
<jcastro> Ensemble Long Presentation is the one I'm having a hard time with, it's like 30 slides but I am wondering if it needs to be so many slides vs. just doing a demo and going through it step by step
<m_3> ok, I'll look
<jcastro> m_3: I just TODOed myself making a presentation of your blog post basically
<jcastro> it's nice to have something other than mediawiki to talk about, heh
<SpamapS> m_3: no but I think we'll just use ensemble.ubuntu.com/Formulas/Interfaces/(interface-name-here) .. 
<SpamapS> m_3: I was thinking we can use a yaml format.. and leave it open to plugin test scripts that can be run at each point to verify an interface... but.. I wonder if we shouldn't just write them up in a way we can re-use for your cucumber idea.
<SpamapS> jcastro: less slides unless you do them "Identity 2.0" style.. http://www.youtube.com/watch?v=RrpajcAgR1E
<m_3> cucumber can consume yaml and adapt tests to each interface as nec
<m_3> jcastro: I'll try to build out some other stacks and blog about them
<m_3> jcastro: mediawiki full stack is still a very compelling xample
<SpamapS> m_3: more compelling w/ config settings  implemented. :)
<m_3> jcastro: especially if we can connect other aux services we've developed
<m_3> SpamapS: +1
<m_3> jcastro: ok, I personally don't like bullet points on slides... it automatically makes me want to sleep
<SpamapS> m_3: watch identity 2.0
<SpamapS> best presentation I've ever seen
<SpamapS> Its the top of the pyramid, but some elements fit where we will start.. in the middle. :)
<m_3> jcastro: diagrams on 10min-page15 should maybe be flipped over (mysql on bottom)... totally minor though
<jcastro> m_3: someone from design will make us proper stuff  so pretend the blocks and stuff look pretty (but noted)
<m_3> wow I really like the animated parts of that... good story
<m_3> hard to believe though... (everybody says what we're saying here)
<SpamapS> Cloud 2.0....
<SpamapS> ;)
<m_3> it's like... "no, really!"
<m_3> jcastro: I'd expect we'll have different long-form talks depending on audience?
<jcastro> m_3: that's totally ripped off from Nick, but I have no shame
<jcastro> m_3: yeah so I was thinking of having fragments based on audience
<jcastro> and then swap out parts of the long one depending on audience
<jcastro> like, I don't think sysadmins will even want or care about the long presentation, I'd want to just see a terminal and someone typing.
<m_3> jcastro:right, that's sort of the point of services though
<SpamapS> they will care about the long presentation at oscon
<SpamapS> lots of sysadmins and devs there
<m_3> you used to have different perspectives on this
<jcastro> SpamapS: I look forward to stealing your long form one as well.
<jcastro> SpamapS: actually if you have it done and want to jet me a copy that would be swell
<m_3> infrastructure meant very different things to different people
<SpamapS> jcastro: I really like the idea around having 40-50 total slides, and just building presentations from those based on a few clues.
<SpamapS> hahaha done hahahahaha
<m_3> high levefolks thought of services, sysadmins thought of servers
<m_3> we're bringning th two together
<m_3> jcastro: lightning's look great, lemme see what sort of notes I can come up with for longform.
<jcastro> SpamapS: yeah, right now that's exactly what the 10 minute and long form one are, it's a shuffle of what I had around + more added in 
<jcastro> m_3: you can plop your notes right into the slide notes if you want
<m_3> definitely steal from SpamapS... 
<m_3> SpamapS: how long is your talk?
<m_3> (going to be)
<SpamapS> m_3: 40 min
<SpamapS> I have approximately 2 slides done.. in outline form only. ;)
<m_3> jcastro: there's the answer :)
<m_3> nice
<SpamapS> I actually plan to have only about 10, with a couple repeating, if I do it right.
<m_3> that what mornings are for
<m_3> SpamapS: let us know if we can help... (draw a diagram or something)
<SpamapS> Heh.. this was my deck for the last Ubuntu Cloud Days.. http://spamaps.org/files/Ensemble%20Presentation.pdf
<jcastro> SpamapS: I just stole it, got any others around?
<_mup_> ensemble/expose-provision-machines-reexpose r294 committed by jim.baker@canonical.com
<_mup_> Removed debugging
<m_3> SpamapS: great last slide
<SpamapS> m_3: I think there are a few too many arrows and lines.. it was the first ensemble picture I painted. ;)
<jcastro> we'll just get someone to do a proper awesome diagram
<jcastro> once we get the content the way we want it
<jcastro> so hopefully it won't look so complicated, heh
<jcastro> robbiew: I'd like to go over these slides today with you too if that's ok
 * jcastro is looking for any excuse to do a G+ hangout. :p
<robbiew> jcastro: heh...cool...got a 1:1 with jimbaker now...in about 30min?
<jcastro> whenever is fine for me!
<_mup_> ensemble/expose-provision-machines-reexpose r295 committed by jim.baker@canonical.com
<_mup_> Removed sleeps in ensemble.state.tests.test_base and added test for client being disconnected in a watch
<robbiew> jimbaker: ping :)
<jimbaker> robbiew, hi
<robbiew> jcastro: yo
<jcastro> howdy
<robbiew> ready to go, when you are
<jcastro> robbiew: sure, wanna hang out?
<jcastro> er, "hangout"
<robbiew> sure
<jcastro> started
<robbiew> jcastro: so this is my big problem with G+
<robbiew> I can never figure out how to hangout
<robbiew> beyond hanging with a whole damn circle
<bcsaller> oh, are we hanging out? brt
<bcsaller> j/k
<SpamapS> jcastro: hrm, why doesn't u1's little messaging indicator tell me when you have shared something with me? In this case, it only told me that I had accepted a share.. :-P
<SpamapS> seems rather silly
<SpamapS> jcastro: "Ensemble Manges Services, not Machines" .. how baout "Ensemble Manages Services, not Servers"
<jcastro> ooh, I like that
<jcastro> SpamapS: I am heading to dinner, if you can just PM me your feedback that would be great and I can check it out when I get back
<SpamapS> jcastro: Or in the morning.. no rush. :)
<adam_g> that reminds me... 
<adam_g> im going to give a demo of ensemble at a local AWS users group. are there slides anywhere that might be of use?
<jcastro> I've got some
<jcastro> when is your talk?
<adam_g> jcastro: thanks!
<adam_g> tomorrow night :)
<adam_g> SpamapS: regarding your comment on https://bugs.launchpad.net/ensemble/+bug/810808 , is there a way to run debug-hooks so that it can catch exec of the install hook of a yet-to-be-deployed unit  in time?
<_mup_> Bug #810808: Formula install hook never completes <Ensemble:New> < https://launchpad.net/bugs/810808 >
<SpamapS> adam_g: I think if you get debug-hooks going before the install hook it will work.. but that may be tricky as you probably can't run debug-hooks until ssh is working
<adam_g> SpamapS: ssh isn't the tricky part, but deploying the service and getting debug-hooks to run before the install hook fires
<SpamapS> adam_g: looks like we need a --debug-wait option to deploy/add-unit
<adam_g> SpamapS: might be useful. i was actually able to catch the hook by looping while waiting on ec2
<adam_g> and it turns out, if i run the install hook manually it installs rabbit, exits and continues onto start after
 * adam_g puts on his strace goggles
<SpamapS> Haha
 * SpamapS has a vision of a gunner in a Mad Max / Waterworld tank preparing to mow down the unfortunate syscalls..
<adam_g> hehe. 
<adam_g> which process actually execs the hooks? the machine agent or the unit agent?
<SpamapS> unit
<SpamapS> machine execs unit agent..
#ubuntu-ensemble 2011-07-19
<_mup_> ensemble/expose-provision-machines-reexpose r296 committed by jim.baker@canonical.com
<_mup_> Mocked version of verifying connection closed terminates topology watch
<_mup_> ensemble/expose-provision-machines-reexpose r297 committed by jim.baker@canonical.com
<_mup_> Removed need for mocking
<_mup_> ensemble/expose-provision-machines-reexpose r298 committed by jim.baker@canonical.com
<_mup_> Test clean up/comments; also now test what happens when the client is disconnected before the watch is even started
<_mup_> ensemble/expose-provision-machines-reexpose r299 committed by jim.baker@canonical.com
<_mup_> Comment
<_mup_> ensemble/expose-provision-machines-reexpose r300 committed by jim.baker@canonical.com
<_mup_> Removed unused import
<_mup_> Bug #812619 was filed: Re-exposing a service does not work with the provisioning agent <Ensemble:In Progress by jimbaker> < https://launchpad.net/bugs/812619 >
<fwereade> does anyone know anything abut the config parsing in ensemble?
<niemeyer> Morning all
<niemeyer> fwereade: I'm sure someone knows about it :-)
<niemeyer> fwereade: What are you looking for?
<fwereade> niemeyer: I've written a test for the orchestra schema, and I can't get both that and the ec2schema tests to pass at the same time
<fwereade> niemeyer: the problem is that we get an error for every schema we check
<fwereade> niemeyer: and we don't necessarily get the one I'm testing for
<fwereade> niemeyer: (that's controlled by the order in which we specify the provider schemas in the OneOf)
<fwereade> niemeyer: am I missing something obvious? :)
<niemeyer> fwereade: Hard to tell without knowing what the error is
<fwereade> niemeyer: if I'm missing (say) orchestra-server, then we get an exception from coerce()
<fwereade> niemeyer: quite rightly
<niemeyer> fwereade: Can you please paste the error and the code somewhere?
<fwereade> niemeyer: but we also get an exception when we attempt to coerce() to the EC2 schema, because we don't have (say) control-bucket
<fwereade> http://paste.ubuntu.com/647260/
<niemeyer> fwereade: Well, we shouldn't attempt to coerce an orchestra config to an EC2 schema
<fwereade> niemeyer: agreed
<niemeyer> fwereade: They have to be separate schemas
<fwereade> niemeyer: hm, ok, so we just do a first pass to determine the type
<fwereade> niemeyer: and then try to apply the actual schema we've detected
<fwereade> niemeyer: it's not intractable, but I didn't want to lay heavily into that section of code without checking with someone
<niemeyer> fwereade: Hmmm.. just a sec
<fwereade> niemeyer: no rush, I was just about to have lunch anyway
<fwereade> niemeyer: and I've got some unrelated issue with my VMs that I need to sort out before I can be confident it's worth proposing a merge
<fwereade> niemeyer: (I have the cobbler communication tested, and I can boot machines, but they don't actually manage to install)
<fwereade> niemeyer: talk later?
<niemeyer> fwereade: I see what you mean
<niemeyer> fwereade: This is a shortcoming in our schema library
<niemeyer> fwereade: Let me ponder for a moment on the issue and we catch up later
<niemeyer> fwereade: You're not blocked on this, right?
<niemeyer> fwereade: I think you're doing the right thing, FWIW.. it's just the schema library itself that should be fixed
<lynxman> hey guys, I have a question for you :)
<lynxman> got ensemble module on Mac but when I execute it says
<lynxman> ImportError: No module named zookeeper
<lynxman> that I assume is not the txzookeper python library
<lynxman> so which one should I look to import?
<lynxman> zookeeper itself?
<lynxman> ello? :)
<niemeyer> lynxman: Hey
<niemeyer> lynxman: Yeah, that's the C extension that is provided by zookeeper itself
<niemeyer> lynxman: txzookeeper is a wrapper on top of that which "Twistifies" (ugh) the library
<fwereade> niemeyer: that was my reading of the issue, but I thought it was worth checking ;)
<fwereade> niemeyer: and no, I'm not blocked yet, but I should be in a little while
<fwereade> niemeyer: I'll let you know when it bcomes urgent :)
<lynxman> niemeyer: cool, so I'll just create the python extensions for that :)
<niemeyer> fwereade: Thanks :)
<niemeyer> lynxman: Super :)
<RoAkSoAx> fwereade: howdy! do you think I should do something like this?: http://paste.ubuntu.com/647347/
<fwereade> RoAkSoAx: sounds sensible to me
<fwereade> RoAkSoAx: will make it easy to pull out whatever's common
<fwereade> RoAkSoAx: btw, you remember we got everything running on my machine
<fwereade> RoAkSoAx: seems we didn't quite, was wondering if you had a few minutes to point me in the right direction
<fwereade> RoAkSoAx: I can launch a machine but the install doesn't complete
<RoAkSoAx> fwereade: you need to change the IP of the proxy
<RoAkSoAx> fwereade: in the preseed
<RoAkSoAx> fwereade: that's what comes to my mind
<fwereade> RoAkSoAx: the first errors I see are a couple of "debconf-copydb: cannot open question file"s
<fwereade> RoAkSoAx: but it keeps doing stuff after that
<fwereade> RoAkSoAx: it starts to look fatal around "cannot stat /etc/default/puppet", while setting up ubuntu-orchestra-client
<lynxman> niemeyer: just FYI, it's done https://trac.macports.org/ticket/30237
<RoAkSoAx> fwereade: remove orchestra-client from the preseed
<RoAkSoAx> fwereade: or extra packages
<RoAkSoAx> that is from the preseed *inside* the cobbler VM
<RoAkSoAx> /var/lib/cobbler/kickstarts/ensemble.preseed I think
<fwereade> RoAkSoAx: thanks: it took me an embarrassingly long time peering at the kickstarts dir on the host system before the penny dropped there
<hazmat> lynxman, awesome
<hazmat> lynxman, does the live check stuff mean it will auto pull the latest?
<hazmat> hmm. probably not with the checksums
<lynxman> hazmat: nope, we'll have to update manually for now
<lynxman> hazmat: macports is like BSD ports, it requires static versions, but once it's there updating it is easier
<_mup_> ensemble/trunk r276 committed by kapil.thangavelu@canonical.com
<_mup_> merge deploy-tolerates-broken-formulas-bug-810765 [r=fwereade,jimbaker][f=810765]
<_mup_> Ensemble deploy will behave nicely when formulas in a repository have broken metadatas, by ignoring them.
<_mup_> ensemble/trunk-merge r270 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
<_mup_> ensemble/security-node-policy-def r276 committed by kapil.thangavelu@canonical.com
<_mup_> address review comments, inline owner_ace func, additional doc comments on rule signature.
<RoAkSoAx> fwereade: heh it happens
<niemeyer> lynxman: Woot!
<lynxman> niemeyer: \o/
<lynxman> niemeyer: as soon as it's live mac developers will be able to use and deploy ensemble as well
<lynxman> (client side of course)
<niemeyer> lynxman: Please keep us updated.. quite curious to see how that'll go
<lynxman> niemeyer: will do :)
<niemeyer> Alright.. I'll step out for lunch
<niemeyer> Just off of a meeting
<niemeyer> Oh
<niemeyer> Btw,
<niemeyer> For those who didn't receive it, we have a developer (all formula and core are welcome) meeting today at 17:30UTC for sync up
<niemeyer> (voice)
<niemeyer> In Google+
<niemeyer> Please be here if you're interested
 * niemeyer => lunch
<_mup_> Bug #810808 was filed: Formula install hook never completes <Ensemble:In Progress> < https://launchpad.net/bugs/810808 >
<kirkland> niemeyer: how does one attend this meeting?
<kirkland> niemeyer: is it irc only?
<kirkland> niemeyer: i see "(voice)" and "In Google+", but I don't see how to join
<bcsaller> kirkland: I think someone has to start it which hasn't happened yet as far as I can see
<kirkland> bcsaller: kthx
<_mup_> Bug #812996 was filed: Deploy should support config options <Ensemble:In Progress by bcsaller> < https://launchpad.net/bugs/812996 >
<hazmat> bcsaller, it looks like you have a conflict against trunk in that merge proposal
<hazmat> kirkland, are you on google plus?
<kirkland> hazmat: sadly, yes
<robbiew> hangouts are cool...but setting them up sucks
<robbiew> unless you create a damn circle for each individual friend...or always hangout with the same group </rant>
<SpamapS> You don't have to "hangout" .. you can just video chat with somebody
<fwereade> SpamapS: where's the fun in that?
<robbiew> but if I want to video chat with 2 people...I need to hangout, right?
<SpamapS> Probably
<robbiew> if those two people are in my circles, i should be able to just select them and hang...but you can't 
<robbiew> you have to invite them..and hope you have the right address for them that they've used with google +
<robbiew> Beta crap
<SpamapS> your worlds... will collide
<robbiew> </rant..for real>
<robbiew> I guess I'm getting what I paid for
<robbiew> lol
<kirkland> robbiew: you're paying with your own soul
<hazmat> fwereade, niemeyer, jimbaker  .. we're all hanging out in hangout if your interested
<jimbaker> hazmat, i've been following this discussion about the hangout - but is there a URL for it?
<hazmat> jimbaker, you have to be logged into google plus, and it should appear there
<jimbaker> let me check it again... i just joined yesterday
<jimbaker> hazmat, looked around the page, just see something about creating a hangout
<hazmat> jimbaker, i'll send another link
<hazmat> jimbaker, what's your icon.. there are lots on google +
<jimbaker> hazmat, my icon? ;)
<hazmat> jimbaker, its hard to figure out which of the many is you
<jimbaker> hazmat, i guess i'm the one who has not yet associated a picture w/ my account yet
<hazmat> jimbaker, there are lots of those.. do you have profile info filled out?
<hazmat> jimbaker, actually easier might just be sending me your gmail address via priv msg
<jimbaker> hazmat, i just added two bits - my picture, and employment at canonical
<jimbaker> btw, my desktop doesn't have audio input
<jimbaker> i will have to switch over to my laptop instead
<_mup_> ensemble/trunk r277 committed by kapil.thangavelu@canonical.com
<_mup_> merge security-node-policy-def [r=niemeyer,bcsaller][f=810510]
<_mup_> A security policy implementation that returns ACLs for zk nodes by path.
<niemeyer> hazmat: Oh, wasn't it 17:30?
<niemeyer> I'll join
<niemeyer> Hmm
<niemeyer> bcsaller, fwereade, jimbaker, m_3: ping?
<jimbaker> niemeyer, hi
<bcsaller> niemeyer: ready when you are
<fwereade> niemeyer: heyhey
<fwereade> niemeyer: we're all hanging out
<niemeyer> fwereade: Where?
<niemeyer> I don't have any invitations here
<fwereade> niemeyer: google plus somewhere... not sure who set it up
<niemeyer> fwereade: Someone has to invite me
<niemeyer> fwereade: and bcsaller/jimbaker, I suppose
<fwereade> niemeyer: done, I think
<jimbaker> hazmat is the person who set up the hangout
<jimbaker> i think the easiest way is for every person who wants to join the meeting to pm hazmat their gmail address, assuming they have one
<niemeyer> fwereade: No, nothing here
<fwereade> heh, bizarre
<m_3> ney niemeyer... we're "hanging"
<fwereade> niemeyer: just tried again, just in case
<kirkland> hazmat: how do i get in?
<fwereade> it's a shame niemeyer's not there to enjoy our cool party
<fwereade> :p
<niemeyer> Ah, got from bcsaller now
<m_3> I think ben set it up... it appeared in my stream
<bcsaller> just now, yeah
<niemeyer> kirkland: We have our own party now! :-)
<kirkland> bummer
<m_3> hazmat: we switched from that hangout to a new one
<niemeyer> SpamapS: ping
<kirkland> robbiew: SpamapS: you guys coming?
<SpamapS> Where do I go?
<jimbaker> looks like i have a problem joining from my laptop... will try again
<adam_g> can someone send invite to gandelman.a@gmail.com plzz?
 * robbiew has no delusions about the lack of value he will provide to the discussion
<niemeyer> hazmat: ping?
<niemeyer> hazmat: Where are you?
<hazmat> niemeyer, networking hicckup
<SpamapS> Sorry guys, I want to be there, but I don't know where there is
<m_3> SpamapS: invited
<m_3> adam_g: invited
<SpamapS> m_3: I have no evidence of any invitations. :p
<bcsaller> SpamapS: not in your stream?
<m_3> it shows up on the stream
<m_3> go to ben's profile
<SpamapS> well hidden
<niemeyer> fwereade: I can hear you
<fwereade> niemeyer: looks like it was actually foretelling the near future
<m_3> SpamapS: bad audio
<fwereade> trying to rejoin
<fwereade> not looking good :/
<m_3> SpamapS: bad audio again
<SpamapS> my net connection has basically failed
<RoAkSoAx> fwereade: still arou nd?
<fwereade> RoAkSoAx: yep
<fwereade> RoAkSoAx: what can I do for you?
<RoAkSoAx> fwereade: want you to review something real quick
<fwereade> RoAkSoAx: sure :)
<RoAkSoAx> fwereade: http://paste.ubuntu.com/647512/
<RoAkSoAx> fwereade: so from OrchestraSaveState __init__ I'm instancing OrchestraStorage which basically obtains the connection to the storage
<RoAkSoAx> fwereade: so my concerns are, instancing this : # FIXME: Should this be done like this?
<RoAkSoAx>         self._storage = OrchestraStorage(self)
<RoAkSoAx> and in OrchestraStorage
<RoAkSoAx> obtain the config like this:
<RoAkSoAx> self.config = provider._provider.config
<fwereade> RoAkSoAx: I think that should be a self._provider.get_file_storage
<fwereade> RoAkSoAx: in general I think if we can do things via the provider we should
<RoAkSoAx> fwereade: ok so I keep the get_file_storage function on the MachineProvider then
<fwereade> RoAkSoAx: I think so
<RoAkSoAx> fwereade: alright, cool, thanks!
<fwereade> RoAkSoAx: a pleasure :)
<adam_g> is this error message meaningful to anyone? i haven't seen it before:
<adam_g> http://paste.ubuntu.com/647521/
<SpamapS> whoa that does look confusing
<SpamapS> are the hooks copied to some tmp dir?
 * SpamapS prepares an upload of ensemble that runs the test suite during build
<adam_g> SpamapS: the file in /tmp is some template script generated by ensemble
<adam_g> http://paste.ubuntu.com/647525/
<SpamapS> Interesting.. here are the failures when running the test suite ina clean chroot...
<SpamapS> http://paste.ubuntu.com/647530/
<SpamapS> exceptions.ValueError: Could not find AWS_ACCESS_KEY_ID
<m_3> adam_g: the kill not finding the pid just means the hook exited without removing its pid file
<m_3> just an unclean or abnormal exit for some reason?
<adam_g> hopefully
<m_3> what happens in non debug-hooks mode?
<adam_g> same
<adam_g> or at least i think.. let me check once more
<_mup_> ensemble/expose-provider-ec2 r293 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<_mup_> ensemble/expose-provider-ec2 r294 committed by jim.baker@canonical.com
<_mup_> Merged upstream expose-provision-machines-reexpose and resolved conflicts
<_mup_> ensemble/expose-provider-ec2 r295 committed by jim.baker@canonical.com
<_mup_> Default to NotExposed for watched_units
<_mup_> Bug #813112 was filed: test suite cannot run without an ssh key <Ensemble:New> < https://launchpad.net/bugs/813112 >
<niemeyer> Alright!
<niemeyer> That was a long call
<niemeyer> fwereade: ping
<jimbaker> good to see the firewall actually opening/closing ports in response to the use of ensemble expose/ensemble unexpose
<jimbaker> niemeyer, the only issue with dividing up the branches is that there's no ec2 provider implementation in expose-provision-machines; this doesn't come into place until the branch i'm currently working on, expose-provider-ec2
<niemeyer> jimbaker: Dividing which branches in this case?
<jimbaker> the two branches i just mentioned, expose-provision-machines and expose-provider-ec2
<jimbaker> given that MachineProvider is a subclass of object, we don't currently have a way of defaulting say an empty implementation
<jimbaker> expose-provision-machines works fine with the dummy provider of course
<niemeyer> jimbaker: Sorry, ECONTEXT
<niemeyer> jimbaker: What's the issue you're trying to address?
<jimbaker> niemeyer, the MachineProvider implementation is required to implement open_port, close_port, get_opened_ports
<jimbaker> this is done with the dummy provider
<niemeyer> jimbaker: Yes, the implementation is required.. for the implementation? :-)
<jimbaker> and is changed with the ec2 provider in expose-provider-ec2
<jimbaker> niemeyer, maybe i should have a branch called transitional that adds these empty methods, possibly throwing some sort of error like a provider error?
<jimbaker> alternatively, this can be merged as a group into trunk
<niemeyer> jimbaker: Probably not, if that's not required
<niemeyer> jimbaker: Having just the dummy implementation, and then adding the ec2 one in a follow up sounds ok
<jimbaker> niemeyer, sounds good
<niemeyer> jimbaker: Unless you're detecting  a problem with doing it that I'm still missing?
<jimbaker> niemeyer, the problem is that if you run expose-provision-machines against an ec2 provider, it will fail, due to the lack of implementation
<jimbaker> so i think the cleanest way of addressing this is to just merge these branches as a group
<jimbaker> once they are all approved
<jimbaker> niemeyer, the other thing is that once in trunk, this will force some formula writers and presumably all formula deployers to consider the firewall
<jimbaker> niemeyer, so this will be an important change to communicate on my part :)
<niemeyer> jimbaker: That's doable too
<niemeyer> jimbaker: Yes, that sounds reasonable, on both counts
<niemeyer> jimbaker: We should also discuss the change
<jimbaker> niemeyer, cool
<niemeyer> jimbaker: To see if we have to introduce a migration path or not
<jimbaker> niemeyer, my feeling is that in discussing the migration, it would be best to combine the one line + comments change for the wordpress example formula with the doc changes (mostly from draft to main)
<jimbaker> that would be the branch after the one i'm currently working on
<niemeyer> jimbaker: Hmm.. the example formulas don't really worry me.. what we have to discuss with SpamapS, m_3, adam_g, negronjl, and lynxman is the fact that their formulas will stop working
<jimbaker> niemeyer, after expose-provider-ec2 and expose-migration (the branch just discussed), there's a firewall solution in place for ec2
<niemeyer> Well.. not really stop working per se
<jimbaker> niemeyer, correct
<niemeyer> They'll just not open the firewall
<niemeyer> Which doesn't sound like a big issue, but we have to talk
<niemeyer> jimbaker: Can you please bring up that conversation in the mailing list?
<jimbaker> niemeyer, right, they can always manually open the firewall using their favorite tool
<niemeyer> Yep
<jimbaker> niemeyer, i will do the mailing list discussion
<jimbaker> or at least kick it off ;)
<niemeyer> jimbaker: Thanks
<jimbaker> niemeyer, then after all this dust settles, i can then work on supporting unexpose and expose hooks. but that's more for completeness than what i would imagine is a common use case
 * hazmat goes into review mode
<niemeyer> hazmat?
<negronjl> from my point of view guys, change what needs to be changed in ensemble and let me know what changed.  I'll adapt the formulas to match the work.
<negronjl> in other words...don't worry about my formulas.  Just let me know what changed and I'll adapt them to work.
 * negronjl is out to a meeting
<jimbaker> negronjl, it boils down to: open-port PORT[/PROTOCOL] in a hook script; ensemble expose SERVICE_TO_BE_EXPOSED
<jimbaker> ensemble does the rest of the work
<niemeyer> hazmat: Looking at that can isn't fun
<hazmat> niemeyer, back
<adam_g> when using add-unit, is formula uploaded from locall repository to new unit or is it fetched from bootstrap node such that all units are running the same revision of a formula?
<m_3> jimbaker, niemeyer: +1 on what negronjl said (different answer later when things are in production tho!)
<SpamapS> I wonder if there is some sort of kernel level event service that can be tapped to find out what ports a process is using.
<m_3> adam_g: dunno, never tried to add-unit without upgrading changes first... big issue though
<SpamapS> like inotify but for sockets instead of files
<m_3> SpamapS: doesn't netstat have a mode where it continues watching sockets?
<SpamapS> m_3: --continuous .. that polls tho
<adam_g> m_3: i'm trying to add new units that have modified configuration. is this doable?
<SpamapS> configuration?
<SpamapS> do we have config settings in trunk yet?
<SpamapS> I really want to start mucking with that
<adam_g> i need each unit deployed to have a zone # assigned to it, and i'd like to be able to decide which zone a member joins when i deploy
<m_3> adam_g: I've only been able to do that by adding the config info to an interface and letting another related service manage it
<adam_g> i was thinking, in the meantime, of just hard-coding and updating it each time i deploy a new unit, but that doesn't work if they all share the same revision
<m_3> other service can do 'relation-list' on a relation-joined and then send out new info across the board
<m_3> can peer solve this?
<adam_g> i'd like to decide which zones new units join, not the formula
<m_3> yeah, that's just totally config
<m_3> we'll get a 'deploy --config local.yaml' but I don't know if that'll be an option on add-unit
<adam_g> i think another way would be to deploy the service once for each zone, and add-units to that services/zone
<m_3> zone_manager service, or just curl it from somewhere... bad, worse
<m_3> separate formulas?
<adam_g> deploy --config zone1.yaml swift-storage swift-zone1
<adam_g> deploy --config zone2.yaml swift-storage swift-zone2
<adam_g> add-unit swift-zone1
<m_3> absolutely if that's in there yet
<adam_g> not yet, im just using some hax for now.
<m_3> that's probably the best... I've mocked up some config-get calls already
<m_3> we'll have to see what bcsaller did with add-unit and --config
<bcsaller> I didn't, you configure the service, not the units
<m_3> so in adam_g's previous example, will 'add-unit swift-zone1' pick up the config?
<m_3> (that came from a local yaml?)
<SpamapS> add-unit, no
<SpamapS> oops, was scrolled back
<m_3> I'm confused then... I'll dig trhough the branch to see what's up
<bcsaller> m_3: service options are available available in all units and are the same 
<bcsaller> all units of a given service 
<_mup_> ensemble/expose-provider-ec2 r296 committed by jim.baker@canonical.com
<_mup_> Doc strings, comments, test on EC2 error when accessing instances
<_mup_> ensemble/expose-provider-ec2 r297 committed by jim.baker@canonical.com
<_mup_> Docstrings for test_securitygroup
<_mup_> ensemble/expose-provider-ec2 r298 committed by jim.baker@canonical.com
<_mup_> PEP8
<niemeyer> Extremely sleepy right now..
<niemeyer> Will step out and lay down a bit
 * negronjl is back
<_mup_> ensemble/expose-provider-ec2 r299 committed by jim.baker@canonical.com
<_mup_> Test for EC2 errors in security group ops
<_mup_> ensemble/expose-provider-ec2 r300 committed by jim.baker@canonical.com
<_mup_> PEP8/PyFlakes
<hazmat> adam_g, formulas are uploaded to formula storage (s3) and downloaded from there for each deployed unit
<hazmat> adam_g, environment's don't span ec2 regions 
<hazmat> we'll need to work out some sort of capability for that in the future
<hazmat> SpamapS, service config is almost there, but not yet fully in trunk, last bits are in review
 * SpamapS will continue meditating in the corner until said review is complete
<hazmat> SpamapS, i'm going to be looking at the lxc stuff and your branch next week, had a long chat with niemeyer about it an hr ago.
<SpamapS> hazmat: I'm at your disposal if you have questions
<hazmat> SpamapS, i definitely will, thanks
<jimbaker> ok, i just proposed a branch that pulls all the pieces together for working with the firewall
<jimbaker> in terms of doing this on ec2, that is
<m_3> jimbaker: do you have a rough estimate of when ensemble will default to all ports closed behavior?
<jimbaker> m_3, the branch i just proposed implements exactly that
<jimbaker> m_3, it might be worthwhile checking it out so you can see the impact - but i think it's going to be super minimal
<m_3> ok, I'll keep and eye out
<jimbaker> m_3, re "when", only when it gets through the review process of course ;) - but the only big dependency has been approved by niemeyer and is awaiting a review by hazmat
<jimbaker> (that would be expose-provision-machines)
<m_3> thanks!
 * hazmat digs back into reviews
#ubuntu-ensemble 2011-07-20
<hazmat> g'morning
<niemeyer> Good morning!
<hazmat> niemeyer, g'morning
<niemeyer> hazmat: Hey man
<niemeyer> hazmat: How did reviews go yesterday?
<niemeyer> I'll dive in myself now
<niemeyer> Beautiful queue
<niemeyer> fwereade: So
<niemeyer> fwereade: The problem we have in the schema issue is error reporting
<niemeyer> fwereade: and it happens because all of the cases in the OneOf fail to match
<niemeyer> fwereade: So it just picks the last one randomly
<fwereade> niemeyer: yep
<niemeyer> fwereade: Which is bad
<niemeyer> fwereade: I think we can do something much better, and very simple too
<niemeyer> fwereade: I suggest we introduce a new kind of "case selection"
<fwereade> niemeyer: cool
<niemeyer> fwereade: Which is based on picking an entry from the enclosed dictionary
<fwereade> niemeyer: yep, sounds sensible, I think
<niemeyer> fwereade: Let me come up with an example to make it clear.. just looking at the code
<niemeyer> fwereade: SelectDict("type", {"ec2": subschema, "orchestra": otherschema})
<fwereade> niemeyer: was doing almost exactly the same as you :)
<fwereade> fwereade: well then, I'll roll that in with the boot-only branch and hopefully sumbit shortly
<niemeyer> fwereade: Support for that in schema is a nice independent branch (hint! hint! ;_)
<fwereade> niemeyer: I was attempting to talk to you there :/
<fwereade> niemeyer: ah-ha
<fwereade> niemeyer: sounds like a plan
<fwereade> niemeyer: and the other one just waits on that?
<niemeyer> fwereade: Doesn't have to wait
<niemeyer> fwereade: Merge it into the new branch
<niemeyer> fwereade: I mean, merge the pre-requisite into the follow up
<fwereade> niemeyer: ...and then the followup's diff-with-trunk just gets smaller once the prerequisite is in?
<niemeyer> fwereade: Precisely
<fwereade> niemeyer: cool, so long as nobody ends up spending time reviewing the bits of the followup that come from the prerequisite before the prerequisite gets merged
<fwereade> niemeyer: thanks :)
<niemeyer> fwereade: _If_ the pre-requisite hasn't been merged, you can point out in the merge proposal that there is a pre-requsite branch
<niemeyer> fwereade: There's a special field for it
<fwereade> niemeyer: I saw that, I wasn't clear what actual effect it had
<niemeyer> fwereade: It's informational, and gets the diff correctly
<fwereade> niemeyer: the implication seemed to me to be that prereq *wasn't* merged into followup, but that it wouldn't make sense to merge followup into trunk until prereq was in
<niemeyer> (in Launchpad)
<fwereade> niemeyer: awesome
<niemeyer> Absolutely
<fwereade> niemeyer: thanks again :)
<niemeyer> Merging the follow up into trunk without the pre-req being reviewed would be awkward
<niemeyer> fwereade: No problem
<fwereade> niemeyer: a thought
<fwereade> niemeyer: a SelectDict demands that the values in the initialisation dict actually be dict-like schemas
<niemeyer> fwereade: Yeah
<fwereade> niemeyer: I'm inclined to create a SchemaMakesNoSenseError (or something)
<niemeyer> fwereade: That's why it's called SelectDict (select which dict schema)
<niemeyer> fwereade: That MakesNoSenseError to me as well :-)
<fwereade> niemeyer: but I'm not sure it fits with the existing style, which seems to let you put anything in and just blow up on coerce
<niemeyer> fwereade: Sorry, I don't actually get your original point
<hazmat> fwereade, an explicit validate method might be nice
<niemeyer> fwereade: coerce() is a fine place to validate that they are dictionaries for SelectDict as well?
<niemeyer> fwereade: Not sure if that's your question
<hazmat> hmm is_valid i guess
<fwereade> niemeyer: not quite -- it's that the schema itself can be broken if we SelectDict("name", Int())
<niemeyer> hazmat: Changing the way validation works in schema is an unrelated issue to what fwereade is trying to address
<hazmat> Nicke, not validation, but validity of the schema
<hazmat> er. niemeyer ^
<fwereade> niemeyer: (for example), but I don't think we'd detect that made no sense until we tried to parse an actual schema
<fwereade> niemeyer: just as hazmat says
<niemeyer> fwereade: Tons of things will break if we pass arbitrary garbage in
<fwereade> niemeyer: true :)
<fwereade> niemeyer: which I guess is why I sought guidance before following my initial instinct to validate on schema creation
<niemeyer> fwereade: Yeah, don't worry about it
<fwereade> niemeyer, hazmat: I do still quite like the idea of schema validation but -- if nothing else -- I'll skip it in the interest of keeping the branch small ;)
<niemeyer> fwereade: Schemas are static
<niemeyer> fwereade: If you create a bad schema, testing will catch it
<fwereade> niemeyer: cool
<fwereade> niemeyer: thanks :)
<niemeyer> fwereade: The whole point of schema.py is schema validation
<niemeyer> fwereade: It does that through coerce
<niemeyer> fwereade: Which is elegant,IMO, as it does both things at once
<negronjl> hi guys:  quick question re: open-port/close-port new hook commands.... can they be executed from the install/start/stop scripts ( I am more interested in the install script )?
<fwereade> niemeyer: true :)
<niemeyer> fwereade: Well, I guess validating that the schema validator is valid
<niemeyer> fwereade: SchemaSchema? ;-)
<fwereade> niemeyer: haha
<fwereade> niemeyer: that's what I was trying to get at ;)
<niemeyer> fwereade: That's only a good idea if you also implement SchemaSchemaSchema
 * niemeyer runs
<niemeyer> negronjl: Yes, that must work
<niemeyer> negronjl: If it doesn't, we're very interested on it
<SpamapS> I'd like for schema validation to be more flexible
<fwereade> niemeyer: yeah, I've been meaning to ask, where are the tests for our tests, hmm? :p
<negronjl> niemeyer:  perfect!  I'll start modifying formulas and will provide feedback  :)
<SpamapS> while working on different providers, one can't use the same environments.yaml because the sections are invalid
<SpamapS> it should just ignore these invalid sections
<niemeyer> SpamapS: Not really.. it should validate these different providers
<niemeyer> negronjl: Thanks!
<SpamapS> niemeyer: but lxc and orchestra don't exist in trunk.. so I have to mv environments.yaml to environments.yaml.lxc .. mv environments.yaml.ec2 environments.yaml .. every time I switch
<niemeyer> SpamapS: I see what you mean, yeah, that's painful
<SpamapS> Whereas its really of no concern.. so I could see a warning, but a complete failure seems like coarse error handling.
<niemeyer> SpamapS: This is pretty easy to sort out, though.. we just need a small branch being committed to trunk early on, which adds validation
<SpamapS> What about people who want to write plugins?
<niemeyer> SpamapS: That's a separate problem.. plugins are not supported right now
 * SpamapS intends to change that soon if it pleases the court. ;)
<SpamapS> Have a friend who wants to write a jclouds provider to enable rackspace.
<niemeyer> SpamapS: That's not a plugin.. we can very happily take contributions that implement new providers
<SpamapS> He's been watching dev and isn't really interested in going through the song and dance to get it into Ensemble.. he just wants to *use* it.
<SpamapS> And throw it up on github somewhere
<SpamapS> I'm sure we could then suck it into Ensemble at some point..
<SpamapS> but de-coupling his dev process from ours should create a vibrant ecosystem of plugins
<niemeyer> SpamapS: We certainly want to support plugins at some point, but right now that's not a priority
<niemeyer> SpamapS: Plugins isn't just "you can throw stuff in"
<SpamapS> I'm also interested in maintaining my LXC provider outside of Ensemble if its at all possible.
<niemeyer> SpamapS: We need a stable API that people can trust
<niemeyer> SpamapS: Otherwise next week his plugin is broken
<SpamapS> Yeah, hard to write plugins w/o releases :)
<niemeyer> SpamapS: Heh
<niemeyer> SpamapS: Having a release is not the same as having a stable API plugin
<SpamapS> An API release I mean.
<SpamapS> And releases help w/ that too.. Wordpress plugins work that way.. they simply list the
<SpamapS> releases of wordpress that they work with
<SpamapS> when new releases come out.. people try it, and verify it, and add it to the metadata in their plugin repo
<niemeyer> SpamapS: No, they don't, really..
<niemeyer> SpamapS: Ask the bzr guys
<SpamapS> Are you suggesting wordpress plugins don't work?
<niemeyer> SpamapS: I'm suggesting they actually care about the API for plugins, rather than simply having releases
 * SpamapS has to run.. but will think on that. :)
<RoAkSoAx> fwereade: ping
<fwereade> RoAkSoAx: pong
<RoAkSoAx> fwereade: howdy! How's it going today?
<fwereade> RoAkSoAx: not too bad, once I figured out squid-deb-proxy was the source of my woes :)
<RoAkSoAx> fwereade: hehe yeah!! So anyways, I';m gonna start working on the deploying machine stuff
<RoAkSoAx> fwereade: but before, I wanted to see if you wanted to work on getting the bootstrap validatiosn working first
<fwereade> RoAkSoAx: I'll want to grab you later to figure out a couple of things but I'm focusing on something else atm
<RoAkSoAx> fwereade: or shall we leave that for later
<niemeyer> fwereade: The new changes you pushed into the tweaks branch look awesome!
<niemeyer> So much cleaner overall
<hazmat> SpamapS, cli plugins are easy.. server side plugins need a bit of work
<fwereade> RoAkSoAx: sorry, expand "bootstrap validation"
<fwereade> RoAkSoAx: thanks, all credit due to hazmat :)
<fwereade> er, that should have been "niemeyer: thanks"
<fwereade> :D
<niemeyer> fwereade: This is ready for merging
<RoAkSoAx> fwereade: checking if there's another zoopeeker running and things like that
<niemeyer> fwereade: Yeah, hazmat's idea of putting bootstrap within LaunchMachine itself was great
<niemeyer> I'll have lunch
<niemeyer> Had a meeting with Robbie, but he's not around
<fwereade> RoAkSoAx: ah, cool
<niemeyer> (and I was the one screwing our original timing..)
<fwereade> RoAkSoAx: that sounds great
<fwereade> niemeyer: sweet :D
<RoAkSoAx> fwereade: I mean if we should work on that before I start working on getting machines deployed
<fwereade> RoAkSoAx: ah I see
 * fwereade thinks
<fwereade> RoAkSoAx: yeah, please work on the bootstrap validation -- I think that will be most helpful to me
<fwereade> RoAkSoAx: thanks :)
<RoAkSoAx> fwereade: ok, I'll look into it ;) but wanna test what will happen if I try to deploy first
<RoAkSoAx> fwereade: because, as I can recall, we discussing on setting a global variable to obtain the zookeeper instance
<RoAkSoAx> and stuff like that
<RoAkSoAx> fwereade: so that will probably involved re-doing part of the start machine code. But anyways, let me look at it first
<fwereade> RoAkSoAx: as you like -- they're fundamentally independent, but keeping the actual deployment broken will prevent me from indulging my tendency to grow branches further than I should ;)
<fwereade> RoAkSoAx: I'll just have to maintain my own discipline :p
<RoAkSoAx> fwereade: hehe ok gonna start working on testing some stuff first as I need it before continue with the coding
<jcastro> robbiew: any feedback on those graphs?
<robbiew> jcastro: which ones...only saw the PDF you added
<robbiew> are there more
<jcastro> three graphs in one PDF.
<jcastro> which style should I go with and is the content in each circle right?
<jcastro> (I was thinking the concentric circles)
<robbiew> jcastro: oh, heh
<robbiew> let me look again
<robbiew> jcastro: man...these are nice
<jcastro> I know. :)
<jcastro> total time, 10 minutes.
<robbiew> I kinda like #2
<jcastro> I just need the details you would want in each box, I made it kind of generic
<jcastro> dunno if you want it that generic or not
<jcastro> yeah each view has a nice feel for it
<robbiew> jcastro: so I converted #1 into an odp format
<robbiew> for editing
<robbiew> can also do #2
<robbiew> then folks can adjust as needed
<jcastro> that would be swell, I couldn't figure out the conversion
<RoAkSoAx> fwereade: is everything on tweak-launch-machine already approved into trunk?
<robbiew> jcastro: no worries...I'll sort that while on this call...multitask
<fwereade> RoAkSoAx: just merged it recently
<robbiew> ;)
<RoAkSoAx> fwereade: nice.. did you do any further changes to what's in ensemble-bootstrap branch?
<fwereade> RoAkSoAx: I have another branch which is basically a copy, but uses twisted's xmlrpc instead of xmlrpclib
<fwereade> RoAkSoAx: ...and has a failed translation of the deployment stuff we had so far
<fwereade> I won't get it to a mergeable state today, but it should be ready earlyish tomorrow
<fwereade> RoAkSoAx: and, in fact, I think I'll be proposing a merge of just enough to launch a machine, without expecting it to run or even necessarily complete an install
<RoAkSoAx> fwereade: for cobbler/orchestra stuff?
<fwereade> RoAkSoAx: yep
<RoAkSoAx> fwereade: ok, well I think we should be in sync on this
<RoAkSoAx> as I dont really wanna do stuff
<RoAkSoAx> that is later gonna be changed :)
<fwereade> RoAkSoAx: I can understand that
<fwereade> RoAkSoAx: but my understanding was that niemeyer was asking me to take your stuff, in small pieces, and integrate those individually
<RoAkSoAx> fwereade: I mean, as I understand, you are gonna require to do changes to get the API interaction with twisted xmlrpc instead of xmlrpclib right?
<fwereade> RoAkSoAx: yes, changes, but I think not really serious changes
<fwereade> RoAkSoAx: I still absolutely depend on your stuff to show me how to do the right thing
<fwereade> RoAkSoAx: I just need to put it in a different coat
<fwereade> RoAkSoAx: ...as it were ;)
<RoAkSoAx> fwereade: right, so what I'm syaing is instead of you doing that, why didn't we do that from the beginning :)
<RoAkSoAx> as in get things working with twisted xmlrpc directly
<fwereade> RoAkSoAx: heh
<RoAkSoAx> :)
<fwereade> RoAkSoAx: I think niemeyer's reading of the situation was that your skills were better spent on the actual interaction, while I'd be best placed wrapping it in tests and, er, local flavour
<RoAkSoAx> fwereade: alright, I guess we can continue like that for now then ;)
<fwereade> RoAkSoAx: to be fair, the actual interaction is non-trivial
<jimbaker> trunk is broken: http://paste.ubuntu.com/648385/
<fwereade> jimbaker: curses, odds are that's me... I swear I tested :/
<fwereade> jimbaker: ah, no, I think you have a stale .pyc
<RoAkSoAx> fwereade: fair enough, let's just keep things as is until I finish merging all the stuff then ;)
<jimbaker> fwereade, no worries. just wanted to try out your refactoring against some branches
<jimbaker> fwereade, ahh, thanks for that reminder
<fwereade> phew :)
<jimbaker> let me try again
 * fwereade retracts the "phew" until he hears the tests have *actually* passed ;)
<jimbaker> i should have bzr merge always do a clean code
<jimbaker> the pyc problem only bites frequently enough that i remember it after the fact ;)
<niemeyer> RoAkSoAx: Yeah, the point about xmlrpclib vs. Twisted's xmlrpc is that SpamapS's code was already using xmlrpclib, so it was trivial to move forward that way, and boring to refactor just to learn about the interaction
<niemeyer> RoAkSoAx: fwereade can now easily port things over into Twisted lingo while doing the integration
<jimbaker> fwereade, not looking good: http://paste.ubuntu.com/648389/
<jimbaker> i wonder if the previous error was a pyc problem that was not allowing this bad cascade
<niemeyer> fwereade: Have you run tests during merge?
<niemeyer> fwereade: s/during/before/
<fwereade> niemeyer: yes I did
<fwereade> niemeyer: ...but I can't swear that my *trunk* didn't have its own stale pycs
<niemeyer> fwereade: Doesn't look like import errors now
<fwereade> niemeyer: they look a bit like transient errors I *think* I've seen on fresh code before
<hazmat> hmm.. odd i ran the full suite against it
<jimbaker> niemeyer, fwereade - not certain what's going on. i'm seeing a lack of repeatibility
<RoAkSoAx> niemeyer: fair enough
<fwereade> niemeyer: I am well aware that suites should be green 100% of the time but I felt I should hold off on fixing that sort of thing until I had a clearer idea how everything works together
<jimbaker> never mind, it does come up with -u looping
<niemeyer> fwereade: As long as it's not your branch introducing the issues
<fwereade> niemeyer, jimbaker: ...and it's green for me on a clean trunk
<jimbaker> ./test -u ensemble.providers.ec2.tests.test_bootstrap will eventually fail it seems
<niemeyer> jimbaker: Can you reproduce that on trunk?
<jimbaker> niemeyer, that's against trunk
<niemeyer> jimbaker: Sorry, on trunk before the branch
<jimbaker> niemeyer, yeah, i will do that now
<niemeyer> jimbaker: Thanks
<fwereade> jimbaker: ooh, I didn't know about -u
<fwereade> that's really cool :)
<jimbaker> fwereade, no it's not
<jimbaker> :)
<fwereade> jimbaker: heh :)
<fwereade> jimbaker: I am past 180 successful runs here
<jimbaker> the fact that we need to test with it is a very good reason for golang
<fwereade> jimbaker: sorry, expand please
<jimbaker> fwereade, this is the arg that niemeyer made to convince me why golang was a good direction for this project
<jimbaker> to summarize: deterministic tests are good
<fwereade> jimbaker: oh yes indeed -- I didn't realise go would give us that
<niemeyer> It doesn't come for free.. it's just easier to get there
<fwereade> niemeyer: would you look at http://paste.ubuntu.com/648398/ and tell me if the assertion against the str of the error makes sense to you?
<jimbaker> fwereade, niemeyer - i'm running a test now against trunk, but i can't obviously loop ensemble.providers.ec2.tests.test_bootstrap now, since that's new
<jimbaker> sorry trunk as of r277
<jimbaker> and that just passed fine
<niemeyer> jimbaker: The bootstrap logic isn't changing significantly
<niemeyer> jimbaker: Can you loop on ec2.tests?
<fwereade> niemeyer: SAMPLE_ENV in http://paste.ubuntu.com/648405/
<niemeyer> jimbaker: Also, you have some experience debugging this sort of issue.. would you mind to give fwereade a hand on it?
<jimbaker> niemeyer, yeah, i'm seeing the failure now in r277 too. so test_bootstrap is a refactoring of code in launch, as i now recall
<niemeyer> fwereade: Not really, that doesn't make much sense to me
<niemeyer> fwereade: It should say something like environments.myfirstenv has no type
<jimbaker> niemeyer, i can help out fwereade on this
<niemeyer> jimbaker: THanks a lot
<fwereade> niemeyer: cool, I'll make it do something like that :)
<niemeyer> fwereade: Thanks
<fwereade> niemeyer: also: am I right in thinking that "still running after 5.0 seconds" was code you wrote with me on the sprint?
<niemeyer> fwereade: Huh!?
<fwereade> niemeyer: perhaps not then, I didn't quite follow what you did but it was tangential and I didn't want to crash my stack with a new discussion
<SpamapS> http://spamaps.org/files/mediawiki-plus-slave.svg
<niemeyer> fwereade: That's an error message from a runaway test, in general
<fwereade> niemeyer: ...and a closer look reveals that it can't possibly be
<SpamapS> We need to work on this so it has a more suitable orientation.. way too wide
<fwereade> niemeyer: so I just saw ;)
<niemeyer> fwereade: So not code.. are you seeing this when you run tests?
<fwereade> niemeyer: I have seen similar things cropping up, very rarely, in random tests at random times
<SpamapS> haproxy is providing something to demo-wiki .. so I don't think it should be on the same level as things demo-wiki is consuming.. would be good if providers went above consumers
<fwereade> probably once every couple of days
<SpamapS> err
<niemeyer> SpamapS: That's really dot generating it.. if you have a patch to the formatter, we can easily put it in
<SpamapS> demo-wiki is providing something to haproxy I mean
<jimbaker> fwereade, first thing, need to ensure that we are both getting these random failures
<niemeyer> fwereade: Ok, that's not something to be taken lightly
<niemeyer> fwereade: It's usually a big warning that there is uncontrolled logic running
<niemeyer> fwereade: and hooks into the determinism comments jimbaker made earlier
<fwereade> niemeyer: indeed, but I didn't feel competent to investigate that in my first few days
<niemeyer> fwereade: Again, you don't have to, as long as it's not your branch introducing it
<niemeyer> fwereade: If it's your branch, you can ask for a hand to understand what is going on and someone else can dive in with you
<niemeyer> fwereade: But don't just let it go through unnoticed, or we'll figure it's broken much later, and will be much harder to debug
<fwereade> niemeyer: I'm 90% sure I've seen it elsewhere, but that 10% makes me want to take responsibility for now ;)
<fwereade> niemeyer: besides, I can think of few more effective baptisms by fire :p
<niemeyer> fwereade: You may have seen it elsewhere, but it may be a new issue
<niemeyer> fwereade: If you want to be sure, it's easy
<niemeyer> fwereade: Get the original test in trunk, run it in a loop with -u
<niemeyer> fwereade: If it's already broken, you don't have to stop what you're doing to fix it
<niemeyer> fwereade: If you don't see it breaking in trunk, heads up
<fwereade> niemeyer: assuming there's no pollution between tests ;)
<niemeyer> fwereade: The advice is the same
<fwereade> niemeyer: but, yep, I'll get that running now
<fwereade> niemeyer: and leave it overnight... I need to be off very shortly, I'm afraid
<niemeyer> fwereade: Cool, thanks for today
<fwereade> niemeyer: a pleasure, as always :)
<niemeyer> fwereade: jimbaker will give you a hand on this one
<fwereade> niemeyer, jimbaker: awesome, if I get a chance to talk tonight I will: otherwise I look forward to figuring it out tomorrow
<fwereade> nn all :)
<RoAkSoAx> niemeyer: clear me a doubt... does ensemble need to be installed in the zookeeper?
<niemeyer> RoAkSoAx: You mean the zk machine, right?
<RoAkSoAx> niemeyer: yes, the zk machine
<niemeyer> RoAkSoAx: If so, yes, some of its bits get installed there
<RoAkSoAx> niemeyer: how?
<RoAkSoAx> bzr?
<niemeyer> RoAkSoAx: cloud-init :-)
<niemeyer> RoAkSoAx: bzr/apt, depending on conf
<RoAkSoAx> niemeyer: right but I mean how does cloud-init install those bits that needs for ensemble (such as the provider stuff)
<niemeyer> RoAkSoAx: bzr/apt, depending on conf
<RoAkSoAx> niemeyer: so how can I tell it to install those bits from an specific branch?
<niemeyer> RoAkSoAx: You add an option to your environments.conf file under ~/.ensemble
<niemeyer> RoAkSoAx: Named ...
<niemeyer> RoAkSoAx: ensemble-branch
<niemeyer> RoAkSoAx: next to type: orchestra
<niemeyer> RoAkSoAx: ensemble-branch: lp:~RoAkSoAx/ensemble/...
<RoAkSoAx> niemeyer: cool thanks!!
<niemeyer> RoAkSoAx: You're welcome
<SpamapS> http://spamaps.org/files/mediawiki-circo.svg
<SpamapS> zoom *way* out ;)
<SpamapS> +    dot.set_prog('circo')
<SpamapS> one line patch FTW ;)
<niemeyer> SpamapS: Why is it so spread out>
<niemeyer> ?
<niemeyer> dot graphs usually look a lot nicer than both versions
<niemeyer> It's not clear to me what we're doing wrong there
<SpamapS> not sure, you could definitely shorten some of the lines. :)
<niemeyer> SpamapS: I'm used to reading performance graphs with dozens of edges and nodes
<niemeyer> SpamapS: and it's tight, easier to follow
<niemeyer> SpamapS: We must be using some crazy setting that puts things aside that way
<niemeyer> SpamapS: http://labix.org/tmp/pprof.svg
<SpamapS> ahh, with the circo layout, it specifies a minimum distance of 1.0 .. which is rather far I think
<niemeyer> SpamapS: The concept of using a circumference looks like a good idea for that sort of map indeed
<niemeyer> SpamapS: If we can make that tighter, it will look quite nice
<niemeyer> SpamapS: Can we just disable the minimum distance and let it sort it out by itself?
<SpamapS> niemeyer: and circo naturally pushes the more connected nodes to the center
<niemeyer> SpamapS: Exactly
<SpamapS> niemeyer: thats what I'm looking at doing
<niemeyer> SpamapS: It's an awesome idea
<jimbaker> i'm having problems with the networking card on my laptop
<jimbaker> unfortunately today i also have my car being repaired... anyway, i should have my MBP in use as a spare shortly
<hazmat> jimbaker, i noticed that those provider tests fail badly if your not connected
<hazmat> the mocks for the image query aren't in place
 * hazmat curses his isp for dropping the connection every 20m
<niemeyer> Any bash scripting masters around?
<RoAkSoAx> niemeyer: ideas? http://pastebin.ubuntu.com/648521/
<niemeyer> RoAkSoAx: Are you using the webdav hack?
<niemeyer> RoAkSoAx: Or is it S3?
<RoAkSoAx> niemeyer: webdav
<niemeyer> RoAkSoAx: It's refusing the PUT
<RoAkSoAx> niemeyer: cool, will look into it further
<robbiew> niemeyer: google+?
<dannf> niemeyer: i've been known to write a script or two - what you need?
<niemeyer> dannf: Oh neat
<niemeyer> dannf: A review..
<niemeyer> dannf: Let me send you the link
<niemeyer> dannf: https://code.launchpad.net/~bcsaller/ensemble/bash-completion/+merge/68124
<dannf> niemeyer: so i tend to stick to posix sh, so i'm unlikely to spot issues w/ bashisms like arrays. agreed spacing needs cleanup, though i don't personally have an issue with the [[ ]] && [[ ]] construct or the lack of ';;' in the case - though the latter is internally inconsistent, and a a single switch in a case seems odd
<_mup_> ensemble/security-policy-rules r275 committed by kapil.thangavelu@canonical.com
<_mup_> security policy rules for formula, machine, and relations.
<niemeyer_> dannf: Cool, does it look good to you overall
<niemeyer_> ?
<_mup_> ensemble/security-policy-rules r276 committed by kapil.thangavelu@canonical.com
<_mup_> additional peer relation test for relation security rule.
<_mup_> Bug #813773 was filed: Ensemble should have security rules/acls for every path in zk <Ensemble:In Progress by hazmat> < https://launchpad.net/bugs/813773 >
<dannf> niemeyer_: i'd suggest quoting around $cur $curOpt references to be more liberal wrt user input
<dannf> other than that, yeah - i think its fine
<niemeyer_> dannf: Would you mind to note that in the review, and mark it as Approve?
<dannf> niemeyer_: sure
<niemeyer_> dannf: Under the comment box, that is
<dannf> right
<niemeyer_> dannf: Thanks a lot for your help
<dannf> np
<SpamapS> aha!
<SpamapS> fdp FTW
<SpamapS> http://spamaps.org/files/mediawiki-slave-fdp.svg
<SpamapS> nice *3 tier architecture* :-D
<m_3> cool man
<SpamapS> I think that makes sense as a default output
<SpamapS> Also I think we should actually not call .create when outputting the dot format.. since that adds the positioning..
<SpamapS> we should just write out the 'to_string()'
<_mup_> Bug #813794 was filed: ensemble status fails with "ERROR unhashable type: 'list' " when scope is passed <Ensemble:New> < https://launchpad.net/bugs/813794 >
<hazmat> SpamapS, do you have any ideas how to reproduce that status bug?
<SpamapS> hazmat: pushing a fix now ;)
<hazmat> SpamapS, even better :-)
<SpamapS> w/ a test case
<SpamapS> hazmat: but basically .. 'ensemble status service-name' ..
<hazmat> ugh
<hazmat> that should have had a test
<hazmat> sad if not
<SpamapS>         state = yield status.collect(
<SpamapS>             "wordpress", self.provider, self.client, None)
<SpamapS> it does.. not sure why that is passing
<SpamapS> ahh
<SpamapS> the tests are only *simulating* what they think argparse passes to collect()
<SpamapS> argparse actually passes a list
<SpamapS> the things I do to get a good graph for my presentation.. sheesh
<robbiew> jcastro:  who did the DevOps grows up picture...that's great...T-shirt worthy
<m_3> robbiew: that was me
<m_3> jcastro and I were chatting about message stuff and I remembered the classic "evolution" pic
<jcastro> I've tossed it in the preso folder
<robbiew> yeah...sweet
<m_3> the last guy should have an arnold-head though
 * robbiew might play around with this concept a bit...I'm known for cool t-shirts ;)
<robbiew> jcastro STILL wants a foundations shirt
<robbiew> lol
<m_3> I'll upload the xcf
 * robbiew heads out to cloud camp....
 * jcastro heads off to florida
<jcastro> catch you guys on tuesday!
<_mup_> ensemble/security-policy-rules r277 committed by kapil.thangavelu@canonical.com
<_mup_> services and units security rules
<_mup_> Bug #813831 was filed: errors can be misleading when environments are specified incorrectly <Ensemble:In Progress by fwereade> < https://launchpad.net/bugs/813831 >
<hallyn> smoser: hm, the jaunty ec2 image.  i can't ssh into it. known?
<_mup_> ensemble/security-policy-rules r278 committed by kapil.thangavelu@canonical.com
<_mup_> additional unit and service subtree nodes
<_mup_> ensemble/security-connection r280 committed by kapil.thangavelu@canonical.com
<_mup_> add service and unit rules to the default security rules list
<_mup_> ensemble/security-policy-rules r280 committed by kapil.thangavelu@canonical.com
<_mup_> add service and unit rules to the default security rules list
<_mup_> ensemble/security-connection r281 committed by kapil.thangavelu@canonical.com
<_mup_> security policy connection integration
<RoAkSoAx> niemeyer_: ping?
<_mup_> Bug #813847 was filed: status format "dot" should not actually call 'create' so users can process the graph with dot <Ensemble:In Progress by clint-fewbar> < https://launchpad.net/bugs/813847 >
#ubuntu-ensemble 2011-07-21
<SpamapS> RoAkSoAx: you've been quiet.. I trust things are going well?
<_mup_> ensemble/security-connection r282 committed by kapil.thangavelu@canonical.com
<_mup_> additional tests for security policy connection integration.
<_mup_> Bug #813856 was filed: Security policies need to be integrated into zookeeper connections <Ensemble:In Progress by hazmat> < https://launchpad.net/bugs/813856 >
<_mup_> ensemble/security-connection r283 committed by kapil.thangavelu@canonical.com
<_mup_> mixin the security policy connection integration into the existing ssh zk client.
<adam_g> any reason why relation-changed would fire two or more times in a row? is a relation-changed fired for every relation-set on the other side of the relationship?
<hazmat> adam_g, relation changed gets fired after relation-joined as well
<adam_g> hazmat: in all cases, on both ends? 
<hazmat> adam_g, yes.. it also gets fired when a hook does a relation-set (although only once for the hook, regardless of how many times it does relation-set in the hook)
<adam_g> hmm. ok
<hazmat> we used not to do after joined automatically, but noticed some of the formulas where getting their behavior wrong, as they where waiting on values in relation-changed.. but those values may have already been set by the remote end by the time the relation-joined hook fired locally so the relation-changed wouldn't have ever fired... ideally the hooks should have checked for the values in relation-joined and we could have avoided the spurious relation-
<hazmat> change, but we figured it would be easier for formula authors this way (fire the relation-changed after a relation-join)
<hazmat> re relation-set multiple times within a hook, the sets are all buffered to after the hook is executed so there's a single change event for all the related units.
<smoser> hallyn, did you really just say "jaunty" ?
<RoAkSoAx> SpampS yeah everything is going well...  changes made by william william have improved some things and some others have tp be rewritt
<RoAkSoAx> argh from cell phonr 
<hallyn> smoser: I did indeed
<fwereade> does anyone recall the details of the test failure in which "<ClientEvent changed at blah state:connected>" != "<ClientEvent changed at blah>"
<fwereade> I seem to recall it's something to do with the packages we're using
<hazmat> fwereade, it needs an update of txzookeeper
<hazmat> fwereade, the __repr__ of client event changed
<fwereade> hazmat: I appear to be up to date
<hazmat> fwereade, you should check if that's the version/checkout being used
<fwereade> hazmat: I have my PYTHONPATH set; is there anything else I should do?
<hazmat> fwereade, you can startup python, import txzookeeper, print txzookeeper it will show the on disk location, verify that version is the one you think and is the latest
<hazmat> sometimes if your mix packages, or do setup.py install you'll end up with a different copy on your system
<hazmat> fwereade, you installed txzookeeper from?
<hazmat> ppa or bzr?
<fwereade> hazmat: ppa
<fwereade> should it be in /usr/lib/pymodules/python2.7?
<fwereade> or does that indicate I have the wrong one again somehow?
<hazmat> fwereade, what version of the package do you have?
<fwereade> hazmat: 0.8.0
<fwereade> hazmat: is there somewhere I can check what version I *should* have without asking people?
<hazmat> fwereade, the bzr revision number should be in  the package versionthere.. so as far being able to determine this by yourself, the package version bzr revision should match that of txzookeeper trunk (rev 43)
<fwereade> hazmat: hm, good to know
<fwereade> hazmat: so my life would probably be easier if I just kept a checkout of txzookeeper and installed that whenever it changed?
<fwereade> hazmat: or should I actually be watching trunk for many things?
<hazmat> fwereade, txzookeeper is pretty stable, maybe one change every couple months at this point. i use virtualenvs and run it with python setup.py develop which links the source tree into the python package dir, so updating the source tree is sufficient.
<hazmat> fwereade, can you confirm the exact package version ?
<hazmat> s/exact/full
<fwereade> hazmat: 0.8.0 is what I get from txzookeeper.version
<hazmat> fwereade, apt-cache show txzookeeper will show the package version
 * fwereade raises eyebrow ("no packages found")
<hazmat> fwereade, so that would imply it wasn't installed by ppa
<fwereade> hazmat: hmm
<fwereade> hazmat: I just re-added the repository and updated again
<hazmat> or maybe its python-txzookeeper
<fwereade> hazmat: ooh, that rings a bell
<hazmat> doesn't look like it from here.. https://launchpad.net/~ensemble/+archive/ppa?field.series_filter=natty but it sounds sensible ;-)
<fwereade> hazmat: 0.8.0-0ensemble43~natty1
<hazmat> hmm.. that's the latest
<fwereade> hazmat: which looks right based on what you've been saying
<hazmat> fwereade, which test is it that's failing
<fwereade> hazmat: sorry, bad time for system to fall over
<fwereade> hazmat: let me recover my state...
<hazmat> fwereade, no worries
<fwereade> hazmat: rerunning suite now, but you can find it by grepping for ClientEvent
<fwereade> ensemble.state.tests.test_relation.???.test_watch_user_callback_invocation_delays_node_watch
<fwereade> UnitRelationStateTest
<fwereade> hazmat: trunk works for you then?
<hazmat> fwereade, yeah.. works fine for me
<fwereade> hazmat: bah :)
<fwereade> hazmat: well, not to worry for now, I'll poke around and see if I can figure something out
<fwereade> hazmat: thanks for your help
<hazmat> fwereade, yeah.. i'm not sure what the issue is.. i'm firing up an ensemble instance to verify the unit tests there
<hazmat> fwereade, just verified the failure using txzk package on ec2
<hazmat> hmm
<hazmat> it looks like i'm pushing and pulling from different branches
<hazmat> they have the same content though
<daker> hey i am blocked at preseed conf file for my formula, can anyone explain how it works ?
<_mup_> ensemble/trunk-merge r271 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
<fwereade> hazmat: hey, soory I missed you
<fwereade> hazmat: yeah, I verified that from my perspective it really is the tests that are wrong (all the txzookeepers I can find repr with state: connected), and thought I'd wait to hear back
<hazmat> fwereade, it looks like that test is broken
<hazmat> fwereade, the repr of the client event now has the connection status
<hazmat> the test needs an update to reflect that
<fwereade> hazmat: cool (well, not cool, but at least it's not my own mistake :p)
<fwereade> hazmat: I already have a local fix, shall I create a bug and propose a merge?
<fwereade> hazmat: or should we complain to whoever last merged?
<hazmat> fwereade, its a trivial, typically just paste the diff, get an ack, and commit to trunk
<fwereade> hazmat: http://paste.ubuntu.com/649098/ look ok to you?
<hazmat> hmm.. looks like ben merged the failed test in last rev
<hazmat> fwereade, looks good +1
<fwereade> hazmat: cheers
<fwereade> hazmat: just rerunning the tests for paranoia's sake
<hazmat> it looks like some other test logic got removed in that merge
<hazmat> unrelated to this fix
<RoAkSoAx> fwereade: o/ I have a few issues to show you in just a second
<fwereade> RoAkSoAx: cool, just shout when you need me
<RoAkSoAx> fwereade: will do in just a few secs
<RoAkSoAx> or minutes :)
<RoAkSoAx> fwereade: can I start using trunk to test the orchestra integration instead of the bootstrap branch?
<RoAkSoAx> fwereade: as all your change shave already been merged tright?
<RoAkSoAx> and I think I might use of a couple of bugfixes
<fwereade> RoAkSoAx: trunk has the big refactoring
<fwereade> RoAkSoAx: there's some stuff in lp:~fwereade/ensemble/cobbler-connect-production that might be of use to you
<RoAkSoAx> fwereade: so is it safe to use my work with that branch?
<fwereade> RoAkSoAx: it's a hopefully-useful CobblerConnect implemented with twisted
<fwereade> RoAkSoAx: and it should be just enough to maunch a machine, without passing anything useful to it whatsoever
<RoAkSoAx> fwereade: right now im getting erros that doesn't seem related to the refactoring
<RoAkSoAx> that's why I'm asking
<fwereade> RoAkSoAx: I'm going to give it a final once-over and then propose a merge
<fwereade> RoAkSoAx: like what?
<RoAkSoAx> fwereade: unhasable dict
<RoAkSoAx> error
<RoAkSoAx> when trying to deploy 
<fwereade> RoAkSoAx: sorry, doesn't ring a bell :(
<fwereade> RoAkSoAx: paste a traceback?
<RoAkSoAx> fwereade: yeah I'm gonna reproduce with a clean zookeeper machine
<RoAkSoAx> I'm deploying it atm
<RoAkSoAx> fwereade: now I can't even bootstrap
<RoAkSoAx> http://pastebin.ubuntu.com/649146/
<fwereade> RoAkSoAx: is that on trunk?
<RoAkSoAx> fwereade: bot, ensemble-bootstrap
<fwereade> RoAkSoAx: what it (probably) means is that OrchestraLaunchMachine is returning a string (name?) instead of a deferred list of OrchestraMachines
<Daviey> eek
<RoAkSoAx> fwereade: yeah, and the thing is that from yesterday till today nothing changed and yesterday there was no error :)
<fwereade> RoAkSoAx: wait, I misread
<fwereade> RoAkSoAx: that's the result of get_zookeeper_machines
<RoAkSoAx> fwereade: heh never mind found the error. the same thing that's giving error when launching machines, is the thing that's giving error when bootstrapping if I comment out the first thing
<RoAkSoAx> fwereade: but yeah, I've bumped into stuff that seems to still be ec2 specific
<RoAkSoAx> let me re-check
<fwereade> RoAkSoAx: very probable... I did my best, but it's still all a bit new to me ;) what in particular
<RoAkSoAx> fwereade: yeah we are both learning our way here:)
<daker> fwereade, yo i have just pushed the changes https://code.launchpad.net/~daker/ensemble/small-fix/+merge/68038
<fwereade> daker: cool, thanks, I'll take a look
<fwereade> daker: tiny tweaks :)
<fwereade> RoAkSoAx: I'm just firming up my tests a little, shout if I can help
<daker> fwereade, ah you mean that between the two sub_parser.add_argument there should a break, right ?
<fwereade> daker: sorry, no
<fwereade> daker: I just meant that the original source line was over 80 chars
<daker> and ? the help text should be reduced ?
<fwereade> daker: and so a line break, probably just before the "help=", would make the code nice without needing to touch the text
<fwereade> daker: the text looks fine to me
<fwereade> daker: if the source is still over 80 characters even with that break, you can just do something like the following (give me a moment...)
<daker> fwereade, is it good like that http://paste.ubuntu.com/649169/ ?
<fwereade> daker: that just breaks 80 chars, how about http://paste.ubuntu.com/649177/
<daker> ah ok i understand now!
<hazmat> bcsaller, the merge of config-set-lifecycle looks like it had some testing regressions added
<hazmat> bcsaller, fwereade took care of the failing test, but it looks like (as noted in review) that some other tests also lost some of their logic
<RoAkSoAx> fwereade: what should __check_state from findzookeepers.py return?
<fwereade> RoAkSoAx: thinking
<fwereade> RoAkSoAx: it should *eventually* return a list of OrchestraMachines
<fwereade> RoAkSoAx: (with just one element for now)
<fwereade> RoAkSoAx: ...ok, I put that badly
<fwereade> RoAkSoAx: it can return whatever it wants, but the callback should be able to turn *that* into a list of etc
<fwereade> RoAkSoAx: the callback chain should end up returning a list of machines
<fwereade> RoAkSoAx: am I making sense?
<RoAkSoAx> fwereade: i think so :) yes!
<fwereade> RoAkSoAx: cool :)
<fwereade> RoAkSoAx: as long as we get the expected results/errors from .run(), how you do it is up to you
<fwereade> jimbaker: hi
<fwereade> jimbaker: I haven't managed to induce any errors in EC2BootstrapTest, try as I might
<fwereade> jimbaker: any pointers?
<fwereade> jimbaker: however, I *have* found a different test that will fail pretty soon if run with -u
<fwereade> RoAkSoAx: you know the kickstart file on the cobbler server?
<fwereade> RoAkSoAx: that I had to make a little change to to get past the system install step?
<fwereade> RoAkSoAx: such a file must have a certain set of properties before we can actually deploy an ensemble machine running it, right?
<fwereade> RoAkSoAx: ...but those files are not actually part of ensemble
<bcsaller> hazmat: all the tests on trunk are passing for me. I thought I did it properly, though I did have to clear .pyc files before it worked
<bcsaller> hazmat: maybe you can point me at what you're seeing?
<fwereade> bcsaller: is your txzookeeper up to date?
<daker> fwereade, can you take a look now https://code.launchpad.net/~daker/ensemble/small-fix/+merge/68038 ?
<fwereade> bcsaller: if you search the source for ClientEvent you'll find the test
<bcsaller> fwereade: ahh, the state stuff in the repr
<fwereade> bcsaller: yep
<fwereade> bcsaller: I know there was something funny -- I hit something similar on the sprint -- and niemeyer went off, discussed packaging with someone, and magically fixed it
<fwereade> bcsaller: ...but I don't know the details
<fwereade> bcsaller: regardless: the current version does have "state: connected" in the __str__, so I changed the test to expect it
<bcsaller> fwereade: I updated my txzk, I'll give it another pass
<fwereade> bcsaller: that one's already fixed, I think hazmat was concerned about something else, but I didn't review so I don't know what it is
<jimbaker> fwereade, which test is that?
<hazmat> bzr qdiff -c -1 on trunk will show the fix from fwereade, the fix that remains is for the tests that have lost logic.. as per bzr qdiff -c -2
<fwereade> jimbaker: ensemble.state.tests.test_agent.AgentDomainTest.test_connect_agent
<jimbaker> fwereade, yes that one fails on a regular basis
<fwereade> jimbaker: ah, cool :)
<fwereade> jimbaker: well, not cool :p
<jimbaker> fwereade, just wonder why the bootstrap tests fail for me
<hazmat> most of these bits are missing from test_hook https://pastebin.canonical.com/50069/
<hazmat> bcsaller, ^
<bcsaller> fwereade: yeah, didn't realize there was a txzk version issue there, its working on trunk now. 
<bcsaller> hazmat: those bits don't apply, we inlined and removed that api
<fwereade> jimbaker: how often are you seeing the bootstrap failure?
<hazmat> bcsaller, the flush logic at the top does
<jimbaker> if i run -u, it will eventually fail
<hazmat> fwereade, also the tests now fail if you run them disconnected
<jimbaker> seems like less than 10 iterations
<hazmat> the calls to determine the image id aren't being mocked
<fwereade> jimbaker: for that specific test method?
<jimbaker> fwereade, when i ran just the bootstrap tests, for example
<fwereade> hazmat: I have noticed that, I didn't think it was me, but given that I was hitting that code most recently I guess it was
<fwereade> jimbaker: weord, I've tried with all variants from .providers on up
<fwereade> jimbaker: quick thought
<jimbaker> fwereade, which is generally how i will isolate: run a section of tests w/ -u (test/test suite/script if possible)
<fwereade> jimbaker: is your machine under noticeably low or high load?
<jimbaker> fwereade, low load
<fwereade> jimbaker: bah, so has mine been
<jimbaker> fwereade, however i will retry now, on both my laptop and desktop
<fwereade> hazmat: shall I pick that issue up?
<jimbaker> fwereade, curiously i was seeing it fail in the same place on my macbook pro yesterday
<fwereade> jimbaker: I'm sure it's a real problem, I just wish I could figure out how to repro it
<hazmat> fwereade, if you want/time allows, else just file an issue for it
<hazmat> fwereade, the cobbler work is higher priority
<jimbaker> SpamapS, you had asked earlier about osx, the 2.7ism that apparently snuck in was around argparse
<fwereade> hazmat: true
<fwereade> hazmat: cheers
<fwereade> bbs, diconnecting to get the error list
<jimbaker> SpamapS, this shouldn't impact actualy client usage however - it's just testing against specific error messages, and these got changed in 2.7
<jimbaker> probably need to change the tests
<RoAkSoAx> SpamapS: ping
<jimbaker> the actual mac failures were all about different temp file paths
<jimbaker> and also uninteresting
<_mup_> Bug #814144 was filed: test errors when no network available <Ensemble:New> < https://launchpad.net/bugs/814144 >
<jimbaker> fwereade, given the network dependency in bug 814144 on the same tests mentioned earlier as having looping problems (./test -u ensemble.providers.ec2.tests.test_bootstrap), maybe this theory explains things
<_mup_> Bug #814144: test errors when no network available <Ensemble:New> < https://launchpad.net/bugs/814144 >
<jimbaker> 1. yesterday i was running the tests at a coffee shop w/ a so-so network, and they were eventually failing, maybe because of additional timing
<jimbaker> 2. today, i'm running them at home w/ a good low-latency network, no failures
<fwereade> jimbaker: ah-ha!
<fwereade> jimbaker: very plausible
<fwereade> jimbaker: consider me on it
<jimbaker> fwereade, sounds good
<daker> fwereade, is my MP good now ?
<fwereade> daker: so sorry, I think I lost you in a context switch
<daker> no worries https://code.launchpad.net/~daker/ensemble/small-fix/+merge/68038
<fwereade> daker: would be good to lose the spaces before the colons, sorry I notied them late
<fwereade> if confirmed..., if value.strip()...
<fwereade> but lgtm otherwise
<jimbaker> shang, we met in budapest, right?
<daker> fwereade, i am lost :/
<fwereade> daker: sorry again
<fwereade> if confirmed is False :
<fwereade> should read
<fwereade> if confirmed is False :
<fwereade> if confirmed is False:
 * fwereade winces
<fwereade> ...and there's another one a couple of lines later
<fwereade> it's just for consistency's sake ;)
<jimbaker> i'm pretty certain anything like that would be picked up by the pep8 tool
<jimbaker> i recommend always running pep8/pyflakes (reviewers certainly do)
<jimbaker> also, in this particular case, best to say something like: if confirmed: ...
<jimbaker> no need to say is False, certainly the argparse setting ensures it's True or False, and if even if it didn't, i don't see that it would matter if it were 0 or None
<SpamapS> RoAkSoAx: pong, sup?
<RoAkSoAx> SpamapS: howdy!! I was working on the load_state stuff, and doing that, http://pastebin.ubuntu.com/649258/, returns something like: exceptions.AttributeError: 'str' object has no attribute 'get' 
<RoAkSoAx> any ideas?
<SpamapS> RoAkSoAx: reading now
<hazmat> RoAkSoAx, the state being returned by load_state is a string it appears, does the error happen if you remove the bottom line
<hazmat> RoAkSoAx, you also can't do this interactively since there are deferreds in the mix
<daker> fwereade, sorry again, can you give a final look https://code.launchpad.net/~daker/ensemble/small-fix/+merge/68038 ?
<SpamapS> Its possible the state that was saved wasn't properly serialized
<hazmat> the reactor needs to be started and the result processed either with a callback or with @inlineCallbacks and yield
<RoAkSoAx> hazmat: yeah I'm working on that ;)
<fwereade> daker: looks great to me, my approve stands :)
<hazmat> daker, just added a review comment to it
<daker> hazmat, i hope it's good now ã
 * daker crosse his fingers
<hazmat> daker, looks good, except for the missing test
<fwereade> hazmat: golly, we even have tests for the command line, I didn't even think to demand them...
<fwereade> hazmat: I think the laxity at my last place has damaged my brain :/
<fwereade> hazmat: I used to test *everything*
<fwereade> hazmat: ...I'm so happy to be here :D
<hazmat> fwereade, in this case the cli is our primary interface, if we're not testing it.... then bad things ;-)
<fwereade> hazmat: quite... I just got used to being happy when tests for *anything* showed up ;)
<fwereade> hazmat: ok, that's not fair, but we didn't test anything like as much as we did at the place before
<fwereade> anyway, I think I'm degenerating into friday evening rambling, and I should be off
<fwereade> bye all :)
<robbiew> Friday?
<robbiew> lol
 * robbiew notes that fwereade is on holiday tomorrow
<_mup_> Bug #814194 was filed: Implement support for expose and unexpose hooks <Ensemble:New> < https://launchpad.net/bugs/814194 >
<hazmat> daker, here's a test for it btw, if you could add this to your branch in ensemble/control/tests/test_shutdown the branch should be good.. https://pastebin.canonical.com/50078/
<daker> oh good thanls hazmat 
<hazmat> daker, np, thanks for working on it
<daker> hazmat, would be good if you can put it on paste.u.c, i don't have permission  access that site
<daker> to*
<hazmat> daker, http://pastebin.ubuntu.com/649319/
<daker> hazmat, one last question, how can i run the test ?
<hazmat> daker, the easiest option assuming you have the zookeeper package installed (and the zk server running) is just trial ensemble/control/tests/test_shutdown.py
<daker> ok
<daker> any thoughts http://paste.ubuntu.com/649326/ ?
<daker> hazmat, ^
<hazmat> PYTHONPATH=/home/daker/Projects/small-fix/ trial ensemble/control/tests/test_shutdown.py
<hazmat> daker, ^
<hazmat> trial is from the twisted pkg
<daker> hazmat, is the test includes the -y argument ?
<hazmat> daker, the test i pasted does
<daker> ok hazmat the test succeed 
<hazmat> daker, the error is because the test is being run directly, which A) isn't going to work as the test runner won't be invoked b) relative imports need a fully qualified reference to work correctly when the script is being run this way the current module doesn't really know its package
<hazmat> daker, cool, i tested before i pasted, but its good to have confirmation ;-)
<daker> hazmat, done! https://code.launchpad.net/~daker/ensemble/small-fix/+merge/68038
<niemeyer> Yo all
<jimbaker> niemeyer, hi
<_mup_> ensemble/security-groups r285 committed by kapil.thangavelu@canonical.com
<_mup_> security groups
<_mup_> ensemble/security-groups r286 committed by kapil.thangavelu@canonical.com
<_mup_> group principal membership management and connection attaching.
<_mup_> Bug #814260 was filed: Ensemble security group principals <Ensemble:New for hazmat> < https://launchpad.net/bugs/814260 >
<_mup_> ensemble/security-groups r287 committed by kapil.thangavelu@canonical.com
<_mup_> add get_token to groupprincipal for parity with normal principal
<_mup_> ensemble/trunk-merge r272 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
<_mup_> ensemble/security-otp-principal r289 committed by kapil.thangavelu@canonical.com
<_mup_> otp principal for passing credentials via insecure channels
<_mup_> ensemble/robust-hook-exit r281 committed by jim.baker@canonical.com
<_mup_> Use processExited as the normal end of the process, while making processEnded available for purposes like testing
<_mup_> ensemble/robust-hook-exit r282 committed by jim.baker@canonical.com
<_mup_> PEP8
<_mup_> ensemble/robust-hook-exit r283 committed by jim.baker@canonical.com
<_mup_> Doc string
<_mup_> ensemble/security-otp-principal r290 committed by kapil.thangavelu@canonical.com
<_mup_> minimal principal interface for otp, using the otp consumes it
<_mup_> Bug #814320 was filed: Need OTP for passing credentials to an external process <Ensemble:In Progress by hazmat> < https://launchpad.net/bugs/814320 >
<hazmat> phew.. i think thats all the core security machinery needed
<hazmat> sigh.. "Launchpad encountered an internal error during the following operation: generating the diff for a merge proposal.  It was logged with id OOPS-2028MPJ3.  Sorry for the inconvenience.
<hazmat> "
<hazmat> oh well
#ubuntu-ensemble 2011-07-22
<shang> jimbaker: yes we did :)
<_mup_> ensemble/security-acl r291 committed by kapil.thangavelu@canonical.com
<_mup_> an acl abstraction allowing grants/removals by principal name on nodes
<SpamapS> shhhhhhhhhhhhhhhhhhhhhhh
<jimbaker> SpamapS, yes quiet. heads down all around i guess
<SpamapS> thats a very good thing. :)
<Aram> hi.
<Aram> happen to know where niemeyer or robbiew are? :-).
<jimbaker> Aram, robbiew is on vacation; niemeyer is at a conf
<Aram> thanks.
<adam_g> jimbaker: is there a PPA somewhere that contains your hookProtocol fix?
<jimbaker> adam_g, no, i could push the branch however. still needs some more testing btw
<jimbaker> adam_g, then you can just use environments.yaml to use that branch, using the ensemble-branch option
<adam_g> jimbaker: that sounds doable. where is the branch?
<jimbaker> adam_g, just pushed it to lp:~jimbaker/ensemble/robust-hook-exit
<jimbaker> adam_g, hope that works for you!
<adam_g> jimbaker: great, thanks ill give it a shot later
<jimbaker> adam_g, cool
<adam_g> well, openstack via ensemble is coming together quite nicely: http://paste.ubuntu.com/650191/
<adam_g> you have anyway of turning that into a graph, SpamapS ?
<SpamapS> adam_g: yes!
<SpamapS> adam_g: ensemble status --format=svg --output=file.svg will produce something..
<SpamapS> adam_g: if you use the branch I made yesterday you can try out a few different layouts (fdp seems the most consistently useful)
<adam_g> neat
<SpamapS> adam_g: lp:~clint-fewbar/ensemble/dot-raw-output
<SpamapS> if you use tha tone you can do ensemble status --format=dot | fdp -Tsvg > file.svg 
<SpamapS> adam_g: and btw, w00t.
<RoAkSoAx> any ensemble core dev around?
<SpamapS> hmm.. new analogy for what Ensemble is..
<SpamapS> Ensemble is to config management as Home Depot is to your toolbox at home.
<adam_g> SpamapS: when you get into town?
<SpamapS> adam_g: Thursday morning
<SpamapS> adam_g: I unfortunately have 3 other family obligations M,T, and Wed.. :-/
<SpamapS> adam_g: and Fri night.. so leaving around 6pm Friday. :-P
<fwereade> hi all
<fwereade> I need a quick second opinion about test coverage
<fwereade> you may or may not be aware that there's a problem in trunk, with certain tests that fail or timeout depending on the absence or flakiness of the network
<fwereade> it emerges that there was a commented-out line in the mocking setup, and that reinstating it (with a minor tweak) fixes the issue
<fwereade> however, doing this leaves the coverage only *slightly* improved
<fwereade> please confirm that the appropriate response is to write the necessary battery of additional tests
<fwereade> and that just reinstating the tweaked line is a pitiful and weaksauce dodge of the real issue, that I should be ashamed of myself for considering
<SpamapS> fwereade: tests that work w/o the network are critical.. eventually we want the Ubuntu buildd's to run the test suite .. and they have no network connectivity.
<fwereade> SpamapS: agreed
<fwereade> SpamapS: the question is, do I reinstate weak tests that work without a network or do I write a bunch more that also work without the network, and give us better coverage?
<SpamapS> fwereade: so I'd say that if you have the time to do the additional testing.. it is a noble and high cloud that awaits you in programmer heaven. But if you are pressed for time.. making the tests run w/o the network would get the tests run more often, which may be more important.
<fwereade> SpamapS: cheers, I won't spend too much time on it
<SpamapS> Do we graph test coverage anywhere?
<fwereade> SpamapS: but I'll have a go at least ;)
<fwereade> SpamapS: not that I'm aware of, but that would be interesting
<SpamapS> My last employer had a graph over time of test coverage. Bonuses were attached to significant increases.
<SpamapS> Which did very little for the graph going up.. because programming isn't motivated by the carrot and stick.
<SpamapS> Still, the graph was quite interesting when drilling down into a specific subsystem.. you knew how risky the change was..
<fwereade> SpamapS: that's a shame
<fwereade> SpamapS: even if the carrot itself wouldn't work, perhaps they hoped that being seen to care about such things might be a sort of, er, motivational meta-carrot
<fwereade> SpamapS: ...as it were
<SpamapS> fwereade: well certainly, new stuff got more tests because you didn't want the percentage to go *down*
<bcsaller> there is a `make coverage` target which will generate a table 
<SpamapS> So in theory, we could run back through all the commits and run it for each one, and graph that, and then do that on an ongoing basis?
<SpamapS> bcsaller: did I see service config settings land this week?
<bcsaller> SpamapS: yeah, there is still an approved branch I have to merge for `deploy` but the `set` version of things work. I expect the other will go in today and it will be ready 
#ubuntu-ensemble 2011-07-23
<SpamapS> bcsaller: sweeeeet
<SpamapS> bcsaller: curious, why are the config options not part of the main metadata.yaml ?
<bcsaller> SpamapS: I don't think there is a strong reason, but for example when you look at a packages metadata its different that looking at what it might put in /etc/ to tune it 
<SpamapS> Yeah I think I like it better this way actually
<SpamapS> metadata could get out of hand fast if everything about the formula went into it
<SpamapS> bcsaller: so... this is going to force a move toward a general direction that I think will become common to most formulas..
<SpamapS> bcsaller: it appears that most hooks pretty much just need to update a local store of relation settings and config settings..
<SpamapS> bcsaller: and then run a regeneration script.
<bcsaller> SpamapS: vs data scoped at the service, you're talking about scoped to the unit?
<SpamapS> I'm just thinking out loud mostly.
<SpamapS> I like the way I did it with memcached.. with an include.. basically when memcached relations in mediawiki change, you just regenerate the memcached settings..
<SpamapS> but some services don't have includes..
<SpamapS> So for those, each hook just has to store some data locally ... "facts" if you will, and then regenerate from a template that queries said facts.
<bcsaller> SpamapS: what example are you thinking of that shouldn't apply to all units of a service?
<SpamapS> Err, we are totally on different pages
<SpamapS> I'm talking about how hooks work together to configure the service.
<SpamapS> bcsaller: FYI, adding config options is actually pretty fun.. the formula feels far more useful with them.
 * SpamapS signs off for the weekend.
<bcsaller> SpamapS: thanks, have a good one 
<RoAkSoAx> fwreade ok so.bootstrap and deploy work and needs 2 fixes will email u later im not home
<RoAkSoAx> fwereade  ^^
<hazmat> fwereade, tests that need external factors are functional tests
<hazmat> SpamapS, we're around 98% last i checked, which is a while ago granted
<SpamapS> hrm.. very confusing
<SpamapS> so if I have a default value for an option, it should be considered optional
<SpamapS> but I get
<SpamapS> Invalid options specification: options.wiki-title.validator: required value not found in /home/clint/src/principia/principia-tools/formulas/mediawiki/config.yaml
<SpamapS> That seems rather broken
<_mup_> Bug #814974 was filed: config options need a "file" type <Ensemble:New> < https://launchpad.net/bugs/814974 >
<_mup_> Bug #814977 was filed: ImportError when running ensemble-make-image <Ensemble:New> < https://launchpad.net/bugs/814977 >
<_mup_> Bug #814987 was filed: config-changed hook is retried on 'resolved' even when --retry is not passed <Ensemble:New> < https://launchpad.net/bugs/814987 >
<SpamapS> bcsaller: reminder, the next upload to Ubuntu is Tuesday.. would be great to have the deploy bits in there too. :)
<hazmat> SpamapS, is there a bug filed for the default value should be used for required values?
 * hazmat goes back to setting up his new laptop
