#ubuntu-ensemble 2011-08-22
<_mup_> Bug #830995 was filed: default-instance-type and default-image-id are unhelpful <Ensemble:New> < https://launchpad.net/bugs/830995 >
<_mup_> Bug #830999 was filed: cannot specify machine constraints <Ensemble:New> < https://launchpad.net/bugs/830999 >
<fwereade> I need to learn to pay attention to my status :/
<_mup_> Bug #831058 was filed: cobbler FileStorage assumes no authentication required <Ensemble:New> < https://launchpad.net/bugs/831058 >
<niemeyer> Hello all
<niemeyer> Good week kickoffs for everybody
<fwereade> niemeyer: heyhey!
<fwereade> niemeyer: how's it going?
<niemeyer> fwereade: Hey!
<niemeyer> fwereade: Very nice
<niemeyer> fwereade: Had a good relaxing weekend
<niemeyer> fwereade: You?
<fwereade> niemeyer: pretty good thanks
<fwereade> niemeyer: keeping half an eye on libya, what with it being right in my back yard
<niemeyer> fwereade: :(
<fwereade> niemeyer: it seems to me like a tentative ":)" at the moment, but who knows what's coming :/
<fwereade> niemeyer: and we didn't give back his fighters, so I doubt we have much to fear from the new lot :p
<fwereade> niemeyer: anyway :)
<fwereade> niemeyer: I've been wondering about bug importance and milestones
<niemeyer> fwereade: Yeah.. worth paying some attention to
<niemeyer> fwereade: Ok
<fwereade> niemeyer: there are a few bugs marked "high" but not assigned to eureka -- is that explicit notice that we shouldn't be worrying about them now?
<niemeyer> fwereade: It's a good hint they might either a) be misplaced; or b) be miscategorized
<fwereade> niemeyer: hm, ok :)
<fwereade> niemeyer: I've been a little reluctant to go around assigning priorities and suchlike, what with still being pretty new
<fwereade> niemeyer: which is why 90% of my branches have been fixes to "uncategorized" bugs
<fwereade> niemeyer: should I perhaps be taking a different approach?
<niemeyer> fwereade: I would guess this happens due to our overuse of bugs to mean "task"
<niemeyer> fwereade: You're working on a single High priority problem
<niemeyer> fwereade: Getting physical deployments working
<fwereade> niemeyer: similarly, when I report a bug, I don't usually assign a milestone; I only do that when I'm actually working on something, but I'm suddenly worrying that they'll just fly under the radar
<niemeyer> fwereade: You can tons of uncategorized bugs because they are simply a path to get pieces of that work landed
<niemeyer> s/You can/You get/
<fwereade> niemeyer: makes sense
<niemeyer> fwereade: The bug priority is most interesting when it's an actual issue we want to stash away without an assignee
<niemeyer> fwereade: Because it tells the next guy what he should be paying attention to
<fwereade> niemeyer: I probably kinda mostly see
<fwereade> niemeyer: as an example, there's this idea that we should be able to specify more about what sort of machines are used in certain circumstances
<fwereade> niemeyer: a bug for that didn't leap out at me, so I filed one this morning, as a general "we should do this sometime" note
<niemeyer> fwereade: That sentence gives such a positive feeling :-)
<fwereade> niemeyer: haha :)
<fwereade> that's lp:830999, which has no importance, assignee, or milestone
<fwereade> niemeyer: should it maybe be High, None, Eureka?
<niemeyer> fwereade: Definitely not Eureka
<niemeyer> fwereade: I'd personally qualify that as a Medium, non-Eureka
<niemeyer> fwereade: Eureka has a short time span (as usual ;-)..
<fwereade> niemeyer: I feel vindicated in my reluctance to go around categorizing things myself ;)
<fwereade> niemeyer: I don't think I know when eureka actually is targeted to... LP thinks it was 1/1/2011, which I suspect to be incorrect (:p)
<niemeyer> fwereade: As far as physical deployment goes, Eureka is about finishing the barebones
<niemeyer> fwereade: Making it work reliably
<niemeyer> fwereade: Rather than getting fancy features in
<niemeyer> fwereade: Eureka is 11.10
<fwereade> niemeyer: that's fine by me ;)
<niemeyer> fwereade: End of September, pretty much
<fwereade> niemeyer: ok, makes sense, and we have some sort of special dispensation to release whatever we have done (while most things have already passed the deadline to get in) ...right?
<niemeyer> fwereade: Kind of.. this is actually the real deadline
<niemeyer> fwereade: For everyone
<niemeyer> fwereade: Or rather.. for everyone in similar circumstances
<niemeyer> fwereade: E.g. Ensemble is in universe, it's already there right now, etc
<hazmat> i think the close out of the milestone is scheduled right before the final testing freeze
<hazmat> hmm.. actually its not, i should move it up
<hazmat> the final freeze is sept 22nd
<hazmat> er 29th
<hazmat> 22nd is beta 2
<niemeyer> hazmat: SpamapS requested us to land things by Sep 22nd for testing..
<niemeyer> hazmat: We can close the milestone with 29th
<niemeyer> hazmat: and have that week for testing on our side too
<niemeyer> Still within Eureka
<jcastro> negronjl: ok, I tried a few times over the weekend and just now, still getting this "error: { "$err" : "not master", "code" : 10107 }" every time
<niemeyer> Lunch time..
<jcastro> Ensemble is on devops cafe! http://devopscafe.org/show/2011/8/22/devops-drop-004.html
<robbiew> nice
<robbiew> jcastro: did they talk about it?
<jcastro> yeah
<jcastro> it's a short podcast, worth a listen
<robbiew> jcastro: interesting..thanks
<_mup_> ensemble/expose-cleanup r328 committed by jim.baker@canonical.com
<_mup_> Initial refactoring per review
<_mup_> ensemble/expose-cleanup r329 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<negronjl> jcastro:  I'll look into it.
<jcastro> negronjl: ok I tried some other things and that didn't work either
<jcastro> I have the instance up and running now if you want to check it out
<negronjl> sure.  add my ssh-id ( negronjl ) 
<SpamapS> lol... John Willis
<SpamapS> love his monitoring lightning talk
<niemeyer> jcastro: How is it in the podcast?
<SpamapS> starts around 4:50 in
 * niemeyer downloads
<niemeyer> Mostly curious about what the take was
<niemeyer> and what's the "ironic twist" :)
<SpamapS> Not sure
<SpamapS> still listening
<jcastro> I have like a minute of silence in mine
<SpamapS> yeah
<SpamapS> thats a quote
<SpamapS> turn it up
<SpamapS> its from Fast Times @ Ridgemont High
<SpamapS> whoa I got a silence at 8:49 too
<SpamapS> right at the juicy part ;)
<SpamapS> its like the missing fuzz on the nixon tapes. ;)
<SpamapS> yeah that missing minute is really weird
<niemeyer> Nice.. quite positive overall
<jcastro> clearly he'll have to do another one with more detail. :)
<SpamapS> Maybe one of his listeners will write a formula w/ Chef. ;)
<hazmat> ensemble team, wikimedia, check emails
<hazmat> happening now
<_mup_> ensemble/expose-cleanup r330 committed by jim.baker@canonical.com
<_mup_> Added mock tests with respect to refactoring of remove_security_groups
<RoAkSoAx> fwereade: ping...
<RoAkSoAx> fwereade: has something change whithin the code that generates the cloud-init user data?
<adam_g> w/in 14
 * SpamapS cheers for adam_g's w/inning
<_mup_> ensemble/expose-cleanup r331 committed by jim.baker@canonical.com
<_mup_> PEP8, PyFlakes, call remove_security_groups more robustly
<robbiew> so dumb question:  have we tried testing ensemble against Oneiric images in EC2/OpenStack...or are we still using Natty-based amis
<hazmat> robbiew, we've tried it, its not the default
<SpamapS> bug 816169 has broken a few peoples' formulas..
<_mup_> Bug #816169: When using Ensemble, add-apt-repository no longer functions properly <ensemble (Ubuntu):Confirmed> <python2.7 (Ubuntu):Confirmed> <software-properties (Ubuntu):Confirmed> < https://launchpad.net/bugs/816169 >
<SpamapS> need to tak a look at that pronto
<hazmat> we should probably switch that to oneiric, there's a bug open about using the default package distros instead of the ppa
<hazmat> SpamapS, i thought i merged the fix for that
<hazmat> oh.. nevermind
<robbiew> might be good to switch it once beta is out
<hazmat> SpamapS, there was another bug about pythonpath env variable leaking to executed hooks, and causing issues with package installation. that's no longer the case though
<niemeyer> SpamapS: Do you know why they broke?
<niemeyer> SpamapS: There's a thread with some questioning around that
<niemeyer> adam_g mentioned the interactivity, but Michael Vogt mentioned it actually checks for the tty presence
<niemeyer> Not sure about what's happening there
<adam_g> niemeyer: its actually cloud-init that was screwing with the tty check
<SpamapS> I have not looked at it, just followed the thread.
<adam_g> SpamapS: just fixed that cloud-init bug, will check on the other python problems soon
<adam_g> niemeyer: the interactivity change is unrelated to #816169 that SpamapS mentioned
<_mup_> Bug #816169: When using Ensemble, add-apt-repository no longer functions properly <ensemble (Ubuntu):Confirmed> <python2.7 (Ubuntu):Confirmed> <software-properties (Ubuntu):Confirmed> < https://launchpad.net/bugs/816169 >
<adam_g> SpamapS: looks like that is no longer an issue. seems there was some weirndess in the python environment that has since settled? im going to wait for cloud-init to be updated in images so i can bootstrap+deploy without hacking before confirming 100%
<SpamapS> adam_g: so since we don't really know where the problem was.. maybe just close all the tasks as Invalid once you've confirmed it doesn't exist anymore?
<adam_g> SpamapS: will do
<SpamapS> adam_g: sweet thx!
 * robbiew needs to eat...back in a bit
<RoAkSoAx> fwereade: when you are around pelase give me a shout
<hazmat> adam_g, yeah.. it was probably fixed as part of bug 816264
<_mup_> Bug #816264: PYTHONPATH causing corrupt environment <Ensemble:Fix Released by hazmat> < https://launchpad.net/bugs/816264 >
<adam_g> hazmat: ah, nice
<niemeyer> I'll break for some coffee
<niemeyer> biab
<robbiew> niemeyer: your Google GOpher just arrived at my house
<niemeyer> robbiew: Hah :)
<robbiew> niemeyer: I'll be sure to bring it to UDS
<robbiew> squishable.com....never heard of them before
<robbiew> but apparently they are pretty popular
<robbiew> the GOpher must be "special"
<robbiew> lol
<niemeyer> robbiew: They are, somehow.. :-)
<niemeyer> robbiew: Thanks for feeding it, meanwhile ;)
<robbiew> lol...sure thing
<niemeyer> I thought they were going to send the gopher here as they asked for my home address, but if they actually sent the gopher to the original address I provided, I wonder what they've put in the mail now..
<niemeyer> Today was remarkably unproductive
<_mup_> ensemble/lxc-lib-merge r323 committed by kapil.thangavelu@canonical.com
<_mup_> merge lxc-lib
<_mup_> ensemble/formula-state-with-url r315 committed by kapil.thangavelu@canonical.com
<_mup_> parameterize trunk in ENSEMBLE_TRUNK env variable for makefile
<_mup_> ensemble/formula-state-with-url r316 committed by kapil.thangavelu@canonical.com
<_mup_> link reference doc for s3 auth url
<_mup_> ensemble/formula-state-with-url r317 committed by kapil.thangavelu@canonical.com
<_mup_> formula urls are required, also seed ec2 test provider base class with fake credentials
<_mup_> ensemble/formula-state-with-url r318 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
<_mup_> ensemble/trunk r323 committed by kapil.thangavelu@canonical.com
<_mup_> merge formula-state-with-url [r=niemeyer,bcsaller][f=761053]
<_mup_> Formula states are now created in an enviroment with a url to their
<_mup_> location in provider storage. This is to enable machine agents not to
<_mup_> need access to the provider for deploying formulas.
<_mup_> ensemble/machine-agent-uses-formula-url r316 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
<_mup_> ensemble/trunk r324 committed by kapil.thangavelu@canonical.com
<_mup_> merge machine-agent-uses-formula-url [r=niemeyer,fwereade][f=828189]
<_mup_> Machine agents download formulas using the formula url, instead of 
<_mup_> retrieving them directly from provider storage, when deploying units.
<_mup_> Bug #831688 was filed: Openstack compatibility is needed <Ensemble:New> < https://launchpad.net/bugs/831688 >
<niemeyer> Can someone please have a second review on this: https://code.launchpad.net/~fwereade/ensemble/cobbler-zk-connect-error-messages/+merge/72033?
<_mup_> ensemble/stack-crack r326 committed by kapil.thangavelu@canonical.com
<_mup_> formulas are published with a formulas- prefix instead of a formulas/, per workaround for bug 829880
<_mup_> ensemble/flex-origin r323 committed by jim.baker@canonical.com
<_mup_> Fix test that was depending on hash ordering in dict
#ubuntu-ensemble 2011-08-23
<jimbaker> niemeyer, re point #4 in your review of cobbler-zk-connect-error-message, we have been using RST in docstrings for a while. not consistently of course. but this did address a review point i made in what this branch is stacked on
<jimbaker> of course it would be nice to use sphinx to generate api docs
<niemeyer> jimbaker: The format used by William there is not any format I personally recognize
<niemeyer> jimbaker: It looks very much like the syntax pointed out by hazmat, but not quite
<niemeyer> jimbaker: I'm personally happy with anything, though
<niemeyer> jimbaker: Let's just agree on a standard and move on with it
<jimbaker> niemeyer, sure, i guess if it's slightly tweaked it would conform
<niemeyer> jimbaker: Interestingly, in Go there's actually no special syntax
<niemeyer> jimbaker: Besides a convention to start the sentence with the function name itself + verb..
<jimbaker> niemeyer, i think that's the point of rst docs is to move in that direction
<niemeyer> jimbaker: Works quite well
<jimbaker> niemeyer, albeit w/ a small amount of markup compared to go. but not as rigid as the javadoc/epydoc style
<jimbaker> anyway, as i understand it, we agree on this point, so that's good :)
<niemeyer> jimbaker: Right, I don't mind the touch of RST in Python, even though I wouldn't do it in Go as the standard tools do not use it
<niemeyer> jimbaker: +1
<sidnei> so is there a puppet formula yet? /me hides
<hazmat> sidnei, nope
<hazmat> sidnei, notionally a puppet formula would be a colo located service, the implementation of which is post oneiric
<sidnei> hazmat, ah, right. makes sense.
<_mup_> ensemble/stack-crack r327 committed by kapil.thangavelu@canonical.com
<_mup_> less granular s3 file storage error checking
<h0lyRS> hey guys... is it normal that a bootstrap instance is being created without a key-pair?
<h0lyRS> i think i just found the answer here: http://irclogs.ubuntu.com/2011/08/02/%23ubuntu-ensemble.html
<siwos> hi.
<siwos> anyone using ensemble with openstack?
<siwos> I keep getting this message with openstack
<siwos> ERROR Error Message: AWS authorization header is invalid.  Expected AwsAccessKeyId:signature
<siwos> anyone?
<siwos> please - note the fact I run only computing part of openstack right now
<_mup_> Bug #831805 was filed: wildly inconsistent docstring style <Ensemble:New> < https://launchpad.net/bugs/831805 >
<_mup_> Bug #831883 was filed: weak error message testing <Ensemble:New for fwereade> < https://launchpad.net/bugs/831883 >
<hazmat> siwos, its a known issue atm, i've got some work in-progress for openstack compatibility.
<siwos> hazmat - thanks for clarifying this ;-)
<siwos> I keep an eye on this project ;-)
<hazmat> siwos, if you really want to try it, i can give some instructions, but its using a few branches atm.
<hazmat> siwos, out of curiosity, are your instances routable from the machine your using the ensemble cli on?
<hazmat> in your openstack setup
<siwos> well - I am stress testing openstack compute (over 100 vm-s currently running) - so definitely I am interested in orchestrating them ;-)
<siwos> instances are routable
<siwos> I've gote ensemble running on my api node
<hazmat> siwos, cool, that should work well
<siwos> if you find some time - just tell me what to do ;-)
<hazmat> siwos, i can't really provide any help on it, i'm trying to get into the ppa for next week but here are the instructions if you want to try  it out. http://pastebin.ubuntu.com/673083/
<siwos> great - thanks! I'll try to get this running
<niemeyer> Hello all
<fwereade> niemeyer: morning/afternoon
<niemeyer> fwereade: Hey!
<hazmat> fwereade, g'afternoon
<fwereade> niemeyer, hazmat: hey guys :) how's life?
<hazmat> peachy :-)
<niemeyer> hazmat: How's the LXC work coming?
<hazmat> niemeyer, just getting into it, i had a talk with ben yesterday to catch up, i'm starting in on the lxc-provider today.. wanted to try and get the openstack branch (stack-crack) cleaned up and into the review queue as well
<niemeyer> hazmat: Ok.. I'm a bit concerned because things have been on the quiet side
<hazmat> niemeyer, its going to be tight, but things will heat up this week, and i can give a more accurate assessment of concern.. afaics both openstack and lxc are top level priorities for the release
<niemeyer> hazmat: Sounds good, thanks
<niemeyer> hazmat: They are top priorities, and we have to try to coordinate efforts
<fwereade> niemeyer: speaking of review queues... cobbler-shutdown has apparently not been touched in the week since you approved it; should I be picking on people and hassling them myself, or...?
<niemeyer> hazmat: My original plan was to have jimbaker pushing the OpenStack fixes
<niemeyer> hazmat: But your work on it is hugely appreciated
<niemeyer> hazmat: The problem is just that it means the other side is less covered
<hazmat> niemeyer, yeah.. i know i asked him about it, but then i realized i had told spamas i'd try to get some time on it last week.. and then i got it working ;-)
<niemeyer> fwereade: Yes, that's always a good idea :)
<niemeyer> hazmat: Right.. and we haven't heard much from Jim on the topic, so can't blame you
<niemeyer> jimbaker: Haven't heard much on the topic of testing either.. have to catch up with you
<hazmat> niemeyer, i was inspired by robbie's JFDI for the web stuff
 * hazmat likes JFDI
<hazmat> can i get some more ;-)
<fwereade> yeah, it pleases me that there's a somewhat widely understood acronym for it
<niemeyer> hazmat: Yes, JFD the local development feature now, please :-)
<niemeyer> Meanwhile, I have to JFD the store
<hazmat> niemeyer, so we don't want to have a separate package for local dev?
<niemeyer> hazmat: Not unless necessary
<hazmat> i'll need to apt-get install zk and jdk in bootstrap.. to be sure there present otherwise afaics 
<hazmat> niemeyer, the notion is zk and machine agent are running on the host
<hazmat> offset port for zk
<niemeyer> hazmat: zk is already packaged.. machine agent is already packaged too
<hazmat> niemeyer, ? packaged != installed on the client side
<niemeyer> hazmat: Sorry, I'm still missing some context.. installed on the client side == apt-get install ensemble
<hazmat> niemeyer, right that doesn't install zk locally, just the admin client tools.
<hazmat> niemeyer, for local dev we need zk locally in addition to the admin client tools
<niemeyer> hazmat: apt-get install zookeeper installs zk..
<hazmat> niemeyer, right.. but if i have the client tools, i don't know if "apt-get install zookeeper" equivalent has been done or not
<hazmat> i guess i can detect it.. the question is do i do it for the user or ask them to do it
<niemeyer> hazmat: I'd just try to run it as usual, and report an error if the command isn't found
<hazmat> i'm also going to leave off debootstrap progress reporting, the additional lxc template script layer would need some changes, bcsaller this might make a nice optional thing in the future, if we can get the lxc-template to take it as an optional parameter when debootstraping, we can get back a progress stream from it ala https://github.com/jolicloud/base-installer (see documentation at bottom of readme)
<RoAkSoAx> fwereade: ping
<fwereade> RoAkSoAx: pong
<RoAkSoAx> fwereade: howdy!! how's it going man
<fwereade> RoAkSoAx: pretty good thanks, and you?
<RoAkSoAx> fwereade: pretty good
<fwereade> RoAkSoAx: any joy on the installer?
<RoAkSoAx> fwereade: oh yeah!
<_mup_> Bug #832041 was filed: orchestra FileStorage can't handle unicode paths <Ensemble:New> < https://launchpad.net/bugs/832041 >
<_mup_> Bug #832043 was filed: can't deploy on orchestra <Ensemble:New for fwereade> < https://launchpad.net/bugs/832043 >
<hazmat> bcsaller, re lxc-lib lxc-ls has different output depending on container status
<bcsaller> hazmat: lxc_ls isn't part of the public api anyway
<hazmat> bcsaller, it needs to be
<bcsaller> why? a set of container objects isn't enough? there could easily be containers unrelated to ensemble returned by lxc-ls
<hazmat> bcsaller, to satsify the get_machines interface of the provider
<hazmat> bcsaller, i'm tracking which machines belong to ensemble by recording name on start
<bcsaller> hazmat: so its not ls so much as container status?
<bcsaller> container.running might be what you want
<jimbaker> niemeyer, hi, you wanted to catch up?
<hazmat> bcsaller, that's not in the published version.. and i want to get all the running containers. 
<niemeyer> jimbaker: I wrote a message to the list about it.. can you please answer it there?
<hazmat> bcsaller, did you push the lib again?
<jimbaker> cool, i was just catching up on emails etc
<hazmat> i branched yesterday
<bcsaller> hazmat: I'll push a new version this morning, I expect within the hour
<niemeyer> Hmm.. interesting.. in Go we don't need the schema library to _coerce_ the values.
<niemeyer> We need it for validation purposes only
<niemeyer> Since the yaml/json support will automatically coerce values as necessary for actual use
<niemeyer> I guess I'll just mimic the implementation even then.. it won't really hurt to have coercion working
<niemeyer> But now, I'll have lunch instead
<niemeyer> biab
<niemeyer> wrtp: Hey, btw :)
<wrtp> niemeyer: hiya!
<fwereade> later all, I'll try to pop back on later but no guarantees
<_mup_> Bug #832183 was filed: HA - Ensemble needs to handle a failed bootstrap. <Ensemble:New> < https://launchpad.net/bugs/832183 >
<bcsaller> hazmat: can you pull from the branch again,  the container stuff is more inline with what we talked about
<hazmat> bcsaller, cool, just doing  a review, i'll pull shortly
<bcsaller> hazmat: I'll fetch some coffee then, I'll be around in a few
<jimbaker> adam_g, thanks for putting that bug in so it's not just in our heads
<jimbaker> adam_g, i still think that we can get to that without rewriting the provisioning agent, although that's preferable for the long-term solution
<hazmat> its a fairly simple change
<hazmat> structuring it so that we can use ensemble commands to expand it is what needs work
<jimbaker> hazmat, glad to hear the concurrence so to speak
<jimbaker> hazmat, indeed, that's part of the long-term reimpl
<jimbaker> simply have one active provisioning agent at a time, with 1 or more on standby. just ZK to determine who is active (standard leader stuff). that should be enough
<niemeyer> jimbaker: I don't think that's necessary
<niemeyer> jimbaker: Original plan discussed with hazmat back then allowed for multiple implementations working in parallel
<jimbaker> niemeyer, sure, just my opinion here
<jimbaker> back to functional testing stuff ;)
<niemeyer> jimbaker: Sure, just mine there
<hazmat> jimbaker, yeah just a queue with multiple providers agents active
<hazmat> alternatively a lock structure, but a queue seems more appropriate
<hazmat> wtf,  i think just got hit by an earthquake
<jimbaker> hazmat, in dc?
<hazmat> jimbaker, yeah.. big one.. house rattled pretty heavy
<niemeyer> hazmat: Woah!
<jimbaker> i do remember an earthquake in providence rhode island, but it was remarkable only because it was perceptible
<hazmat> i got some broken glass to cleanup, bbiab
<jimbaker> i got a stomach to take care of, biab
<niemeyer> hazmat: Ouch.. is everyone ok there?
<bcsaller> hazmat: looks like they estimate a 5.8, thats a pretty good sized one
<hazmat> niemeyer, everyone on the block seems okay, not sure about folks on metro, sirens are starting
<niemeyer> hazmat: :-(
<hazmat> bcsaller, i moved out sf to get away from the earthquakes ;-)
<hazmat> all phone lines jammed. evac in progress on pentagon and capital
<niemeyer> hazmat: Ugh.. everyone calling their loved ones
<hazmat> indeed
<bcsaller> people could feel it all the way in NYC
<hazmat> everything seems okay, some structural damage, no major injury scenes per the police scanner
<jimbaker> reminds me of when i was in an earthquake in seattle in 2001 iirc
<SpamapS> http://earthquake.usgs.gov/earthquakes/recenteqsus/Maps/US10/32.42.-85.-75.php
<SpamapS> 5.9, thats a real shaker!
<SpamapS> very shallow
<SpamapS> 1km
<hazmat> fwereade, review in
<fwereade> hazmat: you rock :)
<hazmat> fwereade, you just missed the context that makes that an understatement
 * fwereade scrolls up...
<hazmat> fwereade, http://news.blogs.cnn.com/2011/08/23/quake-hits-near-washington-d-c/?hpt=us_c2
<fwereade> holy shit :)
<fwereade> all ok?
<hazmat> fwereade, so far so good, i've got police scanner on in the background, mostly minor structural damage
<fwereade> hazmat: phew
<fwereade> hazmat: well then, let me restate: you shake violently, and cause minor structural damage, but in a good way
<niemeyer> hazmat: Was just thinking.. a lot of people must have panicked there thinking it was something else
<hazmat> niemeyer, dc rolls like that
<niemeyer> I'm also slightly surprised twitter didn't fall down :-) 
<hazmat> niemeyer, the sad part is landline and cell did. twitter is *the* source of breaking news as much as anything else
<niemeyer> hazmat: Better tell the family to check twitter next time.. :)
<niemeyer> How ironic..
<jimbaker> interesting, we also had an earthquake here in colorado: http://www.cnn.com/2011/US/08/23/colorado.quake/
<niemeyer> jimbaker: Would you mind to respond to the message in the ML?
<jimbaker> niemeyer, i'm writing up a reply
<niemeyer> jimbaker: Thanks
<jimbaker> niemeyer, i haven't had a chance to work on it until now, but as i understand it, it's now my #1 priority. so that's fine
<niemeyer> SpamapS: ping
<SpamapS> niemeyer: pong! good afternoon!
<niemeyer> SpamapS: Hey man!
<adam_g> SpamapS: did pushing formulas to arbitrary branches ever get figured out?
<niemeyer> adam_g: Yeah
<niemeyer> adam_g: Should be working already
<SpamapS> adam_g: I've tested it recently, seems to work fine.
<adam_g> oh, neat
<adam_g> just push to lp:principia/whatever?
<SpamapS> yes but
<SpamapS> make sure its lp:~ensemble-composers/principia/oneiric/whatever/trunk *first*
<SpamapS> If you push your branch there , then only you will own the official branch
<adam_g> hmm. ok
<SpamapS> adam_g: it really doesn't matter too much, as we can always take over the official branch if we need to make a change.. but its better if they're just all owned by ensemble-composers
<adam_g> SpamapS:  still seing a no such source package. does this require a bzr upgrade?
<adam_g> http://paste.ubuntu.com/673370/
<SpamapS> hm
<niemeyer> Hmm
<niemeyer> bcsaller: Wondering about the regex part of our validator 
<niemeyer> (in the context of schema0
<niemeyer> )
<SpamapS> adam_g: maybe
<SpamapS> $ bzr push lp:~ensemble-composers/principia/oneiric/nova-cloud-controller/trunk
<SpamapS> Created new branch.  
<bcsaller> niemeyer: what about it?
<niemeyer> bcsaller: We have a couple of issues.. one short term, one long term
<niemeyer> bcsaller: long term is compatibility.. if we move out of Python, the regex syntax will change
<bcsaller> you want something that uses a regex to validate rather than validates something is a regex
<m_3> adam_g: I've got notes for how to create a dummy source package that gets around this... lemme dig them up
<bcsaller> ahh, right, you're porting to go... 
<adam_g> SpamapS: hmm
<SpamapS> m_3: shouldn't be necessary anymore tho
<niemeyer> bcsaller: The other issue, which would affect us even in the short term, is that a Python regex is a wonderful way to DoS a system :)
<niemeyer> bcsaller: Which means even if it was Python across the board, I can't validate them on the server
<bcsaller> wonderful how? its used to validate the config stuff before its sent over the network now. I mean sure it could be, but its like saying an install script could fork-bomb
<bcsaller> right now its only used in the environment and config parsing I think
<niemeyer> bcsaller: Hmm.. true.. I guess as long as we don't apply the regex, that'd be fine
<SpamapS> is pcre available in python? Thats pretty much the lingua franca for regexes.. and usually the faster implementation
<niemeyer> bcsaller: It'll be an issue for GUI systems, though
<niemeyer> bcsaller: e.g. any kind of hosted management interface
<niemeyer> bcsaller: So it's not exactly like a fork-bombing install script
<niemeyer> bcsaller: The regex is run in the management interface
<niemeyer> bcsaller: Isn't it?
<bcsaller> in that case, no, I agree, but it something we can work around
<bcsaller> niemeyer: do golang use re2?
<niemeyer> bcsaller: Kind of..
<niemeyer> bcsaller: re2 was written by the authors of golang, conveniently..
<niemeyer> bcsaller: It's being ported to Go
<SpamapS> adam_g: what version of bzr do you have?
<m_3> adam_g: http://pastebin.com/aZG0NQ8n they're rough and shouldn't be needed anymore...
<niemeyer> bcsaller: I think we should restrict the syntax accepted to a more common ground regex, for the moment
<bcsaller> niemeyer: I don't know that there is a single common regex def that we could validate in JS in a GUI, but the validator could run in a time bound thread if you're really worried about it 
<niemeyer> bcsaller: and then open up as we understand how the future looks like
<bcsaller> its a really strange attack vector as you need admin perms to use it 
<niemeyer> bcsaller: It's a trivial DoS.. I can't be not-worried :)
<bcsaller> heh, I managed :)
<SpamapS> m_3: I just tested this though, and you shouldn't need to do a fake package anymore.. the fix landed.
<niemeyer> bcsaller: Sure, give the address of your web site accepting Python regexes please
<niemeyer> bcsaller: ;)
<bcsaller> seriously, rather than parsing around and re-writing regex for a UI we don't have yet I think we can just keep a bug report
<niemeyer> bcsaller: I'll make you more worried. :)
<bcsaller> niemeyer: again, if the admin wants to hose the system its done
<niemeyer> bcsaller: Hose _our_ system
<niemeyer> bcsaller: Any _hosted_ management interface could be attacked
<bcsaller> I assumed any web-ui to an ensemble system run in the cloud on their nodes... but I guess not
<m_3> SpamapS: cool
<niemeyer> bcsaller: Not necessarily, no
<SpamapS> Why don't we just not compile other peoples' regexs?
<niemeyer> SpamapS: Because we have a feature right now that enables its use.. even though your statement makes more comfortable in the sense we can remove it and you'd not notice ;-)
<niemeyer> makes me
<SpamapS> I would have always assumed that those regexes are only used on the client.
<bcsaller> SpamapS: currently, yes
<SpamapS> they fall into the same category as maintainer scripts really.. you shouldn't use them if they're not from a trusted source.
<adam_g> SpamapS: needed to upgrade bzr, works now
<SpamapS> adam_g: well that seems a little.. odd.
<niemeyer> SpamapS: That's not the case.. regexes are used for validation of settings
<niemeyer> SpamapS: any management interface, including hosted, would make use of it
<niemeyer> SpamapS, bcsaller: My proposal, though, is not to remove them entirely, but to use a subset we know there are alternative implementations that do not suffer from the problem
<SpamapS> Yeah a subset would probably be best.
<niemeyer> It's also a sensible thing to do in general..
<niemeyer> Otherwise we're unconsciously binding the _metadata_ of the whole project to a particular language
<SpamapS> The most complicated things I'd expect to see are things like   [\d,]+
<niemeyer> SpamapS: Exactly..  this should be [0-9,]+ or similar..
<niemeyer> Which is compatible with most extended POSIX engines
<niemeyer> Including grep
<bcsaller> niemeyer: supposedly re2 guarantees O(length_of_regex) runtime and configurable memory-use limit
<niemeyer> bcsaller: Yeah, it does
<niemeyer> bcsaller: O(len of string), actually
<niemeyer> bcsaller: But as long as it's O(n), we're goofd
<niemeyer> LOL.. goofd is an awesome typo
<bcsaller> that could be good or bad :)
<bcsaller> so maybe including that makes having to filter it a non-issue
<niemeyer> bcsaller: It does, but I'm a little reluctant still because we'd be binding it to a particular implementation
<niemeyer> bcsaller: My suggestion is that we restrict the regex to a common subset, and if people start claiming for more we can gradually introduce, or even go full blown re2/similar if really necessary
<bcsaller> this seems like a very easy place to have an opinion 
<niemeyer> bcsaller: You're right.. let's filter.
<bcsaller> filtering it down using a regex? ;)
<niemeyer> bcsaller: http://commandcenter.blogspot.com/2011/08/regular-expressions-in-lexing-and.html
<hazmat> a ui can do sensible length filtering on an input
<hazmat> are there are circular exponential cases in under 1k?
<hazmat> for a dos
<niemeyer> hazmat: Oh yeah
<niemeyer> hazmat: Either way, it doesn't really matter.. we shouldn't be binding the _metadata_ for the whole project to Python, even if the implementation is in Python.
<hazmat> oh.. its a formula specified regex
<niemeyer> hazmat: Right
<hazmat> i suggest we drop it then,  unless there's a feature request for it, there's some missing coverage in that implementation (floats, missing type error handling) as is
<niemeyer> hazmat: I find the feature interesting.. it's quite handy to do validation upfront rather than reporting errors on execution
<niemeyer> hazmat: When that's easily doable
<niemeyer> Is Python able to do POSIX regex easily?
 * niemeyer can't remember
<hazmat> niemeyer, evaluate of an arbitrary regex against unknown input?
<niemeyer> Doesn't look like so.. sucks
<niemeyer> hazmat: Hm?
<niemeyer> hazmat: Sorry, I'm not sure about what the question is?
<hazmat> niemeyer, you mean validate an arbitrary regex against unknown input.. switching the regex language gives more options, s/pcre/posix.. we could also use re2 in python
<niemeyer> hazmat: Yeah, posix (or extended posix, more likely) would be good
<niemeyer> hazmat: I mean validate the user input using a regex
<niemeyer> hazmat: Exactly like bcsaller implemented
<niemeyer> hazmat: I think feature is a good idea.. the problem is just the complete lack of portability right now
<hazmat> niemeyer, there's also fnmatch.py (stdlib).. translated shell regex
<niemeyer> Python's re is unlike anything else
<hazmat> niemeyer, its closest to pcre afaik
<niemeyer> hazmat: That sounds too poor.. can't even differentiate digits etc
<_mup_> Bug #832384 was filed: ensemble.state.tests.test_security.PrincipalTests.test_activate sometimes hangs <Ensemble:New> < https://launchpad.net/bugs/832384 >
<niemeyer> Ok, most of the schema stuff is in place.. just need to port some of the more complex types now.. (List and Dicts)
<jimbaker> bcsaller, i'm sitting w/ m_3 here in boulder and he's wondering about the status of config-defaults. the branch is marked as being approved, but it's not under the needs release column in the eureka kanban
<jimbaker> i think that's because the bug is still "new", not some other state
<jimbaker> (or at least that's one possibility)
<hazmat> jimbaker, yeah. i needs to be labeled in-progress at least
<hazmat> surprised it even showed up in the review queue without that label
<m_3> hazmat: thanks... I'll update the bug
#ubuntu-ensemble 2011-08-24
<jimbaker> m_3, is the corresponding branch actually merged in yet? it's not marked as being so in bug 828152
<_mup_> Bug #828152: default formula config values not available to hooks <Ensemble:Fix Released by bcsaller> < https://launchpad.net/bugs/828152 >
<jimbaker> bcsaller, ^^^
<bcsaller> jimbaker: yeah, missed the first comment I see
<jimbaker> bcsaller, thanks for merging it in
<jimbaker> m_3, enjoy!
<bcsaller> jimbaker, m_3: sorry, forgot to go back to it. That was important as far as the feature is concerned too
<jimbaker> bcsaller, np, just unfortunate it had been hiding in plain sight in the kanban
<bcsaller> jimbaker: I think what happened was I committed with --fixes (trying that out) and it got the status out of sync
<jimbaker> bcsaller, you mean --fixes against something besides trunk?
<jimbaker> (i use --fixes all the time)
<jimbaker> looking at the log not too closely, apparently i'm the only one who does so ;)
<jimbaker> or at least consistently. i did see usage by hazmat at r298
<jimbaker> bcsaller, i don't --fixes actually changes anything in lp however. i always have to go and still mark it as "fix released"
<bcsaller> I though it set it to 'fix committed', not sure
<jimbaker> again, that's something one would set manually as i understand it
<jimbaker> but at least the fix is now in!
<hazmat> jimbaker, i tried it but it had an effect
<hazmat> odd one that is
<hazmat> it would link trunk to the bug report
<jimbaker> hazmat, i believe it simply and only adds the metadata to the log 
<jimbaker> so it's redundant with respect to [f=XXX], but possibly useful once we always use it
<hazmat> jimbaker, i've seen it happen more than once
<hazmat> jimbaker, take a look at one of your merge proposals https://bugs.launchpad.net/ensemble/+bug/825398
<_mup_> Bug #825398: Store Ensemble version number in topology node and verify ops <Ensemble:Fix Released by jimbaker> < https://launchpad.net/bugs/825398 >
<hazmat> see how's it linked to trunk
<hazmat> and linked to it on the kanban
<hazmat> i find that an anti-feature
<hazmat> i can see its use when we use fix-committed and have actual releases though
<hazmat> but for our current usage modes, its more interesting to me, to see the merge proposal branch linked
<hazmat> so i stopped using --fixes
<jimbaker> hazmat, ok, so it linked trunk into the bug, changing the kanban to report lp:ensemble for the fixing branch?
<hazmat> jimbaker, its the metadata gets processed by launchpad
<jimbaker> yes, that would be anti-feature
<hazmat> once the commit that has --fixes lands in trunk, it updates the branch to link to the bug
<jimbaker> so possibly more useful in a more complex scenario because we can see where the fix is
<hazmat> jimbaker, its a different use case it solves, we typically want to see which branch the bug fix/feature impl landed in, and the merge proposal discussion
<hazmat> the --fixes use case is to show someone looking at the bug the current location of the fix
<hazmat> we already have fix committed bug status which would denote that its on trunk, and fixed release to show  its been released..  on bug status
<jimbaker> hazmat, yes, it makes sense now
<hazmat> jimbaker, i'd suggest we avoid using --fixes, it doesn't match our use case
<jimbaker> it would be sort of nice if the kanban showed the originating branch
<jimbaker> but in the meantime, we can just stop using --fixes
<hazmat> bcsaller, not sure why i was so dense about cloud init, looking at the updates its very clear why not.
<bcsaller> hazmat: so you did get a chance to see it, let me know if you need changes. In the meantime I am improving the script
<hazmat> i suppose either would have worked.. but this seems cleaner
<bcsaller> hazmat: also it currently doesn't arrange for the agent to run but I expect that will be triggered again by data in /etc
<hazmat> bcsaller, looks good, a couple over 80 lines
<hazmat> bcsaller, what's /etc/ensemble supposed to be again? 
<bcsaller> hazmat: its the variables that get sourced by the script to control its behavior. Cloud init can write stuff there as well and then resuse the same scripts
<bcsaller> maybe... I think, we'll see, thats what I'm going for anyway
<hazmat> make sense though, we can drop some useful immutable conf there like agent ids
<hazmat> not clear if we can source the file in an upstart job though
<hazmat> ah. the script section is an entire block
<hazmat> sounds good
<_mup_> ensemble/lxc-lib-merge r324 committed by kapil.thangavelu@canonical.com
<_mup_> local provider skeleton and utils for setting up network and zk
<_mup_> ensemble/local-ubuntu-provider r324 committed by kapil.thangavelu@canonical.com
<_mup_> local provider skeleton and utils for setting up network and zk
<_mup_> ensemble/lxc-lib-merge r324 committed by kapil.thangavelu@canonical.com
<_mup_> merge lxc-lib
<niemeyer> Man.. long day
<niemeyer> Time to sit in the couch and not move for a while
<niemeyer> Have good EOD everyone
<hazmat> niemeyer, bcsaller do we want to support multiple lxc environments simultatenously for oneiric?
<hazmat> niemeyer, cheers
<jimbaker> niemeyer, take care
<hazmat> i was thinking it would be better to do it with chroots, but i'm trying to keep support for it, but if we wanted to do it, it means namespacing things with the env name, which means the env name in the configuration.yaml becomes immutable after use
<bcsaller> hazmat: like many different network bridges at the same time and all that... it shouldn't be that much worse I think, but it shouldn't be that much needed (I hope)
<hazmat> bcsaller, indeed, although its not absolutely clear we really need multiple bridges, i structured it that way for now
<bcsaller> and the interface to that I gave you seemed ok?
<adam_g> cool. ensemble on oneiric should be bootstrapable again. hail smoser.
<smoser> yeah, with 20110822.2
<smoser> adam_g, is anyone testing/using canonistack?
<adam_g> smoser: testing what? ensemble? cloud-init?
<adam_g> i haven't touched canonistack, honestly, but i believe hazmat had gotten it working
<adam_g> (ensemble bootstrap)
<hazmat> smoser, i've done a bootstrap and deploy on canonistack
<hazmat> smoser, it needs some branches though
<hazmat> till its landed on trunk, instructions for the brave.. for ensemble+openstack.. http://pastebin.ubuntu.com/673083/
<RoAkSoAx> fwereade: howdy!!
<fwereade> RoAkSoAx: heyhey
<fwereade> RoAkSoAx: how's it going?
<hazmat> g'morning folks
<niemeyer> Hey folks
<niemeyer> How's life?
<niemeyer> hazmat: Shaken up?
 * niemeyer hides
<hazmat> niemeyer, no stirred not shaken ;-)
<hazmat> there was a funny picture on twitter.. 'damage from the earthquake' and showed a some plastic chairs on a lawn, with one fallen down.
<RoAkSoAx> fwereade: good good
<RoAkSoAx> fwereade: did you update the cobbler-complete branch with the latest fixes?
<fwereade> RoAkSoAx: let me do that for you now
<fwereade> RoAkSoAx: blech, conflicts, might take a few minutes
<niemeyer> hazmat: I was quite surprised with the lack major damage with a 5.8
<niemeyer> hazmat: That region doesn't sit on top of a failure, does it?
<hazmat> niemeyer, no  its actually a very inert area, its in the middle of a plate, not an edge
<RoAkSoAx> fwereade: hehe ok no worries
<niemeyer> hazmat: Creepy :(
<hazmat> niemeyer, so i'm trying to figure out if we want to support multiple environments running simulatenously in the local dev story
<hazmat> and if so should they be isolated (separate network bridge per env) or be able to talk to each other.. at this point i'm thinking yes to the former, and no to the latter. ideally we'd be using a single zk with a chroot, but for now it could work with multiple zks
<hazmat> the real issue is that it would end up codifying the env name into persistent datastructures (unit dirs for example) that would mean the environment name in the ensemble yaml config would become immutable
<niemeyer> hazmat: +1 on yes to the former and no to the latter
<hazmat> niemeyer, so single network bridge for all local envs?
<niemeyer> hazmat: Uh.. that's what you said above :)
<hazmat> i can't think of any good reason to restrict it for the local case ;-)
<hazmat> niemeyer, indeed, but i added an 'or' clause which made the whole thing ambigious
<hazmat> to a simple yes/no
<niemeyer> hazmat: Ok.. I misunderstood then
<niemeyer> hazmat: I think they should be isolated.. there's no good reason to make them communicate with each other, and isolating at this point is just organizational
 * robbiew is getting loads of DevOpsolution shirt requests
<jimbaker> fwereade, thanks for the headsup on the refactoring conflicts
<_mup_> Bug #833064 was filed: orchestra has no firewalls <Ensemble:New> < https://launchpad.net/bugs/833064 >
<niemeyer> fwereade: Btw, I know the review queue is a bit on the upper side right now.. I'll take care of it tomorrow, if that's ok
<fwereade> niemeyer: no worries
<niemeyer> fwereade: I'm trying to put some focused work on the store, or we won't have it
<fwereade> niemeyer: there are at least a couple of little ones, and it'll be a little while before it grows again
<_mup_> Bug #833113 was filed: test should support a JUnit XML reporter  <Ensemble:New> < https://launchpad.net/bugs/833113 >
<hazmat> niemeyer, re the multi-environment local support, so immutability of env names in config.yaml is a consequence, which i'm not sure about.. i'd rather avoid it, but i don't see any good solutions outside of some additional mapping/identifier for environments.. the problem arises in networks/disk structures referencing the environment name as part of qualified unit name resolution.. multi-env on one machine breaks the unit id is unique which we curren
<hazmat> tly use for unit state on disk.
<_mup_> Bug #833119 was filed: Deploy Jenkins with support for EC2 functional testing <Ensemble:New for jimbaker> < https://launchpad.net/bugs/833119 >
<niemeyer> jimbaker: My message in the list is still unanswered.. that's really not how a distributed team works.
<jimbaker> niemeyer, as i mentioned yesterday, i started working on this yesterday. please expect me to report on my findings shortly
<niemeyer> jimbaker: That's not how a distributed team works..
<jimbaker> niemeyer, thanks for the clarification
<niemeyer> jimbaker: Email is for communication
<niemeyer> jimbaker: I've made a simple question to open a conversation
<niemeyer> jimbaker: I didn't expect a whitepaper back 
<jimbaker> niemeyer, understood
<niemeyer> jimbaker: XML report for Jenkins seems overblown.. let's start simple.  1 or 0 exit code is failure/success.. dump text output, and make available.
<niemeyer> hazmat: Hmm
<hazmat> bcsaller, afaics we're just using the standard lxc-ubuntu template now with lxc, and we don't need any upstream changes 
<hazmat> is that correct?
<niemeyer> hazmat: It sounds fine to namespace according to env name
<niemeyer> hazmat: Perhaps user name (or id) + env name, though
<jimbaker> niemeyer, the xml output is quite simple (as seen in XmlTestRunner), and it has the advantage that this is what CI systems in general are looking for.  the only issue here is enabling an appropriate plugin reporter that integrates with twisted trial
<hazmat> niemeyer, i just worry about new users, who change an env name, which works fine for ec2, orchestra, but not local-lxc
<niemeyer> jimbaker: I'll take your word for it.. I'd like to see this working this week still.
<hazmat> niemeyer, good point re + user
<niemeyer> jimbaker: Not in a month
<jimbaker> niemeyer, cool, i understand the JDFI mandate
<niemeyer> jimbaker: Not a JFDI, not a mandate.. we have a simple problem, requiring a simple solution
<niemeyer> jimbaker: If you do it next month, we don't need it anymore
<jimbaker> niemeyer, understood
<bcsaller> hazmat: don't need, no, could still use, but its not a blocker
<hazmat> bcsaller, awesome
<niemeyer> Lunch time, biab
<hazmat> fwereade, the cloud-init-class-replace branch mp depends on cloud-init-class-add  branch but the latter is not in review?
<hazmat> and without a merge proposal
<fwereade> hazmat: explained in the -replace mp
<hazmat> ah
<fwereade> hazmat: I'm probably Doing It Wrong, but the -add is just to make life easier for the reviewers so you get a clear break between two reaosnably self-contained diffs
<hazmat> fwereade, indeed the merge proposal explains it well, thanks
<fwereade> hazmat: cool, cheers
<fwereade> hazmat, btw, thank you very much for all the reviewing you've been doing
<fwereade> and, everyone, see you tomorrow :)
<hazmat> fwereade, np, the others where small and easy, have a good one
<hazmat> bcsaller, what do you think of using link local addresses for connecting units to zk?
<hazmat> localhost is shadowed, and ip addresses otherwise are subject to drift
<hazmat> hmm
<bcsaller> hazmat: I'd have to experiment to guess whats best there
<hazmat> bcsaller, yeah.. i've been playing around, the question is having an immutable network reference, and ip addresses are subject to change and recycling
<bcsaller> hazmat: it seems like dns would work for this though
<hazmat> just not sure if link local addresses are stable and always present
<hazmat> bcsaller, i
<hazmat> hmm
<bcsaller> it also doesn't seem like this is a problem we wouldn't have in other environments 
<bcsaller> or rather if this is an issue we already have it, no?
<hazmat> bcsaller, dns works well from the host to the container, most of our communication is in reverse, how does the host past a network reference the container that's resolvable from the container
<hazmat> ^to the
<hazmat> interesting
<hazmat> bcsaller, dns resolve on the host name from the container returns 127.0.1.1
<m_3> libvirt creates a virbr0, assigns private (not link-local) addresses, runs dnsmasq for dhcp, and NATs through the host via iptables
<hazmat> bcsaller, that works well
<m_3> that dnsmasq might resolve names... dunno
<hazmat> m_3, that works well for host resolving containers, in this case the host (runnning zk) wants to pass an address to the contains they can connect to zk on
<m_3> right... the host is addressable(sp?) from the guests via that virbr0 interface (usually 192.168.122.1)
<hazmat> m_3, at what address from the container ?
<hazmat> m_3, it looks like 127.0.1.1 works.. although in truth that might be an effect from a desktop installation
<m_3> guests are dhcp-assigned via dnsmasq (running bound to the virbr0 interface of the host on 192.168.122.0/24)
<m_3> that's the 'default' setup
<m_3> dnsmasq is the crucial key for the libvirt networking
<m_3> (with iptables if you need external bridging... not sure if you do)
<m_3> dnsmasq is more moving parts, I understand... but don't know of a lighter way to do what you're trying to do
<_mup_> Bug #833248 was filed: Deploying formulas no longer works <Ensemble:New> < https://launchpad.net/bugs/833248 >
<hazmat> m_3, i understand the bridge and the ip addresses, and resolving names onto the bridge, as well as what dnsmasq provides for outside world connectivity, but the question is specifically addressing the host machine from the container.. i think i've got a good solution just need to some verification, else i'll use hostname resolution
<hazmat> ^to do
<m_3> cool, gotcha
<m_3> that'd be nice and lightweight
<hazmat> i'm concerned that 127.0.1.1 is created as a consequence of installing a desktop package
<hazmat> per http://qref.sourceforge.net/quick/ch-gateway.en.html  section 10.4
<niemeyer> hazmat: Can you please have a look at #833248?
<_mup_> Bug #833248: Deploying formulas no longer works <Ensemble:New> < https://launchpad.net/bugs/833248 >
<hazmat> niemeyer, woah.. yeah... i'm on it
<niemeyer> hazmat: Thanks!
<niemeyer> adam_g: Will be working again in a moment.. sorry about that
<adam_g> niemeyer: thanx & np
<hazmat> it looks like the s3 signature is wrong
 * hazmat can't wait for functional tests
<SpamapS> https://ensemble.ubuntu.com/kanban/ .. no eureka?
<hazmat> hmm.. the aws documentatoin is wrong
<hazmat> someone is wrong on the internet ;-)
<hazmat> niemeyer, i can revert back the machine-agent-uses-formula-url branch merge for now and trigger the ppa build, the s3 documentation i was using appears to be wrong, http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html ... i've copy and pasted the example code and it gets the same error
<hazmat> i'm probably getting something wrong
<niemeyer> hazmat: Have you checked the goamz implementation?
<niemeyer> hazmat: It has integration tests against S3 itself, so it must be right
<niemeyer> hazmat: Well.. I suppose txaws should be too
<hazmat> niemeyer, they don't generate signing urls, they use additional headers afaicr
<hazmat> ie. with Authorization: AWS XXX header 
<niemeyer> hazmat: Indeed
<hazmat> comparing against boto
<niemeyer> hazmat: Quite surprising
<niemeyer> hazmat: The algorithm is very close to the normal one, though
<hazmat> niemeyer, it is exactly the same just with additional quoting
<niemeyer> hazmat: and the Expires
<niemeyer> hazmat: Is it including the Expires as a query parameter? Also, are you sure the canonicalization is right?
<niemeyer> hazmat: Would be worth taking another signer algorithm, and trying to pass it in a URL for testing purposes
<hazmat> niemeyer, its using expires, i'm comparing against boto right now
<hazmat> niemeyer, ah.. the bucket key also needs quoting
<niemeyer> hazmat: Phew, cool
<hazmat> hmm
<hazmat> niemeyer, bcsaller, jimbaker can i get a +1 on https://pastebin.canonical.com/51668/
 * niemeyer checks
<jimbaker> taking a look
<hazmat> some reformatting, but the basic change is urllib.quote(path)
<hazmat> s/basic/functional
<niemeyer> hazmat: I think the strip may be dropped
<niemeyer> hazmat: It looks good in principle.. I have no idea if the actual algorithm is right or not, though, so my review isn't really worth much in this case
<niemeyer> hazmat: +1 with real interaction working
<hazmat> niemeyer, yeah.. in progress on testing that
 * hazmat gets some caffiene
<jimbaker> hazmat, looks reasonable, and strip is unnecessary - it's already padded with =
<hazmat> ah.. that's why testing is odd
<hazmat> the old formulas stick around
<niemeyer> hazmat: Bump ensemble.state.topology.VERSION?
<hazmat> niemeyer, in s3.. still digging through it, i'm not hitting pdbs and prints that i'd expect to
<hazmat> i'm not concerned about zk compatibility yet
<lynxman> hazmat: ping
<susanb> hi.
<susanb> have a quick question re: hadoop formula - where do I find it??
<hazmat> lynxman, pong
<_mup_> ensemble/trunk r327 committed by kapil.thangavelu@canonical.com
<_mup_> [trivial] fix for broken s3 signed url mechanism [r=niemeyer,jimbaker][f=833248]
<_mup_> The s3 bucket and key name paths needed to be quoted as well for the signed url signature.
<_mup_> This prevented formula deployment.
<niemeyer> hazmat: Woohay!
<hazmat> new ppa build requested
<lynxman> hazmat: hey, I'm trying to deploy formulas with ensemble on Canonistack, just as a test to make sure that all is in place for our Ensemble + Openstack sprint next week
<hazmat> jimbaker, looking forward to functional tests, there's also the corbertura plugin for coverage (which has a compatible xml report option)
<lynxman> hazmat: unfortunately the problem reported by TREllis is still there, is there any update on this from your side?
<hazmat> lynxman, i've got some branches that allowed me to deploy formulas successfully.. i should have the branches into review this week, but it might be next week b4 their merged, trying to focus on lxc atm, but i hadn't heard about the sprint.. the instructions for using the branches are at . http://pastebin.ubuntu.com/673083/ .. 
<hazmat> lynxman, is there any info on the ensemble+openstack sprint out there?
<lynxman> hazmat: yeah I've done exactly that, it doesn't work :)
<hazmat> lynxman, cool, so what's the problem?
<lynxman> hazmat: This https://pastebin.canonical.com/51673/
<lynxman> hazmat: which is exactly the same problem that TREllis reported a couple days ago
<lynxman> hazmat: the Sprint is about writing a white paper for deploying openstack and workloads over it using orchestra + ensemble
<hazmat> lynxman, can you paste  your redacted environments.yaml file?
<lynxman> hazmat: so we use ensemble to deploy openstack then ensemble again to deploy workloads over that openstack :)
<lynxman> sure
<hazmat> lynxman, turtles all the way down, cover page of the white paper ;-)
<lynxman> hazmat: lol
<lynxman> hazmat: https://pastebin.canonical.com/51674/
<jimbaker> hazmat, i will definitely take a look at adding in corbertura after we get solid functional testing in place. right now just finalizing the jenkins setup, the junit xml stuff is working fine
<hazmat> lynxman, hmm.. so i suspect the problem is not having the branches on your pythonpath on the admin side
<hazmat> lynxman, you might need to set PYTHONPATH=$aws_branch_path:$ensemble_branch_path:$PYTHONPATH before  running
<lynxman> hazmat: aah I see, ye oldie pythonpath problem, I'll create a .pth for that, gimme just 2 mins :D
<hazmat> lynxman, you can confirm by using.. $ python,  >>> import txaws,  >>> print txaws 
<hazmat> and verify its at the branch path
<lynxman> hazmat: yeah lack of pythonpath https://pastebin.canonical.com/51675/
<lynxman> hazmat: will create the .pth and get back to you asap
<hazmat> lynxman, virtualenv ftw
<jimbaker> hazmat, of course when used with caution. but fine against trunk
<lynxman> hazmat: haven't played much with virtualenv :)
<lynxman> hazmat: been trying to for months, never got to it
 * lynxman clearly needs 27 hour days
<jimbaker> lynxman, it can create quite inscrutable errors i have found
<jimbaker> but usually it works well
<hazmat> lynxman, if you find one, let me know ;-).. 
<lynxman> hazmat: oh I will, hehe
<hazmat> jimbaker, the multiple directory branch model not  working well with setup.py develop (stale directory pointers in easy-install.pth), is a little different than virtualenv issues, that would happen regardless of virtualenv.
<lynxman> hazmat: actually I compiled the eggs and put them in the new path and it works well now, just a minor misconfig issue
<lynxman> hazmat: you da man
<hazmat> lynxman, awesome
 * lynxman high fives hazmat
 * hazmat grabs some more caffiene
<jimbaker> hazmat, sure just more places for the problem to hide i think
<hazmat> back to lxc
<lynxman> hazmat: best of lucks
<hazmat> jimbaker, true
<hazmat> lynxman, let me know if you have any problems
<lynxman> hazmat: I will, I'm still hammering this and it's only 10PM, plenty of time still :D
<niemeyer> Almost done with the schema..
<lynxman> hazmat: okay one last hurdle, I got it to a point where it doesn't find the ssh keypair
<lynxman> hazmat: exceptions.LookupError: SSH authorized/public key not found.
<hazmat> lynxman, you need an ssh-keygen on that machine
<hazmat> lynxman, ensemble sets up an ssh public key from your homedir onto the remote machines
<lynxman> hazmat: cool, this being a pristine instance didn't have one, will try that, ty again :)
<hazmat> lynxman, you can specify it by path or contents in enviroments.yaml.. else it looks for some common ones, this should be in the docs
<hazmat> its become a faq
<hazmat> https://ensemble.ubuntu.com/docs/provider-configuration-ec2.html bottom of
<lynxman> hazmat: excellent, thank you :)
<fwereade> http digest auth is surprisingly disconcerting to work with
<fwereade> mainly because I understand the word "nonce" to mean, broadly speaking, "sex offender"
<hazmat> fwereade, interesting colloquialism 
<fwereade> hazmat: yeah, not a regularly used part of my vocabulary, but one it's suprisingly hard to ignore
<niemeyer> Schema in Go is finished
<niemeyer> Will put some docs, and push for review..
<niemeyer> But will do a break first
<hazmat> fwereade, niemeyer, bcsaller, jimbaker quick reminder.. team meeting tomorrow @utc17:00
<niemeyer> hazmat: Cheers
<jimbaker> hazmat, thanks, it's on my calendar
<SpamapS> Oddly enough.. at this very moment, I am using ensemble for something work related for the first time.. and its *really* quite helpful.
<SpamapS> Just being able to say "give me jenkins in the cloud" .. *saweet*
<niemeyer> SpamapS: Woohay! :)
<_mup_> Bug #833432 was filed: Feature Request: allow freezing/thawing an environment <Ensemble:New> < https://launchpad.net/bugs/833432 >
<jimbaker> SpamapS, that's very nice to hear about jenkins, we will be doing the same for ensemble itself shortly
<_mup_> ensemble/go-schema r2 committed by gustavo@niemeyer.net
<_mup_> Ported schema verification and coercion logic from Python. (!)
<SpamapS> jimbaker: I should have OpenID support added to the formula shortly. :)
#ubuntu-ensemble 2011-08-25
<jimbaker> SpamapS, nice, sounds very complementary to what i need
<SpamapS> That way we can give everybody in the ensemble team access to modify things.
 * SpamapS just made it exposable by adding open-port
<jimbaker> good to see that functionality working out nicely
<jimbaker> although it will be nicer once expose-cleanup lands
<_mup_> Bug #833446 was filed: Schema library is needed in Go <Ensemble:New> < https://launchpad.net/bugs/833446 >
<_mup_> ensemble/local-ubuntu-provider r326 committed by kapil.thangavelu@canonical.com
<_mup_> refactor net/zk utils into classes, add local machine
 * hazmat yawns
 * hazmat bashes head against java classpath
<niemeyer> hazmat: Don't waste your head on that one ;)
<niemeyer> SpamapS, hazmat: What's the state of the fix-s3-port branch/change?
<niemeyer> Robert provided some feedback on the merge proposal and it's sitting idle for a couple of weeks now
<niemeyer> https://code.launchpad.net/~clint-fewbar/txaws/fix-s3-port/+merge/71289
<SpamapS> I forgot to respond to him.. need to add tests but did actually address his initial concern.
<hazmat> i built on SpamapS's branch with additional fixes (post the additional tests afaicr)
<niemeyer> Lunch before meeting
<niemeyer> biab
<jimbaker> niemeyer, see you then
<fwereade> failure on trunk: trivial fix http://paste.ubuntu.com/674572/
<fwereade> someone approve?
<fwereade> jimbaker: ping
<hazmat> fwereade, +1
<jimbaker> fwereade, hi
<jimbaker> that was super annoying, i stepped outside i got stung by some sort of wasp
<fwereade> jimbaker: just pinged yu because you'd spoken most recently 
<fwereade> ouch :(
<fwereade> jimbaker: I have my +1 now
<fwereade> hazmat: thanks
<jimbaker> fwereade, sounds good
<hazmat> i missed that when i was doing the trivial yesterday :-(
<hazmat> i reordered the params while doing output diffs for ftests
<jimbaker> interesting, it of course worked for me ;)
<jimbaker> i wasn't going to approve it otherwise
<fwereade> hazmat: no worries, these cursed ordering things happen ;)
<jimbaker> there are certainly more in our code
<jimbaker> or specifically, our tests
<jimbaker> ordering issues like these is but one more reason that doctests are flawed (but potentially nice for documenting apis, if not too strict)
<niemeyer> Meeting time?
<niemeyer> robbiew: are you joining today?
<niemeyer> Daviey?
<fwereade> niemeyer: sounds good
<robbiew> can't...otp
<robbiew> could I get a calendar invite? then I won't schedule over it ;)
<niemeyer> robbiew: Same bat-time every week.. we'll try to sort out invites
<robbiew> cool
<_mup_> Bug #833906 was filed: go and python schema implementations could drift out of sync <Ensemble:New> < https://launchpad.net/bugs/833906 >
<Daviey> niemeyer: I'm here watching :)
<jimbaker> just waiting for g+
<Daviey> ah, g+.. :/
<Daviey> I can't do that this week.. Sorry.
<niemeyer> Daviey: No worries.. pinged mostly because you demonstrated interest recently
<Daviey> niemeyer: Oh, very much so... beta freeze today making my life interesting.
<niemeyer> hazmat, fwereade, jimbaker: Should I start?
<hazmat> niemeyer, got it
<jimbaker> niemeyer, sounds good
<niemeyer> hazmat: got it means you've already sent an invite?
<jimbaker> but i still don't see the hangout
<hazmat> invite sent
<niemeyer> Yeah, cool
<niemeyer> hazmat: We can hear you
<niemeyer> hazmat: Can you hear us?
<hazmat> niemeyer, no
<jimbaker> are we on g+, because i don't see any active hangouts for hazmat or niemeyer 
<hazmat> jimbaker, i might have the wrong email addr for you
<jimbaker> james.edward.baker@gmail.com
<jimbaker> hazmat, ^^^
<niemeyer> jimbaker: Sent
<hazmat> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_configuration
<niemeyer> hazmat: 
<niemeyer>             } else if (key.equals("clientPort")) {
<niemeyer>                 clientPort = Integer.parseInt(value);
<niemeyer>             } else if (key.equals("clientPortAddress")) {
<niemeyer>                 clientPortAddress = value.trim();
<fwereade> g+ doesn't like me suddenly :/
<fwereade> nope, it really doesn't want me to rejoin
<hazmat> 127.0.1.1
<niemeyer> hazmat: 127.0.0.1/8 <= loopback
<niemeyer> 127.*.*.*
<niemeyer> 10.0.0.0/8
<niemeyer> 192.168.0.0/16
<hazmat> 127.0.0.1 is bound in the container to the container, 127.0.1.1 is boudn to the host
<niemeyer> 10.1.2.3
<hazmat> 192.168.1*
<hazmat> 192.168.122.*
<hazmat> https://bugs.launchpad.net/nova/+bug/829880
<_mup_> Bug #829880: object store doesn't like key with '/'  <Ensemble:Triaged by hazmat> <OpenStack Compute (nova):Confirmed> <ensemble (Ubuntu):New> < https://launchpad.net/bugs/829880 >
<niemeyer> fwereade: 
<niemeyer> -    user_data = format_cloud_init(
<niemeyer> -        ssh_key, packages=packages, scripts=INSTALL_SCRIPTS)
<niemeyer> +    cloud_init = CloudInit()
<niemeyer> +    cloud_init.authorized_keys = [ssh_key]
<niemeyer> +    cloud_init.packages = packages
<niemeyer> +    cloud_init.scripts = INSTALL_SCRIPTS
<niemeyer> +    user_data = str(cloud_init)
<niemeyer> http://goneat.org/lp/gozk
<niemeyer> http://goneat.org/pkg/launchpad.net/gozk/#ZooKeeper.GetW
<niemeyer> machine.AddUnit
<niemeyer> hazmat: func (c listC) Coerce(v interface{}, path []string) (interface{}, os.Error) {
<niemeyer> func (e error) String() string {
<niemeyer> func (e error) String() string {
<niemeyer>         var path string
<niemeyer>         if e.path[0] == "." {
<niemeyer> type Checker interface {
<niemeyer>         Coerce(v interface{}, path []string) (newv interface{}, err os.Error)
<niemeyer> }
<niemeyer>         if reflect.TypeOf(v).Kind() == reflect.Bool {
<SpamapS> I noticed bug 832043 was just marked 'Fix Released' .. does that mean full orchestra support has landed int runk?
<_mup_> Bug #832043: can't deploy on orchestra <Ensemble:Fix Released by fwereade> < https://launchpad.net/bugs/832043 >
<SpamapS> If so, thats reason enough for a feature freeze exception.
<SpamapS> can somebody login here and let me know if they're allowed to administer jenkins or not: http://ec2-107-20-64-136.compute-1.amazonaws.com:8080/
<jimbaker> SpamapS, checking
<jimbaker> SpamapS, i see it, but i cannot administer
<jimbaker> that would be a good thing ;)
<SpamapS> did you login?
<SpamapS> (top right link)
<jimbaker> SpamapS, i don't have a login, and yes i tried
<SpamapS> You don't have a launchpad account?
<SpamapS> I find that hard to believe. :)
<SpamapS> So it didn't send you to launchpad? :-/
<jimbaker> SpamapS, ahh, i see this is in the context of open id integration
<jimbaker> SpamapS, no that's not working
<jimbaker> (and yes, i do have an lp account ;) )
<SpamapS> whats your lp username?
<SpamapS> jimbaker: when you say it did not work, it dodn't even try to redirect you to LP for openid, or it gave an error?
<hazmat> SpamapS, login link fails.. Error
<hazmat> Failed to login: null
<jimbaker> SpamapS, to be precise - i'm seeing what hazmat just posted
<jimbaker> and there's some brief activity where my browser is talking to launchpad
<jimbaker> in terms of status flying by
<SpamapS> tailing logs, can somebody try again?
<jimbaker> i tried, i actually went through open id this time, but it still failed with Failed to login: null
<SpamapS> jimbaker: it added you
<SpamapS> jimbaker: you exist in Jenkins now
<SpamapS> but you may not have any rights
<jimbaker> you're right, i'm logged in
<jimbaker> i can add projects
<SpamapS> You can probably do anything
<jimbaker> apparently so
<SpamapS> If you have the 'Manage Jenkins' link then I have achieved great success. :)
<SpamapS> jimbaker: now to figure out how to put that in the formula. :)
<jimbaker> just installed the bzr plugin, so yeah it looks mostly there
<jimbaker> just need to fix the failed to login noise
<SpamapS> Not sure where thats coming from
<SpamapS> I don't get it.
<jimbaker> (when actually successful)
<jimbaker> neither do i
<SpamapS> well anybody in ~ensemble can play with that jenkins now. :)
<jimbaker> SpamapS, sweet, i will definitely do so
<SpamapS> jimbaker: I imported your ssh key, so you can also ssh to the box. But please document anything that you wouldn't consider "data" so I can put it in the formula.
<jimbaker> SpamapS, sounds good
<SpamapS> niemeyer: so, given Jay Pipes' response to the objectstore/swift question.. maybe we need an optional 'filestore-protocol:' parameter for the ec2 provider which will make file storage use swift and/or nova-objectstore instead?
<niemeyer> SpamapS: Man.. I've been looking through my email to read whatever he said but I can't move fast enough
<niemeyer> SpamapS: I've been answering things before reaching it :)
<SpamapS> niemeyer: one more chainsaw to keep in the air
<niemeyer> Yeah, I clearly misunderstood the goals of OpenStack in terms of AWS compatibility
<SpamapS> feature parity is goal #1 .. AWS compatibility is counter to Rackspace's goals.
<Daviey> SpamapS: what bug # is that?
<SpamapS> bug 829880
<_mup_> Bug #829880: object store doesn't like key with '/'  <Ensemble:Triaged by hazmat> <OpenStack Compute (nova):Confirmed> <ensemble (Ubuntu):New> < https://launchpad.net/bugs/829880 >
<Daviey> *sigh*
<SpamapS> Daviey: btw, what time is beta freeze? I am trying to debug this LVM issue.. takes a while to iterate tho.. :-P
<Daviey> SpamapS: 21:00 UTC .. so 2 hours.
<Daviey> err, 1.5
<Daviey> SpamapS: I would suggest doing it gently is better than rushing it.. It's probably not going to miss beta 1 if done by end of week.
<SpamapS> Takes a good 10 minutes just to reproduce.. :-P
<SpamapS> Actually it didn't happen
<SpamapS> that was a pretty old image.. maybe its been fixed
<niemeyer> jimbaker, bcsaller, hazmat: We need another review on this: https://code.launchpad.net/~fwereade/ensemble/cobbler-zk-connect-error-messages/+merge/72033
<niemeyer> It's a bit large but there isn't much logic there, so shouldn't take too long
<jimbaker> niemeyer, i will take a look after walking my dog
<niemeyer> jimbaker: Cheers!
<_mup_> ensemble/local-ubuntu-provider r327 committed by kapil.thangavelu@canonical.com
<_mup_> better zookeeper management
<_mup_> ensemble/local-ubuntu-provider r328 committed by kapil.thangavelu@canonical.com
<_mup_> local-dev file storage
<niemeyer> jimbaker: On expose-cleanup, you got [9] in the opposite direction
<niemeyer> jimbaker: I was pointing out that you don't have to wait for deletion to finish on all of the removed groups before running another round
<niemeyer> jimbaker: since it kills parallelism unnecessarily
<niemeyer> jimbaker: You can wait for them to finish after _all_ of the groups have been deleted
<niemeyer> jimbaker: Instead, you've moved backwards, and made it wait on each individual group's deletion
<niemeyer> jimbaker: I've put these comments back in the review with [11] and [12]
<niemeyer> jimbaker: Still +1 on it
<niemeyer> Then, I just need a second review on go-schema, and that's eureka review-clean
<jimbaker> niemeyer, thanks, sorry for my confusion there. but i will change it accordingly
<niemeyer> jimbaker: No worries, it's looking very good overall
<niemeyer> Coffee break
<jimbaker> i will also review go-schema
<_mup_> ensemble/local-ubuntu-provider r329 committed by kapil.thangavelu@canonical.com
<_mup_> additional managed zk test, optional filtering if zk not installed via package
<adam_g> hey, when using config.yaml and adding units to a service, is it possible to have different service units using differnet configuration values?
<adam_g> ie, config.yaml contains 'some_parameter: value1' for the first and second service unit, but the third time add-unit, is it possible to have 'some_parameter: value2' either by editting config.yaml before 'add-unit' or setting it on command line?
<niemeyer> adam_g: No.. I'm curious to understand the use case you have in mind, though
<niemeyer> adam_g: Service units are supposed to be deployments of the same service, so the whole logic in terms of configuration and in terms of what the admin is presented goes around that conceptual representation
<niemeyer> adam_g: Naturally, units can contain differences (e.g. one may be a master, etc).. but as far as the admin and other units are concerned, they are a cluster sharing a configuration that can be maniupated as a single block
<niemeyer> adam_g: If you see this is somehow creating issues for you, it'd be very interesting for us to understand how that's the case
<niemeyer> That said, I have to step out for dinner with friends right now
<niemeyer> adam_g: If you have some notes on the topic, would you mind to please sparkle some debate in the list on it?
<adam_g> niemeyer: sure
<adam_g> my main use case for this:
<adam_g> swift-storage nodes use a local block device, format it, and mount it somewhere to be used by the service.  as such, config.yaml has a 'block_device' argument that lets it know what storage to use
 * Daviey glazes over whilst he tries to work out what is going on.
<adam_g> if i'm creating a pool of storage nodes across 100 nodes, its likely the storage layout would differ from node to node
<adam_g> but, i suppose being able to modify that on-the-fly during add-unit would be useless unless i could also specify the machine i'd like to use for the new unit
#ubuntu-ensemble 2011-08-26
<SpamapS> adam_g: If it gets super custom per node.. then ensemble becomes a bit pointless.
<SpamapS> adam_g: whatever is creating these one-off nodes (I'm assuming this is orchestra not ec2) should just make sure each node has the block device at the specified location.. maybe make a symlink.
<adam_g> SpamapS: i wouldnt consider the existence of a specific blockdev to be super custom. :) but yeah, im thinking it should be left up to cobbler/orchestra/admin.  even if that is the case, though, and we can declare something similar to m1.large with specific storage layout, it would be useful to be able to direct add-unit to a specific machine type/class 
<SpamapS> yeah, --placement=available is pretty inadequate for this.. we probably need a better one.
<niemeyer> adam_g: Yeah, I agree with SpamapS.. we need to find a way to scale that.. if the admin has to put info per node, it becomes harder to manage
<niemeyer> adam_g: I understand your point, though.. we can dig a bit deeper to find a way to manage that need without introducing individual entries per node in the config
<Whiz> Hi, I'm having a hard time on finding information about whether it is possible to override default-image-id somehow per service-unit/formula or something. I.e. I wan't to use small instances for everything but when it comes to db-nodes I'd like'd them to be high-mem instances. My google-fu isn't strong enough :)
<SpamapS> Whiz: there's an open bug for that..
<Whiz> SpamapS, ok. so it was my google-fu :D thank you
<SpamapS> Whiz: you can set it in the environments.yaml just before you deploy right now.. and it will be "the image id" for that service as you add units
<SpamapS> there are actually two bugs.. tho they might need to be merged..
<SpamapS> bug 830995 and bug 808969
<_mup_> Bug #830995: default-instance-type and default-image-id are inadequate <Ensemble:New> < https://launchpad.net/bugs/830995 >
<_mup_> Bug #808969: determine the instance type at deploy <Ensemble:Confirmed> < https://launchpad.net/bugs/808969 >
<Whiz> okay :)
<Whiz> thank you
<hazmat> morning folks
<lynxman> hazmat: ping, whenever you're around :)
<hazmat> lynxman, pong
 * hazmat contemplates an oneiric upgrade
<hazmat> niemeyer, how's oneiric been treating you?
<niemeyer> hazmat: Haven't been using it yet, but was just pondering yesterday about how it _might_ treat me :)
<hazmat> niemeyer, i'm considering taking the plunge this morning, although bug 818830 concerns me
<_mup_> Bug #818830: [Sandy Bridge] serious power regression from kernel 3.0.0-6 to 3.0.0-7 (rc6 disabled) <3.0> <kernel> <oneiric> <power> <linux (Ubuntu):Triaged> < https://launchpad.net/bugs/818830 >
<_mup_> ensemble/stack-crack r328 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
<niemeyer> hazmat: Looks like there's a trivial way to fix it?
<hazmat> niemeyer, doesn't work for everyone
<niemeyer> Ah, gotcha
<_mup_> ensemble/local-ubuntu-provider r330 committed by kapil.thangavelu@canonical.com
<_mup_> refactor get class path utility function into managed zk method
 * hazmat commits work in progress and pushes pre upgrade
<niemeyer> hazmat: Good choice ;-)
<_mup_> txzookeeper/errors-with-path r44 committed by kapil.foss@gmail.com
<_mup_> zk exceptions w/ path information
<hazmat> niemeyer, apparently i scored in the race to get a touchpad as well :-)
<hazmat> took a week to get a confirmation email of my order out of hp, which is sad
<niemeyer> hazmat: Neat
<niemeyer> hazmat: I'm curious about these machines
<hazmat> niemeyer, their dual core qualcomms, 1.2 ghz.. overclockable to 1.5ghz... seem reasonably capable, there's a couple of bounties out to get android on them, wifi only + usb. even if nothing else they should make decent xmas gifts
<niemeyer> hazmat: They are neat Ubuntu machines it seems :)
<hazmat> niemeyer, mee+go+ubuntu :-)
<hazmat> niemeyer, update-manager failed
<niemeyer> hazmat: Ugh
<niemeyer> hazmat: Let me know how it goes
<niemeyer> hazmat: Depending on your results I could get "brave" ;-)
<hazmat> niemeyer, looks like it rolled back properly, i think i might wait till the official beta on sept 1st
<niemeyer> hazmat: Oh, seriously?
<niemeyer> hazmat: That's quite amazing on itself
<niemeyer> hazmat: Well.. depending on where it failed
<hazmat> niemeyer, it looks like it adjusted packages sources, updated, calculated the upgrade, failed to do so, and restored the sources
<niemeyer> hazmat: Ah, ok
<niemeyer> hazmat: That's less impressive
<hazmat> niemeyer, do you still use smart for dist-upgrades?
<niemeyer> hazmat: I know mvogt is looking at doing magic for rolling back for quite a while
<niemeyer> hazmat: Like, _real_ rollbacks :)
<niemeyer> hazmat: I never used Smart for dist-upgrades
<niemeyer> hazmat: update-manager is not just apt.. it's full of hand-made fixes
<hazmat> ugh.. post earthquake.. hurricane.. just got a call from the power company to expect multi-day outages
<robbiew> hazmat: damn
<Daviey> http://t.co/2fNBu7M
<robbiew> heh
<niemeyer> hazmat: Good time for a sprint!
<niemeyer> Daviey: LOL
<RoAkSoAx> fwereade: howdy!!
<fwereade> heya RoAkSoAx
<fwereade> how's life?
<RoAkSoAx> fwereade: pretty good, testing ensemble/orchestra on real HW
<fwereade> RoAkSoAx: sweet
<fwereade> RoAkSoAx: all that's missing in trunk is the unicode-key thing now
<RoAkSoAx> fwereade: alrgith, great then
<fwereade> RoAkSoAx: not mped yet though, haven't actually finished it yet
<fwereade> RoAkSoAx: ...and damn, I need to be off a little early today and might not quite get there
<RoAkSoAx> fwereade: that's fine
<RoAkSoAx> fwereade: at least it is just one more to go
<fwereade> RoAkSoAx: *might* be able to polish it off this evening but don't count on it :)
<RoAkSoAx> fwereade: no worries
<RoAkSoAx> I'm testing with cobbler-compelte-fixes now
<RoAkSoAx> fwereade: but have a few things that might already sound familiar
<fwereade> RoAkSoAx: oh ah?
<RoAkSoAx> fwereade: 1. bootstrap doesn't check if there's ssh keys apparently, so when trying to do status it fails due to not having a ssh key in the zk node. 2. which is related to 1, is throwing and exception when it should probably throw just an error message: http://pastebin.ubuntu.com/675290/ 3. When we bootstrap/deploy, we need to check that power management is configured in the system, cause it is bootstrapping/depoying without actually checking
<fwereade> 1/2: is that just saying that bootstrap fails if you don't have a public key in ~/.ssh?
<RoAkSoAx> fwereade: 3 (cont'd) which means that enmsemble will think it started a machine but since cobbler doesn't have power management configured nothing really happened
<fwereade> 3: hm, I guess that's set in the system dict somewhere, I can just filter on that I presume?
<lynxman> hazmat: ping?
<RoAkSoAx> fwereade: 1/2. not exactlyu. it bootstraps correctly, but since there are no keys on ~/.ssh then, we cannot connect to the zk (I think there's a bug report already)
<RoAkSoAx> fwereade: 3. yes, if I have time later today I'll do it myself so don';t worry just yet ;)
<fwereade> RoAkSoAx: yeah, it rings a bell
<fwereade> RoAkSoAx: cool
<fwereade> RoAkSoAx: nothing showstopping then :)
<hazmat> lynxman, pong
<RoAkSoAx> fwereade: not really, just wanted to make you aware in case you stumbled on them
<lynxman> hazmat: I have an issue with Ensemble + Openstack
<RoAkSoAx> s/on/upon/
<hazmat> lynxman, details?
<lynxman> hazmat: I can bootstrap machines but it looks that no matter how I try the ssh keys are not copied over, the console output logs don't help either :/
<RoAkSoAx> fwereade: speaking of which ^^
<lynxman> hazmat: tried to a) specify public key with authorized-keys variable in environments.yaml b) create default id_rsa and let Ensemble choose it
<fwereade> RoAkSoAx: hmm :)
<hazmat> lynxman, i had that problem as well.. i worked around with specifying a euca key pair
<lynxman> hazmat: in both cases Ensemble bootstraps successfully but then it complains about incorrect SSH key, even ssh'ing manually says that the key is not valid
<fwereade> RoAkSoAx: ...or maybe a different issue
<lynxman> hazmat: cool, any details in how to do so in environments.yaml ?
<RoAkSoAx> fwereade: yeah but related as there was a similar bug report anyways, will test and see what happens ;)
<hazmat> lynxman, its a new option on that branch,  from the paste its under ec2-key-name http://pastebin.ubuntu.com/673083/
<lynxman> hazmat: when I tried that ensemble complained at bootstrap that no SSH was found :)
 * lynxman comes full circle
<hazmat> lynxman, effectively euca-add-keypair and then use the name
<lynxman> hazmat: it's when you suggested to use authorized-keys
<lynxman> hazmat: I'll try again
<hazmat> lynxman, its not authorized keys from your local machine, its the provider managed keys
<lynxman> hazmat: I know :)
<lynxman> hazmat: and that's the one I specified
<hazmat> hmm
<RoAkSoAx> fwereade: nothing to worry from our side though
<lynxman> hazmat: as a matter of fact copy/pasted the key name from euca-describe-keys
<lynxman> hazmat: I'll quickly produce a pastebin so you can see the issue
<hazmat> lynxman, sounds good, just trying to figure out how to debug it
<lynxman> jcastro: excess flood? wow
<lynxman> hazmat: it worked perfectly now *shrugs* thanks :)
<hazmat> lynxman, magic :-)
<hazmat> lynxman, yeah... i had some transient issues with keys as well, i'm not even entirely sure the ec2-key-name has any purpose
<lynxman> hazmat: looks like my version of the line before wasn't loved by Ensemble
<lynxman> hazmat: or yeah, transient issues as you say :)
<hazmat> lynxman, i think it might be cloud-init related (delay in key installation) but its hard to say without more investigation
<_mup_> ensemble/go r2 committed by gustavo@niemeyer.net
<_mup_> Merged go-schema branch into go [r=fwereade,jimbaker]
<_mup_> Ported schema verification and coercion logic from Python.
<niemeyer> http://goneat.org/lp/ensemble/go/schema
<fwereade> niemeyer: very neat :)
<fwereade> I need to skip out a bit early today -- happy weekends all :)
<niemeyer> fwereade: Cheers man!
<RoAkSoAx> fwereade: see ya my frioend
<hazmat> fwereade, have a good one
<hazmat> niemeyer, 503 on http://goneat.org/lp/ensemble/go/schema
<niemeyer> hazmat: Try again
<niemeyer> hazmat: I was updating it so that searching would work too
<niemeyer> hazmat: http://goneat.org/search?q=FieldMap
<hazmat> niemeyer, looks good
<niemeyer> On that note, I'm stepping out for lunch.. I hope to nail down Formulas this afternoon, given that we have a clean Eureka review queue (!)
<hazmat> ugh.. that can't be allowed to continue ;-)
<hazmat> niemeyer, have a good one
<niemeyer> hazmat: Thanks! :-)
<SpamapS> $ ./unit2machine.py master-jenkins/0
<SpamapS> ec2-107-20-64-136.compute-1.amazonaws.com
<SpamapS> :-D
 * SpamapS is in ur tezts, writing new codez
<hazmat> :-)
<hazmat> this looks pretty interesting http://wiki.perfkit.org/index.php?title=Main_Page
<hazmat> a nice gui to profiling tools
<hazmat> ala osx instruments
<SpamapS> hm.. how does ensemble work on python 2.6 if we use argparse?
<SpamapS> is it just an external module?
<jimbaker> SpamapS, correct
<SpamapS> so using argparse won't cause my code to be 2.7 only?
<jimbaker> there are some small differences in error messages iirc, so that breaks a couple of tests
<jimbaker> SpamapS, that's right
<jimbaker> SpamapS, i was running the ensemble test suite for a while on os x (don't tell anyone), it worked just fine except for a few trivial issues
<jimbaker> SpamapS, it would certainly be useful to have a jenkins slave testing the client code, as packaged by macports
<kim0> um, can a formula request to run on oneiric specifically ?
<SpamapS> jimbaker: yeah we should do that. I'm sure we can find an old mac mini to use as a slave. :)
<SpamapS> kim0: not yet no
<kim0> okie .. ami-id is env.yaml then for now
<SpamapS> kim0: yeah, thats being discussed in a couple of bugs tho
<hazmat> SpamapS, python 2.6 and ensemble work, there was one minor incompatibility recently that jim fixed
<SpamapS> We'll find out for sure.. running lots of examples on lucid in my test script.. :)
<m_3> SpamapS: I do lots of ensemble cli on lucid VMs... couple of warnings lately, but it's generally worked
<m_3> never spun up lucid amis tho
<SpamapS> jamespage: go back to your vacation! ;)
<niemeyer> hazmat: ping
<hazmat> niemeyer, pong
<niemeyer> hazmat: Can I get a quick review on this simplification: http://paste.ubuntu.com/675388/
<hazmat> niemeyer, hmm.. doesn't the existing use of resphaper inline to the formula need to be removed as well if its directly in the schema
<niemeyer> hazmat: How do you mean?
<hazmat> oh.. nm. its all in the schema
<niemeyer> Right
<niemeyer> hazmat: The only logical  change is removing the OneOf
<hazmat> niemeyer, +1
<niemeyer> hazmat: Thanks!
<hazmat> np
<_mup_> ensemble/trunk r332 committed by gustavo@niemeyer.net
<_mup_> Minor simplification in metadata schema [r=hazmat]
<_mup_> The OneOf was redundant since the interface reshaper/expander also
<_mup_> takes care of a complete interface.
<_mup_> Also took the chance for improving naming and docs a bit.
<niemeyer> Me and Ale teaching our future kids: http://www.smbc-comics.com/index.php?db=comics&id=2348#comic
<hazmat> cute
<niemeyer> if not formula_id.count(":") >= 1:
<niemeyer> That's an awesome way to say == 0 :-)
<SpamapS> hmm.. is there a way to add an SSH key to an ensemble environment currently?
<hazmat> SpamapS, ? there already is one
<SpamapS> Yeah, so say I bootstrapped from my laptop...
<hazmat> SpamapS, the authorized key specified ( or default ssh key found if not specified) is installed on all the machines on the env
<SpamapS> then I want to hand off maintenance to Bob
<SpamapS> I'm not going to give Bob my key. :)
<SpamapS> Needs to be a way to add/remove keys.
<hazmat> SpamapS, atm no. we haven't really tackled multi-user management.. better would be to specify a shared key when creating the env, but your right we should probably support changing
<SpamapS> better, but not realistic
<SpamapS> s/should probably/must/ IMO
<hazmat> indeed
<SpamapS> It shouldn't be too hard really
<SpamapS> ensemble is root.. so it can just twiddle the keys as needed
<hazmat> SpamapS, its straight forward, its just a question of handling multiple users.. i think we'll might start walking down a rest api road for multi-user support
<SpamapS> I will file a bug.
<hazmat> sounds good
<SpamapS> hazmat: that will mitigate some of it, but you still need to be able to ssh to the machines.
<hazmat> true
<_mup_> Bug #834930 was filed: Need a way to manage SSH keys in an ensemble environment. <Ensemble:New> < https://launchpad.net/bugs/834930 >
<_mup_> ensemble/go-formulas r3 committed by gustavo@niemeyer.net
<_mup_> Implemented ParseId.
<negronjl> has the behavior of ensemble debug-hooks <blah> changed?  The hooks are being executed ( apparently ) but, I don't see anything happening in the debug-hooks windows 
<niemeyer> negronjl: There was a minor change by kirkland to use byobu, but this shouldn't have affected anything
<niemeyer> negronjl: Which hook are you trying to debug?
<negronjl> niemeyer: mysql-admin-relation-joined/changed on cloudfoundry-server and cf-mysql formulas
<niemeyer> negronjl: Ok, this should definitely be working
<niemeyer> SpamapS, m_3: Any comments?
<negronjl> niemeyer:  I bootstrapped, deployed both formulas, debug-hooks on both and then add-relation.....and I get nothing on the debug-hooks .... it's stuck on "bash"
<niemeyer> negronjl: Yeah, this scenario sounds fine
<negronjl> btw. I deployed....waited for them to be successfully deployed ( started ), then debug-hooks on them...
<niemeyer> negronjl: Does the relation get succesfully established afterwards?
<niemeyer> negronjl: According to status
<negronjl> niemyer:  according to status yes but, none of the changes get applied.
<niemeyer> negronjl: Hmm
<niemeyer> negronjl: So the hooks do not run either
<niemeyer> ?
<niemeyer> negronjl: ?
<negronjl> niemeyer:  not consistenly.  they apparently run on one of the two deployed formulas but not on both of them.
<negronjl> niemeyer:  this doesn't always/nor never happens....it's all erratic.
<niemeyer> negronjl: Very supect..
<niemeyer> jimbaker: Are you able to investigate this?
<niemeyer> jimbaker
<hazmat> negronjl, do you have a debug-log output
<negronjl> I am writing up exactly what I am doing ( down to the environment.yaml file ) so I can pass it on....
<hazmat> negronjl, awesome
<hazmat> negronjl, i can't think of anything directly related to debug-hooks which has changed recently outside of the byobu changes
<kirkland> hallyn: negronjl: hmm, I'm here, if there's any issue to look at, wrt to the byobu/debug-hooks changes
<negronjl> hazmat:  yeah... I tend to agree plus, I don't see byobu breaking debug-hooks either.... I am just hoping that I am missing something here and a fresh set of eyes on this issue can just clear it all up for me :)
<m_3> negronjl: send your config... haven't seen this, but I was on a pretty stale version until yesterday
<negronjl> m_3:  I will... I am also documenting everything ( environment.yaml, steps, debug-log, formula.log, etc. ) so I can give you a complete picture on what's going on and the necessary resources for you to reproduce if you can .
 * SpamapS starts another test run .. 25th time is the charm!
<jimbaker> m_3, are you looking into this with negronjl? just ping me if you need some help from the core perspective
<m_3> jimbaker: will do
<negronjl> m_3, hazmat, kirkland, niemeyer, jimbaker, bcsaller, anyone :) :  Here is more or less my issue with as much documentation as I can remember.  Please let me know if I missed anything here.
<negronjl> m_3, hazmat, kirkland, niemeyer, jimbaker, bcsaller, anyone :) :  Here is more or less my issue with as much documentation as I can remember.  Please let me know if I missed anything here:  It would help if I paste it: http://paste.ubuntu.com/675480/
<niemeyer> negronjl: Sweeet
<niemeyer> negronjl: Thanks!
<niemeyer> Now, who's looking at this?  It won't help us if everyone jumps into it :)
<negronjl> niemeyer:  Please let me know if there is anything at all I can do to help solve this :)
<niemeyer> negronjl: Will do, thanks again
<niemeyer> hazmat: Are you interested in looking?
<hazmat> sure
 * negronjl going to grab a quick byte while you guys look at this....
 * negronjl will be back in a few minutes.
<hazmat> negronjl, cool, i should  have some more info soon
<negronjl> hazmat: thx
<niemeyer> hazmat: Thanks!
<SpamapS> Hrm.. I need shutdown --yes-really-I-want-that
<m_3> SpamapS: +1  I so want a '-y' option to shutdown
<SpamapS> currently I'm walking through and just torching all services/machines now because it doesn't exist
<SpamapS> python experts.. halp
<SpamapS> http://paste.ubuntu.com/675490/
<SpamapS> if my eyes don't deceive me, those two things equal
<bcsaller> p type(service)
<SpamapS> <type 'str'>
<SpamapS> (Pdb) type(status['services'].keys()[2])
<SpamapS> <type 'str'>
<bcsaller> I thought it might be a __str__ method on the state object
<bcsaller> wordpess
<bcsaller> spelling
<SpamapS> No I'm actually just parsing ensemble status via subprocess and yaml.load() ;)
<SpamapS> wordpess ? hahahahahaha
<SpamapS> thats what I get for copy/pasting
<bcsaller> I can see why they didn't go with that name
<SpamapS> instead of using a durn variable
<SpamapS> well wordpuke was taken
<bcsaller> by a slam poet?
<m_3> slampress
<bcsaller> heh
<hazmat> negronjl, is that cloud foundry formula the latest, on my deploy it fails to start
<hazmat> although i don't see anything obvious in the error log
<hazmat> odd it seems like its hanging on config-changed
<hazmat> but there isn't a config-change hook here
<hazmat> but it never finishes the transition to start, and stays installed, and resolved doesn't help :-(
<hazmat> i've got two debug windows up though
<m_3> hazmat: heck of a long install
<hazmat> indeed
<hazmat> still its a bug that's not in debug-log output..
<_mup_> ensemble/go-formulas r4 committed by gustavo@niemeyer.net
<_mup_> The schema package now consistently outputs values of type schema.M
<_mup_> for maps and schema.L for lists.
<SpamapS> heh.. well at least my tests legitimately failed for a good reason
<SpamapS> service names can only be like 16 chars long w/ the example formulas
 * negronjl is back
<negronjl> hazmat:  I'll make sure you get the latest one....committing, pushing now
<negronjl> hazmat: cloudfoundry-server and cf-mysql both updated.
<hazmat> negronjl, thanks, redeploying
<negronjl> hazmat:  make sure you deploy on oneiric and not natty.  I put the ami and type in the environment file there.
<hazmat> negronjl, indeed i'm using those, i don't see any new revs on cf-mysql.. latest revno is 2
<m_3> so I get joined and two changes firing
<m_3> walking through the logic
<negronjl> hazmat:  I pushed revno 3
<hazmat> negronjl, to where? https://pastebin.canonical.com/51828/
<negronjl> hazmat: cf-mysql is here: bzr+ssh://bazaar.launchpad.net/~negronjl/%2Bjunk/cf-mysql/
<negronjl> hazmat: cloudfoundry-server is here: bzr+ssh://bazaar.launchpad.net/~canonical-sig/%2Bjunk/cloudfoundry-server/
<hazmat> negronjl, cool
<hazmat> hmm
<negronjl> hazmat: hmm...doesn't sound so good :)
<hazmat> negronjl, so it looks like debug-hooks isn't working, i haven't seen a hook go by yet, ( ic new hooks being executed while sitting in the tmux session), and it seems to active be preventing successful transition recording.. quite odd.. 
 * hazmat baselines against trunk
<hazmat> and the example formulas
<negronjl> hazmat:  ahah!!  I finally got someone to see it too ....YEAH!!!  I am not crazy :)
<hazmat> negronjl, indeed, not i'm going to see if this also occurs with the example formulas and take it from there
<hazmat> s/not/
<_mup_> ensemble/go-formulas r5 committed by gustavo@niemeyer.net
<_mup_> Fixed and tested schema coercing of nil values in several cases.
<m_3> I'm running default amis (already had other services running) and saw debug-hooks fire between cf and cf-mysql
<m_3> install failed though
<negronjl> m_3:  install fails on natty....works only on oneiric
<m_3> (exited cleanly but the service wasn't up)
<_mup_> ensemble/go-formulas r6 committed by gustavo@niemeyer.net
<_mup_> Ported formula id parsing and interface schema normalization.
<hazmat> seems like the easiest way to debug is to setup an ssh into python interpreter on the agents
 * hazmat goes back to trying it from the cli
<negronjl> hazmat: FWIW.  I deploy both formulas ( cloudfoundry-server and cf-mysql ), ensemble ssh to both of them and tail -f formula.log... 
<negronjl> hazmat: then I add-relation etc.  At least I get to see more or less what's going on 
<_mup_> Bug #835037 was filed: Go port must handle interface schemas <Ensemble:New> < https://launchpad.net/bugs/835037 >
<hazmat> negronjl, indeed that's a good thing to do, i'm just focusing on debug-hook functionality and forensics there is really about getting into the runtime state in the process to try and see what's going wrong.. before getting a local failing test case i can bisect against.. i have some suspicions but evidence is better
<negronjl> hazmat: indeed :)
<_mup_> ensemble/simplify-iface-schema r333 committed by gustavo@niemeyer.net
<_mup_> Simplify the interface schema further.
<niemeyer> hazmat++
<hazmat> reproduced against example formulas
<_mup_> Bug #835044 was filed: Interface schema handling is too complex <Ensemble:In Progress by niemeyer> < https://launchpad.net/bugs/835044 >
<niemeyer> hazmat**2
<hazmat> niemeyer, so i deployed mysql, and then did an immediate debug-hooks.. install hook is still running, completes and then executes start transition, config-change hook tries to execute.. no new debug-hook window in tmux session.. and it looks like the debug hook watcher is still running..
<hazmat> unit agent log.. https://pastebin.canonical.com/51830/
<hazmat> hook.pid never shows up in the debug-hook dir
<hazmat> so creating the new window seems to have the problem
<niemeyer> hazmat: Yeah, looks stuck in config-changed
<niemeyer> hazmat: What's the deal there?
<hazmat> niemeyer, its not really config-changed, its any hook, config-changed doesn't exist on these
<niemeyer> hazmat: Ok, but I supposed you've hooked in with debug
<hazmat> niemeyer, the debug hook script that executes something into tmux and waits for it, the launch in tmux portion is failing
<hazmat> and the script just waits around forever
<negronjl> hazmat, niemeyer:  unrelated question... config-change seems to be called when anything changes on the instances.   Is there a way to ( from inside the config-change hook ) figure out which hook changed or some indication of what changed?  This way I know more or less what needs to be updated .
<niemeyer> hazmat: Any idea of how's it failing?
<hazmat> niemeyer, still looking bbiam
<niemeyer> negronjl: Not yet.. we've discussed a few ideas around that
<niemeyer> negronjl: But still have to evolve a bit more
<negronjl> niemeyer: cool...thx
<niemeyer> negronjl: np
<hazmat> niemeyer, what's this line do exec > $ENSEMBLE_DEBUG/debug.log >&1
<hazmat> its just execing the rest of the script and sending stdout/sterr to the debug.log file?
<niemeyer> hazmat: That's right
<hazmat> niemeyer, hmm.. executing the script by hand works.. creates the new tmux window
<niemeyer> Hmmm
<niemeyer> hazmat: Is the log empty?
<hazmat> niemeyer, it is
<niemeyer> Feels like it hasn't run
<hazmat> i'm adding a bunch of echoes to it, and going to run it again
<hazmat> niemeyer, its running, its in the process list
<hazmat> when the hook 'hangs'
<niemeyer> Cool, that's good at least
<hazmat> niemeyer, found it.. unit-name isn't being exported to debug hook
<niemeyer> hazmat: Woah?
<hazmat> niemeyer, tmux command doesn't get captured as error (there is a set -e at the top), and then goes onto wait for hook.pid
<hazmat> tmux fails to start the new window because of the invalid session name
<niemeyer> hazmat: set -e is the opposite.. it forces everything to error out
<hazmat> ah
<hazmat> niemeyer, i got it wrong about the env var i think.. i was using another window in the session, and its env variables got exported overwriting the original debug hook ones
<niemeyer> Hmm, ok
 * hazmat goes back to drawing board
<hazmat> hmm tmux versions went from 1.3 in natty to 1.5 in oneiric
<niemeyer> Nice bump.. I hope they didn't introduce the screen bug on the way
 * niemeyer looks at the release notes
<hazmat> yeah. was just checking those out
<hazmat> niemeyer, this interesting comparison of screen 2 tmux.. makes why 2 use tmux pretty clear.. http://www.techrepublic.com/blog/opensource/is-tmux-the-gnu-screen-killer/1901
<niemeyer> hazmat: Can't see anything obvious at least in the changelog
<hazmat> i'm going to try backing the byobu work to see
<niemeyer> hazmat: What about the echos?
<hazmat> niemeyer, yeah.. me neither.. lots of couple of new session starting option
<hazmat> niemeyer, just executing the debug hook script with echoes inserted to figure out the failure point
<hazmat> i'm going to back out to a higher level and just try it sans the byobu work
<hazmat> one more try on the cli b4 that
<niemeyer> hazmat: Note that I've screwed up when merging the byobu work..
<hazmat> niemeyer, yeah.. i saw the two commits
<niemeyer> hazmat: Cool
<hazmat> niemeyer, this is the line that locks up it looks like -> http://www.techrepublic.com/blog/opensource/is-tmux-the-gnu-screen-killer/1901
<hazmat> whoops.. copy paste error
<hazmat> tmux new-session -d -s $ENSEMBLE_UNIT_NAME 2> /dev/null || true
<niemeyer> hazmat: Oh my.. not again :-)
<niemeyer> hazmat: Ok.. what happens if you run two tmux with the same name on Oneiric?
<niemeyer> hazmat: Using that line
<niemeyer> This would also explain why sometimes it happens and sometimes it doesn't, since it depends on who wins the race to get the session created
<hazmat> niemeyer, root@domU-12-31-39-05-75-E1:/tmp# tmux new-session -d -s wordpress/0 
<hazmat> duplicate session: wordpress/0
<niemeyer> hazmat: Phew.. alright, that's good
<niemeyer> hazmat: No locks or anything
<niemeyer> ?
<hazmat> niemeyer,  but there's serveral processes in the process listing that are on that same command
<hazmat> its definitely locked for some of them
<niemeyer> hazmat: What's the behavior if you use a new name
<niemeyer> ?
<niemeyer> hazmat: That "-d" is from detach.. it should never stay
<hazmat> hmmm.. i should probably try it outside of tmux ;-)
<niemeyer> hazmat: Yeah, sounds like an idea :)
<hazmat> hmm. on login via ssh ERROR: [/home/ubuntu/.byobu] is not writable by the current user
<niemeyer> hazmat: That sounds like a hit
<niemeyer> hazmat: Let's try to roll byobu back
<hazmat> i'm not entirely convinced, but i'm not sure what else to try
<hazmat> its wierd i can execute the debug hook new-session command from the cli and get the duplicate output, but it hangs if i execute the script on that line
 * hazmat tries the rollback
<niemeyer> hazmat: If you execute in which line?
<hazmat> niemeyer, the tmux new-session -d -s unit_name 2   line
<niemeyer> hazmat: Sorry, I'm not sure I understand
<niemeyer> hazmat: You say it hangs if you execute the script on that line
<niemeyer> hazmat: How are the working and the non-working executions different/
<niemeyer> ?
<hazmat> niemeyer, i can execute "tmux new-session -d -s unit_name" on the cli, works fine.. but the debug-hook hangs on that line
<niemeyer> hazmat: Ok.. are you able to reproduce a hanging tmux somehow?
<niemeyer> hazmat: Or is it just the outcome of the debug-hooks context itself that gets there?
<hazmat> niemeyer, its a new ssh shell executing it, and it is reproducible.. executing the debug-hook script hangs
<hazmat> this is with some previous context of the debug-hook session already being active
<niemeyer> hazmat: Ok
<hazmat> anyways.. going to rollback byobu and see if that helps
<niemeyer> hazmat: If you get rid of ~/.tmux.conf, does it hang?
<hazmat> niemeyer, checking
<niemeyer> hazmat: "Getting rid" is just removing the tmux.conf..
<niemeyer> hazmat: If byobu is at fault, the hanging script should be free as in speech
<hazmat> niemeyer, still hangs
<niemeyer> Ok, that's not it then
<hazmat> yup
<niemeyer> hazmat: Can you strace that?
<hazmat> suer
<niemeyer> hazmat: strace f
<niemeyer> hazmat: strace -f
<hazmat> sitting in epoll
<niemeyer> hazmat: On what fds?
<hazmat> niemeyer, hold on grabbing a trace output for pastebin
<hazmat> niemeyer, https://pastebin.canonical.com/51834/
<niemeyer> hazmat: Super
<niemeyer> hazmat: 23695 connect(7, {sa_family=AF_FILE, path="/tmp/tmux-0/default"}, 21) = 0
<niemeyer> hazmat: That's what it's waiting on
<hazmat> somethings wonkey.. i started killing off the new-session processes that are hanging now i'm a tmux refresh loop
<niemeyer> hazmat: I'm connecting some dots here
<negronjl> hazmat, niemeyer: fwiw....I think byobu is installed and enabled by default on oneiric.. not sure how it could affect what's going on but, worth mentioning
<hazmat> niemeyer, so the byobu session was still active actually in the existing session after the conf file
<niemeyer> hazmat: Hmm
<niemeyer> hazmat: Makes sense
<niemeyer> negronjl: Cool.. we still have to source the file I think, but from hazmat's comment we were actually still with it
<niemeyer> hazmat: I'm starting to suspect this may actually be a new internal race too
<niemeyer> hazmat: Worth testing without byobu
<hazmat> niemeyer, in progress
<niemeyer> hazmat: If it happens again, try to see what are the initial tmux processes that are active 
<niemeyer> hazmat: It looks like a lock is being left acquired
<niemeyer> hazmat: I have a workaround in mind, but we should probably try to understand a bit better what's going on before that 
<niemeyer> hazmat: I read it wrong.. it's not waiting for a lock.. it's waiting for data
<niemeyer> Makes it yet more strange
<hazmat> hmm. ah i see what negronjl was saying.. oneiric defaults you into a byobu screen shell
<niemeyer> hazmat: How do you mean?
<hazmat> ssh ec2-instance.. -> byobu shell
<hazmat> ec2 is being slow today..
<hazmat> at instance creation for me
<_mup_> ensemble/fix-functional-testing r332 committed by jim.baker@canonical.com
<_mup_> Fixes for a variety of bitrot in ftests
<niemeyer> hazmat: Wow, really?
<niemeyer> hazmat: So we get straight into a screen?
<hazmat> niemeyer, yeah.
<hazmat> on debug-hooks into a new machine "To launch in a nested session, run: byobu" is at the top
<niemeyer> hazmat: Oh my
<niemeyer> hazmat: Well, wait.. but that's because we're within a tmux session, right?
<hazmat> it is in that case
<niemeyer> hazmat: What does CTRL-A-C do?
<hazmat> new window
<niemeyer> hazmat: Does it create something into the internal tmux, or in an outside screen?
<hazmat> its in tmux
<hazmat> i think
<niemeyer> hazmat: Ok, so there must be no outside screen
<niemeyer> hazmat: Or tmux
<niemeyer> hazmat: Isn't that simply reflecting the fact we're reading the byobu conf?
<hazmat> niemeyer, we're using the old conf, not sourcing byobu
<hazmat> for tmux
<niemeyer> hazmat: So where is byobu being loaded in? I don't get it..
<niemeyer> kirkland: ping
<negronjl> niemeyer:  byobu is being loaded from $HOME/.profile
<hazmat> hmm. there both tmux sessions
<niemeyer> negronjl: Aha
<niemeyer> Ok
<niemeyer> hazmat: Hm?
<hazmat> oh.. i'm on different machines nevermind
<hazmat> okay debug-hooks goes into tmux with the proper config
<hazmat> and still hangs
<hazmat> https://pastebin.canonical.com/51835/
<niemeyer> WOah
<niemeyer> SO, how come
<niemeyer> hazmat: Is that the initial state?
<hazmat> niemeyer, that's debug-hooks into a new machine, service unit finishes install, and tries to execute config-changed
<hazmat> otherwise clean
<niemeyer> hazmat: I don't understand how there are two pts's there
<niemeyer> hazmat: Ah, I get it I think
<niemeyer> hazmat: What does tmux list-sessions show?
<hazmat> mysql/0: 2 windows (created Fri Aug 26 22:01:05 2011) [168x36] (attached)
<niemeyer> hazmat: The pstree has one tmux
<niemeyer> hazmat: The ps auxw has three..
<niemeyer> hazmat: Sorry, my mistake
<hazmat> negronjl, looks like byobu is activated by  /etc/profile.d/Z98-byobu.sh 
<niemeyer> hazmat: Seriously, there are two sessions somehow :/
<kirkland> hazmat: right, that's the package-enabled way
<negronjl> hazmat:  you're right....sorry for the misinformation.  I checked MY box and not the oneiric instance.
<kirkland> hazmat: there's also a per-user method
<niemeyer> hazmat: Can you get a list of files that each of those tmux new-session processes are using?
<niemeyer> hazmat: lsof -p 1634
<niemeyer> hazmat: lsof -p 3667
<niemeyer> hazmat?
<hazmat> niemeyer, 1634 -> https://pastebin.canonical.com/51837/ 3667 -> https://pastebin.canonical.com/51836/   tmux-attach / 1636 -> https://pastebin.canonical.com/51838/
<niemeyer> hazmat: Is there more than one /tmp/tmux-* dir?
<hazmat> what's that command to install someones launchpad key on a machine?
<hazmat> niemeyer, one tmux dir 
<hazmat> ssh-import-lp-id
<hazmat> niemeyer, ubuntu@ec2-174-129-118-179.compute-1.amazonaws.com
<niemeyer> hazmat: Cheers!
<niemeyer> kirkland: This is starting to feel a bit too intrusive
<niemeyer> ubuntu@ip-10-46-54-97:/tmp$ sudo su -
<niemeyer> To launch in a nested session, run: byobu
<niemeyer> root@ip-10-46-54-97:~#
<kirkland> niemeyer: right, do you want a nested session, or not?
<niemeyer> kirkland: I want to run sudo
<kirkland> niemeyer: then run sudo
<niemeyer> kirkland: I did.. and the system tells me completely unrelated facts all the time
<kirkland> niemeyer: what are "unrelated facts"
<niemeyer> kirkland: I've just pasted above
<SpamapS> /usr/lib/pymodules/python2.6/ensemble/providers/ec2/files.py:8: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
<SpamapS> Is there a way to suppress that?
<kirkland> niemeyer: the fact you're speaking of being, "To launch in a nested session, run: byobu"
<kirkland> ?
<niemeyer> kirkland: Getting output from "sudo su -" every time telling me to run something else is a bit obnoxious
<niemeyer> SpamapS: Ugh.. yeah, we should follow the advice :)
<niemeyer> hazmat: Was that a clean run?
<niemeyer> hazmat: There are 4 different tmp directories in /tmp
<niemeyer> hazmat: With hook data, that is
<hazmat> niemeyer, outside of me killing processes yes ;-)
<niemeyer> Oh man :)
<hazmat> niemeyer, its pretty clean though, those are the original processes competing
<SpamapS> a good, clean kill
<kirkland> niemeyer: that's a one liner to comment out or remove, file a bug and i'll fix it
<kirkland> ./usr/bin/byobu-launcher:                       printf "$(gettext 'To launch in a nested session, run: byobu')\n"
<kirkland> niemeyer: i put that there at the request of users, however
<niemeyer> kirkland: Probably of users of byobu.. I'm concerned about people that are used to using sudo
<kirkland> niemeyer: does it do that if you do "sudo true" ?
<kirkland> niemeyer: in my experience, "no, it does not";  it does that when it establishes a new login shell, a la 'su -'
<niemeyer> kirkland: No, that doesn't load the env
<kirkland> niemeyer: right
<kirkland> niemeyer: fine.  if you file a bug at https://bugs.launchpad.net/ubuntu/+source/byobu/+filebug, I'll silence that now
<niemeyer> kirkland: Cheers!
<kirkland> niemeyer: cheers
<kirkland> niemeyer: paste me the bug number and i'll commit and push
<niemeyer> kirkland: #835130
<_mup_> Bug #835130: byobu is too verbose <byobu (Ubuntu):New> < https://launchpad.net/bugs/835130 >
<kirkland> niemeyer: Committed revision 1622.                                                                                                                     
<kirkland> niemeyer: thanks for playing ;-)
<niemeyer> kirkland: Interestingly, users seem to have asked the feature because it was even more disruptive to the workflow
<niemeyer> kirkland: #747649
<_mup_> Bug #747649: opening a nested session from byobu to byobu should not ask, but rather should remind the user... <byobu (Ubuntu):Fix Released by kirkland> < https://launchpad.net/bugs/747649 >
<niemeyer> kirkland: This kind of tool should really not kick in before the user decides to use, IMO
<kirkland> niemeyer: it's silently exiting now, as of this commit
<niemeyer> kirkland: Have you seen this one:
<niemeyer> ERROR: [/home/ubuntu/.byobu] is not writable by the current user
<niemeyer> ubuntu@ip-10-46-54-97:~$ 
<kirkland> niemeyer: yes
<niemeyer> kirkland: Supa
<kirkland> niemeyer: this happens if you sudo byobu
<kirkland> niemeyer: or run byobu while under sudo
<kirkland> niemeyer: the byobu directory gets created under the root user
<kirkland> niemeyer: rather than the $SUDO_USER
<niemeyer> kirkland: Yeah, I understand why it happened
<niemeyer> Now I just have to understand tmux
<kirkland> niemeyer: let me see if I can come up with a suitable fix to that right quick ....
<niemeyer> hazmat: I have little doubts that this is a race introduced in tmux
<niemeyer> hazmat: Oh ho ho
<niemeyer> hazmat: 
<niemeyer> # tmux new-session -d -s mysql/0 < /dev/null 2> /dev/null
<niemeyer> ^C^\Quit (core dumped)
 * SpamapS wonders if core is feeling vulnerable and lonely after being dumped..
<hazmat> niemeyer, hmm
<hazmat> so what are options..
<hazmat> SpamapS, probably just lazing around getting fat ;-)
<niemeyer> hazmat: 1) Fixing the bug in tmux; 2) Changing the logic so it doesn't hit the bug
<hazmat> niemeyer, the race seems odd, there are seconds between them
<niemeyer> hazmat: It's not even a race
<niemeyer> hazmat: Plain bug :(
<hazmat> niemeyer, back to screen :-)
<kirkland> SpamapS: probably relieved, stomachs feeling much better
<niemeyer> hazmat: :-)
<niemeyer> hazmat: Let's use dash instead
<niemeyer> kirkland: You're in oneiric right?
<kirkland> niemeyer: yup
<niemeyer> kirkland: Would you mind to help us with a quick test
<kirkland> niemeyer: i've got a fix for your ownerships issue;  file another bug and i'll commit
<kirkland> niemeyer: sure, whaddaya need?
<niemeyer> kirkland: tmux new-session -d -s foo
<niemeyer> kirkland: Run that a couple of times in different non-nested terminals
<kirkland> Nicke: k
<kirkland> kirkland@x201:~$ tmux new-session -d -s foo
<kirkland> kirkland@x201:~$ tmux new-session -d -s foo
<kirkland> duplicate session: foo
<niemeyer> kirkland: Should blow up the second time with duplicated session error
<niemeyer> Cool
<niemeyer> kirkland: Now, one more
<niemeyer> kirkland: tmux new-session -d -s foo 2> /dev/null
<kirkland> kirkland@x201:~$ tmux new-session -d -s foo 2> /dev/null
<kirkland> ......
<kirkland> <hangs>
<niemeyer> Man.. I'm getting out of this software business
<niemeyer> kirkland: Thanks!
<kirkland> niemeyer: np
<kirkland> niemeyer: checking my $SUDO_USER fix with the security team
<niemeyer> kirkland: Thanks man
<hazmat> niemeyer, http://sourceforge.net/mailarchive/forum.php?thread_name=20110826161855.GC13865%40smoon&forum_name=tmux-users
<niemeyer> hazmat: WOAH!
<niemeyer> hazmat: Sadly, the workaround there doesn't work
<kirkland> niemeyer: security team says: <kees> run sudo with -H :)
<niemeyer> kirkland: Heh
<kirkland> niemeyer: i'm negotiating with him ;-)
<niemeyer> hazmat: Let's make sure we don't do anything absurd with tmux like redirecting its output to /dev/null
<niemeyer> hazmat: tmux new-session -d -s mysql/0 2>&1 | cat > /dev/null
<niemeyer> !!!
<niemeyer> negronjl: So.. there's a bug in tmux that we're hitting.  We should have a solution in a moment
<negronjl> niemeyer:  no worries.  I've been monitoring this :)
<kirkland> Nicke: do you have a bug for the $SUDO_USER issue?
<kirkland> niemeyer: ^
<niemeyer> kirkland: Nope, haven't created one for that
<SpamapS> hrm.. debug-log seems to not be as useful as it once was
<kirkland> niemeyer: 835152
<kirkland> niemeyer: i created it for you
<niemeyer> kirkland: Thanks!
<hazmat> niemeyer, yeah.. works for me
<niemeyer> hazmat, kirkland, negronjl: http://paste.ubuntu.com/675631
<hazmat> niemeyer, http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=609444
<hazmat> niemeyer, that looks fine minus all the formula/* stuff
<kirkland> +tmux new-session -d -s {unit_name} 2>&1 | cat > /dev/null || true
<kirkland> wowzer
#ubuntu-ensemble 2011-08-27
<niemeyer> Haha
<niemeyer> hazmat: Sorry about that
<niemeyer> :)
<hazmat> kirkland, yeah.. its something in libevent/epoll handling
<niemeyer> kirkland: Blame linux, apparently :-(
<SpamapS> DOH
<SpamapS> ensemble depends on ssh
<SpamapS> gotta love what testing in clean chroots does
<_mup_> ensemble/trunk r333 committed by gustavo@niemeyer.net
<_mup_> Workaround lock up bug in tmux/epoll/whatever. [r=hazmat]
<_mup_> Thanks a lot to Kapil and Juan for debugging this issue for hours.
<niemeyer> negronjl: Just asked for a build in the ppa
<kirkland> niemeyer: et al: I'm outta here... talk to you folks later
<kirkland> niemeyer: hit me with any others you hit
<kirkland> later
<niemeyer> kirkland: Thanks for your help
<SpamapS> hm.. trying to figure out the best way to include AWS creds on this jenkins box without giving away everything.
<niemeyer> kirkland: Have a good EOW :)
<negronjl> niemeyer: thx.  I'll work with it when I see it.  Thank you and hazmat and kirkland for working on this.
<niemeyer> negronjl: My pleasure
<niemeyer> negronjl: I'm glad we managed to find it
<negronjl> niemeyer: me too :)
<niemeyer> It's certainly going towards the obscure side :-) 
<negronjl> heheh....ensemble bugs are going towards the dark side ...lol
<niemeyer> Worse than the problem was a friend asking on G+ "Why would you redirect stderr to /dev/null?"
<negronjl> the more and more people start playing with ensemble's source, you'll start getting these kinds of questions more and more.... be ready :P
<niemeyer> negronjl: Hehe :)
<hazmat> niemeyer, afaics it is a bug in tmux, i don't really see any concrete pointers to why they think its libevent/epoll
<niemeyer> hazmat: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=631984
<hazmat> niemeyer, not very concrete, the author says there is a bug.. its not reported/linked to the bug anywhere..
<niemeyer> hazmat: There's evidence that using select rather than epoll works
<hazmat> fair enough... niemeyer have a good weekend
<niemeyer> hazmat: Thanks dude, have a good one
<_mup_> ensemble/fix-functional-testing r333 committed by jim.baker@canonical.com
<_mup_> More robust test_bootstrap_and_connect testing
<_mup_> ensemble/expose-cleanup r332 committed by jim.baker@canonical.com
<_mup_> Delete security groups in the background
<_mup_> ensemble/expose-cleanup r333 committed by jim.baker@canonical.com
<_mup_> Removed debugging
<_mup_> ensemble/go-formulas r7 committed by gustavo@niemeyer.net
<_mup_> Implemented initial metadata.yaml reading and parsing.
<_mup_> Bug #835479 was filed: Parsing metadata.yaml is needed in Go <Ensemble:New> < https://launchpad.net/bugs/835479 >
#ubuntu-ensemble 2011-08-28
<_mup_> ensemble/expose-cleanup r334 committed by jim.baker@canonical.com
<_mup_> Minor cleanup
<_mup_> ensemble/expose-cleanup r335 committed by jim.baker@canonical.com
<_mup_> Only remove env security group in context of provider destroy_environment
<_mup_> ensemble/expose-cleanup r336 committed by jim.baker@canonical.com
<_mup_> Reworked remaining mocks
<botchagalupe> Is there a way to pre-seed the environments.yaml before you run the ensamble command.  I am trying to great a simple #chef recipe and I want a way to set some of the key/values in the file.
