/srv/irclogs.ubuntu.com/2011/08/23/#ubuntu-ensemble.txt

jimbakerniemeyer, re point #4 in your review of cobbler-zk-connect-error-message, we have been using RST in docstrings for a while. not consistently of course. but this did address a review point i made in what this branch is stacked on00:14
jimbakerof course it would be nice to use sphinx to generate api docs00:14
niemeyerjimbaker: The format used by William there is not any format I personally recognize00:16
niemeyerjimbaker: It looks very much like the syntax pointed out by hazmat, but not quite00:17
niemeyerjimbaker: I'm personally happy with anything, though00:17
niemeyerjimbaker: Let's just agree on a standard and move on with it00:17
jimbakerniemeyer, sure, i guess if it's slightly tweaked it would conform00:17
niemeyerjimbaker: Interestingly, in Go there's actually no special syntax00:18
niemeyerjimbaker: Besides a convention to start the sentence with the function name itself + verb..00:18
jimbakerniemeyer, i think that's the point of rst docs is to move in that direction00:18
niemeyerjimbaker: Works quite well00:18
jimbakerniemeyer, albeit w/ a small amount of markup compared to go. but not as rigid as the javadoc/epydoc style00:19
jimbakeranyway, as i understand it, we agree on this point, so that's good :)00:19
niemeyerjimbaker: Right, I don't mind the touch of RST in Python, even though I wouldn't do it in Go as the standard tools do not use it00:19
niemeyerjimbaker: +100:19
sidneiso is there a puppet formula yet? /me hides00:39
hazmatsidnei, nope01:18
hazmatsidnei, notionally a puppet formula would be a colo located service, the implementation of which is post oneiric01:19
sidneihazmat, ah, right. makes sense.01:36
_mup_ensemble/stack-crack r327 committed by kapil.thangavelu@canonical.com01:37
_mup_less granular s3 file storage error checking01:37
h0lyRShey guys... is it normal that a bootstrap instance is being created without a key-pair?04:04
h0lyRSi think i just found the answer here: http://irclogs.ubuntu.com/2011/08/02/%23ubuntu-ensemble.html04:46
siwoshi.07:05
siwosanyone using ensemble with openstack?07:05
siwosI keep getting this message with openstack07:34
siwosERROR Error Message: AWS authorization header is invalid.  Expected AwsAccessKeyId:signature07:34
siwosanyone?07:34
siwosplease - note the fact I run only computing part of openstack right now07:37
_mup_Bug #831805 was filed: wildly inconsistent docstring style <Ensemble:New> < https://launchpad.net/bugs/831805 >08:00
_mup_Bug #831883 was filed: weak error message testing <Ensemble:New for fwereade> < https://launchpad.net/bugs/831883 >10:02
hazmatsiwos, its a known issue atm, i've got some work in-progress for openstack compatibility.11:46
siwoshazmat - thanks for clarifying this ;-)11:47
siwosI keep an eye on this project ;-)11:47
hazmatsiwos, if you really want to try it, i can give some instructions, but its using a few branches atm.11:47
hazmatsiwos, out of curiosity, are your instances routable from the machine your using the ensemble cli on?11:48
hazmatin your openstack setup11:48
siwoswell - I am stress testing openstack compute (over 100 vm-s currently running) - so definitely I am interested in orchestrating them ;-)11:49
siwosinstances are routable11:49
siwosI've gote ensemble running on my api node11:49
hazmatsiwos, cool, that should work well11:49
siwosif you find some time - just tell me what to do ;-)11:50
hazmatsiwos, i can't really provide any help on it, i'm trying to get into the ppa for next week but here are the instructions if you want to try  it out. http://pastebin.ubuntu.com/673083/11:58
siwosgreat - thanks! I'll try to get this running11:59
=== kim0_ is now known as kim0
niemeyerHello all12:20
fwereadeniemeyer: morning/afternoon12:22
=== otubo[AFK] is now known as otubo
niemeyerfwereade: Hey!12:27
hazmatfwereade, g'afternoon12:28
fwereadeniemeyer, hazmat: hey guys :) how's life?12:29
hazmatpeachy :-)12:29
niemeyerhazmat: How's the LXC work coming?12:30
hazmatniemeyer, just getting into it, i had a talk with ben yesterday to catch up, i'm starting in on the lxc-provider today.. wanted to try and get the openstack branch (stack-crack) cleaned up and into the review queue as well12:35
niemeyerhazmat: Ok.. I'm a bit concerned because things have been on the quiet side12:37
hazmatniemeyer, its going to be tight, but things will heat up this week, and i can give a more accurate assessment of concern.. afaics both openstack and lxc are top level priorities for the release12:39
niemeyerhazmat: Sounds good, thanks12:40
niemeyerhazmat: They are top priorities, and we have to try to coordinate efforts12:40
fwereadeniemeyer: speaking of review queues... cobbler-shutdown has apparently not been touched in the week since you approved it; should I be picking on people and hassling them myself, or...?12:40
niemeyerhazmat: My original plan was to have jimbaker pushing the OpenStack fixes12:41
niemeyerhazmat: But your work on it is hugely appreciated12:41
niemeyerhazmat: The problem is just that it means the other side is less covered12:41
hazmatniemeyer, yeah.. i know i asked him about it, but then i realized i had told spamas i'd try to get some time on it last week.. and then i got it working ;-)12:41
niemeyerfwereade: Yes, that's always a good idea :)12:41
niemeyerhazmat: Right.. and we haven't heard much from Jim on the topic, so can't blame you12:42
niemeyerjimbaker: Haven't heard much on the topic of testing either.. have to catch up with you12:42
hazmatniemeyer, i was inspired by robbie's JFDI for the web stuff12:42
* hazmat likes JFDI12:43
hazmatcan i get some more ;-)12:43
fwereadeyeah, it pleases me that there's a somewhat widely understood acronym for it12:43
niemeyerhazmat: Yes, JFD the local development feature now, please :-)12:47
niemeyerMeanwhile, I have to JFD the store12:47
hazmatniemeyer, so we don't want to have a separate package for local dev?12:48
niemeyerhazmat: Not unless necessary12:49
hazmati'll need to apt-get install zk and jdk in bootstrap.. to be sure there present otherwise afaics 12:49
hazmatniemeyer, the notion is zk and machine agent are running on the host12:50
hazmatoffset port for zk12:50
niemeyerhazmat: zk is already packaged.. machine agent is already packaged too12:50
hazmatniemeyer, ? packaged != installed on the client side12:50
niemeyerhazmat: Sorry, I'm still missing some context.. installed on the client side == apt-get install ensemble12:51
hazmatniemeyer, right that doesn't install zk locally, just the admin client tools.12:51
hazmatniemeyer, for local dev we need zk locally in addition to the admin client tools12:51
niemeyerhazmat: apt-get install zookeeper installs zk..12:51
hazmatniemeyer, right.. but if i have the client tools, i don't know if "apt-get install zookeeper" equivalent has been done or not12:52
hazmati guess i can detect it.. the question is do i do it for the user or ask them to do it12:52
niemeyerhazmat: I'd just try to run it as usual, and report an error if the command isn't found12:53
hazmati'm also going to leave off debootstrap progress reporting, the additional lxc template script layer would need some changes, bcsaller this might make a nice optional thing in the future, if we can get the lxc-template to take it as an optional parameter when debootstraping, we can get back a progress stream from it ala https://github.com/jolicloud/base-installer (see documentation at bottom of readme)13:01
RoAkSoAxfwereade: ping13:40
fwereadeRoAkSoAx: pong13:41
RoAkSoAxfwereade: howdy!! how's it going man13:41
fwereadeRoAkSoAx: pretty good thanks, and you?13:41
RoAkSoAxfwereade: pretty good13:41
fwereadeRoAkSoAx: any joy on the installer?13:41
RoAkSoAxfwereade: oh yeah!13:42
_mup_Bug #832041 was filed: orchestra FileStorage can't handle unicode paths <Ensemble:New> < https://launchpad.net/bugs/832041 >14:21
_mup_Bug #832043 was filed: can't deploy on orchestra <Ensemble:New for fwereade> < https://launchpad.net/bugs/832043 >14:25
hazmatbcsaller, re lxc-lib lxc-ls has different output depending on container status15:12
bcsallerhazmat: lxc_ls isn't part of the public api anyway15:17
=== otubo is now known as otubo[AFK]
hazmatbcsaller, it needs to be15:18
bcsallerwhy? a set of container objects isn't enough? there could easily be containers unrelated to ensemble returned by lxc-ls15:19
hazmatbcsaller, to satsify the get_machines interface of the provider15:20
hazmatbcsaller, i'm tracking which machines belong to ensemble by recording name on start15:20
bcsallerhazmat: so its not ls so much as container status?15:21
bcsallercontainer.running might be what you want15:21
jimbakerniemeyer, hi, you wanted to catch up?15:22
hazmatbcsaller, that's not in the published version.. and i want to get all the running containers. 15:23
niemeyerjimbaker: I wrote a message to the list about it.. can you please answer it there?15:23
hazmatbcsaller, did you push the lib again?15:23
jimbakercool, i was just catching up on emails etc15:23
hazmati branched yesterday15:23
bcsallerhazmat: I'll push a new version this morning, I expect within the hour15:23
niemeyerHmm.. interesting.. in Go we don't need the schema library to _coerce_ the values.15:30
niemeyerWe need it for validation purposes only15:30
niemeyerSince the yaml/json support will automatically coerce values as necessary for actual use15:31
niemeyerI guess I'll just mimic the implementation even then.. it won't really hurt to have coercion working15:32
niemeyerBut now, I'll have lunch instead15:34
niemeyerbiab15:34
niemeyerwrtp: Hey, btw :)15:34
=== m_3_ is now known as m_3
=== daker is now known as daker_
wrtpniemeyer: hiya!16:09
fwereadelater all, I'll try to pop back on later but no guarantees16:21
=== otubo[AFK] is now known as otubo
_mup_Bug #832183 was filed: HA - Ensemble needs to handle a failed bootstrap. <Ensemble:New> < https://launchpad.net/bugs/832183 >17:02
bcsallerhazmat: can you pull from the branch again,  the container stuff is more inline with what we talked about17:11
hazmatbcsaller, cool, just doing  a review, i'll pull shortly17:11
bcsallerhazmat: I'll fetch some coffee then, I'll be around in a few17:12
jimbakeradam_g, thanks for putting that bug in so it's not just in our heads17:39
jimbakeradam_g, i still think that we can get to that without rewriting the provisioning agent, although that's preferable for the long-term solution17:40
hazmatits a fairly simple change17:41
hazmatstructuring it so that we can use ensemble commands to expand it is what needs work17:42
jimbakerhazmat, glad to hear the concurrence so to speak17:42
jimbakerhazmat, indeed, that's part of the long-term reimpl17:42
jimbakersimply have one active provisioning agent at a time, with 1 or more on standby. just ZK to determine who is active (standard leader stuff). that should be enough17:44
niemeyerjimbaker: I don't think that's necessary17:49
niemeyerjimbaker: Original plan discussed with hazmat back then allowed for multiple implementations working in parallel17:49
jimbakerniemeyer, sure, just my opinion here17:49
jimbakerback to functional testing stuff ;)17:49
niemeyerjimbaker: Sure, just mine there17:49
hazmatjimbaker, yeah just a queue with multiple providers agents active17:50
hazmatalternatively a lock structure, but a queue seems more appropriate17:51
hazmatwtf,  i think just got hit by an earthquake17:52
jimbakerhazmat, in dc?17:52
hazmatjimbaker, yeah.. big one.. house rattled pretty heavy17:52
niemeyerhazmat: Woah!17:52
jimbakeri do remember an earthquake in providence rhode island, but it was remarkable only because it was perceptible17:53
hazmati got some broken glass to cleanup, bbiab17:55
jimbakeri got a stomach to take care of, biab17:55
niemeyerhazmat: Ouch.. is everyone ok there?17:55
bcsallerhazmat: looks like they estimate a 5.8, thats a pretty good sized one17:55
hazmatniemeyer, everyone on the block seems okay, not sure about folks on metro, sirens are starting17:57
niemeyerhazmat: :-(17:57
hazmatbcsaller, i moved out sf to get away from the earthquakes ;-)18:00
hazmatall phone lines jammed. evac in progress on pentagon and capital18:01
niemeyerhazmat: Ugh.. everyone calling their loved ones18:02
hazmatindeed18:02
bcsallerpeople could feel it all the way in NYC18:06
hazmateverything seems okay, some structural damage, no major injury scenes per the police scanner18:09
jimbakerreminds me of when i was in an earthquake in seattle in 2001 iirc18:10
SpamapShttp://earthquake.usgs.gov/earthquakes/recenteqsus/Maps/US10/32.42.-85.-75.php18:10
SpamapS5.9, thats a real shaker!18:11
SpamapSvery shallow18:11
SpamapS1km18:11
hazmatfwereade, review in18:15
fwereadehazmat: you rock :)18:21
hazmatfwereade, you just missed the context that makes that an understatement18:22
* fwereade scrolls up...18:22
hazmatfwereade, http://news.blogs.cnn.com/2011/08/23/quake-hits-near-washington-d-c/?hpt=us_c218:22
fwereadeholy shit :)18:22
fwereadeall ok?18:22
hazmatfwereade, so far so good, i've got police scanner on in the background, mostly minor structural damage18:23
fwereadehazmat: phew18:23
fwereadehazmat: well then, let me restate: you shake violently, and cause minor structural damage, but in a good way18:24
niemeyerhazmat: Was just thinking.. a lot of people must have panicked there thinking it was something else18:32
hazmatniemeyer, dc rolls like that18:33
niemeyerI'm also slightly surprised twitter didn't fall down :-) 18:33
hazmatniemeyer, the sad part is landline and cell did. twitter is *the* source of breaking news as much as anything else18:34
niemeyerhazmat: Better tell the family to check twitter next time.. :)18:45
niemeyerHow ironic..18:46
jimbakerinteresting, we also had an earthquake here in colorado: http://www.cnn.com/2011/US/08/23/colorado.quake/19:27
=== otubo is now known as otubo[AFK]
niemeyerjimbaker: Would you mind to respond to the message in the ML?19:52
jimbakerniemeyer, i'm writing up a reply20:04
niemeyerjimbaker: Thanks20:04
jimbakerniemeyer, i haven't had a chance to work on it until now, but as i understand it, it's now my #1 priority. so that's fine20:05
niemeyerSpamapS: ping20:28
SpamapSniemeyer: pong! good afternoon!20:29
niemeyerSpamapS: Hey man!20:30
=== pfibiger` is now known as pfibiger
adam_gSpamapS: did pushing formulas to arbitrary branches ever get figured out?20:42
niemeyeradam_g: Yeah20:47
niemeyeradam_g: Should be working already20:47
SpamapSadam_g: I've tested it recently, seems to work fine.20:47
adam_goh, neat20:49
adam_gjust push to lp:principia/whatever?20:49
SpamapSyes but20:50
SpamapSmake sure its lp:~ensemble-composers/principia/oneiric/whatever/trunk *first*20:50
SpamapSIf you push your branch there , then only you will own the official branch20:50
=== otubo[AFK] is now known as otubo
adam_ghmm. ok20:51
SpamapSadam_g: it really doesn't matter too much, as we can always take over the official branch if we need to make a change.. but its better if they're just all owned by ensemble-composers20:52
adam_gSpamapS:  still seing a no such source package. does this require a bzr upgrade?20:54
adam_ghttp://paste.ubuntu.com/673370/20:54
SpamapShm20:58
niemeyerHmm20:58
niemeyerbcsaller: Wondering about the regex part of our validator 20:59
niemeyer(in the context of schema020:59
niemeyer)20:59
SpamapSadam_g: maybe20:59
SpamapS$ bzr push lp:~ensemble-composers/principia/oneiric/nova-cloud-controller/trunk20:59
SpamapSCreated new branch.  20:59
bcsallerniemeyer: what about it?21:00
niemeyerbcsaller: We have a couple of issues.. one short term, one long term21:00
niemeyerbcsaller: long term is compatibility.. if we move out of Python, the regex syntax will change21:00
bcsalleryou want something that uses a regex to validate rather than validates something is a regex21:00
m_3adam_g: I've got notes for how to create a dummy source package that gets around this... lemme dig them up21:01
bcsallerahh, right, you're porting to go... 21:01
adam_gSpamapS: hmm21:01
SpamapSm_3: shouldn't be necessary anymore tho21:01
niemeyerbcsaller: The other issue, which would affect us even in the short term, is that a Python regex is a wonderful way to DoS a system :)21:01
niemeyerbcsaller: Which means even if it was Python across the board, I can't validate them on the server21:02
bcsallerwonderful how? its used to validate the config stuff before its sent over the network now. I mean sure it could be, but its like saying an install script could fork-bomb21:02
bcsallerright now its only used in the environment and config parsing I think21:03
niemeyerbcsaller: Hmm.. true.. I guess as long as we don't apply the regex, that'd be fine21:03
SpamapSis pcre available in python? Thats pretty much the lingua franca for regexes.. and usually the faster implementation21:03
niemeyerbcsaller: It'll be an issue for GUI systems, though21:03
niemeyerbcsaller: e.g. any kind of hosted management interface21:04
niemeyerbcsaller: So it's not exactly like a fork-bombing install script21:04
niemeyerbcsaller: The regex is run in the management interface21:05
niemeyerbcsaller: Isn't it?21:05
bcsallerin that case, no, I agree, but it something we can work around21:05
bcsallerniemeyer: do golang use re2?21:05
niemeyerbcsaller: Kind of..21:06
niemeyerbcsaller: re2 was written by the authors of golang, conveniently..21:06
niemeyerbcsaller: It's being ported to Go21:06
SpamapSadam_g: what version of bzr do you have?21:06
m_3adam_g: http://pastebin.com/aZG0NQ8n they're rough and shouldn't be needed anymore...21:07
niemeyerbcsaller: I think we should restrict the syntax accepted to a more common ground regex, for the moment21:07
bcsallerniemeyer: I don't know that there is a single common regex def that we could validate in JS in a GUI, but the validator could run in a time bound thread if you're really worried about it 21:07
niemeyerbcsaller: and then open up as we understand how the future looks like21:07
bcsallerits a really strange attack vector as you need admin perms to use it 21:08
niemeyerbcsaller: It's a trivial DoS.. I can't be not-worried :)21:08
bcsallerheh, I managed :)21:09
SpamapSm_3: I just tested this though, and you shouldn't need to do a fake package anymore.. the fix landed.21:09
niemeyerbcsaller: Sure, give the address of your web site accepting Python regexes please21:09
niemeyerbcsaller: ;)21:09
bcsallerseriously, rather than parsing around and re-writing regex for a UI we don't have yet I think we can just keep a bug report21:09
niemeyerbcsaller: I'll make you more worried. :)21:09
bcsallerniemeyer: again, if the admin wants to hose the system its done21:10
niemeyerbcsaller: Hose _our_ system21:10
niemeyerbcsaller: Any _hosted_ management interface could be attacked21:10
bcsallerI assumed any web-ui to an ensemble system run in the cloud on their nodes... but I guess not21:10
m_3SpamapS: cool21:11
niemeyerbcsaller: Not necessarily, no21:11
SpamapSWhy don't we just not compile other peoples' regexs?21:11
niemeyerSpamapS: Because we have a feature right now that enables its use.. even though your statement makes more comfortable in the sense we can remove it and you'd not notice ;-)21:11
niemeyermakes me21:12
SpamapSI would have always assumed that those regexes are only used on the client.21:12
bcsallerSpamapS: currently, yes21:12
SpamapSthey fall into the same category as maintainer scripts really.. you shouldn't use them if they're not from a trusted source.21:13
adam_gSpamapS: needed to upgrade bzr, works now21:13
SpamapSadam_g: well that seems a little.. odd.21:13
niemeyerSpamapS: That's not the case.. regexes are used for validation of settings21:13
niemeyerSpamapS: any management interface, including hosted, would make use of it21:14
niemeyerSpamapS, bcsaller: My proposal, though, is not to remove them entirely, but to use a subset we know there are alternative implementations that do not suffer from the problem21:15
=== otubo is now known as otubo[AFK]
SpamapSYeah a subset would probably be best.21:15
niemeyerIt's also a sensible thing to do in general..21:15
niemeyerOtherwise we're unconsciously binding the _metadata_ of the whole project to a particular language21:16
SpamapSThe most complicated things I'd expect to see are things like   [\d,]+21:16
niemeyerSpamapS: Exactly..  this should be [0-9,]+ or similar..21:17
niemeyerWhich is compatible with most extended POSIX engines21:18
niemeyerIncluding grep21:18
bcsallerniemeyer: supposedly re2 guarantees O(length_of_regex) runtime and configurable memory-use limit21:19
niemeyerbcsaller: Yeah, it does21:19
niemeyerbcsaller: O(len of string), actually21:19
niemeyerbcsaller: But as long as it's O(n), we're goofd21:19
niemeyerLOL.. goofd is an awesome typo21:19
bcsallerthat could be good or bad :)21:19
bcsallerso maybe including that makes having to filter it a non-issue21:20
niemeyerbcsaller: It does, but I'm a little reluctant still because we'd be binding it to a particular implementation21:20
niemeyerbcsaller: My suggestion is that we restrict the regex to a common subset, and if people start claiming for more we can gradually introduce, or even go full blown re2/similar if really necessary21:21
bcsallerthis seems like a very easy place to have an opinion 21:21
niemeyerbcsaller: You're right.. let's filter.21:22
bcsallerfiltering it down using a regex? ;)21:22
niemeyerbcsaller: http://commandcenter.blogspot.com/2011/08/regular-expressions-in-lexing-and.html21:23
hazmata ui can do sensible length filtering on an input21:27
hazmatare there are circular exponential cases in under 1k?21:28
hazmatfor a dos21:28
niemeyerhazmat: Oh yeah21:28
niemeyerhazmat: Either way, it doesn't really matter.. we shouldn't be binding the _metadata_ for the whole project to Python, even if the implementation is in Python.21:29
hazmatoh.. its a formula specified regex21:30
niemeyerhazmat: Right21:31
hazmati suggest we drop it then,  unless there's a feature request for it, there's some missing coverage in that implementation (floats, missing type error handling) as is21:32
niemeyerhazmat: I find the feature interesting.. it's quite handy to do validation upfront rather than reporting errors on execution21:34
niemeyerhazmat: When that's easily doable21:34
niemeyerIs Python able to do POSIX regex easily?21:34
* niemeyer can't remember21:34
hazmatniemeyer, evaluate of an arbitrary regex against unknown input?21:35
niemeyerDoesn't look like so.. sucks21:35
niemeyerhazmat: Hm?21:35
niemeyerhazmat: Sorry, I'm not sure about what the question is?21:35
hazmatniemeyer, you mean validate an arbitrary regex against unknown input.. switching the regex language gives more options, s/pcre/posix.. we could also use re2 in python21:36
niemeyerhazmat: Yeah, posix (or extended posix, more likely) would be good21:37
niemeyerhazmat: I mean validate the user input using a regex21:38
niemeyerhazmat: Exactly like bcsaller implemented21:38
niemeyerhazmat: I think feature is a good idea.. the problem is just the complete lack of portability right now21:38
hazmatniemeyer, there's also fnmatch.py (stdlib).. translated shell regex21:38
niemeyerPython's re is unlike anything else21:38
hazmatniemeyer, its closest to pcre afaik21:39
niemeyerhazmat: That sounds too poor.. can't even differentiate digits etc21:39
_mup_Bug #832384 was filed: ensemble.state.tests.test_security.PrincipalTests.test_activate sometimes hangs <Ensemble:New> < https://launchpad.net/bugs/832384 >21:54
niemeyerOk, most of the schema stuff is in place.. just need to port some of the more complex types now.. (List and Dicts)22:16
jimbakerbcsaller, i'm sitting w/ m_3 here in boulder and he's wondering about the status of config-defaults. the branch is marked as being approved, but it's not under the needs release column in the eureka kanban22:47
jimbakeri think that's because the bug is still "new", not some other state22:48
jimbaker(or at least that's one possibility)22:48
hazmatjimbaker, yeah. i needs to be labeled in-progress at least23:12
hazmatsurprised it even showed up in the review queue without that label23:12
m_3hazmat: thanks... I'll update the bug23:17

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!