[00:14] niemeyer, re point #4 in your review of cobbler-zk-connect-error-message, we have been using RST in docstrings for a while. not consistently of course. but this did address a review point i made in what this branch is stacked on [00:14] of course it would be nice to use sphinx to generate api docs [00:16] jimbaker: The format used by William there is not any format I personally recognize [00:17] jimbaker: It looks very much like the syntax pointed out by hazmat, but not quite [00:17] jimbaker: I'm personally happy with anything, though [00:17] jimbaker: Let's just agree on a standard and move on with it [00:17] niemeyer, sure, i guess if it's slightly tweaked it would conform [00:18] jimbaker: Interestingly, in Go there's actually no special syntax [00:18] jimbaker: Besides a convention to start the sentence with the function name itself + verb.. [00:18] niemeyer, i think that's the point of rst docs is to move in that direction [00:18] jimbaker: Works quite well [00:19] niemeyer, albeit w/ a small amount of markup compared to go. but not as rigid as the javadoc/epydoc style [00:19] anyway, as i understand it, we agree on this point, so that's good :) [00:19] jimbaker: Right, I don't mind the touch of RST in Python, even though I wouldn't do it in Go as the standard tools do not use it [00:19] jimbaker: +1 [00:39] so is there a puppet formula yet? /me hides [01:18] sidnei, nope [01:19] sidnei, notionally a puppet formula would be a colo located service, the implementation of which is post oneiric [01:36] hazmat, ah, right. makes sense. [01:37] <_mup_> ensemble/stack-crack r327 committed by kapil.thangavelu@canonical.com [01:37] <_mup_> less granular s3 file storage error checking [04:04] hey guys... is it normal that a bootstrap instance is being created without a key-pair? [04:46] i think i just found the answer here: http://irclogs.ubuntu.com/2011/08/02/%23ubuntu-ensemble.html [07:05] hi. [07:05] anyone using ensemble with openstack? [07:34] I keep getting this message with openstack [07:34] ERROR Error Message: AWS authorization header is invalid. Expected AwsAccessKeyId:signature [07:34] anyone? [07:37] please - note the fact I run only computing part of openstack right now [08:00] <_mup_> Bug #831805 was filed: wildly inconsistent docstring style < https://launchpad.net/bugs/831805 > [10:02] <_mup_> Bug #831883 was filed: weak error message testing < https://launchpad.net/bugs/831883 > [11:46] siwos, its a known issue atm, i've got some work in-progress for openstack compatibility. [11:47] hazmat - thanks for clarifying this ;-) [11:47] I keep an eye on this project ;-) [11:47] siwos, if you really want to try it, i can give some instructions, but its using a few branches atm. [11:48] siwos, out of curiosity, are your instances routable from the machine your using the ensemble cli on? [11:48] in your openstack setup [11:49] well - I am stress testing openstack compute (over 100 vm-s currently running) - so definitely I am interested in orchestrating them ;-) [11:49] instances are routable [11:49] I've gote ensemble running on my api node [11:49] siwos, cool, that should work well [11:50] if you find some time - just tell me what to do ;-) [11:58] siwos, i can't really provide any help on it, i'm trying to get into the ppa for next week but here are the instructions if you want to try it out. http://pastebin.ubuntu.com/673083/ [11:59] great - thanks! I'll try to get this running === kim0_ is now known as kim0 [12:20] Hello all [12:22] niemeyer: morning/afternoon === otubo[AFK] is now known as otubo [12:27] fwereade: Hey! [12:28] fwereade, g'afternoon [12:29] niemeyer, hazmat: hey guys :) how's life? [12:29] peachy :-) [12:30] hazmat: How's the LXC work coming? [12:35] niemeyer, just getting into it, i had a talk with ben yesterday to catch up, i'm starting in on the lxc-provider today.. wanted to try and get the openstack branch (stack-crack) cleaned up and into the review queue as well [12:37] hazmat: Ok.. I'm a bit concerned because things have been on the quiet side [12:39] niemeyer, its going to be tight, but things will heat up this week, and i can give a more accurate assessment of concern.. afaics both openstack and lxc are top level priorities for the release [12:40] hazmat: Sounds good, thanks [12:40] hazmat: They are top priorities, and we have to try to coordinate efforts [12:40] niemeyer: speaking of review queues... cobbler-shutdown has apparently not been touched in the week since you approved it; should I be picking on people and hassling them myself, or...? [12:41] hazmat: My original plan was to have jimbaker pushing the OpenStack fixes [12:41] hazmat: But your work on it is hugely appreciated [12:41] hazmat: The problem is just that it means the other side is less covered [12:41] niemeyer, yeah.. i know i asked him about it, but then i realized i had told spamas i'd try to get some time on it last week.. and then i got it working ;-) [12:41] fwereade: Yes, that's always a good idea :) [12:42] hazmat: Right.. and we haven't heard much from Jim on the topic, so can't blame you [12:42] jimbaker: Haven't heard much on the topic of testing either.. have to catch up with you [12:42] niemeyer, i was inspired by robbie's JFDI for the web stuff [12:43] * hazmat likes JFDI [12:43] can i get some more ;-) [12:43] yeah, it pleases me that there's a somewhat widely understood acronym for it [12:47] hazmat: Yes, JFD the local development feature now, please :-) [12:47] Meanwhile, I have to JFD the store [12:48] niemeyer, so we don't want to have a separate package for local dev? [12:49] hazmat: Not unless necessary [12:49] i'll need to apt-get install zk and jdk in bootstrap.. to be sure there present otherwise afaics [12:50] niemeyer, the notion is zk and machine agent are running on the host [12:50] offset port for zk [12:50] hazmat: zk is already packaged.. machine agent is already packaged too [12:50] niemeyer, ? packaged != installed on the client side [12:51] hazmat: Sorry, I'm still missing some context.. installed on the client side == apt-get install ensemble [12:51] niemeyer, right that doesn't install zk locally, just the admin client tools. [12:51] niemeyer, for local dev we need zk locally in addition to the admin client tools [12:51] hazmat: apt-get install zookeeper installs zk.. [12:52] niemeyer, right.. but if i have the client tools, i don't know if "apt-get install zookeeper" equivalent has been done or not [12:52] i guess i can detect it.. the question is do i do it for the user or ask them to do it [12:53] hazmat: I'd just try to run it as usual, and report an error if the command isn't found [13:01] i'm also going to leave off debootstrap progress reporting, the additional lxc template script layer would need some changes, bcsaller this might make a nice optional thing in the future, if we can get the lxc-template to take it as an optional parameter when debootstraping, we can get back a progress stream from it ala https://github.com/jolicloud/base-installer (see documentation at bottom of readme) [13:40] fwereade: ping [13:41] RoAkSoAx: pong [13:41] fwereade: howdy!! how's it going man [13:41] RoAkSoAx: pretty good thanks, and you? [13:41] fwereade: pretty good [13:41] RoAkSoAx: any joy on the installer? [13:42] fwereade: oh yeah! [14:21] <_mup_> Bug #832041 was filed: orchestra FileStorage can't handle unicode paths < https://launchpad.net/bugs/832041 > [14:25] <_mup_> Bug #832043 was filed: can't deploy on orchestra < https://launchpad.net/bugs/832043 > [15:12] bcsaller, re lxc-lib lxc-ls has different output depending on container status [15:17] hazmat: lxc_ls isn't part of the public api anyway === otubo is now known as otubo[AFK] [15:18] bcsaller, it needs to be [15:19] why? a set of container objects isn't enough? there could easily be containers unrelated to ensemble returned by lxc-ls [15:20] bcsaller, to satsify the get_machines interface of the provider [15:20] bcsaller, i'm tracking which machines belong to ensemble by recording name on start [15:21] hazmat: so its not ls so much as container status? [15:21] container.running might be what you want [15:22] niemeyer, hi, you wanted to catch up? [15:23] bcsaller, that's not in the published version.. and i want to get all the running containers. [15:23] jimbaker: I wrote a message to the list about it.. can you please answer it there? [15:23] bcsaller, did you push the lib again? [15:23] cool, i was just catching up on emails etc [15:23] i branched yesterday [15:23] hazmat: I'll push a new version this morning, I expect within the hour [15:30] Hmm.. interesting.. in Go we don't need the schema library to _coerce_ the values. [15:30] We need it for validation purposes only [15:31] Since the yaml/json support will automatically coerce values as necessary for actual use [15:32] I guess I'll just mimic the implementation even then.. it won't really hurt to have coercion working [15:34] But now, I'll have lunch instead [15:34] biab [15:34] wrtp: Hey, btw :) === m_3_ is now known as m_3 === daker is now known as daker_ [16:09] niemeyer: hiya! [16:21] later all, I'll try to pop back on later but no guarantees === otubo[AFK] is now known as otubo [17:02] <_mup_> Bug #832183 was filed: HA - Ensemble needs to handle a failed bootstrap. < https://launchpad.net/bugs/832183 > [17:11] hazmat: can you pull from the branch again, the container stuff is more inline with what we talked about [17:11] bcsaller, cool, just doing a review, i'll pull shortly [17:12] hazmat: I'll fetch some coffee then, I'll be around in a few [17:39] adam_g, thanks for putting that bug in so it's not just in our heads [17:40] adam_g, i still think that we can get to that without rewriting the provisioning agent, although that's preferable for the long-term solution [17:41] its a fairly simple change [17:42] structuring it so that we can use ensemble commands to expand it is what needs work [17:42] hazmat, glad to hear the concurrence so to speak [17:42] hazmat, indeed, that's part of the long-term reimpl [17:44] simply have one active provisioning agent at a time, with 1 or more on standby. just ZK to determine who is active (standard leader stuff). that should be enough [17:49] jimbaker: I don't think that's necessary [17:49] jimbaker: Original plan discussed with hazmat back then allowed for multiple implementations working in parallel [17:49] niemeyer, sure, just my opinion here [17:49] back to functional testing stuff ;) [17:49] jimbaker: Sure, just mine there [17:50] jimbaker, yeah just a queue with multiple providers agents active [17:51] alternatively a lock structure, but a queue seems more appropriate [17:52] wtf, i think just got hit by an earthquake [17:52] hazmat, in dc? [17:52] jimbaker, yeah.. big one.. house rattled pretty heavy [17:52] hazmat: Woah! [17:53] i do remember an earthquake in providence rhode island, but it was remarkable only because it was perceptible [17:55] i got some broken glass to cleanup, bbiab [17:55] i got a stomach to take care of, biab [17:55] hazmat: Ouch.. is everyone ok there? [17:55] hazmat: looks like they estimate a 5.8, thats a pretty good sized one [17:57] niemeyer, everyone on the block seems okay, not sure about folks on metro, sirens are starting [17:57] hazmat: :-( [18:00] bcsaller, i moved out sf to get away from the earthquakes ;-) [18:01] all phone lines jammed. evac in progress on pentagon and capital [18:02] hazmat: Ugh.. everyone calling their loved ones [18:02] indeed [18:06] people could feel it all the way in NYC [18:09] everything seems okay, some structural damage, no major injury scenes per the police scanner [18:10] reminds me of when i was in an earthquake in seattle in 2001 iirc [18:10] http://earthquake.usgs.gov/earthquakes/recenteqsus/Maps/US10/32.42.-85.-75.php [18:11] 5.9, thats a real shaker! [18:11] very shallow [18:11] 1km [18:15] fwereade, review in [18:21] hazmat: you rock :) [18:22] fwereade, you just missed the context that makes that an understatement [18:22] * fwereade scrolls up... [18:22] fwereade, http://news.blogs.cnn.com/2011/08/23/quake-hits-near-washington-d-c/?hpt=us_c2 [18:22] holy shit :) [18:22] all ok? [18:23] fwereade, so far so good, i've got police scanner on in the background, mostly minor structural damage [18:23] hazmat: phew [18:24] hazmat: well then, let me restate: you shake violently, and cause minor structural damage, but in a good way [18:32] hazmat: Was just thinking.. a lot of people must have panicked there thinking it was something else [18:33] niemeyer, dc rolls like that [18:33] I'm also slightly surprised twitter didn't fall down :-) [18:34] niemeyer, the sad part is landline and cell did. twitter is *the* source of breaking news as much as anything else [18:45] hazmat: Better tell the family to check twitter next time.. :) [18:46] How ironic.. [19:27] interesting, we also had an earthquake here in colorado: http://www.cnn.com/2011/US/08/23/colorado.quake/ === otubo is now known as otubo[AFK] [19:52] jimbaker: Would you mind to respond to the message in the ML? [20:04] niemeyer, i'm writing up a reply [20:04] jimbaker: Thanks [20:05] niemeyer, i haven't had a chance to work on it until now, but as i understand it, it's now my #1 priority. so that's fine [20:28] SpamapS: ping [20:29] niemeyer: pong! good afternoon! [20:30] SpamapS: Hey man! === pfibiger` is now known as pfibiger [20:42] SpamapS: did pushing formulas to arbitrary branches ever get figured out? [20:47] adam_g: Yeah [20:47] adam_g: Should be working already [20:47] adam_g: I've tested it recently, seems to work fine. [20:49] oh, neat [20:49] just push to lp:principia/whatever? [20:50] yes but [20:50] make sure its lp:~ensemble-composers/principia/oneiric/whatever/trunk *first* [20:50] If you push your branch there , then only you will own the official branch === otubo[AFK] is now known as otubo [20:51] hmm. ok [20:52] adam_g: it really doesn't matter too much, as we can always take over the official branch if we need to make a change.. but its better if they're just all owned by ensemble-composers [20:54] SpamapS: still seing a no such source package. does this require a bzr upgrade? [20:54] http://paste.ubuntu.com/673370/ [20:58] hm [20:58] Hmm [20:59] bcsaller: Wondering about the regex part of our validator [20:59] (in the context of schema0 [20:59] ) [20:59] adam_g: maybe [20:59] $ bzr push lp:~ensemble-composers/principia/oneiric/nova-cloud-controller/trunk [20:59] Created new branch. [21:00] niemeyer: what about it? [21:00] bcsaller: We have a couple of issues.. one short term, one long term [21:00] bcsaller: long term is compatibility.. if we move out of Python, the regex syntax will change [21:00] you want something that uses a regex to validate rather than validates something is a regex [21:01] adam_g: I've got notes for how to create a dummy source package that gets around this... lemme dig them up [21:01] ahh, right, you're porting to go... [21:01] SpamapS: hmm [21:01] m_3: shouldn't be necessary anymore tho [21:01] bcsaller: The other issue, which would affect us even in the short term, is that a Python regex is a wonderful way to DoS a system :) [21:02] bcsaller: Which means even if it was Python across the board, I can't validate them on the server [21:02] wonderful how? its used to validate the config stuff before its sent over the network now. I mean sure it could be, but its like saying an install script could fork-bomb [21:03] right now its only used in the environment and config parsing I think [21:03] bcsaller: Hmm.. true.. I guess as long as we don't apply the regex, that'd be fine [21:03] is pcre available in python? Thats pretty much the lingua franca for regexes.. and usually the faster implementation [21:03] bcsaller: It'll be an issue for GUI systems, though [21:04] bcsaller: e.g. any kind of hosted management interface [21:04] bcsaller: So it's not exactly like a fork-bombing install script [21:05] bcsaller: The regex is run in the management interface [21:05] bcsaller: Isn't it? [21:05] in that case, no, I agree, but it something we can work around [21:05] niemeyer: do golang use re2? [21:06] bcsaller: Kind of.. [21:06] bcsaller: re2 was written by the authors of golang, conveniently.. [21:06] bcsaller: It's being ported to Go [21:06] adam_g: what version of bzr do you have? [21:07] adam_g: http://pastebin.com/aZG0NQ8n they're rough and shouldn't be needed anymore... [21:07] bcsaller: I think we should restrict the syntax accepted to a more common ground regex, for the moment [21:07] niemeyer: I don't know that there is a single common regex def that we could validate in JS in a GUI, but the validator could run in a time bound thread if you're really worried about it [21:07] bcsaller: and then open up as we understand how the future looks like [21:08] its a really strange attack vector as you need admin perms to use it [21:08] bcsaller: It's a trivial DoS.. I can't be not-worried :) [21:09] heh, I managed :) [21:09] m_3: I just tested this though, and you shouldn't need to do a fake package anymore.. the fix landed. [21:09] bcsaller: Sure, give the address of your web site accepting Python regexes please [21:09] bcsaller: ;) [21:09] seriously, rather than parsing around and re-writing regex for a UI we don't have yet I think we can just keep a bug report [21:09] bcsaller: I'll make you more worried. :) [21:10] niemeyer: again, if the admin wants to hose the system its done [21:10] bcsaller: Hose _our_ system [21:10] bcsaller: Any _hosted_ management interface could be attacked [21:10] I assumed any web-ui to an ensemble system run in the cloud on their nodes... but I guess not [21:11] SpamapS: cool [21:11] bcsaller: Not necessarily, no [21:11] Why don't we just not compile other peoples' regexs? [21:11] SpamapS: Because we have a feature right now that enables its use.. even though your statement makes more comfortable in the sense we can remove it and you'd not notice ;-) [21:12] makes me [21:12] I would have always assumed that those regexes are only used on the client. [21:12] SpamapS: currently, yes [21:13] they fall into the same category as maintainer scripts really.. you shouldn't use them if they're not from a trusted source. [21:13] SpamapS: needed to upgrade bzr, works now [21:13] adam_g: well that seems a little.. odd. [21:13] SpamapS: That's not the case.. regexes are used for validation of settings [21:14] SpamapS: any management interface, including hosted, would make use of it [21:15] SpamapS, bcsaller: My proposal, though, is not to remove them entirely, but to use a subset we know there are alternative implementations that do not suffer from the problem === otubo is now known as otubo[AFK] [21:15] Yeah a subset would probably be best. [21:15] It's also a sensible thing to do in general.. [21:16] Otherwise we're unconsciously binding the _metadata_ of the whole project to a particular language [21:16] The most complicated things I'd expect to see are things like [\d,]+ [21:17] SpamapS: Exactly.. this should be [0-9,]+ or similar.. [21:18] Which is compatible with most extended POSIX engines [21:18] Including grep [21:19] niemeyer: supposedly re2 guarantees O(length_of_regex) runtime and configurable memory-use limit [21:19] bcsaller: Yeah, it does [21:19] bcsaller: O(len of string), actually [21:19] bcsaller: But as long as it's O(n), we're goofd [21:19] LOL.. goofd is an awesome typo [21:19] that could be good or bad :) [21:20] so maybe including that makes having to filter it a non-issue [21:20] bcsaller: It does, but I'm a little reluctant still because we'd be binding it to a particular implementation [21:21] bcsaller: My suggestion is that we restrict the regex to a common subset, and if people start claiming for more we can gradually introduce, or even go full blown re2/similar if really necessary [21:21] this seems like a very easy place to have an opinion [21:22] bcsaller: You're right.. let's filter. [21:22] filtering it down using a regex? ;) [21:23] bcsaller: http://commandcenter.blogspot.com/2011/08/regular-expressions-in-lexing-and.html [21:27] a ui can do sensible length filtering on an input [21:28] are there are circular exponential cases in under 1k? [21:28] for a dos [21:28] hazmat: Oh yeah [21:29] hazmat: Either way, it doesn't really matter.. we shouldn't be binding the _metadata_ for the whole project to Python, even if the implementation is in Python. [21:30] oh.. its a formula specified regex [21:31] hazmat: Right [21:32] i suggest we drop it then, unless there's a feature request for it, there's some missing coverage in that implementation (floats, missing type error handling) as is [21:34] hazmat: I find the feature interesting.. it's quite handy to do validation upfront rather than reporting errors on execution [21:34] hazmat: When that's easily doable [21:34] Is Python able to do POSIX regex easily? [21:34] * niemeyer can't remember [21:35] niemeyer, evaluate of an arbitrary regex against unknown input? [21:35] Doesn't look like so.. sucks [21:35] hazmat: Hm? [21:35] hazmat: Sorry, I'm not sure about what the question is? [21:36] niemeyer, you mean validate an arbitrary regex against unknown input.. switching the regex language gives more options, s/pcre/posix.. we could also use re2 in python [21:37] hazmat: Yeah, posix (or extended posix, more likely) would be good [21:38] hazmat: I mean validate the user input using a regex [21:38] hazmat: Exactly like bcsaller implemented [21:38] hazmat: I think feature is a good idea.. the problem is just the complete lack of portability right now [21:38] niemeyer, there's also fnmatch.py (stdlib).. translated shell regex [21:38] Python's re is unlike anything else [21:39] niemeyer, its closest to pcre afaik [21:39] hazmat: That sounds too poor.. can't even differentiate digits etc [21:54] <_mup_> Bug #832384 was filed: ensemble.state.tests.test_security.PrincipalTests.test_activate sometimes hangs < https://launchpad.net/bugs/832384 > [22:16] Ok, most of the schema stuff is in place.. just need to port some of the more complex types now.. (List and Dicts) [22:47] bcsaller, i'm sitting w/ m_3 here in boulder and he's wondering about the status of config-defaults. the branch is marked as being approved, but it's not under the needs release column in the eureka kanban [22:48] i think that's because the bug is still "new", not some other state [22:48] (or at least that's one possibility) [23:12] jimbaker, yeah. i needs to be labeled in-progress at least [23:12] surprised it even showed up in the review queue without that label [23:17] hazmat: thanks... I'll update the bug