[00:02] hazmat: quite common [00:02] hazmat: it actually will come back.. you have to wait for whatever giant operation (probably an image upload) completes [00:02] * SpamapS decides its time to call it a day. [00:26] bcsaller1, is the lxc-lib branch ready to go back into review? [00:35] hazmat: I think I can mark it again, yeah [00:37] <_mup_> ensemble/local-ubuntu-provider r331 committed by kapil.thangavelu@canonical.com [00:37] <_mup_> classify managed zk, add missing test package module [01:17] <_mup_> ensemble/local-ubuntu-provider r332 committed by kapil.thangavelu@canonical.com [01:17] <_mup_> refactor file storage out of dummy into common for lxc reuse [01:18] <_mup_> ensemble/local-ubuntu-provider r333 committed by kapil.thangavelu@canonical.com [01:18] <_mup_> switch lxc provider over to using providers.common.files [01:38] <_mup_> ensemble/local-ubuntu-provider r334 committed by kapil.thangavelu@canonical.com [01:38] <_mup_> local machine using /proc/uptime for launch time [13:59] niemeyer: hiya [13:59] wrtp: Hey! [14:43] niemeyer: https://code.launchpad.net/~niemeyer/ensemble/go-final-formula-meta/+merge/73304 [14:44] niemeyer: is the testrepo copied from the python one? and if so, why? [14:44] fwereade: It is.. and I don't get the question [14:45] niemeyer: there seem to be redundant formulas [14:45] niemeyer: it feels to me as if either the go and the python ones are conceptually different -- in which case we don't need the extra formulas like mysql2 [14:46] niemeyer: ...or fundamentally the same, in which case I feel nervous about the lack of any mechanism to keep them the same [14:46] niemeyer: does that make more sense? [14:46] fwereade: It does, thanks [14:47] fwereade: There's so such thing as "Go formula" and "Python formula" [14:47] fwereade: There is "Ensemble formula" only [14:47] niemeyer: indeed [14:47] fwereade: The repository there contains example Ensemble formulas for test purposes [14:47] What would be a good formula to be beta 1 QA Guinea pig? Ideally expected to work on bare metal.. OpenStack formula is too complicated for the QA team to scratch for this. [14:48] fwereade: We have two self-contained repositories.. one with a Python implementation, and one with a Go implementation [14:48] niemeyer: understood [14:48] fwereade: I'm not sure about what's your proposal.. are you suggesting we should merge both repositories in a single one? [14:49] Daviey: Hmm [14:49] SpamapS: any suggestions? [14:51] niemeyer: ...maybe -- I'm not really proposing anything so much as talking in the hope that I can feel comfortable with a verdict on the review [14:51] Daviey: anything will do.... mysql and mediawiki have the most history, but hadoop-master/slave are easy to install and more apropos to the openstack demo atm [14:52] m_3: Hmm.. mysql and mediawiki might be easier for someone that doesn't have prior experience with hadoop? [14:53] fwereade: No worries really.. I'm just explaining the problem we have [14:53] fwereade: and trying to investigate if you have anything in mind :) [14:53] niemeyer: I guess I would be happier if we had a common place for common stuff [14:54] fwereade: What's the problem you have in mind? [14:54] Daviey: sure... there're more moving parts with the mysql/mediawiki... hadoop has a little more of a mem requirement too [14:55] m_3: I'll present them as either/or then :) [14:55] thanks [14:55] niemeyer: just that someone will, sooner or later, change one but not the other, and I'm not sure what the worst possible consequences could be [14:55] Daviey, m_3: Hadoop feels sexy, if nothing else :) [14:55] Daviey: sure... hadoop's easy though... we have several blog posts about using it with ensemble in cloud.ubuntu.com [14:55] certainly has all the buzz [14:56] fwereade: That's a critical point we have to watch out for that indeed [14:56] s/for that/for/ [14:57] niemeyer / m_3: Heh.. Ok.. Someone might want to have a chat with hggdh about QA effort ongoing. :) [14:57] fwereade: The test repository feels pretty minor, though [14:57] Daviey: Superb! [14:57] fwereade: This is a real issue across the board [14:57] fwereade: We are effectively duplicating logic [14:57] niemeyer: true, but the test repo will be the second thing I approve which could drift out of sync, and I don't want that to become a habit ;) [14:57] fwereade: I'm not sure the burden of separating out into different repositories pays off [14:58] fwereade: Given that we'll still have the burden either way [14:58] fwereade: Care from coders and reviewers is necessary [14:58] fwereade: Heh.. [14:58] fwereade: Every single line in that source code can drift out of sync [14:59] fwereade: The real solution for that is to finish the port. [14:59] niemeyer: heh, yes [14:59] fwereade: After thinking some more, I also don't feel good about the suggestion of having a metadata format that is common for both [15:00] fwereade: for syncing up the schema [15:00] fwereade: Because that's _increasing_ the problem [15:00] fwereade: You're basically pushing the problem to a different level [15:00] fwereade: We'd have to develop a schema metadata format (a schema schema!) [15:00] fwereade: and a parser, etc [15:00] fwereade: and _that_ would have to stay in sync [15:01] fwereade: Doesn't feel like it'd pay off [15:01] niemeyer: well, I never proposed that :) [15:01] fwereade: I think you did? [15:01] niemeyer: not exactly [15:02] niemeyer: well, ok, I did :/ ...but I'm not convinced it actually makes the problem worse [15:02] fwereade: "A language-agnostic schema format (where we define any given schema OAOO, and load it in both Go and Python) would solve the second problem." [15:02] https://bugs.launchpad.net/ensemble/+bug/833906 [15:02] <_mup_> Bug #833906: go and python schema implementations could drift out of sync < https://launchpad.net/bugs/833906 > [15:03] fwereade: That's a schema schema [15:03] niemeyer: hmm, I wrote it badly: I mentioned that, and went on to describe something similar but less heavyweight [15:04] fwereade: The schema is in place.. formula metadata is in review [15:04] fwereade: Very small amount of code [15:04] niemeyer: just a common way of specifying tests -- ie we keep schemas native and readable, but ensure we're running the same tests for each implementation [15:05] fwereade: Diving into a significantly more complex implementation in *both* languages to solve a drift-away problem feels like going into the opposite direction [15:05] niemeyer: I don't believe that what we have is unworkable, otherwise I'd be making a lot more noise about it [15:06] niemeyer: but I'm fretting that the amount of things we need to remember to keep in sync will grow and grow [15:06] fwereade: Again, the way to avoid that is to not have duplication [15:06] fwereade: You seem to be worried about the metadata, but that's the simplest case [15:06] fwereade: Just cp -a solves it.. the real problem is in logic [15:07] fwereade: I'm also concerned for sure, but I'm more concerned about logic and thinking about approaches to solve that [15:07] niemeyer: I'm worried about the duplication, and the metadata is what set me off :) [15:08] fwereade: That's what I just said I think [15:08] fwereade: I want help on that.. how can we avoid duplication of _logic_ [15:09] niemeyer: clearly we need a third language, which will generate both go and python for us [15:09] * fwereade ducks [15:09] LOL [15:09] fwereade: The interesting thing is that this is exactly what the bug above is about ;-) [15:10] niemeyer: kind of, I don't think I've expressed myself very effectively there, I might have another go at it in a little while [15:10] niemeyer: anyway, thanks for the discussion, my nerves are soothed ;) [15:11] fwereade: :-) [15:11] fwereade: Mine are not.. I'm still concerned about duplication [15:12] fwereade: I just don't see a big deal in test data.. this can easily be copied [15:12] niemeyer: I'm still worried about that, but I'm reassured that you are too [15:12] fwereade: I've been thinking about potential ways to speed up a migration [15:13] fwereade: The real solution would really be to finish the port [15:13] fwereade: We can have intermediate wins, though [15:14] fwereade: and port bits in a way we can kill the other side [15:14] fwereade: I have also been thinking about the possibility of _integrating_ both ports (!) [15:14] fwereade: There may be a way to do it nicely [15:15] fwereade: and I want to talk to you about that at some point.. I know you've made some very interesting work on that area before [15:15] niemeyer: hmm, that does sound interesting actually [15:15] gopython ? [15:15] hazmat: Yep :) [15:15] fwereade, hazmat: http://labix.org/lunatic-python [15:16] niemeyer: nice :D [15:16] I'm still not entirely sure about it, since it might get so involved that simply porting things over could be easier/faster/simpler [15:17] But, it's an idea.. [15:18] niemeyer: > =python.eval("lua.eval('python.eval(\"lua.eval(\\'t\\')\")')") [15:18] niemeyer: you're clearly insane :) [15:18] fwereade: Yeah, it's sick, I know :-) [15:18] niemeyer: (but in a good way ;)) [15:23] niemeyer: re suggestions for a formula.. I'm particularly fond of /usr/share/principia/tests/mediawiki.sh from principia-tools. :) [15:23] SpamapS: Sounds good [15:25] for bare metal that might be too many machines tho [15:28] Daviey: re QA effort.. I've been writing automted tests for the wordpress/mysql example on EC2 and working on the mediawiki example as well. Maybe hggdb can use my scripts? [15:28] hggdh even :-P [15:29] SpamapS: -> #ubuntu-testing ? [15:29] Man.. I gotta /part some channels and close some query windows.. I'm at 106 open. [15:31] 106.. is that all? :) [15:33] <_mup_> Bug #837476 was filed: python2.6: /usr/lib/pymodules/python2.6/ensemble/providers/ec2/files.py:8: DeprecationWarning: the sha module is deprecated < https://launchpad.net/bugs/837476 > [15:52] <_mup_> ensemble/local-ubuntu-provider r335 committed by kapil.thangavelu@canonical.com [15:52] <_mup_> pull ability to use release tarballs and src builds of zk from tests/common.py === daker_ is now known as daker [15:55] SpamapS, interesting it doesn't do the same under 2.7 [16:02] hazmat: yep totally weird [16:02] hazmat: further, python2.6 can't run the test suite. [16:03] http://paste.ubuntu.com/678082/ [16:04] Tho I'm pretty sure thats my fault ;) [16:04] SpamapS, easy enough to fix though [16:04] agreed [16:06] Lunch time.. === daker is now known as daker_ [16:15] <_mup_> ensemble/local-ubuntu-provider r336 committed by kapil.thangavelu@canonical.com [16:15] <_mup_> switch test runner to using lib/zookeeper [16:16] FAILED (failures=5, errors=8, successes=1417) [16:16] Clearly we haven't been running the test suite against python 2.6 [16:18] SpamapS: we mentioned the other day freezing ensemble at something like 305 for oneiric? [16:19] there're changes in 326 that/re critical for config.yaml [16:19] 306 is the version in Oneiric [16:20] can we make an exception? [16:20] If it fixes a critical bug I can either cherry pick it in or go ahead and do a FFE [16:20] either cherr [16:20] gotcha [16:20] it's pretty important... config.yaml defaults aren't picked up until then [16:20] got a bug reference I can mark it as affecting the package (helps build the case for the FFE) [16:20] so every formula that has a config.yaml requires you to pass a --config [16:20] not that we have to build a massive case [16:21] its universe and unseeded.. so there's really no reason for people to complain [16:21] m_3: just give me the bug # :) [16:21] ok, lemme find it... thanks! [16:21] m_3: you can mark it as affecting the package too, I think. [16:24] SpamapS: #828152 [16:24] <_mup_> Bug #828152: default formula config values not available to hooks < https://launchpad.net/bugs/828152 > [16:26] m_3: yeah try clicking "Also affects distribution" [16:26] I think I did [16:26] ubuntu ensemble [16:27] should I 'target to a milestone' too? [16:44] m_3: cool thanks [17:07] jamespage: aha! So it wasn't exposing the port? :) [17:08] well it was broken as well - but should be more stable now upstream have released 1.0 [17:08] ok so should I give it a shot now? [17:08] sure - it should work [17:08] just finished my testing [17:09] cool [17:09] want me to blog it? [17:09] I think people would use the heck out of this [17:35] jamespage: the jenkins formula? [17:35] I added an open-port call last week. [17:35] nope- the etherpad-lite one [17:36] ahh cool. :) [17:36] whats the official position on python 2.6 support? [17:37] Should I file bugs for all the broken stuff? :-P [17:48] SpamapS: https://blueprints.launchpad.net/ensemble/+spec/formula-store [17:49] SpamapS, hazmat: Both of you have demonstrated interest in blueprints several times [17:49] My feeling is that it'll increase the workflow burden without much benefit [17:49] But I'm willing to try it out.. [17:50] I expect it will add a bit of management burden yes. The idea is to provide visibility from outside your team, so that dependent work can carry on without interruptions on either side. [17:51] niemeyer, afaics the main benefit is just tracking features instead of implementation, but that's not an obvious one to the person doing the implementation, because they intuitively know it. [17:51] SpamapS: The kanban provides a lot of visibility already [17:51] SpamapS: and it's not like the major features we're working on change every day [17:51] niemeyer, not at a feature level [17:51] last I looked, there was no kanban for eureka.. but maybe I looked in the wrong place? [17:51] SpamapS: The conversation we had at the sprint is still valid [17:52] SpamapS, link in channel topic [17:52] ah [17:52] http://people.canonical.com/~niemeyer/eureka.html [17:52] Yeah this is good for right now.. it doesn't show anything that is on the "not now but later" list.. [17:53] and it doesn't speak to dependencies.. I can't see there that LXC work has to finish before multi-unit-per-machine [17:53] SpamapS: You can find that here: https://bugs.launchpad.net/ensemble [17:53] Let me step back. The reason BP's are good is just to group efforts that don't fit into one implementation piece like a bug. [17:55] Its just an idea to help get over-arching change implemented.. not something I think is critical to the visibility of the project as a whole. [17:55] The visibility problem that I see isn't real.. I think you're right that the kanban is "whats happening now" and the bugs list is "everything else" [17:56] SpamapS: I understand.. we went through that before. The point I continue to make is that there's a point in the gradient of documentation boilerplate where it's easier to talk to someone on IRC. [17:58] i've experimented with just using tags as easy way to track larger works.. all the security impl bugs are tagged 'security' [17:58] yeah thats probably the lightest weight solution, and could even be visualized very easily [17:58] hazmat: It feels a bit off to be using tags to track chunks of work like that [17:59] hazmat: Tags should continue to be useful across the lifetime of the project [17:59] hazmat: security seems to make sense [17:59] hazmat: But e.g. "handle-ec2-firewall" is not a nice tag [17:59] but perhaps firewall does [17:59] hazmat: Sure, but you're addressing a different problem [17:59] Right, any defects in the firewall handling would be valid years later. [18:00] hazmat: You may have firewall in orchestra and in zookeeper handling of security [18:00] hazmat: It's orthogonal to the concept of features [18:00] and so is security [18:00] Likewise, if it were decided that disconnected operation is important.. one could identify the steps to do that as bugs.. and anything in the future that went against that, can also have said tag. [18:01] I agree that its orthogonal to features.. [18:01] But all the over arching changes that I can think of are actually just wide spread defects .. ;) [18:01] SpamapS: This is a good use for blueprints indeed [18:02] I think it would be worthwhile to use BP's to group bugs and show the dependencies in long term goals. [18:03] but... not worth more than coding.. so.. I thin I'll just stop producing more bytes of text for all of you to consume.. :) [18:03] * SpamapS returns to testing like its 1999 [18:04] SpamapS: I agree, this sounds like a sane approach, as long as we agree on what the blueprints are before starting to dump things on them [18:04] SpamapS: It's also different from the idea of blueprints I was against [18:21] <_mup_> ensemble/lib-lxc-merge r336 committed by kapil.thangavelu@canonical.com [18:21] <_mup_> merge lxc-lib [18:40] <_mup_> ensemble/lib-zookeeper r337 committed by kapil.thangavelu@canonical.com [18:40] <_mup_> extract managed zk into separate branch [18:43] <_mup_> ensemble/lib-zookeeper r338 committed by kapil.thangavelu@canonical.com [18:43] <_mup_> rename managed zk module to avoid name conflict [18:46] <_mup_> Bug #837601 was filed: a zookeeper class for reuse by both local provider and tests < https://launchpad.net/bugs/837601 > [19:16] <_mup_> ensemble/go-formulas r11 committed by gustavo@niemeyer.net [19:16] <_mup_> Reorganized files so meta handling has its own file/tests. [19:52] bcsaller1, got time for a catch-up? [19:58] hazmat, bcsaller1: Quick interface review: http://paste.ubuntu.com/678276/ [20:00] hazmat: yeah [20:00] niemeyer: still leaving out the regex stuff (validator), is that intended? [20:01] bcsaller1: No, I was just going to figure it was missing down the road :) [20:02] The title type is wrong as well, but otherwise the interface will look like this [20:03] I guess default should be anything rather than string [20:03] These issues will be sorted out once I actually implement it, though [20:04] niemeyer, agreed, else looks fine... is the goal to turn the config into a schema? [20:04] for validation? [20:04] hazmat: a schema is already used for that [20:04] hazmat: It is.. will work the same way as the Python side in that regard [20:04] hazmat: Well.. maybe, I guess what you say may be interpreted in a different way [20:04] hazmat: We have actual validator functions to validate the input against the config schema [20:05] hazmat: Against the config, sorry [20:05] hazmat: The config itself has a schema, though [20:05] right, which this is reading, i was just curious if this was going to use a separate mechanism to validate values then the go schema work already done [20:05] hazmat: So there are two levels.. config has a format and a schema to validate it.. the validated config *value* is used to validate the user input using validator functions [20:06] hazmat: That's how it works in Python, at least.. [20:06] hazmat: What you say makes me curious though.. [20:06] hazmat: Maybe we can use the schema to validate the input too [20:06] niemeyer, indeed, but config is a schema, used to validate the user input [20:06] hazmat: Right [20:07] hazmat: It's not how it works today, but I'll see if that may simplify things as I write tests [20:07] cool [20:07] <_mup_> ensemble/go-formulas r12 committed by gustavo@niemeyer.net [20:07] <_mup_> Got config support skeleton in place, and a failing test. [20:07] bcsaller1, phone work for you? [20:07] its fine, I'm on skype too [20:12] bcsaller1: cool if we push our call back to the top of the hour? [20:29] robbiew: was on the phone, anytime is fine [21:00] <_mup_> ensemble/lib-files r339 committed by kapil.thangavelu@canonical.com [21:00] <_mup_> extraact file storange into providers.common for use by local and dummy providers. [21:06] <_mup_> Bug #837692 was filed: A common provider file storage for local and dummy providers < https://launchpad.net/bugs/837692 > [21:10] <_mup_> ensemble/local-machine r340 committed by kapil.thangavelu@canonical.com [21:10] <_mup_> extract local machine into separate branch.. [21:17] >>> sio.truncate() [21:17] >>> sio.getvalue() [21:17] 'foo' [21:17] This is a pretty weird behavior in StringIO [21:18] Ah, I misunderstand the interface [21:18] truncate(0) is what I'm looking for [21:25] <_mup_> ensemble/no-regex-option r336 committed by gustavo@niemeyer.net [21:25] <_mup_> Some consistency clean up in the config handling: [21:25] <_mup_> - Removed regex type. It's uneven with the other types (the value of a [21:25] <_mup_> regex option is not a regex), and it's binding the generic formula [21:25] <_mup_> definition to a specific language. [21:25] <_mup_> - Renamed 'str' type to 'string'. There's no value in saving the 3 chars [21:25] <_mup_> when we have e.g. 'float' already in the same context, and 'string' [21:25] <_mup_> is actually readable while 'str' is not. [21:25] <_mup_> - Introduced backward compatibility handling for 'str' so that it [21:25] <_mup_> continues to work, but the user is warned about the fact that it's [21:25] <_mup_> obsolete so that we can pester authors to rename it timely before it's [21:25] <_mup_> too late and we can't go back. Tested this appropriately. [21:30] <_mup_> Bug #837708 was filed: 'str' in config.yaml should be renamed to 'string' < https://launchpad.net/bugs/837708 > [21:33] <_mup_> Bug #837710 was filed: regex type should be dropped < https://launchpad.net/bugs/837710 > [21:38] <_mup_> ensemble/no-regex-option r337 committed by gustavo@niemeyer.net [21:38] <_mup_> Fixed examples. [21:39] Does anyone know if this tutorial is still valid? https://ensemble.ubuntu.com/docs/user-tutorial.html [21:39] I see the instances in my AWS console, but the relation never occurs. well at least from what I see (rather don't see) in the debug logs [21:41] ameetp: It should be valid, yeah [21:41] ameetp: Note that the debug-log has to be started upfront [21:44] niemeyer, yes, I started the logs in-line with the tutorial. I don't see any relevant output [21:44] ameetp: Cna you please paste it somewhere? [21:46] niemeyer, https://pastebin.canonical.com/51984/ [21:47] ameetp, you started the debug-log before you launched the machines? [21:47] ameetp: That's pretty strange.. it's a completely empty log [21:47] or before adding the relation? [21:47] hazmat: Just asked that [21:48] niemeyer, I was hoping to at least see 'Machine:1: ensemble.agents.machine DEBUG: Downloading formula' when I ran 'ensemble deploy --repository=examples mysql' [21:48] ameetp: Yeah, it's quite weird.. it should definitely be showing that [21:48] ameetp: Unless you deploying and then started the command [21:48] deployed [21:49] * hazmat heads out to dinner, bbiab [21:49] niemeyer, no. I started the command after doing 'ensemble bootstrap' in another console [21:50] ameetp: Ok.. then you kept the command running, and did something else in a different terminal? [21:50] niemeyer, correct [21:51] ameetp: What happens if you deploy mysql again? [21:51] ameetp: Or add the relation? [21:51] hazmat: Enjoy [21:52] niemeyer, https://pastebin.canonical.com/51985/ [21:52] ameetp: Use a different name.. [21:52] ensemble deploy --repository=examples mysql mysql2 [21:53] ameetp: Second name is the service name [21:53] ameetp: It defaults to the same name of the formula, if you don't provide it [21:55] niemeyer, https://pastebin.canonical.com/51986/ [21:55] i see an new instance in AWS console as well. [21:56] ameetp: The debug-log seems completely off somehow.. [21:56] ameetp: Can I access the bootstrap instance to have a look? [21:57] niemeyer, sure. I can't give access to the system itself, but I can give you a file if you would like. Or I can open a bug. I went down this path because the relation never completes for me [21:58] ameetp: What's the status? [21:59] ameetp: You can open a bug, but without further information we won't be able to do much [22:00] niemeyer, state always remains null. Even after 4 hours [22:00] ameetp: :-) [22:00] ameetp: It's quite fast when things are working [22:01] ameetp: Debug log is the way to check this out, but it feels like things are really hosed there [22:01] ameetp: Sorry about that.. can't really do much without further debugging [22:01] niemeyer, is accessing the system the only way to debug? [22:02] ameetp: Telepathy would be the other option, but I'm not very good with that ;-) [22:02] niemeyer, :) Okay let me see if I can get someone to recreate this [22:03] niemeyer, anyway. Thanks for your assistance! [22:04] ameetp: No problem, and sorry for the trouble [22:04] <_mup_> Bug #837724 was filed: Relation already exists error message is bad < https://launchpad.net/bugs/837724 > [22:08] ameetp: Btw, if you manage to run this again and reproduce, please ping me [22:08] niemeyer, sure [22:50] <_mup_> ensemble/go-formulas r13 committed by gustavo@niemeyer.net [22:50] <_mup_> Config parsing begins. [22:50] Alright.. I'm stepping off [22:50] May be back later for other activities [22:50] Cheers everyone === adam_g is now known as foods [23:50] * niemeyer waves