| m_3 | bye baby | 02:12 |
|---|---|---|
| m_3 | ok, wrong focus... sorry | 02:13 |
| dvestal | m_3 or hazmat: Earlier hazmat mentioned the exploration of the notion of co-locating service units in specification form. Is that in reference to the upgrade, etc. documentation at https://ensemble.ubuntu.com/docs/ or are there other specs that I could find to see what's already been discussed about the topic? | 05:16 |
| === dvestal is now known as dvestal|away | ||
| kim0 | Morning folks | 08:20 |
| === koolhead11|Afk is now known as koolhead11 | ||
| hazmat | dvestal|away, re co-location not atm re docs | 10:37 |
| === daker_ is now known as daker | ||
| niemeyer | Hey Ensemblers | 13:34 |
| niemeyer | What are you Formulating today? :-) | 13:34 |
| m_3 | niemeyer: morning | 13:52 |
| m_3 | niemeyer: I'm debugging nfs this morning | 13:52 |
| hazmat | niemeyer, ensemble cli plugins and an ide | 13:52 |
| m_3 | niemeyer: then planning on expanding node.js and starting on greenplum | 13:53 |
| niemeyer | hazmat, m_3: Mornings | 13:53 |
| niemeyer | m_3: Sweet | 13:53 |
| niemeyer | m_3: Wow :) | 13:53 |
| m_3 | having a problem with empty endpoints in add-relation at the moment... probably just naming on my end | 13:54 |
| hazmat | m_3, awesome, which greenplum product out of curiosity? | 13:54 |
| m_3 | hazmat: dunno... just popped on Mark's radar from conversations with Barclay's | 13:55 |
| m_3 | hazmat: that's the first task... ping me with suggestions | 13:55 |
| niemeyer | m_3: What are you planning on node.js? | 13:58 |
| m_3 | niemeyer: well I have a node.js "app" formula written | 13:58 |
| niemeyer | m_3: Interesting | 13:59 |
| m_3 | but it doesn't do much other than deploy a noe.js app from a repo | 13:59 |
| niemeyer | m_3: What does it do to deploy the app? | 13:59 |
| hazmat | m_3, so greenplum has a couple different product lines, they focus on offering analytics w/ ides, on top of warehouse dbs, be it their own version of pg, hadoop, hive.. the latter is typically an ingestion into the former, they have a fairly sophisticated graphical toolchain that the target analyst market expects | 13:59 |
| hazmat | i saw a nice demo of it at a local hadoop meetup | 14:00 |
| m_3 | niemeyer: just installs nodejs packages and then pulls a node.js test app from github | 14:00 |
| m_3 | niemeyer: then spins it up | 14:00 |
| hazmat | niemeyer, m3 i imagine its the same pattern for any language app server deployment, vcs config details, entry points, url mounting, dependency management, and db and proxy options | 14:01 |
| m_3 | hazmat: yup | 14:01 |
| niemeyer | hazmat: Kind of | 14:01 |
| niemeyer | hazmat: It's the same problem, for sure | 14:01 |
| m_3 | next is to figure out how to bind a db service and make it available to the node app | 14:01 |
| niemeyer | hazmat: But I'm curious about how to leverage the environment we're offering | 14:01 |
| niemeyer | hazmat: How to connect dbs etc | 14:01 |
| m_3 | there's a different option too that we should build | 14:01 |
| niemeyer | hazmat: We can literally offer a PaaS for people, without having one | 14:02 |
| hazmat | niemeyer, wsgi, rails, nodejs are pretty similiar aspects here, i think their different formulas, but really they have the same responsibilities and deploy patterns | 14:02 |
| m_3 | a node.js _hosting_ service | 14:02 |
| m_3 | similarly for rails | 14:02 |
| hazmat | niemeyer, sure, that's the rather the point of ensemble. | 14:03 |
| niemeyer | hazmat: Yep.. that's why I'm asking then.. how can we do more than simply "deploy code in a server" with a formula | 14:03 |
| m_3 | nodejitsu has an open source framework for hosting node | 14:03 |
| m_3 | I'll distinguish them by name for now | 14:05 |
| m_3 | 'rails-app' -vs- 'rails'... similarly for node | 14:05 |
| hazmat | m_3, there are literally dozens of the platform specific paas options | 14:05 |
| hazmat | although not many are opensource | 14:05 |
| m_3 | hazmat: yup | 14:06 |
| m_3 | hazmat: again, please feel free to suggest priorities | 14:06 |
| m_3 | as I understand, we need a base set of formulas | 14:07 |
| m_3 | that just keeps building | 14:07 |
| hazmat | m_3, your priorities sound good (nodejs & greenplum), i'd like to see more scalable infrastructure components so glusterfs, cassandra, elasticsearch | 14:09 |
| m_3 | hazmat: yeah, me too | 14:09 |
| m_3 | hazmat: negronjl has cassandra working I think | 14:10 |
| hazmat | hadoop would be interesting to look at as well, especially in comparison to the python whirr implementation (their project specific ec2 scripts).. the biggest missing thing i see there is better volume management | 14:10 |
| hazmat | yeah.. i should check out the latest principia | 14:11 |
| m_3 | db service binding was easy in railsapp, there's pretty much a set way to do it in the app | 14:11 |
| m_3 | harder for node b/c it's a lot more unstructured | 14:11 |
| m_3 | I'll pick _a_ way to show as an example | 14:11 |
| m_3 | hadoop's in there too | 14:11 |
| m_3 | hazmat: I'll check out whirr | 14:12 |
| hazmat | m_3, feel to have a formal entry point into the node app, like have a yaml file with this data | 14:12 |
| m_3 | hazmat: right, one I'll have to structure | 14:13 |
| m_3 | hazmat: maybe using mongo if the peer stuff's up | 14:13 |
| hazmat | m_3, sounds good, which reminds me i should push my mongo formulas | 14:14 |
| m_3 | hazmat: yes please! | 14:14 |
| m_3 | yeah, I'm bad about that... I need to push it all to lp | 14:15 |
| m_3 | branch naming was preventing some of my pushes | 14:15 |
| m_3 | (i.e., my misunderstanding of branch naming rqmts <grin>) | 14:16 |
| m_3 | I'll rtfm | 14:16 |
| m_3 | hazmat: what sort of IDE stuff were you talking about earlier? | 14:17 |
| hazmat | m_3, more of a web interface that would allow for interactive formula development with instrumentation | 14:18 |
| m_3 | hazmat: nic | 14:18 |
| m_3 | hazmat: I want blocks! with wires between them! <grin> | 14:19 |
| hazmat | m_3, i think we'll need cli and server plugins first, but the ide itself would likely come after some sort of rest api server | 14:19 |
| m_3 | hazmat: gotcha | 14:19 |
| m_3 | hazmat: sounds perfect | 14:19 |
| hazmat | m_3, mostly just brainstorming on ideas for the future, i think the cli-plugins are good first start | 14:19 |
| m_3 | hazmat: yes | 14:20 |
| m_3 | hazmat: I was thinking something like a '--block' cli option would allow for formula testing | 14:20 |
| m_3 | hazmat: right now it'd have to have sleeps put in which is... not ideal | 14:21 |
| m_3 | but there're all sorts of cool cli ideas | 14:21 |
| m_3 | love the graphviz status btw | 14:22 |
| === dvestal|away is now known as dvestal | ||
| m_3 | dvestal: re your question from last night | 14:28 |
| m_3 | dvestal: I think the orig idea in colocated services was to provide for 'aspects' like monitoring and log consolidation services to be colocated with a primary service | 14:28 |
| m_3 | dvestal: think mysql with munin-node | 14:29 |
| m_3 | dvestal: I'm looking for other 'specs', but I think the ensemble docs are primary atm | 14:29 |
| dvestal | m_3: Has there been any discussion on how to handle system configuration things outside of the "service" installation and configuration? | 14:30 |
| m_3 | dvestal: there are lots of feature requests too (https://bugs.launchpad.net/ensemble) | 14:30 |
| m_3 | dvestal: yes, lots... I'm not sure the current state of that though | 14:31 |
| dvestal | m_3: For example, I have all of my servers authenticating with ldap instead of just the base AWS ubuntu user auth. | 14:31 |
| alienpulse | kim0, there ? | 14:32 |
| m_3 | dvestal: that one might actually work as a service though... "authentication service" | 14:32 |
| dvestal | m_3: I'll have to dig through the email list archives to try to catch up different things. | 14:32 |
| alienpulse | dvestal, m_3 help plz :) | 14:32 |
| niemeyer | dvestal: Oh, that's quite interesting actually | 14:32 |
| niemeyer | dvestal: We have a few different things in progress ATM | 14:33 |
| m_3 | dvestal: there are things that don't fit as a "service" | 14:33 |
| niemeyer | dvestal: and this one looks like the concept we're pushing in terms of "co-located" formulas | 14:33 |
| m_3 | alienpulse: what's up? | 14:33 |
| niemeyer | dvestal: The idea is that certain formulas will be able to be deployed within the same container as another formulas | 14:33 |
| niemeyer | another formula | 14:34 |
| niemeyer | dvestal: To allow tweaking the environment in a sensible way | 14:34 |
| niemeyer | dvestal: E.g. distributed logging, monitoring, etc | 14:34 |
| niemeyer | dvestal: It sounds like what you're trying to do is related to this as well | 14:34 |
| dvestal | niemeyer: Yeah. | 14:34 |
| alienpulse | m_3, hello..I'm new with every thing like im clueless ,,in Launchpad ,, i'v my own bug but i don't now how to add formula and nodes and etc on it | 14:34 |
| alienpulse | m_3, silly i know ,, but kim0 was my survivor in this kind of things | 14:35 |
| dvestal | niemeyer: I have several small things like that. ldap auth is just the simplest to explain, and the most widely used. | 14:35 |
| niemeyer | dvestal: Ok, do you have another example, just to ensure we're on the same page? | 14:36 |
| m_3 | alienpulse: no problem... wanna send me a url to the lp bug? | 14:36 |
| alienpulse | okay | 14:36 |
| alienpulse | m_3, https://bugs.launchpad.net/~abedeltawil | 14:37 |
| dvestal | niemeyer: <this is not answering your question yet> What I do now is a scripted snapshotting of a machine so that I have an ami to stand up a new machine from. | 14:38 |
| alienpulse | m_3, so how can i remove Admin links drupal .. ? and how to add a new project to start working on it and from where | 14:38 |
| niemeyer | dvestal: Ok, this is a fairly common approach, but at the same time it's the kind of trick we're trying to obsolete with Ensemble | 14:39 |
| m_3 | alienpulse: ah, I think I understand | 14:39 |
| alienpulse | m_3, please advice .. | 14:39 |
| dvestal | niemeyer: Basically things like syslog -> mysql logging. Auto-installation of wars into Tomcat and ears into Glassfish. | 14:40 |
| dvestal | niemeyer: Yeah, I don't like doing it this way... which is how I found Ensemble. :) | 14:40 |
| niemeyer | dvestal: These sound like good examples of co-located formulas | 14:41 |
| niemeyer | dvestal: The idea will be quite simple, actually: you'll be able to describe certain relations as co-located, and Ensemble will know that the related services have to live together | 14:42 |
| niemeyer | dvestal: They will still be normal relations, though | 14:42 |
| niemeyer | dvestal: So, e.g. | 14:42 |
| niemeyer | dvestal: Imagine a relation of type "tomcat-war" | 14:42 |
| niemeyer | dvestal: WIth colocation on | 14:43 |
| niemeyer | dvestal: Ensemble will put both of them in the same container, and will establish the relation | 14:43 |
| niemeyer | dvestal: When tomcat notices the relation was established on war-relation-joined, | 14:43 |
| niemeyer | dvestal: It does "relation-set war-directory=/path/to/wars" | 14:44 |
| niemeyer | dvestal: You see where this is going? | 14:44 |
| m_3 | alienpulse: I can't remove the associated bug... just please ignore that as noise for now | 14:49 |
| m_3 | alienpulse: have you started on the formula itself? | 14:49 |
| alienpulse | m_3, emm okay from where should i start if i wanna start a project for scratch | 14:49 |
| m_3 | alienpulse: what I do to start is copy another formula | 14:50 |
| alienpulse | m_3, i'v my bug but i don't now what should i do next ! :/ | 14:51 |
| m_3 | alienpulse: I don't think there's a need to create a new launchpad project for a formula | 14:51 |
| m_3 | alienpulse: so the first thing to do is get a working directory where your formula will reside | 14:51 |
| alienpulse | m_3, how that ? | 14:51 |
| alienpulse | how can i do that ? | 14:51 |
| m_3 | alienpulse: what platform are you working on? | 14:51 |
| alienpulse | u mean my project ? | 14:52 |
| alienpulse | Drupal ! | 14:52 |
| niemeyer | dvestal fainted with the explanations.. sorry! ;-) | 14:52 |
| m_3 | alienpulse: the first thing we need to do is make a local repo for the formula | 14:53 |
| m_3 | alienpulse: have you used bazaar before? | 14:53 |
| alienpulse | no :( | 14:53 |
| alienpulse | let say i made a local repo | 14:54 |
| alienpulse | should i upload it ,, right ,, but where on the launchpad should i upload ? | 14:54 |
| sladen | rumour has it via the grapevine that somebody has asking for input in doing an Ensemble Logo | 14:55 |
| sladen | who would be the best person who might know more details? | 14:55 |
| m_3 | sladen: kim0 or jcastro maybe? | 14:55 |
| m_3 | alienpulse: it's not bad... something like http://paste.ubuntu.com/639527/ | 14:56 |
| jcastro | not me, I only sat in on the web meeting | 14:57 |
| sladen | jcastro: that's more than I did. kim0: any knowledge? | 14:57 |
| alienpulse | m_3, and should i copy it and past it where !? | 14:57 |
| m_3 | alienpulse: so the formula development will be done entirely on your machine | 14:58 |
| m_3 | alienpulse: once you've gotten it going the way you like | 14:58 |
| m_3 | alienpulse: you can push your repo up to launchpad | 14:59 |
| m_3 | alienpulse: (can really push up at any time... the important point is that the development work is done on your local machine) | 14:59 |
| m_3 | alienpulse: have you installed ensemble on your machine? | 15:00 |
| alienpulse | m_3, yeah sorry for that . | 15:01 |
| m_3 | alienpulse: np | 15:02 |
| alienpulse | m_3, okay you where saying should the Formula development will be done on my laptop and with bzr is linked to my bug on launchpad ? | 15:04 |
| dvestal | niemeyer: Sorry for the delayed response. I had to run upstairs for breakfast with the kids. Yeah, that's a good explanation. I like where it's going too. | 15:05 |
| dvestal | ^ with the wife and kids... | 15:05 |
| niemeyer | dvestal: No worries, was just kidding.. I know how that works too :-) | 15:06 |
| m_3 | alienpulse: yes, develop the formula first, then we can associate the repo you're using with the lp bug | 15:06 |
| niemeyer | dvestal: Nice, so if you have any ideas to suggest or any questions, just pop up and we'll be happy to talk/fix/implement :) | 15:06 |
| dvestal | niemeyer: I definitely will. This is something that I anticipate contributing to. I'm sure that it will need to primarily be things that are specific needs of my company, but I talked to our owner about potentially working on this some as I am working toward automating our infrastructure and he seemed to be onboard with it. | 15:09 |
| m_3 | dvestal: awesome! | 15:09 |
| kim0 | ~> | 15:10 |
| kim0 | that was a hung ssh :) | 15:10 |
| niemeyer | dvestal: That's great news indeed.. it's a good time to join in too, as you can help shaping the face of it to your needs | 15:16 |
| kim0 | alienpulse: sorry my irc was hung .. so are starting on the drupal formula | 15:16 |
| kim0 | alienpulse: did you read my formula writer's tutorial .. it should be easy to follow .. hey it's even for drupal too :) | 15:16 |
| alienpulse | kim0, no :( i want to did u blog it ? | 15:17 |
| alienpulse | ill read it from your blogs | 15:17 |
| * kim0 getting link | 15:17 | |
| kim0 | alienpulse: just this one https://ensemble.ubuntu.com/docs/write-formula.html | 15:17 |
| alienpulse | kim0, okay i was saying i dont know from where should i start ,, i dont want begin wrong .. | 15:19 |
| kim0 | alienpulse: so basically, I'd start by reading that tutorial, and try to execute it step by step till you have drupal running on ec2 | 15:20 |
| kim0 | alienpulse: once it's running, you can start hacking on it | 15:20 |
| kim0 | to make a real formula out of it | 15:20 |
| alienpulse | and then i upload it to launchpad ? | 15:20 |
| kim0 | alienpulse: no need for that .. you can create the formula on localhost .. and use it to deploy .. | 15:21 |
| alienpulse | im almost done with reading the ensemble tutorial .. | 15:21 |
| kim0 | alienpulse: putting it on LP .. is like the final step | 15:21 |
| alienpulse | kim0, aha | 15:21 |
| alienpulse | kim0, okay ,,im getting it all now | 15:21 |
| alienpulse | 1st step ..run drupal on ec2 | 15:22 |
| koolhead11 | okey guys i have filled a bug finally dbconfig-common and phpmyadin | 15:22 |
| kim0 | koolhead11: woohoo \o/ | 15:22 |
| koolhead11 | the preseed made me cry | 15:22 |
| alienpulse | 2nd step hacking it .. 3rd step create a formula on my machine ,, and finally ill upload it to LP | 15:22 |
| alienpulse | kim0, miss any thing ?! | 15:22 |
| kim0 | let me rephrase .. this is how I would do it | 15:23 |
| kim0 | 1- copy formula from tutorial into files | 15:23 |
| kim0 | 2- run it as is, launch drupal on ec2 with it | 15:23 |
| kim0 | 3- understand what is going on | 15:23 |
| kim0 | 4- start changing the formula | 15:23 |
| kim0 | 5- upload to launchpad, share your formula with the world | 15:23 |
| kim0 | alienpulse: that's all | 15:24 |
| alienpulse | kim0, you're a bless from heaven ! | 15:24 |
| kim0 | alienpulse: You know what .. even easier .. I remember now I have pushed my drupal to LP | 15:25 |
| kim0 | alienpulse: so you can skip step 1 .. and instead do → bzr branch lp:~kim0/+junk/drupal | 15:25 |
| alienpulse | kim0, okay | 15:25 |
| niemeyer | fwereade: Would you mind to review this merge proposal when you have a moment: https://code.launchpad.net/~hazmat/ensemble/hooks-with-formula-dir | 15:27 |
| fwereade | niemeyer: a pleasure | 15:27 |
| niemeyer | fwereade: Thanks! | 15:27 |
| kim0 | sladen: o/ | 15:28 |
| kim0 | sladen: so yeah, a community member is interested in working on a logo+mascot for Ensemble | 15:29 |
| kim0 | sladen: he does pencil concepts. He's starting to send me concepts drawings, but I'm really not made to judege these things :) | 15:29 |
| kim0 | sladen: can I link you two up ? so you start hacking those graphics together / | 15:30 |
| * koolhead11 is allowed to ask dumb/stupid questions :P | 15:32 | |
| niemeyer | hazmat: I don't think the env var should be yanked.. please see the latest comment there | 15:33 |
| sladen | kim0: -> sladen@c.com | 15:33 |
| niemeyer | hazmat: Just pointing out here so you don't work on this before seeing the comment | 15:33 |
| niemeyer | koolhead11: Everyone is! ;_) | 15:33 |
| kim0 | sladen: awesome .. linking :) thanks man | 15:33 |
| koolhead11 | niemeyer, :) | 15:34 |
| hazmat | niemeyer, k, thanks | 15:34 |
| hazmat | niemeyer, could you clarify what you meant by [3] | 15:36 |
| niemeyer | hazmat: Just mentioning what is $CWD in the docs | 15:36 |
| niemeyer | hazmat: Next to the hooks themselves | 15:36 |
| hazmat | niemeyer, ah.. sounds good | 15:37 |
| fwereade | couple of questions about what directory we run in | 15:40 |
| fwereade | what makes the unit dir better than the formula dir? | 15:40 |
| fwereade | I haven't been able to see any serious distinction in this context but presumably there's some reason to prefer the unit dir | 15:40 |
| fwereade | hazmat, niemeyer: the docs currently say we run in the formula dir, but it seems we run in the unit dir | 15:42 |
| niemeyer | fwereade: I think we're using the formula dir? | 15:42 |
| hazmat | fwereade, it does run in the formula dir | 15:42 |
| hazmat | it just uses the unit dir as a base reference point | 15:43 |
| fwereade | hm, let me reread the code | 15:43 |
| fwereade | looked like it was using the unit dir | 15:43 |
| niemeyer | In fact, there's an important reason for it to be the formula dir now.. we'll soon have co-located formulas in the same unit dir | 15:44 |
| hazmat | fwereade, its at the bottom of invoker where it spawns the process, and there's a corresponding test | 15:44 |
| fwereade | hazmat: where it calls with "hook_proto, hook, [hook], env, self._unit_path)"? | 15:45 |
| fwereade | hazmat oh damn, wrong rev | 15:45 |
| hazmat | fwereade, are you on the right branch? latest rev is 262 of lp:~hazmat/ensemble/hooks-with-formula-dir | 15:45 |
| hazmat | cool | 15:45 |
| fwereade | forget I said anything | 15:45 |
| fwereade | :) | 15:46 |
| hazmat | niemeyer, the implementation of co-location will need to modify the impl as needed. the formula location will no longer just be /formula | 15:48 |
| niemeyer | hazmat: Right, that was my point | 15:48 |
| * niemeyer => lunch | 16:07 | |
| kim0 | alienpulse: just noticed m_3 did a drupal formula on https://code.launchpad.net/principia .. it's ok, you can still study how things work, then improve on Mark's formula | 16:13 |
| alienpulse | kim0, im working on your now .. | 16:14 |
| sladen | kim0: can you forward me the actual pencil sketches too | 16:14 |
| alienpulse | kim0, but thanks any way | 16:14 |
| kim0 | alienpulse: yeah don't let that stop you .. you need to understand ensemble .. then you can hack any formula :) | 16:14 |
| kim0 | sladen: yep .. getting em | 16:14 |
| kim0 | sladen: emailed | 16:17 |
| hazmat | niemeyer, re formula upgrade and cleanup of files, its not quite read only.. but point taken.. if we assume namespacing in the future, fhs locations in the unit would need qualifications as a result, we can nominate a particular formula directory as inviolate by ensemble and suitable for variable storage (aka /var) or store the manifest at deploy time to compare to the delta, to just remove old formula files but not files created by the formula... | 16:20 |
| sladen | kim0: ta. Do you have an full copy of the mailing message https://lists.ubuntu.com/archives/ensemble/2011-June/000199.html available? | 16:32 |
| sladen | kim0: the one in the archive is missing the attachments | 16:32 |
| kim0 | sladen: it was only links .. no attachments | 16:35 |
| kim0 | sladen: I emailed you the links as well | 16:36 |
| sladen | kim0: nod | 16:44 |
| sladen | niemeyer: would you be able to follow-up to https://bugs.launchpad.net/ubuntu-branding/+bug/807100 ("Ensemble logo (ensemble.ubuntu.com)") with your breathing aperatus quote | 16:45 |
| _mup_ | Bug #807100: Ensemble logo (ensemble.ubuntu.com) <ubuntu-branding:New> < https://launchpad.net/bugs/807100 > | 16:45 |
| niemeyer | sladen: Hey there | 16:57 |
| niemeyer | sladen: Sure, what's up? | 16:57 |
| niemeyer | hazmat: Yeah, storing old manifest + new manifest + diff and remove might be the most expected outcome | 16:58 |
| niemeyer | hazmat: I actually like a bit the var approach too | 16:58 |
| niemeyer | hazmat: Makes it easy to move things around | 16:58 |
| niemeyer | hazmat: I'm a bit concerned that people won't figure it out until too late | 16:58 |
| sladen | niemeyer: Ensemble logo tracking bug | 17:07 |
| sladen | niemeyer: I have a quote that kim0 forwarded me, but I'd rather if you attributed and added it yourself :-) | 17:08 |
| niemeyer | sladen: Done | 17:13 |
| niemeyer | Actually, there's another thread too | 17:15 |
| niemeyer | hazmat: Looks like a recent change has broken txzookeeper: https://launchpadlibrarian.net/74683481/buildlog.txt.gz | 17:18 |
| sladen | niemeyer: did you mean to copy and re-paste 80% of the content from the bug introduction back into the bug report :-) | 17:22 |
| hazmat | niemeyer, yeah.. i'm not sure what the issue is the changelog format looks fine | 17:24 |
| hazmat | niemeyer, is there a static check tool ala lintian for checking deb directories | 17:25 |
| alienpulse | kim0, how can i remove one of my affects !? | 17:30 |
| kim0 | alienpulse: what do you mean by affects | 17:30 |
| alienpulse | kim0, i want to remove admin links Drupal module | 17:30 |
| kim0 | alienpulse: don't install it ? | 17:31 |
| alienpulse | kim0, in my bug i just assigned by mistake :/ | 17:31 |
| kim0 | alienpulse: give me the bug url | 17:31 |
| alienpulse | kim0, https://bugs.launchpad.net/principia/+bug/805955 | 17:31 |
| _mup_ | Bug #805955: Formula needed: Drupal <new-formula> <Admin links Drupal module:New> <Principia Ensemble:In Progress by abedeltawil> < https://launchpad.net/bugs/805955 > | 17:31 |
| niemeyer | sladen: Sorry, I didn't notice you had already added one of the links | 17:32 |
| niemeyer | sladen: It's not clear to me how can I help there? | 17:33 |
| niemeyer | sladen: I've already commented on these graphics in the mailing list thread which I linked there | 17:33 |
| kim0 | alienpulse: I just set it to invalid on that project .. probably good enough | 17:33 |
| niemeyer | sladen, kim0: What sort of action is the bug oriented towards? | 17:34 |
| alienpulse | kim0, emm okay thanks .. | 17:34 |
| niemeyer | hazmat: There is | 17:34 |
| niemeyer | Let me try to find the name | 17:35 |
| m_3 | kim0, alienpulse: I tried to remove this morning with no luck | 17:35 |
| kim0 | yeah, I'm no LP expert either :) | 17:36 |
| sladen | niemeyer: -> see your private message | 17:36 |
| sladen | niemeyer: tracking the logo. I heard on the grape vine that somebody/something wanted help. It's now documenteed in one place, with everything that has been said | 17:38 |
| niemeyer | hazmat: lintian - Debian package checker | 17:42 |
| niemeyer | hazmat: I suspect the date format might be the issue, but not sure | 17:43 |
| niemeyer | hazmat: date -R is the standard | 17:43 |
| * niemeyer playing with glance | 19:10 | |
| niemeyer | fwereade: That may be our S3 service for next week ^ | 19:10 |
| pindonga | SpamapS, you were working on the lxc branch for ensemble, right? | 19:36 |
| niemeyer | SpamapS: We should talk about the LXC work, actually | 19:56 |
| niemeyer | SpamapS: The first step is probably to put a specification in place detailing what we're going after | 19:56 |
| niemeyer | and the impact in the several areas | 19:56 |
| m_3 | niemeyer: +1 for faster formula development/debugging | 19:58 |
| m_3 | jcastro: can I the deck you used for your ensemble lightning talk? | 20:18 |
| adam_g | hazmat: ping | 21:16 |
| adam_g | is bootstrapping stock oneiric ec2 AMIs expected to work? | 22:16 |
| m_3 | adam_g: I was wondering the same thing when pointing to a euca cloud | 22:23 |
| m_3 | adam_g: I don't think so... think zk's gotta be there at least | 22:24 |
| m_3 | adam_g: smoser helped SpamapS bundle up something minimal for lxc | 22:24 |
| m_3 | adam_g: if he's around | 22:24 |
| adam_g | zk looks to be installed via cloud-init | 22:24 |
| adam_g | so does the rest of the bootstrapping | 22:24 |
| niemeyer | adam_g: I haven't tested this myself.. we've been using Natty images just because it makes it easier to have it on both sides | 22:25 |
| niemeyer | adam_g: That said, we should all switch to Oneiric now that we're packing it in 11.10 | 22:25 |
| niemeyer | adam_g: and I can't see why it'd not owrk | 22:25 |
| adam_g | niemeyer: yeah.. i was using the default natty AMIs as well until the archive mirror in us-east-1 decided to fall over | 22:25 |
| niemeyer | wokr | 22:25 |
| adam_g | W: Failed to fetch bzip2:/var/lib/apt/lists/partial/us-east-1.ec2.archive.ubuntu.com_ubuntu_dists_natty-updates_universe_binary-i386_Packages Hash Sum mismatch | 22:26 |
| niemeyer | Ugh | 22:26 |
| Ryan_Lane | howdy | 22:37 |
| kim0 | Ryan_Lane: o/ | 22:37 |
| kim0 | niemeyer: o/ | 22:37 |
| kim0 | Ryan is from the mediawiki ops team | 22:37 |
| Ryan_Lane | kim0: so, ensemble seems like a really great way of pushing changes into an environment that isn't fully formed, and doesn't have something like chef/puppet | 22:38 |
| Ryan_Lane | but it doesn't address long term management of systems | 22:38 |
| kim0 | MediaWiki has a staging cluster, and I was pinging Ryan to test whether we can get some interest to get them to use Ensemble .. Ryan has some questions which I thought we could better answer here | 22:38 |
| Ryan_Lane | the route we are going is to open our puppet repository to everyone | 22:38 |
| adam_g | niemeyer: the run_cmd script in cloud-init for bootstrap doesn't succeed on oneiric. and i cant see how it would work on natty, either. are the default AMIs launched by ensemble stock? | 22:39 |
| Ryan_Lane | and treat our ops like we treat mediawiki | 22:39 |
| kim0 | SpamapS: hazmat m_3 your thoughts are all welcome | 22:39 |
| Ryan_Lane | with code review on the puppet manifests, files, and templates | 22:39 |
| Ryan_Lane | which will then let us directly deploy these changes to our production hardware cluster | 22:39 |
| Ryan_Lane | which isn't cloud based ;) | 22:40 |
| kim0 | Ryan_Lane: let me try to answer the points you've raised | 22:40 |
| kim0 | Ryan_Lane: so Ensemble is not really about cloud only | 22:40 |
| Ryan_Lane | lemme bring up the docs so that I can refresh my knowledge of it | 22:40 |
| kim0 | drivers are being written today to support deploying to LXC containers, as well as hardware machinse and ofcourse openstack private clouds among others | 22:40 |
| kim0 | ec2 was just the fastest way to get started | 22:40 |
| kim0 | Ryan_Lane: regarding the first concern, Ensemble operates at a higher level that puppet or chef | 22:41 |
| kim0 | in the sense that it operates at a "service" level | 22:41 |
| kim0 | for configuration file management .. you _could_ still use puppet if you really wanted to | 22:41 |
| kim0 | or you could use something else | 22:41 |
| kim0 | Ensemble is well suited to long term management of instances | 22:42 |
| kim0 | it simply focuses more on the concept of deployable "services" over machines | 22:42 |
| kim0 | umm .. regarding sharing your deployment recipes with the world | 22:42 |
| kim0 | that is exactly what we're after | 22:43 |
| Ryan_Lane | yeah. I'm not concerned about that part | 22:43 |
| kim0 | We're creating the principia project | 22:43 |
| kim0 | which should cover hopefully all FOSS soon :) | 22:43 |
| kim0 | if you have some more questions or exact use cases .. I'd love to hear them as well | 22:44 |
| m_3 | Ryan_Lane: the thing that really closed ensemble for me was the relationships | 22:44 |
| Ryan_Lane | so run me through a basic idea of installing a new mediawiki instance inside of an already configured mediawiki cluster | 22:44 |
| kim0 | ofc Ensemble is rapidly evolving .. if there's a use case we don't currently support .. things can definitely be tuned | 22:44 |
| m_3 | Ryan_Lane: ensemble provides events when service relationships are changed | 22:45 |
| m_3 | Ryan_Lane: take a look at the mediawiki formula hook for db-relation-joined (maybe db-relation-changed) | 22:45 |
| Ryan_Lane | puppet/chef does the same | 22:45 |
| m_3 | Ryan_Lane: the way the events go out it allows for _really_ easy horiz scaling | 22:45 |
| kim0 | Ryan_Lane: is the rest of the cluster under ensemble management ? | 22:46 |
| Ryan_Lane | let's assume I have a memcached cluster, an apache cluster, a database cluster, a squid cluster, a varnish cluster (for bits), a swift cluster (for media), how will ensemble create just a wiki in that? | 22:46 |
| Ryan_Lane | without deploying new nodes | 22:46 |
| Ryan_Lane | or avoiding too much complexity... | 22:47 |
| Ryan_Lane | an apache cluster, and a database cluster | 22:47 |
| kim0 | are you saying those resources are not under Ensemble's management, or are they | 22:47 |
| Ryan_Lane | I get the idea of adding new nodes to the cluster, that's easy enough :) | 22:47 |
| Ryan_Lane | under ensemble | 22:47 |
| kim0 | Very well | 22:48 |
| Ryan_Lane | created via the screencast example | 22:48 |
| kim0 | m_3: care to explain how that works :) | 22:48 |
| Ryan_Lane | (this is a specific use case that I need, and don't have a solution for yet) | 22:48 |
| kim0 | Ryan_Lane: Would deploying a service inside a new LXC container be acceptable for this use case ? | 22:49 |
| Ryan_Lane | no | 22:49 |
| m_3 | Ryan_Lane: yup, creating a pastebin of the hooks we have implemented for mediawiki | 22:49 |
| Ryan_Lane | it can't add new instances | 22:49 |
| m_3 | Ryan_Lane: so here's the metatdata.yaml file for the formula http://pastebin.com/h8HYzfQ5 | 22:49 |
| m_3 | you can see the attachment with mysql, memcache, munin, and the "website" relation | 22:50 |
| * Ryan_Lane nods | 22:50 | |
| m_3 | the "Website" relation can be directly accessed or consumed in a relation with a reverseproxy | 22:51 |
| m_3 | or load balancer | 22:51 |
| Ryan_Lane | what's the benefit over puppet? | 22:51 |
| m_3 | so there are hooks created for each relation | 22:51 |
| Ryan_Lane | (or chef) | 22:51 |
| m_3 | longer question to answer... lots on the topic though | 22:51 |
| m_3 | lemme get you a list of the hooks here | 22:51 |
| m_3 | http://pastebin.com/aL36P6A5 | 22:52 |
| m_3 | the basic workflow is as follows | 22:52 |
| kim0 | Most people find it an eye openeronce they take some time to understand how Ensemble formulas work under the hood | 22:52 |
| * kim0 makes way for m_3 | 22:53 | |
| m_3 | http://pastebin.com/i2CPibe1 | 22:53 |
| m_3 | kim0: please participate <grin> | 22:53 |
| m_3 | the magic happens in the relation | 22:54 |
| kim0 | :) | 22:54 |
| m_3 | the fact that this is the db relation is just an example | 22:54 |
| m_3 | you'd do the same for memcache, reverseproxy, etc | 22:54 |
| m_3 | so your "stack" would be more complicated than just the script I pasted | 22:54 |
| m_3 | but not _much_ more complicated | 22:54 |
| m_3 | the way the service relation events are fired | 22:55 |
| m_3 | and organized | 22:55 |
| kim0 | I think a main benefit, is that Ensemble is a living system. You are no longer thinking on the level of machines and config files. You're thinking in terms of intelligent "services" that know how to respond to the surrounding environment | 22:55 |
| m_3 | allows you to scale horiz by just 'ensemble add-unit mediawiki' | 22:55 |
| m_3 | rinse and repeat | 22:55 |
| kim0 | and configure themselves, consuming other services, and offering their service to other nodes | 22:55 |
| m_3 | the existing relations between mediawiki and the handful of other services are automatically fired for each new mediawiki unit | 22:56 |
| kim0 | that simplicity of scaling, is a direct result of the fact that services embed the intelligence and know how to react to events | 22:56 |
| adam_g | Ryan_Lane: to answer your question. AFAIK: as it stands right now, when you deploy a new service via ensemble, you're deploying a new instance to host that service. it takes an elastic, cloudy POV that instances are expendable containers of services that will come and go throughout the lifecycle of the service as a whole | 22:56 |
| Ryan_Lane | if I change a service, do the current instances update to reflect that? | 22:57 |
| m_3 | we're working on colocated services within an instance | 22:57 |
| m_3 | Ryan_Lane: that's formula version | 22:57 |
| m_3 | there's upgrade-formula | 22:57 |
| kim0 | Ryan_Lane: yes, you can upgrade formulas on the fly | 22:57 |
| m_3 | and the ability to do rolling upgrades | 22:57 |
| m_3 | issues to be worked out here though! | 22:57 |
| Ryan_Lane | heh | 22:57 |
| Ryan_Lane | so, when upgrading instances, is there some group-based update process? | 22:57 |
| m_3 | rolling upgrades mean different things to different apps, so that's still in dev/testing | 22:58 |
| Ryan_Lane | or do you need to do it per node? | 22:58 |
| m_3 | lemme show you status output | 22:58 |
| m_3 | you can address individual service units (instances) | 22:58 |
| m_3 | but I'm not sure in a upgrade-formula scenario what the options are | 22:59 |
| m_3 | (just started working on it a couple of weeks ago) | 22:59 |
| Ryan_Lane | our staging environment should reflect our production environment as much as possible. | 22:59 |
| Ryan_Lane | including how we do ops | 23:00 |
| m_3 | Ryan_Lane: understood | 23:00 |
| kim0 | again Ensemble is rapidly evolving, so it can adapt to needs | 23:00 |
| m_3 | Ryan_Lane: always particularly fun with a cms! <grin> | 23:00 |
| Ryan_Lane | on our production environment, we may have one box doing a couple things | 23:00 |
| Ryan_Lane | our software is deployed separately from our os related thingd | 23:00 |
| kim0 | Ryan_Lane: and it's also a good ground for experimenting with future ops technology stacks right | 23:00 |
| Ryan_Lane | yeah. if you guys would like to work in the environment, we can do that | 23:01 |
| m_3 | Ryan_Lane: I'd love to talk maybe higher bandwidth or look at docs to learn more about your infra | 23:01 |
| Ryan_Lane | for instance, I can give you guys a project where you can try to model our production environment in ensemble | 23:01 |
| m_3 | yup | 23:01 |
| kim0 | I think that would be very good | 23:01 |
| Ryan_Lane | I think that would be great for both of us | 23:01 |
| * kim0 nods | 23:01 | |
| Ryan_Lane | I like some of the concepts, but it would be very difficult for us to use it now | 23:02 |
| m_3 | gotta run... giving my first ensemble demo at a local LUG | 23:02 |
| m_3 | nice to meet you Ryan! | 23:02 |
| Ryan_Lane | m_3: you too | 23:02 |
| kim0 | m_3: thanks for the help | 23:02 |
| kim0 | Ryan_Lane: Awesome .. we'll both learn a lot by working on this excercise | 23:02 |
| Ryan_Lane | it would be great to see you guys make a similar environment to ours, and show us how we could use it | 23:02 |
| Ryan_Lane | and we can work with you guys to show where things are going to run into problems | 23:03 |
| kim0 | that will surely be needed | 23:03 |
| kim0 | great stuff | 23:03 |
| kim0 | Ryan_Lane: so how do you suggest we get started | 23:03 |
| Ryan_Lane | well, I don't have the cluster fully up yet | 23:04 |
| kim0 | Ryan_Lane: it's openstack right ? | 23:04 |
| Ryan_Lane | also, there's a couple things that I'll likely need to add to openstack for this to properly integrate | 23:04 |
| Ryan_Lane | I think someone is working on it right now | 23:04 |
| Ryan_Lane | the mediawiki extension I wrote adds DNS entries in LDAP when instances are created | 23:05 |
| Ryan_Lane | openstack doesn't have support for adding DNS entries right now | 23:05 |
| Ryan_Lane | for ensemble to work properly, it needs that ;) | 23:05 |
| kim0 | well I'm sure we can beat any technical issues together :) | 23:06 |
| Ryan_Lane | yeah | 23:06 |
| Ryan_Lane | hmm | 23:06 |
| kim0 | perhaps we could even do the work on ec2 for now | 23:06 |
| Ryan_Lane | I may actually be able to give you guys access earlier than others, since you don't need the puppet stuff working | 23:06 |
| kim0 | since that's the most mature provider | 23:06 |
| m_3 | exit | 23:06 |
| kim0 | :) | 23:06 |
| kim0 | Ryan_Lane: yeah we don't | 23:06 |
| m_3 | sorry, wrong focus... doh | 23:06 |
| kim0 | hehe | 23:07 |
| Ryan_Lane | could start, on ec2, yeah. would be good to move you guys over as soon as possible though | 23:07 |
| kim0 | openstack exposes the ec2 api as well, so it should just work | 23:07 |
| Ryan_Lane | yeah. I'm using ec2 api in my extension too | 23:07 |
| Ryan_Lane | you guys will just need the DNS stuff. | 23:07 |
| kim0 | might take some hammering .. but that's what we're good at :) | 23:08 |
| kim0 | Ryan_Lane: so you'll give us access to the cluster, how will we know how your current infrastructure needs to be modelled ? | 23:08 |
| kim0 | like more info on what needs to be done | 23:08 |
| Ryan_Lane | that's the hard part :) | 23:08 |
| kim0 | :) | 23:08 |
| Ryan_Lane | that may be something you need the puppet repo for | 23:09 |
| Ryan_Lane | we have a lot of documentation on wikitech.wikimedia.org | 23:09 |
| Ryan_Lane | but it's not a step-by-step guide | 23:09 |
| Ryan_Lane | we also have configuration files at noc.wikimedia.org | 23:09 |
| Ryan_Lane | for mediawiki/apache | 23:10 |
| kim0 | Ryan_Lane: In the mediawiki community, would there be ops volunteers interested in helping us get this working ? | 23:10 |
| Ryan_Lane | hmm. maybe. | 23:10 |
| kim0 | Cool .. we'll need to broadcast some messages then | 23:10 |
| Ryan_Lane | we are light on ops volunteers. that's one of my goals of the labs project :) | 23:11 |
| kim0 | hehe yeah I know | 23:11 |
| kim0 | very well | 23:11 |
| Ryan_Lane | I can talk to you guys about stuff as you work on it | 23:11 |
| Ryan_Lane | I can give you an overview of how it works | 23:11 |
| Ryan_Lane | you guys located in US? in SF? | 23:11 |
| kim0 | we're everywhere | 23:12 |
| kim0 | :) | 23:12 |
| Ryan_Lane | heh | 23:12 |
| Ryan_Lane | yeah. us too | 23:12 |
| kim0 | some should be close to that .. just no idea who | 23:12 |
| Ryan_Lane | we are going to be doing a hack-a-thon in New Orleans at some point soon | 23:12 |
| Ryan_Lane | it'll be labs focused | 23:12 |
| Ryan_Lane | so that would be a great time to work together if you guys are interested and can come | 23:12 |
| kim0 | Ryan_Lane: I think it would be beneficial to have you on the Ensemble mailing list: http://lists.ubuntu.com/mailman/listinfo/Ensemble | 23:13 |
| Ryan_Lane | otherwise, I can work with you guys online | 23:13 |
| kim0 | sure the hack-athon thing can work too | 23:13 |
| kim0 | I think I'll be sending a summary of this talk to the list, and it would be great to have you over commenting on how to move forward ..etc | 23:13 |
| kim0 | Ryan_Lane: sounds like a plan ? | 23:14 |
| Ryan_Lane | yeah. sounds good | 23:14 |
| kim0 | Ryan_Lane: Great! thanks a lot Ryan .. I really appreciate your time | 23:14 |
| Ryan_Lane | you're welcome. I look forward to working on this with you guys :) | 23:15 |
| kim0 | Awesome, I am very excited as well :) | 23:15 |
| kim0 | We all think Ensemble is bringing a big change, and it rocks to be part of it | 23:15 |
| kim0 | woohoo \o/ | 23:15 |
| kim0 | Okie .. it's getting too late for me now, so I'll catch you guys later | 23:16 |
| Ryan_Lane | kim0: later | 23:16 |
| kim0 | Ryan_Lane: see ya | 23:16 |
| _mup_ | Bug #807281 was filed: Cannot bootstrap stock ubuntu AMIs <Ensemble:New> < https://launchpad.net/bugs/807281 > | 23:20 |
| hazmat | adam_g, not till the sans-ami branch lands, and then it ensemble will work with any cloud-init enabled image ( although we might be dependent on a 10.10 version of cloud-init) | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!