| jimbaker | given that it's easy enough to have the wiki page in rst format, shouldn't really matter in terms of keeping content | 00:22 |
|---|---|---|
| SpamapS | niemeyer: one thing that is proving difficult is parsing out the url in shell scripts. Its a lot easier if the scheme/hostname/port/path are already split up in the relation settings... | 00:29 |
| niemeyer | SpamapS: Hmmm.. that's a nice utility we can provide | 00:47 |
| niemeyer | SpamapS: url -host $FOO | 00:47 |
| devcamcar | hey all, i'm evaluating how ensemble can be used alongside openstack - is there a story around multi-tenancy for ensemble? | 00:47 |
| SpamapS | devcamcar: ensemble *should* work with an openstack implementation, but I don't think anybody has tested it in a while. | 00:48 |
| SpamapS | devcamcar: as far as "multi tenancy" .. each tenant would have their own environment separated by access key ID's | 00:48 |
| SpamapS | niemeyer: it seems a bit redundant to concatenate all of that every time into a URL, just to split it back out half the time into its parts anyway. | 00:49 |
| SpamapS | niemeyer: so far the only two things that require a relation with something that would use the website interface are haproxy and squid reverse proxy.. both of which would need to split it up.. | 00:51 |
| devcamcar | SpamapS: i assumed as much, thanks! | 00:54 |
| niemeyer | SpamapS: A url is by far the most common way to refer to a website.. | 01:11 |
| negronjl | niemeyer: do you have an image-id so I can use m1.large instances ? | 01:26 |
| negronjl | niemeyer: actually does ensemble use a particular image-id or can it use a default ubuntu 11.04 image ? | 01:28 |
| niemeyer | negronjl: We're using a custom base image mostly to speed booting up.. in the future it'll be just a plain Ubuntu image | 01:33 |
| negronjl | niemeyer: thx | 01:33 |
| niemeyer | negronjl: You should be able to just tweak the default-instance-type to m1.large. Is that not working? | 01:34 |
| negronjl | niemeyer: no because m1.large is x86_64 | 01:34 |
| negronjl | niemeyer: the iamge that you are using is i386 | 01:34 |
| niemeyer | negronjl: Hmm.. interesting | 01:35 |
| niemeyer | negronjl: We certainly have one for x86_64 readily available | 01:36 |
| negronjl | niemeyer: if you have it, I can use it. no big deal if it is too much trouble. | 01:37 |
| negronjl | niemeyer: m1.large allows me to work faster ( currently working on tomcat ) | 01:37 |
| niemeyer | negronjl: Yeah, we'll have to generate a new one I think | 01:44 |
| negronjl | niemeyer: no worries | 01:45 |
| niemeyer | negronjl: It's not a big deal, we have a tool to automate it | 01:45 |
| negronjl | niemeyer: if/when you get it, ping me and let me know pleaase | 01:45 |
| niemeyer | negronjl: Will do.. will try to have that available for you tomorrow | 01:45 |
| negronjl | niemeyer: cool. thx | 01:45 |
| _mup_ | ensemble/relation-get-amp-exceptions r259 committed by bcsaller@gmail.com | 01:59 |
| _mup_ | fix for #792071 | 01:59 |
| _mup_ | ensemble/relation-context-required r242 committed by bcsaller@gmail.com | 02:01 |
| _mup_ | Fix for #792071 with proper pre-req branch | 02:01 |
| m_31 | negronjl: ping | 04:53 |
| === _mup__ is now known as _mup_ | ||
| negronjl | m_31: pong | 07:02 |
| m_31 | negronjl: hey man... which orchestra scripts are you porting over to ensemble next? | 07:21 |
| negronjl | tomcat | 07:21 |
| negronjl | I have tomcat working but, I haven't gotten to the clustering part yet. | 07:22 |
| m_31 | negronjl: cool... are you going by some hitlist? | 07:22 |
| negronjl | I'll do that tomorrow or so | 07:22 |
| negronjl | nah....just picking them at random. | 07:22 |
| negronjl | low hanging fruit + interesting stuff | 07:22 |
| negronjl | are you porting any of them ? | 07:22 |
| m_31 | cool. I'm trying to get up to speed on formulas in general | 07:23 |
| m_31 | working on a rails one at the moment | 07:23 |
| negronjl | cool. these guys are pretty helpful. I know I asked a million stupid questions today alone :) | 07:23 |
| m_31 | but I'll dump the hadoop one and try to build some demos on top of yours | 07:23 |
| m_31 | I was just starting on the relations for them, but what you did makes sense | 07:24 |
| === almaisan-away is now known as al-maisan | ||
| negronjl | m_31: didn't mean to step all over your work btw. | 07:26 |
| m_31 | dude, no problem at all... it was cool to see your relations | 07:27 |
| m_31 | I'll coordinate with you before hitting any more orchestra ones | 07:29 |
| negronjl | cool | 07:29 |
| m_31 | gonna hit the sack... talk to you tomorrow | 07:32 |
| negronjl | m_31: gnite | 07:32 |
| === daker_ is now known as daker | ||
| === al-maisan is now known as almaisan-away | ||
| kim0 | I wonder if 'ensemble set' has landed yet | 12:10 |
| _mup_ | Bug #798652 was filed: Ensemble needs bash completion support <Ensemble:New> < https://launchpad.net/bugs/798652 > | 12:46 |
| * niemeyer waves | 14:44 | |
| === almaisan-away is now known as al-maisan | ||
| niemeyer | Such a quiet morning... must be Friday! | 15:54 |
| kim0 | hmm | 16:28 |
| kim0 | memcached_ips.append("'%s'" % getaddrinfo(settings['host'],int(settings['port']))[4][0:2].join(':')) | 16:28 |
| kim0 | breaks when getaddrinfo returns a list | 16:28 |
| kim0 | need to ping SpamapS | 16:28 |
| kim0 | SpamapS: this one seems to work for me → memcached_ips.append("'%s'" % ':'.join(map(str,getaddrinfo(settings['host'],int(settings['port']))[0][4][0:2]))) | 16:36 |
| kim0 | that's in mediawiki/hooks/cache-relation-changed | 16:36 |
| === al-maisan is now known as almaisan-away | ||
| negronjl | good morning all | 16:52 |
| hazmat_ | negronjl: g'morning | 16:52 |
| negronjl | hi hazmat_ | 16:52 |
| m_3 | negronjl: morning | 16:57 |
| negronjl | morning m_3 | 16:57 |
| === almaisan-away is now known as al-maisan | ||
| kim0 | morning | 17:17 |
| niemeyer | negronjl: Morning! | 17:24 |
| negronjl | niemeyer: gmorning | 17:24 |
| SpamapS | kim0: Oh I have that change in my local branch and forgot to push it! | 17:29 |
| SpamapS | kim0: getaddrinfo *always* returns a list | 17:29 |
| kim0 | ah ok then | 17:30 |
| kim0 | SpamapS: is there anything in the default haproxy config that would make it balance unfairly ? (i.e. one machin getting ~100 hits, while the other getting ~20) ? | 17:32 |
| SpamapS | kim0: not that I know of, but I have seen that | 17:32 |
| SpamapS | kim0: I'm, unfortunately, not at all versed in haproxy-fu | 17:33 |
| kim0 | yeah me neither | 17:33 |
| kim0 | I'm testing with: ab -n 1000 -c 100 http://localhost/mediawiki/index.php/Main_Page | 17:33 |
| kim0 | from the haproxy machine | 17:33 |
| SpamapS | was thinking about doing an ipvs formula for that very reason. :-P | 17:33 |
| SpamapS | kim0: be careful, ab may be using keepalives | 17:33 |
| SpamapS | kim0: honestly, ab *sucks* | 17:33 |
| kim0 | hehe | 17:33 |
| SpamapS | its nothing at all like real traffic | 17:33 |
| kim0 | I think it needs -k for keep alives | 17:34 |
| negronjl | SpamapS: I noticed that on the wordpress formula, the website-relation-joined is setting the hostname as hostname -s as opposed to hostname -f. am I missing something ? | 17:34 |
| SpamapS | negronjl: no, thats a bug | 17:34 |
| negronjl | SpamapS: ok. I'll make sure not to do that :) | 17:34 |
| SpamapS | negronjl: though one that doesn't affect anything because all machines have the same search domain (.internal) | 17:34 |
| kim0 | trying to get a nice linear boost in #/sec when adding machine .. real life is complex though ;) | 17:35 |
| SpamapS | negronjl: I hardly ever look at the principia wordpress formula.. mediawiki is much better maintained. :) | 17:35 |
| negronjl | SpamapS: any reason why there isn't any apache formula ( just apache ) ? | 17:35 |
| SpamapS | kim0: we need to enable the admin interface on some port so we can watch it | 17:35 |
| SpamapS | negronjl: because apache is too generic | 17:35 |
| negronjl | SpamapS: k | 17:36 |
| SpamapS | negronjl: where would it get its files to serve? | 17:36 |
| SpamapS | negronjl: I guess we can have one once we have shared filesystem formulas | 17:36 |
| negronjl | SpamapS: you can create the site and then have the user upload the files. | 17:36 |
| SpamapS | negronjl: thats not very interesting though, is it? | 17:36 |
| SpamapS | like.. thats a single line of cloud-config | 17:36 |
| negronjl | SpamapS: I agree with it being generic but, I'm afraid that if you want ensemble to be widely used, you'll have to have something generic like that unless the plan is to have the users generate all of their formulas on their own and just use ensemble to deploy stuff | 17:37 |
| SpamapS | A formula where you also have a webdav interface, authentication of some kind, and rsyncing between nodes via peer relations.. that might be cool. :) | 17:37 |
| SpamapS | negronjl: walk me through the user experience you're envisioning | 17:38 |
| negronjl | SpamapS: .... in orchestra I did a generic apache that creates a site for you and then it leaves it up to you to upload the files. this way you can concentrate on creating your content and not battle apache with the details | 17:38 |
| SpamapS | negronjl: does *anybody* create static sites? | 17:38 |
| SpamapS | negronjl: I'd say that joomla/drupal/{insert cms here} is far more useful for the generic site case. | 17:39 |
| hazmat_ | i see a formula around that being more like deploy generic rails app or deploy wsgi app.. | 17:39 |
| SpamapS | negronjl: Actually a really awesome formula would be a mod_rewrite based box that just aggregates all the other apps into one site. | 17:39 |
| negronjl | SpamapS: so, what if I have my own app. is the expectation that I will create my own formula that will deploy apache, etc. and all of my files ? | 17:39 |
| SpamapS | hazmat_: that just turns into a template.. since to configure generic rails app is not at all the same.. some are db driven, some have queues.. | 17:40 |
| SpamapS | Though templates would be good | 17:40 |
| niemeyer | hazmat_: I was wondering yesterday if we should start using plain Ubuntu images rather than our own customized ones | 17:40 |
| niemeyer | I wonder what the timing impact of that would really be | 17:40 |
| negronjl | +1 on regular images | 17:40 |
| SpamapS | negronjl: well.. you are talking about a single line of bash.. apt-get install apache2 ;) | 17:40 |
| negronjl | try to deploy a hadoop cluster or tomcat on m1.small and you'll see what I mean | 17:41 |
| negronjl | SpamapS: and all of the hooks for haproxy as well | 17:41 |
| negronjl | SpamapS: and all of the mods that go into my app (php, python, etc.) | 17:41 |
| niemeyer | negronjl: We can provide you with a m1.large anyway, and we'll definitely use plain base images in the future no matter what. Just wondering if we should ignore the extra time it takes to do the extra tweaks before we integrate them. | 17:41 |
| SpamapS | negronjl: I like the idea of a template for these. | 17:41 |
| hazmat_ | SpamapS: true, optional relations for queues would be doable, but your right the heart of it is configuration based dependencies, which fall outside ensemble's scope atm. but a lot of the simple rails/django apps are just the core functionality (db server, and optional memcache).. django at least has a reasonably standard way of configuring this stuff | 17:42 |
| negronjl | SpamapS: I see your poing about it being too generic. I just think that we can't possibly know all of the apps out there so, something generic for the rest may not be a a bad idea. | 17:42 |
| SpamapS | hazmat_: +1 for framework formulas | 17:42 |
| negronjl | SpamapS: Is there something like erb that can be used with ensemble ( does it matter what I use )? | 17:43 |
| m_3 | seems like it'd be useful for a rails or django framework that pulls code from a user-specified repo | 17:43 |
| SpamapS | negronjl: You can use *anything* | 17:43 |
| kim0 | I use sed :) | 17:43 |
| negronjl | SpamapS: I like that :D | 17:43 |
| * SpamapS likes augtool when the lenses are available | 17:43 | |
| m_3 | cat > file <<EOS | 17:43 |
| SpamapS | negronjl: you can use *puppet* if you're so inclined. :) | 17:44 |
| * negronjl is going to start using BASIC :D | 17:44 | |
| m_3 | ha | 17:44 |
| kim0 | haha | 17:44 |
| * negronjl is j/k btw | 17:44 | |
| SpamapS | yeah rails-app should actually be a doable formula.. since configuring the known components will be easy. And then people can add to it for their app if they want things different. | 17:44 |
| m_3 | SpamapS: almost done with it now | 17:45 |
| niemeyer | negronjl: Phew! ;-) | 17:45 |
| m_3 | rails3/passenger/pulls the app from github | 17:45 |
| * niemeyer was worried for a second we'd go back to Basic | 17:45 | |
| * SpamapS earned his first programming $$ writing "PickBasic" | 17:47 | |
| SpamapS | not PIC BASIC | 17:47 |
| niemeyer | SpamapS: I started with Basic as well, in the mid-80s | 17:47 |
| m_3 | trs80 | 17:47 |
| m_3 | ugh | 17:47 |
| SpamapS | http://en.wikipedia.org/wiki/Pick_operating_system | 17:48 |
| SpamapS | The OS that you never heard of, but that made DELL successful. :) | 17:49 |
| SpamapS | It was the "ONS" .. as in.. "Original No-SQL" | 17:49 |
| SpamapS | D3 actually had the distinction of being the first commercial database software available on Linux | 17:54 |
| negronjl | trs80 rules! | 17:55 |
| m_3 | negronjl: with the big 8" floppy drives even | 17:55 |
| negronjl | m_3: I didn't have $$ for that so, I had the cassette tapes :) | 17:56 |
| m_3 | negronjl: actually seen a revival of one of those recently where they gutted it and put a modern computer inside | 17:56 |
| m_3 | yeah, they had them at school | 17:56 |
| negronjl | m_3: There is an emulator in the archives: xtrs | 17:57 |
| m_3 | ok, height of sheer laziness: http://paste.ubuntu.com/628504/ | 17:58 |
| m_3 | negronjl: nice | 18:00 |
| m_3 | simple rails formula works now... does a basic install, pulls code during db-relation-joined with mysql, then spins up passenger | 18:04 |
| m_3 | I'll get it into launchpad | 18:04 |
| m_3 | I'm gonna dig through the docs and examples now to figure out the best way to externalize things like "application_name" "application_repo_url" etc | 18:05 |
| SpamapS | bcsaller: did I see you start a branch for fixing the dot output? | 18:11 |
| bcsaller | SpamapS: it works now | 18:12 |
| SpamapS | m_3: config settings might be all thats needed.. 'ensemble set my-rails-app pullcmd="svn co svn://foo /srv/app && /srv/app/setup.rb" | 18:12 |
| m_3 | SpamapS: perfect | 18:13 |
| SpamapS | m_3: though thats where I think, instead, we should just encourage people to make that into a formula for their app | 18:13 |
| bcsaller | SpamapS: it works on that branch rather, it was a minor issue. Just needs reviews | 18:14 |
| SpamapS | Actually, once stacks are available, that will be the right way to do it probably. | 18:14 |
| SpamapS | bcsaller: well, let me see if I can perform a review by testing it. :) | 18:14 |
| negronjl | How do I define dynamic options to a formula? Is there a difference? | 18:14 |
| * SpamapS wants purty graphs | 18:14 | |
| m_3 | SpamapS: right, understand | 18:14 |
| SpamapS | negronjl: thats still in dev IIRC | 18:14 |
| negronjl | SpamapS: k. thx | 18:15 |
| SpamapS | negronjl: I was reading a little more about HDFS and hadoop and stuff. I wonder if we shouldn't go a little further and write Flume and Pig formulas so people get a single way to shove data in and query it. | 18:15 |
| negronjl | SpamapS: I would say hive first. In my experience, it would be hadoop then hive then pig and have no idea what flume is. | 18:16 |
| SpamapS | negronjl: with the map/reduce .. I'm not sure I understand how that scales out ... seems like the slave/master only scales the I/O .. not the compute | 18:16 |
| negronjl | SpamapS: but, I agree with the principle | 18:16 |
| SpamapS | negronjl: flume is an easy way to get data into hadoop | 18:16 |
| * SpamapS reads up on hive | 18:16 | |
| negronjl | SpamapS: hive => SQL interface to hadoop | 18:17 |
| SpamapS | I thought thats what PIG was ;) | 18:17 |
| negronjl | SpamapS: you write things like SELECT * from <TABLE> and hive translates that into hadoop | 18:17 |
| negronjl | SpamapS: pig => easier hive with less options and bells and whistles | 18:17 |
| SpamapS | Ahh cool | 18:18 |
| negronjl | SpamapS: people that use hive look down on the people that use pig :) | 18:18 |
| negronjl | SpamapS: but, I agree to take the formulas a step further.. | 18:18 |
| negronjl | SpamapS: Let me finish tomcat first. | 18:18 |
| negronjl | SpamapS: working on tomcat clustering | 18:18 |
| SpamapS | So.. since all of thise cool stuff is not in Ubuntu yet.. I'm getting more and more tempted to make principia make a separate mrconfig that includes all this PPA coolness.. | 18:19 |
| SpamapS | negronjl: tomcat, in the same sense as m_3's rails? | 18:19 |
| negronjl | SpamapS: for now, in the install, I install the PPA for hadoop. | 18:19 |
| negronjl | SpamapS: for tomcat, it's a bit different because tomcat gives you their admin page that allows you to point and click your way to deploy your WAR files | 18:20 |
| negronjl | SpamapS: so, you can just isntall tomcat and allow it to cluster, put it behind haproxy for LB and you're done. | 18:20 |
| m_3 | hbase too? | 18:20 |
| SpamapS | negronjl: I'm more wondering what tomcat's relations will be other than website. :) | 18:20 |
| negronjl | m_3: hbase too :) | 18:20 |
| negronjl | SpamapS: none | 18:21 |
| negronjl | SpamapS: it also provides another thing: tomcat-cluster ( that where the clustering takes place ) as a peer | 18:21 |
| SpamapS | negronjl: I think we should start exposing an ssh interface for code syncing. | 18:21 |
| negronjl | SpamapS: tomcat has it's own thing on a diff port. | 18:21 |
| SpamapS | tomcat clustering? | 18:21 |
| SpamapS | like, it already will distribute your war file? | 18:22 |
| negronjl | SpamapS: yup | 18:22 |
| negronjl | SpamapS: hence the peer interface | 18:22 |
| SpamapS | *hot* | 18:22 |
| * SpamapS hugs negronjl | 18:22 | |
| SpamapS | go go go | 18:22 |
| * SpamapS brings a case of redbull | 18:22 | |
| SpamapS | GO | 18:22 |
| negronjl | SpamapS: rofl. working from Starbucks so, wired on coffee :D | 18:22 |
| SpamapS | m_3: half those redbulls are for you. :) | 18:22 |
| * SpamapS wishes he had time to write formulas all day.. | 18:23 | |
| m_3 | I like how the debug-log intersperses parallel traffic... has anybody written an log-scanning tools (to separate out units?) | 18:23 |
| negronjl | SpamapS: Hopefully by Dublin, I'll have enough karma coolness to ask you guys to sponsor my UbuntuContributingDeveloper application ( https://wiki.ubuntu.com/JuanNegron/UbuntuContributingDeveloper ) :) | 18:23 |
| negronjl | shameless plug above :D | 18:24 |
| SpamapS | m_3: the logs are available on the individual units in /var/lib/ensemble/units/$service-$unitid/formula.log | 18:24 |
| m_3 | SpamapS: thanks... undercaffeinated(sp?) at the moment | 18:24 |
| m_3 | ah, thanks | 18:24 |
| _mup_ | ensemble/debug-log-relation-settings-changes r259 committed by jim.baker@canonical.com | 18:24 |
| _mup_ | Initial commit | 18:24 |
| SpamapS | negronjl: no way I'm going to trash it so that all you can do is work on formulas. | 18:24 |
| SpamapS | ;) | 18:24 |
| negronjl | SpamapS: perfect! I can see the logs!!!!! :D | 18:24 |
| SpamapS | m_3: but also the debug-log has -x and -i | 18:24 |
| SpamapS | so like 'ensemble debug-log -i 'demo-wiki*' | 18:25 |
| m_3 | this is really a great tool! | 18:26 |
| m_3 | horizontal scaling: for i in {1..10}; do eau rails; done | 18:26 |
| negronjl | SpamapS: ( and anyone else who may know the answer ). what's the way to see as much as possible of what is happening to the formula as it is being deployed ? | 18:26 |
| negronjl | I want to know as much as possible | 18:26 |
| negronjl | about what's happening to the instance as the formula is being deployed | 18:26 |
| jimbaker` | m_3, it might be interesting to analyze the debug log, possibly with alternative formatters, to get at things like swimlanes/sequence diagrams | 18:27 |
| m_3 | jimbaker`: right... all the info's there... just gotta be parsed | 18:28 |
| hazmat_ | m_3: you can specify individual units, machines, services, log channels via the filtering (include/exclude) options on the logger | 18:28 |
| m_3 | hazmat_: yep thanks... need to spend time digging through docs of options | 18:30 |
| SpamapS | negronjl: debug-hooks is probably the best way then | 18:30 |
| jimbaker` | m_3, right, and bug 766317, which i'm working on (albeit it may be controversial), will further enhance this | 18:30 |
| _mup_ | Bug #766317: debug-log should show relation settings changes <Ensemble:In Progress by jimbaker> < https://launchpad.net/bugs/766317 > | 18:30 |
| SpamapS | negronjl: you can run the hook with strace and/or gdb at that point. :) | 18:30 |
| m_3 | SpamapS: yeah, totally wanna dig through debug-hooks next... | 18:30 |
| negronjl | SpamapS: cool. thx | 18:30 |
| SpamapS | its awesome, though I'm bummed that it uses tmux now.. as I'm starting to love byobu's alt-pgup/pgdn | 18:30 |
| m_3 | lots of cleanup to do first in formula writing (idempotency checking, logging, using debconf as much as possible, etc) | 18:31 |
| SpamapS | I do think debug-hooks needs to tell me where the hook is that I'm supposed to be running.. | 18:31 |
| m_3 | SpamapS: never did dig through and figure out the session problems with screen -vs- tmux | 18:31 |
| kirkland | SpamapS: niemeyer and i exchanged a few emails; i think i gave him a solution (using run-one) for the race condition in screen | 18:32 |
| kirkland | SpamapS: it solved it for me, at least | 18:32 |
| m_3 | kirkland: I do love me some screen | 18:33 |
| kirkland | m_3: have you tried byobu? | 18:33 |
| m_3 | kirkland: nope | 18:33 |
| kirkland | m_3: give 'er a shot | 18:33 |
| m_3 | kirkland: actually, I think it installs together now right? | 18:33 |
| kirkland | m_3: screen on steriods | 18:33 |
| kirkland | m_3: yeah, just type 'byobu' instead of 'screen' | 18:33 |
| kirkland | m_3: it's screen under the covers | 18:34 |
| niemeyer | kirkland: It's a solution, but it's one that makes me quite unhappy :( | 18:34 |
| kirkland | m_3: with a bunch of configuration enhancements, status scripts, and keybindings | 18:34 |
| kirkland | niemeyer: and why is that? | 18:34 |
| niemeyer | kirkland: As you have noticed, these scripts are already far from pleasant overall | 18:34 |
| negronjl | I'm looking for something similar to erb ( the ruby template thing ). Is there anything that you guys currently use/recommend for ensemble formulas ? | 18:34 |
| niemeyer | kirkland: Involving additional locks for something that screen should do internally is what's sad | 18:35 |
| m_3 | negronjl: I think the idea is that things are flexible, so use any of them | 18:35 |
| SpamapS | negronjl: I like augeas if the format is a common one (like ini files) .. otherwise cheetah works. | 18:35 |
| m_3 | negronjl: we've got a tradeoff between creating examples that show that a user has options | 18:35 |
| kirkland | niemeyer: would you prefer that I patched screen? | 18:35 |
| m_3 | negronjl: but not confuse them with too much noise | 18:35 |
| SpamapS | negronjl: honestly though, if you like erb, use erb | 18:36 |
| SpamapS | I'd actually say use whatever the most clear and simple thing is to read | 18:36 |
| SpamapS | that may be where cheetah actually fails a bit. :-P | 18:36 |
| negronjl | SpamapS: in my formulas, can I add another directory either besides hooks or under hooks for my template files ? | 18:36 |
| niemeyer | kirkland: Yes, I'd certainly wish that screen implemented this logic by itself in a reliable way | 18:37 |
| SpamapS | negronjl: yes the entire root of the formula is copied | 18:37 |
| negronjl | SpamapS: :D .. tomcat clustering here I come :) | 18:37 |
| m_3 | SpamapS: negronjl: lib! | 18:37 |
| SpamapS | negronjl: you can send binaries, though it is, or at least will be, forbidden until we see a good reason for it. :) | 18:37 |
| negronjl | SpamapS: no binaries...just template files ( text files ) | 18:38 |
| niemeyer | kirkland: I don't know if you have seen the changeset that introduced the tmux/screen replacement | 18:38 |
| kirkland | niemeyer: it was a large one, as i recall | 18:38 |
| SpamapS | how about $formula/hooks/share/templates | 18:38 |
| m_3 | SpamapS: it can be confusing to users if we start using templating tools that have to be installed during the "install" hooks | 18:38 |
| koolhead17 | hi all | 18:39 |
| niemeyer | kirkland: There were several small issues we had to fix there | 18:39 |
| m_3 | koolhead17: hi | 18:39 |
| SpamapS | m_3: I believe we've discussed having required packages be in the metadata at some point. | 18:39 |
| koolhead17 | hi m_3 | 18:39 |
| m_3 | SpamapS: ok, that'd fix it | 18:40 |
| kirkland | niemeyer: okay; here's another approach | 18:40 |
| kirkland | niemeyer: i have begun work on a byobu profile for tmux | 18:40 |
| kirkland | niemeyer: which would duplicate byobu's keybindings and status notifications for tmux | 18:40 |
| koolhead17 | SpamapS: thanks :) | 18:41 |
| kirkland | niemeyer: at your request, i took a look at tmux and I have to say there are some things to really like about it | 18:41 |
| kirkland | niemeyer: not the least of which is the vibrant, active development community (as compared to screen) | 18:41 |
| koolhead17 | niemeyer: hey | 18:41 |
| SpamapS | kim0: fixed the mediawiki formula btw | 18:41 |
| kim0 | SpamapS: great thanks | 18:41 |
| * kim0 starts weekend .. @everyone enjoy | 18:42 | |
| koolhead17 | kim0: :P | 18:42 |
| SpamapS | kim0: cheers! thanks for the intense week! | 18:42 |
| kirkland | niemeyer: if the current byobu/screen/run-one locking workaround is unacceptable to you, then I can generate a byobu profile for tmux | 18:42 |
| m_3 | SpamapS: we might need more than principia... maybe just examples or contrib? | 18:42 |
| niemeyer | kirkland: Oh, sweet | 18:42 |
| niemeyer | kirkland: Yeah, that would definitely be nice | 18:42 |
| kirkland | niemeyer: the main thing I hated seeing was your embedding of tmux configuration into ensemble itself (ie, adding some helpful keybindings, etc.) | 18:42 |
| m_3 | kim0: later... enjoy | 18:43 |
| niemeyer | kirkland: Heh :) | 18:43 |
| kirkland | niemeyer: when ***so*** much of that effort has already been directed elsewhere | 18:43 |
| SpamapS | m_3: the wordpress and mysql examples in ensemble itself are pretty straight forward.. | 18:43 |
| niemeyer | kirkland: Sorry, I don't see what you mean? | 18:43 |
| SpamapS | m_3: as far as contrib .. I'm hoping that can just be a "component" of principia.. one that is still easy to find | 18:43 |
| m_3 | SpamapS: I mean there are some formulas to create (like mahout) that might not belong in the "Principia" | 18:43 |
| niemeyer | kirkland: What you hated specifically, and what effort has been directed where? | 18:43 |
| kirkland | niemeyer: your placement a user friendliness layer on top of tmux | 18:44 |
| m_3 | SpamapS: gotcha... it'd be useful for users if we drew a line... _these_ are sort of blessed formulas that will carefully be maintained | 18:44 |
| niemeyer | kirkland: You're upset I made it user friendly? | 18:44 |
| kirkland | niemeyer: when there's a somewhat common one elsewhere | 18:44 |
| m_3 | SpamapS: and _those_ are just crap examples we're excited about | 18:44 |
| kirkland | niemeyer: i can try to port that over to tmux | 18:44 |
| kirkland | niemeyer: http://paste.ubuntu.com/628519/ | 18:44 |
| kirkland | niemeyer: that's what i have so far | 18:44 |
| kirkland | niemeyer: meager start, but it's committed to byobu's head and in a micro release already | 18:45 |
| SpamapS | m_3: The line I want to draw is whether or not the software that they deploy is from Ubuntu. The formulas, to be in principia, will all be as well maintained regardless of which component. At least, thats how I think it should work. | 18:45 |
| kirkland | niemeyer: i can hack on it more this weekend | 18:45 |
| niemeyer | kirkland: Sorry.. I'm completely missing your point. You're upset because I copy & pasted my tmux.conf rather than using the one you made moments ago? | 18:45 |
| kirkland | niemeyer: of course not | 18:46 |
| SpamapS | m_3: mahout looks completely interesting as a formula. :) | 18:46 |
| m_3 | SpamapS: ok, sound good. I was just thinking of all sorts of random stuff.. not all formulas are the same sort of base level service (like mysql) | 18:47 |
| SpamapS | m_3: sure. I like the idea of creating a formula around a framework. | 18:48 |
| m_3 | SpamapS: I'm thinking we'll eventually have a web interface to manipulate services? we'd have search, and categorizations by provides and requires | 18:48 |
| m_3 | SpamapS: like building blocks that get wired together | 18:49 |
| SpamapS | m_3: there's talk of making that available as part of landscape (as in, for $$) | 18:49 |
| === al-maisan is now known as almaisan-away | ||
| SpamapS | though nobody would be precluded from doing that themselves | 18:49 |
| m_3 | SpamapS: hmmm.. fun stuff over Irish beer | 18:49 |
| kirkland | SpamapS: reviewing txzookeeper | 19:09 |
| kirkland | SpamapS: ./setup.py: license="LGPL", | 19:09 |
| kirkland | SpamapS: while debian/copyright says MIT for *, and BSD for one file | 19:09 |
| kirkland | niemeyer: perhaps you can help | 19:12 |
| kirkland | niemeyer: i'm reviewing txzookeeper for the archive | 19:12 |
| kirkland | niemeyer: https://launchpad.net/txzookeeper says GNU LGPL v3 | 19:13 |
| kirkland | niemeyer: setup.py says license="LGPL" | 19:13 |
| kirkland | niemeyer: debian/copyright says License: MIT | 19:13 |
| kirkland | niemeyer: looks like a bug in debian/copyright, as far as I can tell | 19:14 |
| niemeyer | kirkland: Isn't the debian/copyright talking about the license for things within debian/*? | 19:15 |
| niemeyer | kirkland: txzookeeper is LGPL indeed | 19:15 |
| kirkland | niemeyer: there should be at least 2 stanzas | 19:15 |
| kirkland | niemeyer: one for the packaging (in debian/*) | 19:15 |
| kirkland | niemeyer: but the first one should be for the code that is packaged | 19:16 |
| kirkland | niemeyer: i've rejected it for now, sending a note to SpamapS | 19:16 |
| kirkland | niemeyer: as soon as he fixes that, and re-uploads I'll accept it | 19:16 |
| niemeyer | kirkland: Cool, thanks | 19:16 |
| kirkland | niemeyer: fwiw, i just accepted ensemble into oneiric; congratulations | 19:16 |
| kirkland | niemeyer: i think it'll fail to install until we get txzookeeper too, though | 19:16 |
| kirkland | niemeyer: hopefully we'll sort all of it out today | 19:16 |
| niemeyer | kirkland: Woooo | 19:17 |
| SpamapS | kirkland: yeah setup.py was probably cargo-culted in | 19:18 |
| m_3 | awesome... congrats team | 19:18 |
| SpamapS | the only bit with an actual license text was the upstream debian/copyright | 19:18 |
| SpamapS | kirkland: thanks, I think we need upstream (read: hazmat and niemeyer) to clarify licensing. :) | 19:19 |
| kirkland | SpamapS: niemeyer: true; on both projects, upstream MUST add a LICENSE file with the complete text of the license | 19:19 |
| SpamapS | I think an upstream debian/copyright is authoritative, but it should be made clear in the files. | 19:20 |
| kirkland | SpamapS: i was might have rejected ensemble on that basis, but I know it's trivial and niemeyer will fix it asap ;-) | 19:20 |
| * niemeyer hides | 19:22 | |
| SpamapS | hazmat_: think you could ram some license clarification into txzookeeper ? | 19:22 |
| hazmat_ | SpamapS: sure what do you need... i think we had discussed gpl/lgpl.. but possibly apache if it went upstream.. niemeyer ? | 19:28 |
| hazmat_ | hmm but it has mit license in the copyright deb file | 19:29 |
| SpamapS | hazmat_: well the point is, what is the license "today" ? | 19:29 |
| hazmat_ | SpamapS: lgpl at the moment | 19:31 |
| niemeyer | hazmat_: Yes, it's LGPL, and we can move to whatever the ZooKeeper folks wish if we get it upstream | 19:31 |
| hazmat_ | we've got an okay to relicense it to apache when it heads upstream | 19:31 |
| * negronjl is out to lunch | 19:38 | |
| kirkland | SpamapS: push another upload and i'll approve | 19:54 |
| SpamapS | kirkland: w/o any evidence in the upstream bits? | 20:03 |
| SpamapS | So, can we just add a COPYING file to the bzr tree with the LGPL, and reference to it in at *least* one .py file. Preferrably all that are non-trivial. | 20:07 |
| SpamapS | I can open a bug and do the work myself if you guys prefer. | 20:07 |
| === almaisan-away is now known as al-maisan | ||
| === al-maisan is now known as almaisan-away | ||
| niemeyer | SpamapS: Sounds good to me either way | 21:13 |
| niemeyer | SpamapS: I can push a change, commit, or even add you to the project if you prefer that | 21:13 |
| === daker is now known as daker_ | ||
| * negronjl is back | 22:19 | |
| * niemeyer waves | 22:24 | |
| negronjl | niemeyer: are the formulas and scripts ( install, etc.) run as root ? | 22:26 |
| niemeyer | negronjl: THey are | 22:26 |
| hazmat_ | niemeyer: so the overall plan for session events and connection errors, is to allow user defined callbacks (one for each category) to be defined on the connection. session events are by default ignored, connection errors by default errback on the deferred, if connection level callbacks are specified they'll take precedence.. the notion being we can define a connection error handler at the agent level, which can take care | 22:42 |
| niemeyer | hazmat_: In which way might we take care of the connection error? | 22:43 |
| hazmat_ | niemeyer: restarting the agent | 22:43 |
| niemeyer | hazmat_: I see.. so aborting it all | 22:44 |
| jimbaker` | hazmat, sounds like a reasonable policy - the provisioning agent certainly can be restarted and resynced w/ zk, this should be true of all of our agents | 22:44 |
| hazmat_ | niemeyer: yes, we could be in any arbitrary point in the code, dealing with any arbitrary state change | 22:44 |
| niemeyer | hazmat_: Yes, exactly.. that's my concern | 22:45 |
| hazmat_ | niemeyer: till we have reconcilation and on disk state, we'll get duplicate hook execution for the joins, and potentially miss some changes. | 22:46 |
| niemeyer | hazmat_: You mean it's a bomb? :) | 22:46 |
| hazmat_ | niemeyer: but i don't see how that's avoidable, its a networked system, and subject to disconnect at any time for any period. | 22:46 |
| niemeyer | hazmat_: Error handling in the face of disconnections is a normal problem | 22:46 |
| hazmat_ | niemeyer: alternatives? | 22:47 |
| niemeyer | hazmat_: Error handling? | 22:48 |
| hazmat_ | niemeyer: wrap connection eror handling around every zk interaction? | 22:48 |
| niemeyer | hazmat_: Yes, handle errors they can happen, basically | 22:48 |
| niemeyer | s/they/when they/ | 22:48 |
| niemeyer | hazmat_: Eventually we should have a "disconnected" hook | 22:50 |
| hazmat_ | niemeyer: these aren't normal api errors, their arbitrary connection errors, defining a central place for that logic seems like a win to me, if we want to recover without the spurious hook execution, we need consistent on disk state reflecting what the hooks have been made aware of, to reconcile to the current zk state after the connection is resumed.. | 22:50 |
| niemeyer | hazmat_: My concern is that by "central logic" you mean "crashing" | 22:51 |
| niemeyer | hazmat_: The problem is that arbitrary connection errors are actually normal errors | 22:56 |
| hazmat_ | niemeyer: it doesn't have to be, but anything more intelligent needs resumable components (on disk reconcilliation to zk imo), but we can't assume the unit agent process is always alive or connected at all times, and right now we have volatile memory state that needs to be captured for any process restarts. alternatively though.. it could just reattempt to connect ad naseum... but then we have to determine if our session | 22:56 |
| hazmat_ | niemeyer: their not the same imo.. they effect more than just the current api call, they can effect a slew of out of band/path things | 22:57 |
| niemeyer | hazmat_: and that's normal! | 22:57 |
| niemeyer | hazmat_: The fact we haven't been handling them as such is why it's hard now | 22:58 |
| * hazmat_ ponders | 22:59 | |
| hazmat_ | niemeyer: the problem of watchers and ephemeral nodes re-establishment, isn't a local concern, how would local error handling be able to compensate for a session expiry | 23:05 |
| niemeyer | hazmat_: It's not about compensating.. it's about handling | 23:06 |
| niemeyer | hazmat_: If error, then log problem, wait for reconnection, try agian, whatever | 23:06 |
| hazmat_ | niemeyer: ok.. how would it handle a session expiration | 23:06 |
| hazmat_ | niemeyer: it can do that locally, but the global application state also needs restoration | 23:07 |
| niemeyer | hazmat_: Assuming that the best thing to do in every single path of the application is to crash in case a disconnection happens isn't a good approach IMO | 23:07 |
| hazmat_ | all of the extant watchers and ephemeral nodes | 23:07 |
| niemeyer | hazmat_: Yes, all of that has to be considered.. | 23:08 |
| niemeyer | hazmat_: Or, we change the model completely | 23:09 |
| hazmat_ | niemeyer: such as? | 23:10 |
| niemeyer | hazmat_: Such as a pull model.. | 23:11 |
| niemeyer | hazmat_: Get the state and diff | 23:11 |
| hazmat_ | niemeyer: bingo.. i think that's where we need to go, i think the global recovery can evolve from crash to effecting state diff reconcilliation | 23:11 |
| niemeyer | hazmat_: That's not the same thing | 23:12 |
| niemeyer | hazmat_: This feels like doing things the reversed way | 23:13 |
| hazmat_ | the individual components do the state diff reconcilliation | 23:13 |
| niemeyer | hazmat_: We have a bunch of things running we have no idea, then we stop everything that was running because we don't know how to handle errors, and then try everything again | 23:13 |
| hazmat_ | be it service config, or relation watching, etc | 23:13 |
| niemeyer | hazmat_: The forward approach is: we always know the next step | 23:14 |
| hazmat_ | we stop and reconcile because none of those components are functional if we don't | 23:14 |
| niemeyer | hazmat_: Because they ignore errors! | 23:14 |
| kirkland | SpamapS: i'm about to knock off for a bit; did you get around to fix that txzookeeper copyright issue? | 23:16 |
| kirkland | SpamapS: ensemble is *almost* installable from the archive ;-) | 23:16 |
| hazmat_ | niemeyer: hmm.. so alternatively wrapping communications/interactions into components/protocols and rebroadcasting connection failures (ala pub/sub) to all of them for their individual error handling after the connection has been restablished | 23:17 |
| hazmat_ | niemeyer: i think that's still compatible with a first step of a global connection error handler. | 23:18 |
| niemeyer | hazmat_: Yes, broadcasting errors in case they happen.. I don't see why we need other abstraction layers, though | 23:18 |
| niemeyer | hazmat_: It feels incompatible.. | 23:19 |
| niemeyer | hazmat_: If we crash on errors, when do we start fixing things so that they handle errors? They won't receive the error because it's crashing | 23:20 |
| hazmat_ | a global error handler can implement whatever logic we want, be it rebroadcasting a local exception to all components, etc | 23:21 |
| hazmat_ | niemeyer: a crash/restart error handler would just be a first step, its an incremental solution | 23:21 |
| niemeyer | hazmat_: It's not incremental.. it's stopping the application completely.. we can't implement incremental error handling if the first step is to crash the application entirely. | 23:22 |
| hazmat_ | the first step is to reconnect the application | 23:23 |
| hazmat_ | the application is effectively dead without that | 23:23 |
| niemeyer | hazmat_: Ok, you're right. Let's move forward with the full reconnection then. | 23:25 |
| hazmat_ | niemeyer: i definitely think we can evolve it to better things in the future | 23:26 |
| niemeyer | hazmat_: Let's keep it as simple as possible.. no additional abstraction layers | 23:26 |
| hazmat_ | niemeyer: sounds good to me | 23:26 |
| niemeyer | hazmat_: Let's have something that at least is able to recover itself automatically | 23:27 |
| hazmat_ | niemeyer: recover itself? | 23:27 |
| hazmat_ | niemeyer: could you be more specific? | 23:27 |
| niemeyer | hazmat_: Yeah, restart, reconnect, re-setup | 23:27 |
| hazmat_ | ah.. yeah | 23:28 |
| niemeyer | hazmat_: and let's work towards making that a rare event as much as possible | 23:28 |
| niemeyer | hazmat_: Increased timeouts, then resilient zookeeper nodes, etc | 23:28 |
| hazmat_ | niemeyer: yeah.. the migration is pretty transparent and fast for failovers | 23:28 |
| hazmat_ | i've been running cluster tests the last few days | 23:29 |
| niemeyer | hazmat_: Ensemble 2.0 can then take everything we learned and be better at that. | 23:29 |
| niemeyer | hazmat_: Oh, sweet | 23:29 |
| _mup_ | ensemble/debug-log-relation-settings-changes r260 committed by jim.baker@canonical.com | 23:32 |
| _mup_ | Used namedtuple | 23:32 |
| niemeyer | Will get dinner | 23:34 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!