| imbrandon | to do that it would have to be not just tecnical but they would need to be aware of all the reporcusions and how to debug repotly but be certain its not in vein etc | 00:00 |
|---|---|---|
| imbrandon | and well past what can / should be encapsulated in a first draft charm if at all | 00:01 |
| SpamapS | Man, I really really really want a "pretend to re-negotiate the relations" command | 00:01 |
| imbrandon | oh SpamapS btw, i'm at feature parity with jjc.c now on jujube | 00:02 |
| SpamapS | So many times where I just need to tweak something, and run upgrade-charm .. | 00:02 |
| SpamapS | imbrandon: lol.. of course you are | 00:02 |
| imbrandon | but i did a few thiings that i decided are a little beeter another way so i'm refactoring a little bit of it | 00:02 |
| imbrandon | but even that i should be done tonight too | 00:02 |
| imbrandon | oh and i said that wrong tooo, i'm not at fp with the whole domain , i broke it up into parts in my head and the README but none else know that LOL ... i intended to say i'm there with all of the charm browser aspects of jjc.c | 00:08 |
| imbrandon | includiing both the html output and a .json out out on the same routes if /api/v1/ is prefixed | 00:09 |
| imbrandon | like /charms/precise lists all precise charms and /charms/precise/appflower is details for appflower , then /api/v1/charms/precise and same for appflower are json output and also take a optional ?callback=anything returning jsonp so CORS will work as well | 00:11 |
| imbrandon | with cross origin domain calls, and later other sites could use JS to just make remote api calls for the data thet want ( or what i'm more excuted about is that juju-jitu could as well should it want to maybe it will spark something cool ) | 00:12 |
| SpamapS | http://ec2-54-245-5-73.us-west-2.compute.amazonaws.com/nagios3/ (user:nagiosadmin pass:ohpuipeera) | 00:16 |
| SpamapS | So there's my nagios magic | 00:16 |
| SpamapS | still needs to add host/service groups | 00:17 |
| SpamapS | and I *think* I can do some clever parent/child stuff by inspecting the relationships a bit (provides should be the parent of requires.. perhaps) | 00:18 |
| negronjl | SpamapS: nice re: nagios | 00:41 |
| m_3 | SpamapS: nagios is compelling... can I use this as part of a "mature" stack for Thursday? | 01:15 |
| hazmat | SpamapS, there is no clean up re subs | 01:26 |
| hazmat | SpamapS, thats a significant feature thats needed across the board | 01:26 |
| hazmat | orderly cleanup and instead of parent tree kill | 01:26 |
| hazmat | imbrandon, incidentally i got the oks to push jc.c public | 01:27 |
| hazmat | SpamapS, goju should hopefully have that from the start, it was discussed as part of the mongo storage work | 01:29 |
| hazmat | imbrandon, yes charms are copied on disk for each unit... pls file a bug if its of concern | 01:29 |
| hazmat | for ec2 there's not much to do, it must be present on each machine | 01:34 |
| hazmat | for local.. | 01:34 |
| hazmat | perhaps | 01:34 |
| hazmat | SpamapS, how'd you get juju with ostack working on hpcloud? | 01:35 |
| hazmat | ah.. auth-mode keyapir | 02:27 |
| imbrandon | hazmat: I had to use auth-mode: userpass ... see PM | 02:48 |
| hazmat | imbrandon, i'm making some progress with keypair | 02:48 |
| imbrandon | hazmat: nice, if you do get keypair to work later that would be cool, much less complex than currently i think for the env.y | 03:17 |
| imbrandon | although thats a set and forget thing so ... | 03:17 |
| imbrandon | heh' | 03:18 |
| SpamapS | m_3: this is pretty solid, I'll chat w/ you tomorrow to get you up to speed on deploying it. I think it should be widely consumable. | 07:02 |
| hitexlt | Hi, i am trying to deploy openstack on VM's. I have deployed all required charms, but for most machines in MAAS it shows agent-state: not started, though they are running. Here is output of juju status http://pastebin.com/vAb1mb3r Any ideas why juju is not registering running nodes? | 07:06 |
| SpamapS | hitexlt: the agent state is running on machine 2 | 07:08 |
| SpamapS | hitexlt: and machine 2's rabbitmq seems to be started actually | 07:09 |
| SpamapS | hitexlt: can you look at the console of machine 1 and see if something failed on first boot? | 07:10 |
| === philipballew__ is now known as philballew | ||
| === almaisan-away is now known as al-maisan | ||
| === mrevell_ is now known as mrevell | ||
| jml | :( | 10:27 |
| jamespage | jml, sad today? | 10:34 |
| jml | oh yeah | 10:34 |
| jml | I just had to kill off juju processes that were chewing up all my CPU | 10:34 |
| jamespage | jml, local provider? | 10:34 |
| jml | jamespage: yeah. I pretty much never use ec2 | 10:35 |
| jamespage | :-( | 10:35 |
| mgz | jml: canonistack is actually pretty reasonable currently | 10:36 |
| jml | mgz: I can run precise stuff on that, right? | 10:36 |
| mgz | yup. | 10:37 |
| jamespage | mgz, hows the openstack provider work coming on? | 10:39 |
| mgz | it's is a working state, but there are a few required features left and some cleanup to do. | 10:40 |
| === al-maisan is now known as almaisan-away | ||
| hazmat | mgz, i've spent some time yesterday working it across several providers (rspace, hp, cstack)and doing some cleanups. rspace is very different behavior with v2 api | 11:27 |
| hazmat | no floating ips | 11:27 |
| hazmat | auto public ip | 11:27 |
| mgz | right, hp have also switched to enabling the auto-assign of a public ip | 11:28 |
| mgz | there's an easy enough work around (see if there's a public ip before allocating a floating ip) | 11:28 |
| mgz | but that doesn't get me round the sleep unless I have a way of getting the config state out of a particular deployment | 11:28 |
| hazmat | mgz it would be nice to avoid the wait entirely | 11:28 |
| mgz | what I'd *really* *really* like is juju to move the public ip requirement to the ports handling | 11:29 |
| hazmat | mgz, there's nothing to run that code for the bootstrap node | 11:30 |
| mgz | it wouldn't be that hard to juju ssh to ensure the instance is accessible | 11:30 |
| hazmat | that's a workaround not a solution | 11:30 |
| mgz | well, it's moving the requirement of being able to talk to the machine from instantation to when it's actually needed | 11:31 |
| mgz | any code that currently expects to access public_ip could basically do that lookup with a check that will allocate one on demand | 11:33 |
| hazmat | mgz ah.. so your saying automatically instead of manually. that's reasonable. | 11:33 |
| hazmat | mgz, even distinguishing what's the public ip isn't well defined | 11:33 |
| mgz | right. | 11:33 |
| hazmat | mgz, also for ec2, it no longer waits for the security group shutdown/delete dance, it just reuses the groups | 11:34 |
| hazmat | on trunk | 11:34 |
| hazmat | for ostack this removes the delete dance at shutdown | 11:34 |
| hazmat | all the providers seem to support just killing the machine | 11:34 |
| mgz | that's an improvement, is there any cleanup for the groups? | 11:36 |
| mgz | hazmat: do you have a branch for your changes up anywhere? and did you find my hp tweaks branch? there are a few other differences they have that need properly integrating into the provider | 11:41 |
| hazmat | mgz, it looked like the hp tweaks where already mainlined | 11:48 |
| hazmat | mgz, i'm still in progress on my changes.. but here's the crack in progress ~hazmat/juju/openstack_provider/ | 11:48 |
| hazmat | i need to run an errand, bbiab | 11:49 |
| mgz | right, I merged Clint's, but last week an HP guy did some more testing as they want to use juju in a demo | 11:49 |
| mgz | and turned up some more things | 11:49 |
| hazmat | mgz, rackspace has their own auth form derived from v2 but with different fields. | 11:49 |
| hazmat | mgz, where's that branch? | 11:49 |
| mgz | see lp:~gz/juju/openstack_provider_hp_tweak | 11:49 |
| hazmat | cool, i'll check in when i get back | 11:50 |
| mgz | I'll take a look at your current state too on the rackspace auth things. | 11:50 |
| hazmat | mgz, i still have to parameterize all the security group work there, since all of those apis are 404s on rspace | 11:51 |
| mgz | hm, all of them? they don't support security groups at all? | 11:53 |
| hazmat | mgz, at the moment no | 11:56 |
| hazmat | nor floating ips | 11:56 |
| hazmat | mgz none of those apis is in v2 | 11:57 |
| mgz | okay, those auth changes make sense, shoule be able to integrate that nicely | 11:57 |
| hazmat | there in quantum | 11:57 |
| mgz | well, there are various things moving from nova to other projects | 11:57 |
| mgz | in not very backwards compatible ways | 11:57 |
| mgz | provided detecting can be done sensibly, doing both shouldn't be too hard, just a little annoying | 11:58 |
| === almaisan-away is now known as al-maisan | ||
| === zyga is now known as zyga-afk | ||
| hazmat | mgz, there's some minor capability though inconsistent to detect via service catalog version info | 13:35 |
| hazmat | rspace actually publishes multiple compute endpoints of different versions | 13:35 |
| mgz | hm, probably need to change the endpoint picking logic again then | 13:35 |
| surgemcgee | Hmmm, so the quotes are interpeded as a string now in the yaml. Is there a different type which will evaluate the quotes? | 13:55 |
| mgz | you mean will include the quote marks in the string? | 13:56 |
| surgemcgee | Yes, are we just using the pipe now? | 13:57 |
| surgemcgee | Well, I mean, will evaluate the quotes as a special character (escape sequence) | 14:00 |
| surgemcgee | With the default type of string, the quotes are included. I think this was changed. | 14:00 |
| surgemcgee | Also, referring to the single quote | 14:01 |
| mgz | hm, was about to say it works for me, but turns out I don't use any quotes in my config | 14:03 |
| surgemcgee | I am trial/error'ing it now. I will post my results. | 14:06 |
| smoser | can someone take a look at http://askubuntu.com/questions/162255/does-juju-really-need-two-nodes ? | 14:25 |
| mgz | I looked upon it an despaired | 14:26 |
| mgz | he wants to do an openstack deployment with only one box, the maas demo used 9. | 14:26 |
| hazmat | jitsu deploy-to can lesson it | 14:27 |
| smoser | it seems to me that the JuJu bootstrap node failed to come up. | 14:27 |
| hazmat | smoser, agreed | 14:28 |
| hazmat | it looks like the node is up but not running zk | 14:29 |
| mgz | I've had similar symptoms from cloud-init steps failing | 14:30 |
| mgz | so, sshing to the box and looking at the logs is what I'd do next | 14:30 |
| mgz | I just don't see it's really helping his end goal | 14:31 |
| smoser | mgz, cloud init does not fail. | 14:31 |
| smoser | i'm done talking to you | 14:31 |
| smoser | :) | 14:31 |
| mgz | :D | 14:31 |
| mgz | sorry, 'apt-get fails and cloud-init plows on regardless' :P | 14:31 |
| smoser | mgz, yeah, that is what i woudl have suggested, but i dont know which logs to get. | 14:31 |
| smoser | mgz, thats better. | 14:32 |
| smoser | hazmat, or mgz, could you request posting of some logs that might give information? | 14:32 |
| mgz | /var/log/cloud-init-output.log and the ones under /var/log/juju generally | 14:32 |
| === al-maisan is now known as almaisan-away | ||
| surgemcgee | Is it appropiate for me to start bug reports on juju? E.g. Boolean in config uses python interpratation with capital first letter --> "True". But bash uses lowercase --> "true". | 14:36 |
| surgemcgee | Ahhh crap. Well, if anyone was curious about the quote thing, careful where you put you delimiter settings. IFS="$(echo -e "\n\r")" is not normal for commands. | 15:01 |
| hazmat | surgemcgee, yes re bug reports, that particular one is addressed by some recent changes, but requires the charm to explicitly specify format 2 | 15:04 |
| === zyga-afk is now known as zyga | ||
| hazmat | which makes hook cli api output normalize to yaml | 15:05 |
| SpamapS | hazmat: kind of weird though... format 2 isn't supported by precise... | 15:13 |
| SpamapS | hazmat: I suggest that people just use --format=json | 15:13 |
| hazmat | mgz it looks like the fixes re hp tweaks are to support an older form of sec groups extant there | 15:50 |
| hazmat | it seems like the version identifiers here have no meaning re 1.1 nova compute, as they entail quite a bit differences between extant entities | 15:51 |
| === salgado is now known as salgado-lunch | ||
| marcoceppi | Has openstack support landed yet? | 15:52 |
| hazmat | marcoceppi, almost, just trying to kick the tires across a few providers | 15:53 |
| marcoceppi | Awesome, good to hear | 15:53 |
| mgz | right, the api versioning isn't... er... robust | 15:53 |
| hazmat | considering the size of the branch, i'm sorely tempted to have it just land, and fix per provider incrementally.. | 15:54 |
| hazmat | mgz, does the server respond back with version headers in any meaningful way? | 15:54 |
| hazmat | mgz, the keystone svc catalog doesn't seem to mandate versions | 15:54 |
| hazmat | which makes it hard to utilize that | 15:55 |
| hazmat | mgz, visualizing what v2 compatibility is going to look like with a provider that supports separate network, compute, volume api.. while keeping compatibility with 1.1 feels quite different | 15:56 |
| mgz | nope, though versions should generally be in the catalog... main issue seems to be they're largely ignored | 15:57 |
| hazmat | the encapsulation needs to flow from abstractions inspecting the svc catalog, but i digress.. its probably better to fix up some of the window dressing | 15:57 |
| mgz | so, stuff get broken and no one notices | 15:57 |
| hazmat | mgz, ignored by the current code.. or by the svc provider? | 15:58 |
| hazmat | well the former is true.. just curious about the latter | 15:58 |
| * hazmat openstack's api innovation makes me sad | 15:58 | |
| hazmat | hmm.. s/me/ | 15:58 |
| mgz | for instance, nova-client handles 1.1 and 2 with the same code under v1_1 module | 16:00 |
| mgz | there's pushback now it's starting to get use, but the history so far is messy | 16:00 |
| hazmat | mgz, thats going to fail as soon as people start deploying quantum.. its already a fail using the cli against v2 impl | 16:06 |
| hazmat | with v2 we're going to have construct our facade if we want to support both nova-network and quantum and dispatch by version id | 16:07 |
| hazmat | i'm not as concerned about that at the moment though re getting this into trunk, since those are extant | 16:07 |
| mgz | right, and the volumes code has the reverse problem, I have to use an old version of python-novaclient because the api was moved out of 'compute' into 'volume' and current deployments don't support the new way | 16:08 |
| mgz | I think these problems aren't too hard from the juju side though, given the api usage it pretty light and we should alwayse be able to pick a method that works | 16:09 |
| hazmat | same problem fundamentally.. two versions address the same core api in different ways | 16:09 |
| mgz | right. | 16:09 |
| hazmat | mgz, is volume going to be exposed at its own endpoint in the svc catalog as well? | 16:09 |
| mgz | yes. | 16:09 |
| mgz | and in fact is, but... canonistack gives an unresolvable internal url, and hp gives an endpoint which uses the old naming :) | 16:10 |
| hazmat | argh.. snowflakes | 16:10 |
| mgz | :D | 16:10 |
| SpamapS | doesn't keystone hand back a map.. "volume -> this url, network -> this url" ? | 16:11 |
| hazmat | SpamapS, in future in theory yes | 16:11 |
| hazmat | i believe rspace has both of those in private beta | 16:11 |
| hazmat | rspace is on v2 of the api.. hp is effectively on a forked version of the 1.1 api from diablo time frame which is different than any other impl.. canoistack is on v1.1 but doesn't advertise versions for its svc catalog endpoints (which keystone hands back). | 16:12 |
| SpamapS | I'm surprised HP hasn't spun up an Essex region | 16:18 |
| hazmat | mgz, SpamapS have you guys noticed any oddity about the bootstrap node getting a stringified id in hpcloud stored in zk, and thus not useable for queries to hpcloud about it? | 16:23 |
| SpamapS | hazmat: yes | 16:24 |
| SpamapS | hazmat: the bootstrap node just is wonky w/ hpcloud | 16:24 |
| SpamapS | I haven't looked into it very much | 16:24 |
| hazmat | the provider needs to materialize it back to an int i think before passing it off to the api serialization | 16:24 |
| mgz | hazmat: yeah, that's an issue, I hacked around it in the hp tweak branch I linked earlier | 16:24 |
| hazmat | mgz, it looked like the branch was trying to serialize all the ids to strs | 16:25 |
| hazmat | oh.. this the curl against the main id | 16:26 |
| hazmat | er bootstrap id against fstorage | 16:26 |
| mgz | there's a neater way of doing it in StatusCommand._process_machine but I was trying to avoid changes to other bits of juju | 16:26 |
| hazmat | mgz, i'm thinking just sniff the type in the provider | 16:27 |
| _mup_ | Bug #966577 was filed: add explicit egress 'owner' rule on non-bootstrapping nodes to require root access to zookeeper <rls-p-tracking> <juju:New> <juju (Ubuntu):Triaged> <juju (Ubuntu Precise):Triaged> < https://launchpad.net/bugs/966577 > | 16:27 |
| hazmat | yup.. that does the trick | 16:29 |
| SpamapS | hazmat: ^ the bug above is my hack for preventing non-root users from poking at ZK.. | 16:34 |
| SpamapS | hazmat: No diff yet.. just the idea. But I think its worthwhile even after ACL's land | 16:34 |
| SpamapS | hazmat: which, btw, are we going to land ACLs ? | 16:34 |
| SpamapS | hazmat: or is libzookeeper still an unknown weird fail-generator ? | 16:34 |
| imbrandon | zk dosent handle acl's at all? it expects the client to 100% ? | 16:38 |
| imbrandon | hazmat: btw i found if you use local placement on the bootstrap node the ssh 0 is still failwhale but you can juju ssh service/0 to it fine and juju status us the same at the top but its correct down at the server part with ip and all | 16:40 |
| imbrandon | hazmat: personally i've just been doing juju ( the charm ) deployment or something small just for convience but if that help with the debug path in the code maybe heh | 16:41 |
| imbrandon | hpcloud ^^ btw | 16:41 |
| negronjl | 'morning all | 16:43 |
| imbrandon | ugh, where are unity's global hotkeys stored anyone know right off to save me 15 minutes of looking , please dont say gconfeditor ... | 16:44 |
| imbrandon | heya negronjl | 16:44 |
| negronjl | hi imbrandon | 16:44 |
| hazmat | imbrandon, just fixed that fwiw | 16:44 |
| imbrandon | oh sweet, i was aobut to get lunch bvut after mind if i snag ur branch :) | 16:44 |
| imbrandon | heh | 16:44 |
| hazmat | i guess clint rolled out | 16:44 |
| imbrandon | oh cool cool, i'll catch up when i get back i've been head down in code and only half paying attn to irc today | 16:45 |
| imbrandon | ok back in a bit, foooooood | 16:45 |
| hazmat | the ssh fingerprint problem is more pronounced with ostack providers i'm noticing | 16:47 |
| hazmat | i guess thats the habit of reusing the floating ip | 16:47 |
| hazmat | to a different machine | 16:47 |
| mgz | yup. | 16:48 |
| hazmat | SpamapS, yes stunnel si worthwhile | 16:55 |
| hazmat | SpamapS, i haven't dug into libzk issues | 16:56 |
| SpamapS | stunnel would be a bigger deal | 16:56 |
| SpamapS | just using the iptables rule we at least limit all traffic to the bootstrap node to root | 16:56 |
| SpamapS | imbrandon: not sure what happened while I was split off. ZK has ACL's, but we don't use the (yet) | 16:57 |
| hazmat | imbrandon, just hold down 'windows'/cmd key for a while and the shortcuts appear | 16:58 |
| hazmat | a listing that is | 16:59 |
| SpamapS | ooo somebody is learning unity | 17:00 |
| SpamapS | hazmat: hey, the charm versions in import/export are pretty bogus.. | 17:01 |
| hazmat | SpamapS, ? | 17:01 |
| hazmat | SpamapS, how so? | 17:01 |
| hazmat | SpamapS, pastebin pls | 17:01 |
| SpamapS | hazmat: with local charms at least.. I no longer have "version25" of mysql.. I"m on 31.. but it deployed 31 as local:mysql-25 | 17:01 |
| SpamapS | hazmat: causes no harm, but its confusing | 17:02 |
| hazmat | SpamapS, ah.. i actually added explicit support for that | 17:02 |
| hazmat | SpamapS, otherwise it would fail to deploy at all if the version in a local repo wasn't an exact match | 17:02 |
| SpamapS | It works fantastically, but its a bit weird. ;) | 17:02 |
| SpamapS | hazmat: I'd rather see the actual versions (and warnings). But.. we can do that in the go version, right? ;) | 17:03 |
| hazmat | SpamapS, basically for local charms, the version id isn't a hard requirement, it will look for the specified version and then fallback to namematch | 17:03 |
| marcoceppi | Interesting question, can I change the region during deployment? to have nodes across regions? | 17:03 |
| hazmat | marcoceppi, no.. that's not supported in constraints atm.. twas a long discussion.. | 17:03 |
| hazmat | well.. you can specify az | 17:03 |
| marcoceppi | hazmat: thanks, I figured it would be pretty involved | 17:03 |
| hazmat | for a constraint at a service level | 17:03 |
| hazmat | but if you want a multi region service, the result is you need a service in each that are connected | 17:04 |
| hazmat | that way if you need to add capacity (or a node disappears) replacing it is explicit | 17:04 |
| SpamapS | I'd love to have a service constraint of ec2-availability-zone=balanced | 17:04 |
| SpamapS | meaning, put these in as many different zones as possible | 17:04 |
| * hazmat nods | 17:05 | |
| marcoceppi | +1k | 17:05 |
| hazmat | we went back and forth on that one a few times at uds orlando | 17:05 |
| hazmat | not treating individual nodes as special was the concern for supporting it explicitly within a service | 17:05 |
| hazmat | if i have three nodes in one az and one node in another az, and it dies, what does add-unit do to correct properly | 17:06 |
| hazmat | a juju policy level identifier for zones would work though | 17:06 |
| hazmat | but at the moment the recommendation is to have separate services in each zone and relate them | 17:07 |
| dpb___ | is it possible to reboot an instance inside an install hook (for example)? | 17:08 |
| hazmat | dpb___, not at the moment, the address change doesn't properly propogate if there is one, if the address is stable then yes | 17:09 |
| === salgado-lunch is now known as salgado | ||
| dpb___ | hazmat: ok, you mean the ip address? | 17:10 |
| hazmat | dpb___, yes, both public and private would have to be stable for it to work reliabily | 17:10 |
| dpb___ | hazmat: ah, ok. I guess the private one can change between boots on aws? | 17:11 |
| hazmat | dpb___, both change on reboot in aws | 17:11 |
| dpb___ | hazmat: ok, thx, I did not know that. | 17:11 |
| hazmat | dpb___, there's an unloved branch for it, if your interested, that updates address at boot, and propogates to relations.. with it doing things like suspend/restore an ec2 env become possible | 17:12 |
| dpb___ | hazmat: well, first let me see if I can do some magic to make the reboot uncessary. that would be speed up the process anyway. | 17:13 |
| hazmat | dpb___, absolutely that would be the best way | 17:13 |
| dpb___ | hazmat: thx | 17:13 |
| SpamapS | dpb___: I would defer reboot to config-changed | 17:13 |
| dpb___ | SpamapS: How do you mean? you mean that config_changed can handle a reboot in the middle? | 17:16 |
| SpamapS | dpb___: config-changed is what will be run after the reboot | 17:18 |
| jcastro | SpamapS: heya | 18:10 |
| jcastro | SpamapS: mims is on review this week but I didn't account for oscon | 18:10 |
| jcastro | SpamapS: if it's possible, what do you think about reviewing the queue this week and have mims grab next week, when it was your turn to go. | 18:10 |
| jcastro | basically, wanna swap? | 18:10 |
| SpamapS | Yeah I'm going to take his shift | 18:11 |
| SpamapS | jcastro: I saw that coming yesterday. :) | 18:11 |
| jcastro | <3 | 18:11 |
| jcastro | thanks dude | 18:11 |
| SpamapS | jcastro: which is unfortunate because I was hoping to have Mims review my massive Nagios refactor :) | 18:11 |
| SpamapS | but he probably will anyway as he might use it in his demo :) | 18:11 |
| jcastro | indeed | 18:13 |
| jcastro | also we have decided we want you writing more subordinates. :p | 18:13 |
| SpamapS | ME TOO :) | 18:18 |
| SpamapS | well I already took over james_w's nrpe | 18:18 |
| SpamapS | jcastro: http://ec2-54-245-5-73.us-west-2.compute.amazonaws.com/nagios3/ | 18:18 |
| SpamapS | jcastro: user:nagiosadmin pass:ohpuipeera | 18:19 |
| SpamapS | and no I don't mind if you all pound the crap out of it ;) | 18:19 |
| imbrandon | hazmat: yea i knew that actually, cuz its the ones i keep hitting ( cmd ) and so i want to swap enmase` cmd<->ctl so my hotkeys for osx and ubuntu should line up almost identically, as it is i keep bringing up the hud every time i goto save a file or copy/paste etc etc, finaly had enough and gonna find it even if i got to recompiler unity :) | 18:49 |
| imbrandon | heh | 18:49 |
| jcastro | SpamapS: can you leave that instance up for a few hours? | 18:51 |
| jcastro | that'd be nice to show during our talk | 18:51 |
| SpamapS | jcastro: when are you guys speaking? | 18:53 |
| SpamapS | jcastro: I can give you an export to recreate it. :) | 18:53 |
| jcastro | in 1.5 hours. | 18:53 |
| jcastro | well, we didn't want to go too deep into it, actually, I'll just screenshot it | 18:53 |
| SpamapS | jcastro: are you guys demo'ing at all or just talking? | 18:53 |
| jcastro | demoing a bunch of stuff | 18:53 |
| jcastro | so we didn't want to add another demo | 18:54 |
| SpamapS | hah yeah ok | 18:54 |
| SpamapS | jcastro: let me add thinkup back in and then you can screenshot it | 18:54 |
| jcastro | ok, lmk when | 18:55 |
| SpamapS | jcastro: its there now, just PENDING | 18:55 |
| SpamapS | jcastro: more fun if its all green :) | 18:55 |
| jcastro | I like it with the pending though | 18:57 |
| SpamapS | true | 18:57 |
| jcastro | shows that it all just works, and you don't care, you set the relationship, it WILL work, etc. | 18:57 |
| jcastro | except when it doesn't, but whatever. :p | 18:58 |
| SpamapS | haha | 18:58 |
| SpamapS | jcastro: got a shot yet? I want to refresh one of the relations | 18:58 |
| jcastro | we're good | 18:58 |
| SpamapS | jcastro: Ok re do your snap of services now. MUCH more compelling :) | 19:06 |
| SpamapS | jcastro: and I can give you some reds if you want. :) | 19:07 |
| jcastro | ok one sec | 19:08 |
| jcastro | thanks dude, got it | 19:09 |
| imbrandon | SpamapS: haha ok know what i just realized, ok so like none of this info is new to me ( or you either i dont think ) but it JUST all clicked a few seconds ago | 19:11 |
| SpamapS | imbrandon: you forgot one thing. You forgot, to hookup, the doll | 19:11 |
| imbrandon | so ... Google had another announcement like 2 days ago that they just finished laying the 200th mile of FttH in Kansas City | 19:12 |
| imbrandon | doll ? | 19:12 |
| imbrandon | so ok the fiber rollout is almost complete, its double subsidised once by the city/county to get ppl on it and once by google, so total cost to residents at rollout is $30 per month out of pocket and thats for 100GB SYNC connection , not just to the street but all the way to the net AND 16 static IP ..... | 19:14 |
| imbrandon | on the residential connection ... ok so add that ... pluss i start rounding up some HP blades and Sun NAS boxes heheh | 19:14 |
| imbrandon | toss open stack on it , and start a lil mini Linode in my basemnent | 19:15 |
| imbrandon | likely rivaling if not exceding the quality of many mom and pop operations round the country | 19:15 |
| imbrandon | hahah | 19:15 |
| imbrandon | anyhow i likely will do all that cept the resell part, i dont want that headache :) | 19:16 |
| imbrandon | lol | 19:16 |
| SpamapS | imbrandon: re doll.. http://www.imdb.com/title/tt0090305/quotes .. look for the word 'doll' | 19:16 |
| imbrandon | k | 19:16 |
| SpamapS | I'm like the only one who remembers that quote.. but I use it in my head constantly :p | 19:16 |
| imbrandon | AHHH | 19:17 |
| imbrandon | heh, thats ok i had the audio playing ( very loud too ) for the samuel jakson seen at the start of pulp fiction when my brother came by and dident reconise it | 19:18 |
| imbrandon | i told him i disowned him | 19:18 |
| SpamapS | imbrandon: your power setup will be crap w/o a diesel generator, but otherwise yeah you will have quite a bit of advantage. :) | 19:22 |
| SpamapS | perhaps KC will become us-east-2 ;) | 19:22 |
| imbrandon | hrm yea but that would not be too bad :) | 19:22 |
| imbrandon | hahaha | 19:22 |
| imbrandon | http://gfiber-kansas.appspot.com/fiber/kansascity/news.html | 19:23 |
| imbrandon | cant wait for real, i'm just happy as hell that not only its got 16ips static standard but that are quoted to say something very very close to "KC is the pilot city here is some cool shit now go make the next gen web" | 19:24 |
| imbrandon | PLUS sync both ways :) | 19:24 |
| imbrandon | you should see all the new Tech startup that are already downtown that are newly funded and starting to setup shop now in prep | 19:25 |
| imbrandon | gonna be a wild ride the first year i bet | 19:25 |
| imbrandon | SpamapS: ok, i'm on Time Warner now ... kc.rr.com , and check this pic out http://www.businessinsider.com/google-fiber-speed-2011-8 | 19:27 |
| hazmat | it looks like hpcloud is blocking private traffic by default now | 19:27 |
| imbrandon | they block all by default i thought | 19:27 |
| imbrandon | and you had to add pub and priv to the zones | 19:28 |
| hazmat | huh? | 19:28 |
| imbrandon | i forgot what they call their security groups | 19:28 |
| hazmat | i couldn't deploy services because the agents couldn't connect back to the bootsstrap node | 19:28 |
| hazmat | security groups.. | 19:28 |
| imbrandon | right, juju dont do that on diablo | 19:28 |
| imbrandon | SpamapS: had me do it by hand so i just set it to 0.0.0.0/0 | 19:29 |
| imbrandon | and never destroyed it | 19:29 |
| SpamapS | Oh right I totally forgot about that | 19:29 |
| imbrandon | i did too | 19:29 |
| imbrandon | untill hazmat said it | 19:29 |
| imbrandon | since destroy never took the groups away | 19:30 |
| imbrandon | and i used the same name | 19:30 |
| hazmat | this is going to break most of the charms unless we netmask all internal port traffic | 19:30 |
| imbrandon | to redeploy , never had to mess with it again | 19:30 |
| hazmat | to the env sec group | 19:30 |
| hazmat | i had the same issue on azure although lacking any notion of a private network there | 19:31 |
| imbrandon | right , but you can only do that by name and not ip with the python-novaclient :( | 19:31 |
| SpamapS | hazmat: there's no concept of private traffic being allowed between security group members? | 19:31 |
| imbrandon | SpamapS: they is but its off and docs say its not in the webui | 19:31 |
| hazmat | SpamapS, there should be but it was pretty broken on diablo | 19:31 |
| * hazmat grumbles about snowflakes | 19:32 | |
| imbrandon | snow ? its over 90f here | 19:32 |
| imbrandon | man | 19:32 |
| imbrandon | no effin way | 19:32 |
| hazmat | imbrandon, just referencing the individual snowflake implementations of openstack ;-) its a 100+ here | 19:33 |
| SpamapS | ahh diablo seems to be just making everything ridiculous | 19:33 |
| imbrandon | ohhh ohh ok, i've interntionally stayed clear of the OS guts for now, until it evens out a little | 19:33 |
| imbrandon | and i dont waste a ton of time on something that will be bad before i really "get" it | 19:34 |
| imbrandon | hehe | 19:34 |
| hazmat | SpamapS, the differences between essexes is just as bad.. ie the delta for rackspace and canonistack is huge | 19:35 |
| imbrandon | i figured by the time RAX and that other one rolls em public will be a good time to hunker down and really "get" all the bits that i would need | 19:35 |
| SpamapS | hm, a wise man once told me that the browser wars were the only thing that saved the web | 19:35 |
| imbrandon | to know about , or maybe not but i still like to understand the full stack eventually, kinda sucks lately intentionally staying out of the juju and OS code just for that | 19:36 |
| SpamapS | perhaps we're right at the point where netscape has implemented a billion little extensions, and IE has done the same... | 19:36 |
| imbrandon | i dunno about saved it, but they sure as hell are what drove it and still do | 19:36 |
| imbrandon | SpamapS: we are, even software devlopemnt as a whole is an insanely young disapline let along generations inside of that | 19:37 |
| SpamapS | imbrandon: the theory being that while HTTP RFC's and w3c specs *helped*.. what really kept things honest was users demanding the Netscape/Firefox/etc. interoperate with sites that blindly used IE stuff.. | 19:37 |
| SpamapS | Tho I could make a compelling argument that what really put a nail in the IE-special-snowflake was the iphone | 19:37 |
| imbrandon | who's got money on hp will give in to the OSapi and join the ranks and goog will be the wild card like IE | 19:38 |
| imbrandon | with "alomst" but "better" stuff :) | 19:38 |
| SpamapS | Ugh | 19:38 |
| SpamapS | subordinates should have a guarantee that their primary relations will have fired before any others | 19:39 |
| imbrandon | nah, it was way before the iphone, thats just when everyone realzed it but by that time its tooo late, itunes is what sealed everything | 19:39 |
| SpamapS | nrpe killing me here by relating to nagios before it can talk to its local service | 19:39 |
| imbrandon | SpamapS: no, bad juju , dont force serialization, lets make the hooks smarter to deal with async | 19:39 |
| imbrandon | yea iu know easier said than done | 19:39 |
| SpamapS | imbrandon: I do that | 19:39 |
| SpamapS | but.. we need a 'defer me until later' to make that easy | 19:40 |
| SpamapS | otherwise every hook ends up calling every other hook | 19:40 |
| imbrandon | i know, you do, it was really me just "going by the book" but in reality your right | 19:40 |
| imbrandon | heh , have you seen my hooks, they do call every other one ;) | 19:40 |
| imbrandon | for realz | 19:41 |
| SpamapS | yeah I'm doing that too and its just making me sick how crap thats going to be for some use cases | 19:42 |
| imbrandon | so i want to charm http://c9.io ... but its got a "magic" build system i dont under stand and cant make work on 12.04 but same git checkout works on OSX ... got time / wanna help on it :) heh | 19:42 |
| SpamapS | like nagios.. where I fully expect thousands of units related | 19:42 |
| imbrandon | what thousands ? maybe on the top 3% but the vast majority of deployments will be 20 or less i bet, Penton only had about 100 | 19:43 |
| SpamapS | there's a really big network effect though | 19:43 |
| SpamapS | so even 100 units will be painful to do 4 times every time 1 comes and goes | 19:44 |
| imbrandon | yea but having not looked at it at all i know that the "nagios" way of dealing with that is not the typical setup, we ( read you , i aint dont shit but bitch ) might be rtying to bend nagios into a traditional scale out form factor its not gonna work at ... this is problably going to be the most complex charm i bet , including the OS ones | 19:45 |
| imbrandon | nah that shouldent be | 19:46 |
| imbrandon | those checks are cheap | 19:46 |
| imbrandon | think about how many hit apache a second | 19:46 |
| imbrandon | it just SEEMS like it would | 19:46 |
| SpamapS | its the constant chatter back and forth between juju and the hooks and execing relation-get for every relation-list unit for every relation-id relation | 19:46 |
| imbrandon | that and thats one reason they should not all report to the one | 19:47 |
| SpamapS | its not about *nagios* | 19:47 |
| SpamapS | its about the charm eating up all the CPU cycles just dealing with the churn | 19:47 |
| SpamapS | I can scale out nagios really easily | 19:47 |
| SpamapS | several ways | 19:47 |
| imbrandon | oh well if the charm cant scale thats not nagios's fault | 19:47 |
| SpamapS | my point exactly | 19:48 |
| imbrandon | then we need to look at mq | 19:48 |
| imbrandon | but i thought zk did that for us | 19:48 |
| SpamapS | its not that either | 19:48 |
| SpamapS | its calling all the hooks all the time | 19:48 |
| imbrandon | sure you can slow the queue back | 19:48 |
| SpamapS | nagios is pretty critical.. I'd argue slowing the queue is a major fail | 19:48 |
| imbrandon | right , so how do we make the hooks cheaper ... | 19:49 |
| SpamapS | if I have 60 minute guaranteed response time, getting the check 5 minutes late could cost me money | 19:49 |
| SpamapS | write them in go! | 19:49 |
| SpamapS | ;) | 19:49 |
| imbrandon | nagios will slows its queue too, i'm talking ms not minutes | 19:49 |
| imbrandon | just enought to give the cpu time to slice | 19:49 |
| imbrandon | ok i do see what your saying now tho, and actually | 19:50 |
| imbrandon | thinking aobut it 30 whole seconds heheh | 19:50 |
| imbrandon | its a design problem | 19:50 |
| imbrandon | hooks need to be fat | 19:50 |
| imbrandon | but in that cane they cant | 19:50 |
| imbrandon | cant | 19:50 |
| SpamapS | Its much simpler to write hooks that just regenerate everything every time | 19:51 |
| imbrandon | well its not onlythat | 19:51 |
| SpamapS | BUT that will cost a lot of efficiency | 19:51 |
| imbrandon | you would need to say ok, you only get 250ms of runtime each fire | 19:51 |
| SpamapS | and juju has an event based model that should allow us not to do that | 19:51 |
| imbrandon | do what you need | 19:51 |
| imbrandon | but then | 19:51 |
| imbrandon | you cant garentee | 19:51 |
| imbrandon | so yea, its like conflicting goals | 19:51 |
| SpamapS | I don't think we can guarantee.. | 19:51 |
| marcoceppi | IIRC you can do "push" notifications to Nagios. So nagios doesn't have to poll every time | 19:52 |
| imbrandon | but ... | 19:52 |
| imbrandon | yea we HAVE to | 19:52 |
| SpamapS | but a best effort would be to try and keep hook execution down | 19:52 |
| imbrandon | yok SpamapS i got the solution | 19:52 |
| SpamapS | marcoceppi: indeed, thats a better model I think | 19:52 |
| imbrandon | config mangement, stop going it in bash | 19:52 |
| SpamapS | but.. there are some things you can't push.. like making sure the thing is actually reachable | 19:52 |
| SpamapS | imbrandon: mine are in python :) | 19:52 |
| imbrandon | let the bash fire async off to the nagios boz anddo it | 19:52 |
| SpamapS | at least, a huge portion | 19:52 |
| imbrandon | sure, but you ssee what i mean | 19:53 |
| SpamapS | bash is for the simple stuff | 19:53 |
| marcoceppi | SpamapS: right, we had a similar problem at InMotion hosting. Monitoring 2,500 services in Nagios | 19:53 |
| imbrandon | right i was telling him about that at GSI was the same | 19:54 |
| imbrandon | but its really that 2500 hooks fire not | 19:54 |
| SpamapS | marcoceppi: mod_gearman for nagios can help with scaling out the polling for things that still have to be polled | 19:54 |
| imbrandon | the nagios | 19:54 |
| SpamapS | its not 2500 hooks, lets be clear | 19:54 |
| SpamapS | lets say you have 100 units total | 19:54 |
| SpamapS | all of which are related to nagios | 19:54 |
| imbrandon | i was using marcoceppi example silly | 19:54 |
| imbrandon | but the checks arent a problem | 19:55 |
| SpamapS | and you add or remove at least one every 3 minutes... | 19:55 |
| imbrandon | its that python hook on the juju relations that is the bottle neck | 19:55 |
| SpamapS | departed, joined, and changed all then *must* finish within 3 minutes, and cannot eat all of the CPU for that 3 minutes | 19:55 |
| imbrandon | ok wait a min, we/i'm going in circles, why are the checks and the hooks coupled , i was assuming they wernt | 19:56 |
| imbrandon | untill just now | 19:56 |
| SpamapS | because the checks run on the same box as the hooks | 19:56 |
| imbrandon | why | 19:56 |
| SpamapS | <sigh> | 19:56 |
| SpamapS | where else would they run? another box proxied to? | 19:57 |
| imbrandon | yea only 3 | 19:57 |
| imbrandon | not all of tghenm | 19:57 |
| imbrandon | no | 19:57 |
| imbrandon | but ... ok let me find a way to get what i'm thinking right the first time | 19:57 |
| SpamapS | So my point is that the joins, changes, departs, brokens, have to be moderately efficient | 19:57 |
| imbrandon | you got nagios svr | 19:57 |
| SpamapS | but if they loop over 100 units running relation-* 5 times for each unit.. | 19:58 |
| imbrandon | that solved nothing really tho | 19:58 |
| SpamapS | thats not going to take 3 minutes, but it might take 30 seconds. And thats 30 seconds of CPU backlog | 19:58 |
| imbrandon | because their is nothing stopping another charm form beong ineffecianrt | 19:58 |
| imbrandon | and yea if we're talking the main server running 300 check every few minutes then i think we might be oiptimizing before there is an issue, each nagios check should only take milizeond or 2 so 500 to 1000 milizeonds | 20:00 |
| imbrandon | then again on the callbackl | 20:00 |
| SpamapS | forget | 20:00 |
| SpamapS | the | 20:00 |
| SpamapS | nagios | 20:00 |
| imbrandon | so 2 seocnds max at 100 services | 20:00 |
| SpamapS | checks | 20:00 |
| SpamapS | the hooks alone are the problem | 20:00 |
| imbrandon | oh jesus i just asked why you switched on my and they was coupled above | 20:01 |
| imbrandon | ... | 20:01 |
| SpamapS | They're on the same box, but they're not the problem | 20:01 |
| imbrandon | so the hook runs 3 times ... and the checks run 300 to 500 | 20:01 |
| SpamapS | the problem I'm facing is keeping the nagios changed/departed/broken hooks efficient, thats all. | 20:01 |
| imbrandon | thats still very cheap | 20:01 |
| imbrandon | SpamapS: hadcop did this | 20:01 |
| imbrandon | if its just the hooks then its the same problem | 20:02 |
| imbrandon | the 3000 nodes or /we | 20:02 |
| SpamapS | well anyway, I'm already seeing the hooks take 3 - 5 seconds with 7 units | 20:03 |
| SpamapS | spending their time waiting on relation-* | 20:03 |
| imbrandon | ohhh and wait, for real, cant you put the join/depart ips and stuff your redoing the config for optionally into a mysql/pgsql | 20:03 |
| SpamapS | or just a sqlite :) | 20:04 |
| SpamapS | yes I do some of that | 20:04 |
| imbrandon | SpamapS: that could be zk + python spinup time too | 20:04 |
| imbrandon | about half of the 3 to 7 | 20:04 |
| SpamapS | agreed | 20:04 |
| SpamapS | REST API, I can has, plrease | 20:04 |
| imbrandon | maybe only ship the .pyc files in the /hooks | 20:04 |
| imbrandon | ? | 20:04 |
| === zyga is now known as zyga-afk | ||
| SpamapS | imbrandon: the pyc's for juju.* are already compiled | 20:05 |
| imbrandon | SpamapS: sure http://jujube.websitedevops.com/api/v1/ | 20:05 |
| imbrandon | and i was saying the hooks not juju | 20:05 |
| imbrandon | your hooks are in python | 20:05 |
| SpamapS | the hooks aren't really slow | 20:05 |
| imbrandon | ... | 20:05 |
| imbrandon | >.< | 20:05 |
| SpamapS | measuring them w/ strace -c .. | 20:06 |
| SpamapS | wait() leads the pack | 20:07 |
| imbrandon | marcoceppi: oh btw i did get a full CharmStore API ReST interface for json and jsonp done | 20:07 |
| marcoceppi | imbrandon: cool, saw the link - got a 404 | 20:07 |
| imbrandon | just need to push it up but i need to split out some commits, i'll email ya with the details | 20:07 |
| marcoceppi | <3 | 20:07 |
| imbrandon | that route plus whatever the normal url is will give ya json , and add ?callback=your_callback | 20:08 |
| imbrandon | will give ya jsonp | 20:08 |
| imbrandon | but like i said there are some other little stuff to , i got the readme docs about 3/4 i should be able to get it out for ppl to poke sometime tonight | 20:08 |
| imbrandon | cuz i added a php5-yaml dep as well too and some other things | 20:09 |
| imbrandon | that would be a PITA if you just had to debug why its broke if missing :) | 20:09 |
| imbrandon | SpamapS: what would be calling wait() disk io ? | 20:10 |
| imbrandon | that blocks i assume | 20:10 |
| imbrandon | unless its just a thrad wait | 20:10 |
| imbrandon | or something | 20:10 |
| SpamapS | imbrandon: wait() is waiting on forked processes | 20:11 |
| SpamapS | imbrandon: as in, relation-* | 20:11 |
| imbrandon | oh ... so yea it is the scripts then waiting on the pid to finish | 20:11 |
| imbrandon | err parent waiting on the pid of the scipt etc | 20:12 |
| SpamapS | right | 20:12 |
| SpamapS | I could, as you say, keep all the relation data cached locally | 20:12 |
| SpamapS | and only update it when the hook fires | 20:12 |
| SpamapS | perhaps juju should do that | 20:12 |
| imbrandon | so it should be doing something in the script then durring that but its child ... hrm dtrace ? | 20:12 |
| imbrandon | SpamapS: something this complex i think that might be the most correct way without getting it into productions some and looking again from 30kft | 20:13 |
| SpamapS | imbrandon: the relation-*'s are largely stuck waiting for ZK | 20:14 |
| SpamapS | because the way we interrogate ZK is I think a bit verbose | 20:14 |
| imbrandon | e.g. might comes up with something not occuring to you the last week or whatever after you see it live some afternoon :) | 20:14 |
| imbrandon | SpamapS: imho zk is very slow but i thought that was just the java being slow | 20:15 |
| imbrandon | i was hoping that we'd move to mongo :) | 20:15 |
| SpamapS | mongo will make queries faster | 20:15 |
| SpamapS | but I am not convinced we will be able to observe it as easily | 20:15 |
| imbrandon | we use zk like mongos's gridfs implmentation | 20:15 |
| imbrandon | so it would be identical | 20:15 |
| imbrandon | infact moreso because distributed mongodb is MUCH easier | 20:16 |
| imbrandon | and this dist gridfs | 20:16 |
| imbrandon | a part could be on all nodes reoplication to all the others | 20:17 |
| imbrandon | with like 3 hours of design | 20:17 |
| imbrandon | :) | 20:17 |
| SpamapS | hm, sounds somewhat like my 0mq idea | 20:17 |
| SpamapS | (have relation data live on nodes that own it, not in the central database) | 20:17 |
| SpamapS | I'm about to say screw it and just regenerate the whole config every time.. | 20:18 |
| === medberry is now known as med_ | ||
| imbrandon | and monogo has driveers and gridfs interfaces for everything and its fast and it has auth built in ( something zk also lacks i think ) | 20:18 |
| SpamapS | too many weird cases where I miss one thing and then there's no way to ask juju to re-fire a hook and regenerate the configs | 20:18 |
| imbrandon | heh yea , working is better than correct , we're going to be going back and fixing alot of these charms | 20:18 |
| imbrandon | took me a long time to actually come to grips with that being OK and we're also still so stringent on other things | 20:19 |
| imbrandon | heh | 20:19 |
| SpamapS | I'm also doing something extremely weird.. maybe even wrong.. by relating the primary *and* nrpe to nagios | 20:19 |
| SpamapS | I think I can actually simplify by just relating nrpe to nagios | 20:19 |
| SpamapS | but ugh | 20:19 |
| SpamapS | I've already spent so much damn time on this | 20:19 |
| * SpamapS goes to eat and think | 20:19 | |
| imbrandon | heheh iterate man iterate, do it wrong once or 40 times | 20:20 |
| imbrandon | no one is gonna get this stuff right on the first go and definately not the ones doing it first .... :) | 20:20 |
| imbrandon | but if it works but is just slow at 100 nodes, thats ok :) | 20:21 |
| imbrandon | fo now | 20:21 |
| imbrandon | lol | 20:21 |
| * imbrandon says that like any of the above you did not already know | 20:22 | |
| imbrandon | heh ugh ok i need to ... there was something i neeeded to do before getting back to that charm .... | 20:22 |
| imbrandon | ugh, memory of a dog and the attn span of a gnat ... fml | 20:23 |
| imbrandon | oh hazmat btw , dug this out for ya to poke at later just cuz ... http://caucho.com/products/whitepapers/quercus.pdf | 20:25 |
| imbrandon | but hazmat you konw when i was grabbing that tho, i ran into this and about seriously fell on the floor almost out of breath ... http://morepypy.blogspot.com/ SpamapS marcoceppi m_3 jcastro , yall check that too | 20:26 |
| imbrandon | the second one | 20:26 |
| ejat | hi .. sorry to interrupt .. just want to ask about this email .. is it applicable to all ? https://lists.launchpad.net/lubuntu-qa/msg00787.html | 20:41 |
| imbrandon | ejat: its already began but you can get in touch with jcastro as he is the liason for that promo iirc | 20:43 |
| ejat | imbrandon : thanks | 20:43 |
| imbrandon | ejat: see http://wiki.ubuntu.com/HPCloud iirc ( typed from mem, may be off ) | 20:44 |
| ejat | DEADLINE FOR THE FIRST ROUND OF APPLICATIONS IS 3 JUNE - Sorry this is so short notice, we'll be doing applications in batches. :( i just notice today :( | 20:47 |
| dpb___ | Is there a hook that is called when a service is destroyed (destroy-service) | 20:51 |
| dpb___ | I guess maybe I shouldn't be calling destroy-service, but I'm not sure. | 20:53 |
| ejat | imbrandon : is this the latest from askubuntu http://goo.gl/a0mMX | 20:55 |
| ejat | juju with hpcloud or now already supported ? | 20:55 |
| imbrandon | defined supported ? hehe its somewhat functioning with some known issues but there is a spcific LP branch that can be used and myself , hazmat , SpamapS , mgz and i'm sure a few others have been trying it out and some fixes have landed but its no where near offical yet | 20:59 |
| imbrandon | let me find you the url ... | 21:00 |
| imbrandon | ejat: here is the banch lp:~gz/juju/openstack_provider | 21:05 |
| imbrandon | SpamapS: thats not back in trunk yet correct ? ^ | 21:05 |
| SpamapS | dpb___: no there's no cleanup currently.. you just have to terminate the machines | 21:08 |
| SpamapS | imbrandon: correct, but hazmat is working on it | 21:08 |
| ejat | imbrandon : thanks | 21:10 |
| dpb___ | SpamapS: thx | 21:24 |
| SpamapS | This bit where subs never depart or break really screws me. :-/ | 21:31 |
| SpamapS | hazmat: ^^ is that bug documented somewhere? | 21:32 |
| imbrandon | the hour of irclogs to convince me of it ? lol | 21:32 |
| SpamapS | No this even screws me if I do it the easy way | 21:33 |
| SpamapS | basically I'm trying to simplify | 21:33 |
| SpamapS | I should be able to relate nagios<->nrpe<->* and have * get monitored | 21:34 |
| imbrandon | yea i was trying to be a smrt az | 21:34 |
| SpamapS | but if nrpe units never depart .. | 21:34 |
| imbrandon | no workie tho | 21:34 |
| SpamapS | then I don't know when to tell nagios to regen the configs | 21:34 |
| imbrandon | config-change fire after when it shold have ? | 21:34 |
| SpamapS | So make the admin poke it? | 21:34 |
| SpamapS | that sounds like "the suck" | 21:35 |
| imbrandon | yea but it sounds like that or delay until a patch is pushed | 21:35 |
| SpamapS | there's no patch for this IIRC | 21:35 |
| SpamapS | its fundamentally broken | 21:35 |
| SpamapS | IMO subordinates are kind of just clunky and should be rethought | 21:35 |
| SpamapS | nice first try.. | 21:35 |
| imbrandon | ahh ok i was thinking it was still lot further out | 21:35 |
| SpamapS | but I think they should just be regular services like anything else, just directed to spin up inside the container | 21:36 |
| SpamapS | making them special is making them broken | 21:36 |
| * imbrandon feels that way about all the plugins and hooks too, thet work but feel tied we cant enhance them | 21:36 | |
| imbrandon | :( | 21:36 |
| imbrandon | yea | 21:36 |
| imbrandon | that sounds rihgt | 21:37 |
| SpamapS | I can flip it around and make it work | 21:37 |
| imbrandon | your takergeting the container for the hooks relating and the plane ments should not matter | 21:37 |
| SpamapS | relating nrpe<->*<->nagios | 21:37 |
| SpamapS | and just having nrpe tell * to inject stuff into the monitors relationship | 21:37 |
| SpamapS | but that is *lame* | 21:37 |
| SpamapS | because now I can't have generic nrpe/nagios relations | 21:37 |
| imbrandon | so in theory you could have a subordiate that lives on its own single node but relates to many tings that way | 21:38 |
| SpamapS | I'm going to step away from nagios now | 21:38 |
| SpamapS | Its pissing me off too much. ;) | 21:38 |
| imbrandon | SpamapS: yup i know the feeling, thus how many times i have done the same thing on nginx and others ... maybe not that complex but i get jaded that i cant do something in a way i reason in my head is rweally the correct one | 21:39 |
| imbrandon | so yea ... i feel yea. and now as you pointed out the other day i think lots of mediocre charms just need to get pushed, these are ALL frakiin wrong and bad | 21:40 |
| imbrandon | but they are out base and we just iterate them when each thing lands or is fixed | 21:40 |
| imbrandon | vs not having nginx for a month :) | 21:40 |
| imbrandon | i hate that, i hatre working like that but i dont see another way right nowwith how things are setup, too much stuff is waiting on go, wel really only probably 2 things are but then it cascades and nothing can go, because to "fix" some of this its really implment the feature/provider | 21:42 |
| imbrandon | etc | 21:42 |
| * imbrandon preaches to the chior more | 21:42 | |
| * imbrandon is sadened by his ~/Projects directory ... | 21:43 | |
| SpamapS | :) | 21:46 |
| imbrandon | way tooo many things in to ( as i tend to only keep in progress projects there and once they hit a certain point they live online or a nother perm place on the hss/server .... but SpamapS check this .... | 21:46 |
| imbrandon | bholtsclaw@mini:~$ ls -lR ~/Projects|wc -l | 21:46 |
| imbrandon | 423438 | 21:46 |
| SpamapS | uh | 21:47 |
| SpamapS | imbrandon: randomly delete half of them? ;) | 21:47 |
| imbrandon | well i got 2 big things that are adding to that ... a charm/getall symlink and a OMG full checkout | 21:48 |
| imbrandon | minus those two and it would be a bit more reasonable ... but still | 21:48 |
| imbrandon | its way way past where my OCD/Retardation normaly keep it | 21:49 |
| imbrandon | bholtsclaw@mini:~$ du -sh ~/Projects/ | 21:49 |
| imbrandon | 15G/home/bholtsclaw/Projects/ | 21:49 |
| imbrandon | and crap i just realized that the hostname on this is wrong ... hrm suprised more things are not broken for me ... that i noticed yet least | 21:50 |
| imbrandon | oh yea synergy is broke ... wth , how would my hostname revert ... | 21:51 |
| imbrandon | ahh terminal was left open from prior to it changing ... heh | 21:52 |
| hazmat | SpamapS, generally speaking it is | 22:00 |
| hazmat | i answered that last night.. but you might have not been here.. but basically its the lack of coordination around stop. ie. parent supervision kills children | 22:00 |
| hazmat | in this context (subordinates) its actually a bit worse | 22:01 |
| hazmat | SpamapS, pls file a bug if you have a specific behavior around that you'd like to see | 22:01 |
| hazmat | data jitsu ;-) http://oreilly.com/data/radarreports/data-jujitsu.csp?cmp=tw-strata-books-data-products | 22:14 |
| imbrandon | heh wow | 22:16 |
| imbrandon | hazmat: u seen @http://jit.su ? the nodejs api tool | 22:17 |
| imbrandon | err the nodejitsu api too | 22:17 |
| hazmat | imbrandon, yeah.. i've got an account there | 22:17 |
| hazmat | one of the few paas's to support websockets | 22:18 |
| imbrandon | heh me too and appfog and phpfog and vmc and phpcloud and heroku and about a half dozen others i shiuld use more | 22:18 |
| imbrandon | lol | 22:18 |
| imbrandon | yea i *think* nodester does | 22:18 |
| imbrandon | too, iirc | 22:18 |
| hazmat | SpamapS, imbrandon so to expose services on hpcloud.. did you do it by hand via the control panel? | 22:19 |
| imbrandon | ugh i need to rethink/redo my emails again and unzub from some things ... i hate that gotta do it about once every 2 years | 22:20 |
| imbrandon | hazmat: yea i'm lazy and just set it to 0.0.0.0/0 on bootstrap for the default one | 22:20 |
| imbrandon | and not care about the others | 22:20 |
| imbrandon | cuz not prod services were there yet | 22:20 |
| imbrandon | etc | 22:21 |
| imbrandon | e.g wide open | 22:21 |
| imbrandon | err not default but the service name without a number | 22:21 |
| imbrandon | cuz all of them had it | 22:21 |
| hazmat | actually it looks like the expose functionality works with the tweak branch | 22:21 |
| hazmat | imbrandon, i just setup for internal traffic limited to 10.2.0.0 | 22:21 |
| imbrandon | cool, yea i have it pulled but noy launced a new env yet | 22:22 |
| hazmat | ie. private network addrs | 22:22 |
| imbrandon | yup | 22:22 |
| imbrandon | and the pubs are always 15.185 | 22:22 |
| imbrandon | in all 3 az | 22:22 |
| imbrandon | s | 22:22 |
| imbrandon | just fyi if you wanna limit that for someting later | 22:22 |
| hazmat | its unfortunate but not horrible i guess | 22:22 |
| hazmat | its manual though which is lame and needs docs | 22:23 |
| imbrandon | yea i've seen 15.185.100 .99 .98 and .102 | 22:23 |
| imbrandon | manual ? | 22:23 |
| hazmat | imbrandon, the rule management for the internal traffic | 22:23 |
| hazmat | since hpcloud doesnt' support a rule for a self referential group (ie open group traffic) | 22:24 |
| imbrandon | they have cli tools too , and juju is doing something with the IPs too funky as they dont automaticly get floating ones | 22:24 |
| imbrandon | noramlly | 22:24 |
| imbrandon | yea it does | 22:24 |
| imbrandon | you just have to use the nova client to make it | 22:24 |
| imbrandon | you cant make it with the webui | 22:24 |
| imbrandon | half sec | 22:24 |
| hazmat | imbrandon, you mean the hpcloud tool? | 22:25 |
| imbrandon | damn internet is going slow, but its in the docs, and no i mean python-novaclient | 22:25 |
| imbrandon | eg "nova" onthe cli | 22:25 |
| imbrandon | i'll toss ya my env vars to get it setup | 22:26 |
| imbrandon | this crap was all trial an error | 22:26 |
| imbrandon | its all piecedmealed and fraken stein and wrong | 22:26 |
| imbrandon | hehe | 22:26 |
| SpamapS | hazmat: bug 1025478 btw.. filed yesterday :) | 22:28 |
| _mup_ | Bug #1025478: destroying a primary service does not cause subordinate to depart <juju:New> < https://launchpad.net/bugs/1025478 > | 22:28 |
| hazmat | there nova is some forked bespoke version | 22:28 |
| imbrandon | yup so is their fog | 22:28 |
| imbrandon | aka hpfog | 22:28 |
| hazmat | yeah.. but at least that one works pretty easily | 22:29 |
| hazmat | there nova fork never did auth succesfully for me | 22:29 |
| SpamapS | wow | 22:29 |
| SpamapS | so they just went off in the weeds | 22:29 |
| * hazmat nods | 22:29 | |
| hazmat | SpamapS, it looks like their going down the path of supporting all their own client libs | 22:30 |
| hazmat | and some what amusingly python isn't one of those per se | 22:30 |
| SpamapS | thats just awesome | 22:30 |
| SpamapS | why isn't anyone shaking fists at these people? | 22:30 |
| hazmat | https://docs.hpcloud.com/bindings | 22:30 |
| SpamapS | probably because nobody is actually trying to use them yet | 22:30 |
| hazmat | SpamapS, because everyone is busy congratulating them on validating openstack? | 22:30 |
| SpamapS | I would think the libcloud guys in particular would be a champion for this | 22:31 |
| imbrandon | export NOVA_USERNAME=me@brandonholtsclaw.com | 22:31 |
| imbrandon | export NOVA_PASSWORD= | 22:31 |
| imbrandon | export NOVA_URL=https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/ | 22:31 |
| imbrandon | export NOVA_PROJECT_ID=me@brandonholtscaw.com-default-tenant | 22:31 |
| imbrandon | export NOVA_VERSION=1.1 | 22:31 |
| imbrandon | export NOVA_REGION_NAME=az-3.region-a.geo-1 | 22:31 |
| Daviey | imbrandon: what is the NOVA_PASSWORD ? | 22:31 |
| hazmat | imbrandon, pastebin would have been nicer, but thanks | 22:31 |
| imbrandon | hazmat: that will make their nova work + your webui pass | 22:31 |
| hazmat | imbrandon, tried that didn't work for me though | 22:32 |
| imbrandon | bah was lazy, ohhh crap for got that one Davie, its PinkPoniez23 | 22:32 |
| imbrandon | :) | 22:32 |
| Daviey | ta | 22:32 |
| hazmat | i tried just about every permutation of key/username/password/token auth | 22:32 |
| hazmat | the ruby cli tool works but is limited | 22:32 |
| hazmat | and frankly given how poorly nova worked against rspace.. | 22:32 |
| imbrandon | also with the swift cli tool you need to drop the auth to v1 as well | 22:32 |
| imbrandon | :( | 22:32 |
| SpamapS | I was able to use stock precise nova manage | 22:32 |
| hazmat | i'm not expecting to much | 22:33 |
| SpamapS | IIRC | 22:33 |
| imbrandon | yea i can use the nova | 22:33 |
| hazmat | SpamapS, oh.. interesting.. i'll have to try that .. | 22:33 |
| hazmat | i tried their version of nova in a sandbox | 22:33 |
| imbrandon | nova i installed from repo, the hpfoghpcloud i setup the first time, then it screwed my rvm/rails install | 22:34 |
| imbrandon | and nevew reinstalled it | 22:34 |
| hazmat | most of nova cli doesn't work against rackspace.. volume/secgroups, etc.. because that's all split out into separate services that aren't publicly available atm | 22:34 |
| imbrandon | but did send em a patch so it would work on 1.8.7 :) | 22:34 |
| hazmat | its really a mystery to find out what impl of nova actually supports as far as api | 22:34 |
| hazmat | most of these have some sort of java front end proxy doing request load balancing against the api services and rate limiting it appears (rspace and hp) | 22:35 |
| imbrandon | hp has a knolge base one just for nova cli that lists what does/cont | 22:35 |
| imbrandon | dont* | 22:35 |
| SpamapS | java front say what? | 22:38 |
| imbrandon | i think they had apache traffic server on the GUI console when i first looked | 22:46 |
| imbrandon | but their strange anyhow without much direction i dont think, they have yui and jquery and a few other double duties things i've noticed | 22:47 |
| imbrandon | they tend to grab whatever tool from where ever to do that they need for the moment and customize it a bit then not really fully intergrate any of it | 22:48 |
| imbrandon | cept the php-bindings | 22:48 |
| imbrandon | that is the only exception :) | 22:48 |
| imbrandon | anyhow none is a big deal until after a month or 6 you add them all up | 22:48 |
| imbrandon | and its like wow this really sucks | 22:49 |
| SpamapS | -relname=${JUJU_RELATION_ID##:*} | 22:51 |
| SpamapS | +relname=${JUJU_RELATION_ID%%:*} | 22:51 |
| SpamapS | I hate bash | 22:51 |
| SpamapS | why am I doing things in bash? | 22:51 |
| imbrandon | bwhahaha <?php require('/hooks/lib/common'); | 22:52 |
| imbrandon | you know you wanna :) | 22:52 |
| SpamapS | What I want is for my charms to be 90% declarative | 22:52 |
| imbrandon | ugh i need to push the new stuff for that too .... | 22:52 |
| SpamapS | I wonder if salt's states can be easily used w/o salt's agents | 22:53 |
| imbrandon | nah i like the flexability , i just want first class config mangemwent and raw access to the cloud-init even if hidden normally | 22:53 |
| imbrandon | chefsolo or capistrano or fabric ... ? /me figured ytou'd use fabfiles.py | 22:54 |
| imbrandon | fabfile.py* | 22:54 |
| imbrandon | SpamapS: you know since we require cloud-init on the machines , it bakes capistrano and puppet as dependencies already so | 22:55 |
| imbrandon | they are their to be used :) | 22:56 |
| imbrandon | we just cant use them in cloud-init but could still use them :P | 22:56 |
| SpamapS | how does one break out of puppet into a real language when it is needed? | 22:56 |
| imbrandon | no idea, i hate puppet personally its a DSL not ruby | 22:56 |
| imbrandon | but i know others like em | 22:57 |
| imbrandon | only really seriously used capistrano or homecrown config mgmt for any length of time | 22:57 |
| imbrandon | but i seriously do thing you CAN | 22:57 |
| imbrandon | just not sure HOW | 22:57 |
| SpamapS | Yes I know its a DSL | 22:58 |
| SpamapS | and there are these tiny moments where you need to do something that the DSL can't do | 22:58 |
| SpamapS | I want to know how do I add those capabilities | 22:58 |
| SpamapS | I suspect "with ruby" | 22:58 |
| imbrandon | SpamapS: you might check mozillas github repo for their config stuff they use it alot iirc | 22:59 |
| imbrandon | well all of them | 22:59 |
| imbrandon | diffrently | 22:59 |
| imbrandon | no thats what i'm saying you dont have to add them they are already installed you just need to like call "puppet somefile.pp" | 23:00 |
| imbrandon | in your shell | 23:00 |
| SpamapS | Ok here's another problem with subs.. | 23:00 |
| SpamapS | so my nrpe charm had a bug in the joined hook for the subordinate relation | 23:00 |
| SpamapS | I fixed it | 23:00 |
| SpamapS | upgraded charm | 23:00 |
| SpamapS | but oh noes.. I can't actually apply the fix | 23:00 |
| imbrandon | cloud-init depends on them , thats why i was irritated when we was restricted from them i'm like ummmmm that sux | 23:00 |
| SpamapS | I have to now rewrite that joined charm to be "relation agnostic" | 23:00 |
| SpamapS | err | 23:00 |
| SpamapS | joined hook | 23:00 |
| SpamapS | and then I have to call that hook in upgrade-charm | 23:01 |
| imbrandon | SpamapS: yea when that happeens to me i ssh to the box and edit the files in /var/lib/juju/blahvblhblag | 23:01 |
| SpamapS | except, the error was in the way it grabs JUJU_REMOTE_UNIT | 23:01 |
| imbrandon | then --resolved -retry | 23:01 |
| imbrandon | or whatever | 23:01 |
| SpamapS | which is not available ever again | 23:01 |
| SpamapS | so.. doh .. kill the primary service and start over | 23:02 |
| imbrandon | well i do it local imediately but the alternative is to destroy it | 23:02 |
| SpamapS | imbrandon: it did not error | 23:02 |
| imbrandon | no | 23:02 |
| SpamapS | so there's nothing to retry | 23:02 |
| imbrandon | not debuu=g hooks | 23:02 |
| imbrandon | ok lets step back | 23:02 |
| imbrandon | i know there is nothing .. i'm telling you my hack | 23:02 |
| imbrandon | that works BUT now will get fixed | 23:02 |
| imbrandon | hehe | 23:02 |
| imbrandon | ok soooooo | 23:02 |
| SpamapS | that fix won't even work here | 23:03 |
| SpamapS | unfortunately | 23:03 |
| imbrandon | when it gets to that point that you only have the choice to destroy it | 23:03 |
| imbrandon | i dont debug-hook | 23:03 |
| imbrandon | i ssh TO it | 23:03 |
| imbrandon | real ssh | 23:03 |
| imbrandon | end edit the cached charm | 23:03 |
| SpamapS | there's a value that I can't get | 23:03 |
| SpamapS | its gone | 23:03 |
| imbrandon | then when i run retry --resiolved | 23:03 |
| imbrandon | etc it refiles | 23:03 |
| SpamapS | there is nothing to retry | 23:04 |
| SpamapS | state is up | 23:04 |
| imbrandon | i know there isnt . but it still does it | 23:04 |
| SpamapS | interesting | 23:04 |
| imbrandon | and fixes my problem | 23:04 |
| imbrandon | heh | 23:04 |
| imbrandon | yea i think that unintentionally that "juju resolved --retry service/2" | 23:05 |
| imbrandon | is out "force a hook" | 23:05 |
| imbrandon | our* | 23:05 |
| SpamapS | 2012-07-17 16:05:22,323 WARNING Matched relations are all running | 23:05 |
| imbrandon | sure but watch the log | 23:05 |
| SpamapS | nope | 23:05 |
| SpamapS | nothing running | 23:05 |
| SpamapS | which hook would it even run? | 23:06 |
| imbrandon | it always tells me that it dident need to sdo it and act like it dont then i watch the debug log and hooks start going to town | 23:06 |
| SpamapS | joined? | 23:06 |
| SpamapS | changed? | 23:06 |
| imbrandon | changed | 23:06 |
| SpamapS | I think you been smokin something son | 23:06 |
| SpamapS | that does nothing | 23:06 |
| imbrandon | you could also edit the upgrade temp | 23:06 |
| SpamapS | I could do 100 things | 23:07 |
| SpamapS | but I know juju in and out | 23:07 |
| SpamapS | users might not want to learn that | 23:07 |
| imbrandon | yea they do if there was a way | 23:07 |
| imbrandon | :) | 23:07 |
| imbrandon | no seriously they will tho | 23:07 |
| imbrandon | think about how well the chef users know that | 23:08 |
| imbrandon | etc | 23:08 |
| SpamapS | If you brought me something and said "well sometimes you just have to ssh in and edit random files that you don't understand on disk".. I'd bring you a rather large blunt object to beat me to death with | 23:08 |
| SpamapS | we are not targetting chef users | 23:08 |
| imbrandon | its part of their profession that like most ggeks will take pride in knowing more than the boss | 23:08 |
| imbrandon | :) | 23:08 |
| SpamapS | chef is a big beast | 23:08 |
| SpamapS | so is puppet | 23:08 |
| imbrandon | heh | 23:08 |
| SpamapS | not './configure && make && make install' for the cloud.. *apt-get* for the cloud | 23:08 |
| imbrandon | sure if thats all we needed we have aws-tools in the repo | 23:09 |
| SpamapS | as in, install it, configure it, and get it to the point where you can use it. | 23:09 |
| imbrandon | you all keep saying shit like that and are contradiucting your selfs | 23:09 |
| SpamapS | oh? | 23:09 |
| SpamapS | I know how *I* feel | 23:09 |
| SpamapS | I think I disagree with quite a few on this | 23:10 |
| imbrandon | dude we just had tis exact convo | 23:10 |
| imbrandon | like a week ago | 23:10 |
| imbrandon | heh | 23:10 |
| SpamapS | and I said we want puppet for the cloud? | 23:10 |
| SpamapS | cause, I was drinking a lot last week... | 23:10 |
| imbrandon | well i explained untill shit lands we are puppet | 23:10 |
| SpamapS | oh lol | 23:10 |
| SpamapS | until shit lands we are a ship w/o a rudder | 23:10 |
| imbrandon | yea that was the jest of it | 23:11 |
| imbrandon | :) | 23:11 |
| SpamapS | so lets not talk about our problems, but our intentions | 23:11 |
| SpamapS | intentions == simpler way to get stuff into the cloud | 23:11 |
| imbrandon | no for real tho we went though all this, and you kept telling me how we wernt but we could do this and that and this yet and had to ignbore dry and blah , and i'm like ummm ok so we're not ...... yet :) | 23:11 |
| SpamapS | And a way to share the deployment "formulas"..errr.. charms.. so you don't have to sit down and open your editor and refactor half of the thing to get it tow ork in your infra | 23:12 |
| imbrandon | oh yea i totally see where we're headed , or i wouldent be here | 23:12 |
| SpamapS | What we aren't, and what we want to be, are the same thing. :) | 23:12 |
| SpamapS | So yes, we can ssh in and dick with files | 23:12 |
| SpamapS | but I am kind of tired of throwing bugs on the pile. ;) | 23:12 |
| imbrandon | but i was and am being a realist about right now, we do pretty much aboutr 60% of things worse with plans that are solid to be 200% better | 23:12 |
| imbrandon | the same realist in me says tho once it does all pan out its gonna be sweet tho ! | 23:14 |
| imbrandon | heh and much better than if you bolted this onyo chef or salt stacks | 23:14 |
| imbrandon | or made anseble not use the /etc/hosts file for 7000 nodes | 23:14 |
| imbrandon | ugh | 23:14 |
| SpamapS | haha | 23:15 |
| SpamapS | I see you tried it out? :p | 23:15 |
| imbrandon | you rwealize that btw, ok so here is the 5min explination of anseble == juju s#zookeeper#/etc/hosts+hash in memory no matter how many nodes of ALLLLLL meta data and databags#g | 23:16 |
| imbrandon | yea i try everything , some things more than i should some less ;), thats why i end up with accounts on all services , because unless you try or learn about every part that you might at some point need to know i dunno to me you just should know the options available as much of a pita it si, like your CentOS thing :) | 23:17 |
| SpamapS | yeah I try to keep it limited to the biggest alternatives, not all alts | 23:18 |
| imbrandon | but yea. try to use anseible with 700 nodes launched and another 300ish off but in the host file ... I saw the results and laughed as i unsubscribed from the ML | 23:19 |
| imbrandon | its perfect for 5 machines, does somethings alot better , but ... yea $6 on fuckit | 23:19 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!