| TeTeT | hi, I'm following http://askubuntu.com/questions/65359/how-do-i-configure-juju-for-local-usage and I run into a problem at bootstrap already. the virtual net cannot be set up | 10:23 |
|---|---|---|
| TeTeT | http://pastebin.ubuntu.com/851168/ http://pastebin.ubuntu.com/851170/ | 10:23 |
| TeTeT | I try this in a vm, maybe the network setup does not work with the vm's network emulation? | 10:23 |
| === TheMue_ is now known as TheMue | ||
| koolhead17 | TeTeT: i tried it on a physical machine :P | 11:05 |
| TeTeT | koolhead17: hmm | 11:07 |
| SpamapS | TeTeT: what release of juju? (apt-cache policy juju) | 15:03 |
| TeTeT | SpamapS: Installed: 0.5+bzr457-0ubuntu1 from preicse | 15:05 |
| TeTeT | SpamapS: is it supposed to work on a vm managed through libvirt and using the default virbr0 connected interface? Or does it need a real bridge to the ethernet card of the system? | 15:06 |
| SpamapS | TeTeT: it should work fine inside a VM yes | 15:07 |
| TeTeT | SpamapS: sudo virsh net-start default just does not work, maybe because of vm inside of a vm constraint? | 15:10 |
| SpamapS | TeTeT: no, the networking part of virsh is pretty simple | 15:14 |
| SpamapS | TeTeT: those "virbr" bridges are just bridges with no physical components. | 15:14 |
| TeTeT | SpamapS: ok. any idea how to get this working? | 15:15 |
| SpamapS | TeTeT: whats the problem with virsh net-start default? | 15:16 |
| TeTeT | SpamapS: error: internal error Network is already in use by interface eth0 | 15:16 |
| benji | I'm having a problem running on EC2; the bootstrap appears to work then I run juju status, lie to ssh about verifying the fingerprint and then juju status hangs, never to return | 15:17 |
| SpamapS | TeTeT: sounds like you have it configured to be a "real" bridge instead of a virtual one. | 15:19 |
| SpamapS | benji: does 'juju ssh 0' work? | 15:19 |
| * benji tries | 15:20 | |
| benji | SpamapS: nope, I get "2012-02-21 10:20:05,876 INFO Connecting to environment..." and then a hang | 15:20 |
| TeTeT | SpamapS: doubt it, the vm has the normal 192.168.122.x address for virbr0 on the host | 15:20 |
| SpamapS | benji: interesting... | 15:22 |
| benji | SpamapS: this strace output looks like it might mean something to someone that knows what's going on: http://paste.ubuntu.com/851476/ | 15:23 |
| SpamapS | benji: hmmm.. you are running with default-series: precise ? | 15:24 |
| benji | SpamapS: yep | 15:24 |
| SpamapS | benji: have to go afk for a bit | 15:24 |
| SpamapS | benji: try oneiric.. if that works.. we have a bug in precise | 15:24 |
| benji | will do | 15:24 |
| benji | SpamapS: it works fine with oneiric | 15:34 |
| SpamapS | benji: ok, do you want to open a bug report against juju? I can do it too if you don't want to. | 15:35 |
| benji | SpamapS: I can | 15:36 |
| SpamapS | benji: *thank you*! | 15:38 |
| * hazmat upgrades to precise | 15:39 | |
| hazmat | TeTeT, did you already have a bridge setup? it looks like its just the default libvirt bridge there | 15:41 |
| TeTeT | hazmat: nah, I didn't setup an extra bridge. | 15:41 |
| TeTeT | hazmat: oh, the reboot didn't change anything for me | 15:41 |
| TeTeT | hazmat: which is mentioned in the askubuntu article | 15:42 |
| _mup_ | Bug #937889 was filed: Hang on "juju status" with EC2 and Precise <juju:New> < https://launchpad.net/bugs/937889 > | 15:57 |
| m_3 | benji SpamapS: I had problems with precise juju instances at the end of last week too... it couldn't install packages b/c of dep problems (txzookeeper iirc) | 15:57 |
| m_3 | didn't get a chance to really debug it tho, but it's easily reproducable with just a bootstrap | 16:00 |
| m_3 | (ec2) | 16:00 |
| jcastro | SpamapS: m_3: did you guys get a mail from wordpress for the server blog? | 16:03 |
| benji | SpamapS: https://bugs.launchpad.net/juju/+bug/937889 | 16:04 |
| _mup_ | Bug #937889: Hang on "juju status" with EC2 and Precise <juju:New> < https://launchpad.net/bugs/937889 > | 16:04 |
| m_3 | jcastro: looking now | 16:04 |
| m_3 | benji: thanks, I'll +1 it | 16:04 |
| jcastro | m_3: oh so it worked? awesome, didn't know if it was set up to send mail, nice work! | 16:05 |
| m_3 | jcastro: don't see it | 16:05 |
| m_3 | easiest is probably postfix/gmail to send... I'd bet any direct sends'll be already blacklisted | 16:06 |
| * jcastro nods | 16:06 | |
| jcastro | our first issue, mail! | 16:06 |
| m_3 | jcastro: I've got that config snapshotted somewhere if you want me to dig | 16:06 |
| jcastro | this will be fun | 16:06 |
| m_3 | jcastro: yup! | 16:07 |
| jcastro | m_3: hey so, you think you can charm up lp:summit this week? | 16:08 |
| jcastro | it'd be a nice win | 16:08 |
| m_3 | jcastro: lemme look at it... I'm buried in stuff atm, but it wasn't too bad iirc | 16:10 |
| SpamapS | jcastro: still powering through my email | 16:11 |
| jcastro | SpamapS: it's ok it probably didn't send | 16:11 |
| robbiew | jcastro: you got blog access now | 16:20 |
| jcastro | yep, thanks | 16:20 |
| jcastro | man, wordpress needs FTP to import blogs | 16:20 |
| jcastro | m_3: everytime we run into a problem I'm going to say "octo" | 16:20 |
| m_3 | jcastro: and I'll be happy to "+1 | 16:33 |
| m_3 | " that | 16:33 |
| m_3 | jcastro: zk charm reviewed and ready to promulgate | 16:34 |
| * m_3 rings the promulgate bell | 16:34 | |
| jcastro | oooh cute! | 16:36 |
| * jcastro will blog that one | 16:37 | |
| jcastro | anything special I should care about, or is it in the readme? | 16:37 |
| m_3 | jcastro: readme's on the way still... it's almost eod for jamespage too so prob won't happen until tomorrow | 16:40 |
| jcastro | ok no worries | 16:40 |
| jamespage | \o/ | 16:41 |
| * jamespage is going to charm hbase tomorrow | 16:41 | |
| jamespage | m_3: I was thinking of using dotdee to manage the config for hbase - have you seen any use of it in charms so far? | 16:43 |
| m_3 | jamespage: have not | 16:44 |
| m_3 | I think that's a great idea | 16:44 |
| jamespage | m_3: it would mean that you could generate the part of the config file associated with the event without having knowledge of the rest of the configuration | 16:45 |
| m_3 | some charms have used a combo of config snippets and sed... but dotdee would be a little less manual | 16:45 |
| m_3 | I've been experimenting with calling cheetah from the command line... (exporting vars into the env)... but it's... eh | 16:46 |
| SpamapS | sed is a fail IMO | 16:47 |
| SpamapS | we should be building config files from scratch or using dotdee | 16:47 |
| m_3 | sed can actually be a simpler/cleaner solution for small config changes | 16:49 |
| m_3 | and easier to maintain b/c it's just pure diffs | 16:49 |
| m_3 | but yes, for large-scale or complex config... it sucks | 16:49 |
| jcastro | m_3: hey the IP for that blog instance won't change will it? | 16:51 |
| m_3 | yeah, it totally will... lemme attach an elastic IP real quick... hang on | 16:52 |
| jcastro | thanks | 16:52 |
| jamespage | m_3: I did some cheetah stuff for tomcat7 I think | 16:56 |
| m_3 | jcastro: ok, 23.21.249.196... can you point the server blog url directly to that IP addr? I can add a url like xxx.markmims.com in front of it if we need to | 16:57 |
| jcastro | ok | 16:58 |
| jcastro | hey should juju ssh blog/0 work? | 16:58 |
| SpamapS | hazmat: is that reboot support I see landing in lp:juju ?! | 16:58 |
| jcastro | m_3: hey so does assigning that IP break the existing aws URL? | 17:01 |
| m_3 | jcastro: you might need to specify environment... juju ssh -efido blog/0 | 17:02 |
| m_3 | jcastro: checking on everything now | 17:02 |
| hazmat | SpamapS, yup, and upstartification, and all kinds of yummy goodness | 17:04 |
| jcastro | m_3: I get a connection timed out error | 17:04 |
| m_3 | jcastro: wow... it looks like aws just removed the old dns entry | 17:05 |
| jcastro | and the ec2 url just stopped working for me altogether | 17:05 |
| m_3 | gave it a new one that matches the elastic ip | 17:05 |
| m_3 | ec2-23-21-249-196.compute-1.amazonaws.com | 17:05 |
| jcastro | hah, awesome | 17:06 |
| m_3 | but now it seems like juju's lost | 17:06 |
| jcastro | yep, I was going to ask, did we just find a new bug? | 17:06 |
| m_3 | I've done this before without this problem... lemme poke around and see what happened | 17:07 |
| jcastro | And we're off to a great start! | 17:07 |
| m_3 | in fact, it's up without this problem in another environment | 17:07 |
| jcastro | m_3: save your history, this will be a good post. | 17:07 |
| m_3 | nice, now juju status reports different addresses for the service unit and the machine | 17:08 |
| m_3 | it's trying to ssh to the old one that the service unit shows (and not getting through) | 17:09 |
| jcastro | yep | 17:09 |
| m_3 | jcastro: you can still get in using the explicit machine id... 'juju ssh -efido 2' | 17:10 |
| jcastro | ah! | 17:10 |
| jcastro | jamespage: thanks for the promulgation! | 17:20 |
| jamespage | jcastro, no problemo! | 17:20 |
| _mup_ | Bug #937949 was filed: juju status shows addresses that are out of sync <juju:New> < https://launchpad.net/bugs/937949 > | 17:21 |
| m_3 | jcastro: ^^ | 17:21 |
| mars | Hi guys, I have a question about installing from Launchpad private PPAs: is there a good charm cookbook recipe for doing this? | 17:29 |
| mars | This looks good for a start: http://charms.kapilt.com/~openstack-ubuntu-testing/precise/nova-volume/hooks/nova-volume-common | 17:29 |
| m_3 | mars: https://code.launchpad.net/~james-page/charms/precise/zookeeper/trunk is another example | 17:33 |
| m_3 | it's pretty straightforward once you have packages _in_ the ppa | 17:33 |
| m_3 | the charm's install hook just add-apt-repository and then apt-get update, then apt-get install | 17:34 |
| m_3 | getting packages into the ppa is a whole other ballgame :) | 17:34 |
| mars | m_3, thanks, that is a nice way to do public archives | 17:35 |
| mars | Private ones require a bit more work, what with the custom URL and all. AFAIK add-apt-repository doesn't handle them | 17:36 |
| mars | The two recipes can probably be merged to what I want: install from a private PPA, public PPA, or archive. | 17:38 |
| m_3 | mars: ah, gotcha... sorry | 17:39 |
| m_3 | yeah, any ppas outside of launchpad seems like the tough thing with that would be key exchange | 17:39 |
| m_3 | but I really don't know | 17:39 |
| m_3 | we try to make sure any downloaded payloads can be cryptographically verified... for the charms in the charm store | 17:40 |
| m_3 | for other charms, anything goes... pulling from github for node.js, npm, gems, etc is pretty common | 17:41 |
| SpamapS | m_3: whats the txzookeeper problem exactly? | 17:51 |
| m_3 | SpamapS: boostrap a precise environment... ppa or distro doesn't matter | 17:52 |
| m_3 | then ssh directly to the instance (juju ssh hangs) | 17:53 |
| m_3 | dig through the log... packages aren't installed b/c of a dep problem (I vaguely remember it being txzk...) | 17:54 |
| m_3 | I can reproduce and dig out the logs if you want, just lemme know | 17:55 |
| m_3 | http://paste.ubuntu.com/851637/ | 17:57 |
| m_3 | SpamapS: ^^ | 17:57 |
| m_3 | SpamapS: added your key to ubuntu@ec2-174-129-55-132.compute-1.amazonaws.com | 17:58 |
| SpamapS | m_3: that looks like just plain broken images | 18:00 |
| === grapz_afk is now known as grapz | ||
| m_3 | SpamapS: it's strangely similar to my desktop problem atm: http://paste.ubuntu.com/851649/ | 18:03 |
| SpamapS | m_3: yours looks like an out of sync mirror.. | 18:04 |
| m_3 | oh, nice... /me fixing that! | 18:04 |
| m_3 | very scared I have to reinstall | 18:04 |
| SpamapS | m_3: definitely not | 18:05 |
| SpamapS | m_3: just point at us.archive.ubuntu.com | 18:05 |
| jcastro | SpamapS: m_3: incoming mail about a charm contest, please review by end o business today, so I can launch this badboy | 18:05 |
| m_3 | jcastro: roger roger | 18:05 |
| m_3 | SpamapS: yeah, taking my apt-cacher-ng out of the picture | 18:06 |
| m_3 | SpamapS: no change | 18:08 |
| SpamapS | m_3: have you tried dist-upgrade ? | 18:09 |
| SpamapS | m_3: or apt-get -f install ? | 18:10 |
| m_3 | yup, same | 18:10 |
| SpamapS | m_3: that does not make any sense. | 18:10 |
| m_3 | variations of -f or --fix-xxx didn't seem to do much | 18:11 |
| SpamapS | m_3: you have something damanged.. what version of gnome-control-center? | 18:11 |
| m_3 | I hadn't been using any mirrors.... other than apt-cacher-ng | 18:11 |
| m_3 | gnome-control-center 1:3.2.2-2ubuntu8 | 18:12 |
| SpamapS | m_3: 1:3.3.5-0ubuntu2 is the latest. It should be installing that | 18:12 |
| m_3 | it was an upgrade from oneiric and not a fresh install | 18:12 |
| SpamapS | mine too | 18:12 |
| SpamapS | mine goes back to 10.10 :) | 18:12 |
| m_3 | the juju one is more important though | 18:13 |
| SpamapS | yes I'm looking into the juju one | 18:13 |
| SpamapS | I think thats just broken images | 18:13 |
| m_3 | jcastro: comments in on the charm contest | 18:38 |
| SpamapS | hazmat: https://code.launchpad.net/~clint-fewbar/juju/use-packages-yaml/+merge/94040 | 19:36 |
| SpamapS | hazmat: fixes running juju w/ precise instances | 19:37 |
| SpamapS | hazmat: oddly enough.. I used lbox propose, but I don't see it talking to rietveld. :-P | 19:38 |
| * SpamapS lunches | 19:38 | |
| hazmat | SpamapS, it needs a -cr flag to make it go to reitveld | 19:40 |
| SpamapS | ah, next time.. its trivial anyway | 19:40 |
| SpamapS | benji: hey we found a resolution for the bug you reported this morning | 21:46 |
| SpamapS | err | 21:46 |
| SpamapS | I should say, a few hours ago.. might not have been morning | 21:47 |
| benji | It was morning somewhere. | 21:47 |
| SpamapS | benji: thanks for trying out precise. :) The problem was that libc6 was updated between alpha1 and now, so debconf was prompting for some questions | 21:57 |
| SpamapS | And the real underlying problem was that we were doing apt-get in runcmd instead of listing the package in cloud-init's 'packages' line | 21:58 |
| m_3 | SpamapS: did you merge and kick off a ppa build? | 21:59 |
| benji | SpamapS: interesting; good turn-around time on the fix | 21:59 |
| m_3 | SpamapS: nevermind... I see | 22:03 |
| hazmat | SpamapS, that seems a little strange only because by default will use nightlies builds on ec2, i suppose that's not the case for openstack | 22:08 |
| SpamapS | hazmat: no, we use released builds by default on ec2 | 22:19 |
| SpamapS | def get_current_ami(ubuntu_release="oneiric", architecture="i386", persistent_storage=True, region="us-east-1", daily=False, desktop=False, url_fetch=None): | 22:20 |
| SpamapS | data["version"] = daily and "daily" or "released" | 22:20 |
| hazmat | ugh. | 22:20 |
| SpamapS | hazmat: thats the more conservative approach | 22:20 |
| SpamapS | hazmat: trouble is, there's no way to override it | 22:20 |
| SpamapS | though I suspect that will come with the full implementation of constraints | 22:20 |
| hazmat | SpamapS, we should be using the nightlies, we tell cloud-init to do an update/upgrade | 22:22 |
| hazmat | so we're just wasting bandwidth | 22:22 |
| hazmat | and it also means we get newer kernels | 22:22 |
| SpamapS | hazmat: true! | 22:22 |
| SpamapS | though | 22:22 |
| SpamapS | less repeatable deploys with that strategy | 22:23 |
| SpamapS | can't tell you how many times I had version skews drive me *NUTS* in tracking down problems at 3am | 22:23 |
| hazmat | SpamapS, is the reality any different if we're doing an upgrade on the machine prior to setting up juju? | 22:23 |
| SpamapS | hazmat: I'm suggesting that the upgrade is also a problem | 22:24 |
| SpamapS | hazmat: I think ultimately we just need ways to say "fire all install hooks" on a service so that you can re-assert versions | 22:25 |
| * hazmat nods | 22:25 | |
| hazmat | version skew pincer movement, wiped out entire roman legions, didn't even need the elephants at cannae | 22:25 |
| SpamapS | lol | 22:25 |
| hazmat | yeah.. i can see it both ways | 22:26 |
| hazmat | if your reproducing or adding units to the service, you definitely want the same versions, if your deploying fresh, i'd see wanting to have the latest stable-updates applied | 22:27 |
| hazmat | as a goal | 22:27 |
| hazmat | SpamapS, sounds worth some more discussion on list to poll a larger audience | 22:28 |
| SpamapS | hazmat: perhaps add-unit should not do the update/upgrade | 22:35 |
| SpamapS | hazmat: another thought is to defer updates/upgrades to charms always | 22:35 |
| hazmat | SpamapS, but that could be at a different version delta than the original unit | 22:35 |
| hazmat | SpamapS, ie. if their off releases, and the first unit did the update/upgrade, than the second unit is stuck at the base image version.. doesn't really make sense | 22:36 |
| SpamapS | hazmat: right, so really, perhaps update/upgrade should be pushed off to charms. | 22:37 |
| hazmat | SpamapS, i think its probably more of use to use the nightly unless its an add-unit in which case we use the previously used image, and we don't update/upgrade.. BUT.. there are lots of management tools and probably colo services that might also bbe doing package management | 22:37 |
| SpamapS | hazmat: *or* disconnected from deploy/add-unit | 22:37 |
| SpamapS | hazmat: I think we should just document how it works now, and think about how to improve the "update the whole service" story | 22:38 |
| hazmat | SpamapS, that's fair, although i think it could use some discussion on the wider list as well to help advance the story | 22:39 |
| SpamapS | hazmat: I asked earlier.. but saw no response.. did I see rebooting landing in lp:juju ? | 22:44 |
| jimbaker | SpamapS, hazmat replied at 10:04 MST: SpamapS, yup, and upstartification, and all kinds of yummy goodness | 22:51 |
| jimbaker | nice to see those features land, as i mentioned in #juju-dev at the time :) | 22:52 |
| SpamapS | ahh ok cool! | 22:52 |
| SpamapS | Thats like.. huge | 22:52 |
| jimbaker | SpamapS, indeed! | 22:52 |
| SpamapS | https://bugs.launchpad.net/juju/+bug/863526 | 22:53 |
| _mup_ | Bug #863526: Juju agents do not handle reboots <production> <juju:Triaged> < https://launchpad.net/bugs/863526 > | 22:53 |
| SpamapS | So, thats kind of a meta-bug, but does a reboot work now? | 22:53 |
| * SpamapS tries it | 22:53 | |
| hazmat | SpamapS, you can kill any juju agent and it should do the right thing | 22:53 |
| hazmat | mad props to fwereade_ who did the heavy lifting | 22:53 |
| hazmat | which reminds me, i should circle back to the branches i had waiting on that | 22:54 |
| * SpamapS bootstraps and then reboots to see what happens | 22:54 | |
| hazmat | SpamapS, that might not work ;-) | 22:55 |
| hazmat | SpamapS, its more that agents can be restarted, and killed at arbitrary points and do the right thing | 22:55 |
| * SpamapS would really like to close some 'production' bugs | 22:55 | |
| SpamapS | hazmat: any reason that might not work? | 22:55 |
| hazmat | SpamapS, the session expiration stuff hasn't landed, as it was waiting on the restart work as a mechanism | 22:56 |
| SpamapS | so if the reboot doesn't happen in like, 3 seconds, something bad happens? | 22:56 |
| hazmat | SpamapS, if the zk server advances the clock and expires the session, the agents might not handle that.. i dunno there is some support there for killing old sessions when the process comes backs .. so it might work | 22:57 |
| hazmat | but if the agent is alive when the session expiration happens then they don't do anything for it | 22:57 |
| SpamapS | hazmat: well in this case, I'm rebooting zookeeper too... | 22:57 |
| hazmat | SpamapS, right, but zk will effectively advance the clock on all extant sessions when it comes back up... like i said it might work, i just can't guarantee it yet | 22:58 |
| SpamapS | hazmat: I don't understand what advance the clock means, and I don't understand why an expired session does anything. :-P | 22:59 |
| SpamapS | doesn't the agent just start a new session? | 22:59 |
| hazmat | SpamapS, it will when it starts up, but not if the session is expired while its up | 23:01 |
| SpamapS | Actually the agents are failing on start | 23:01 |
| SpamapS | http://paste.ubuntu.com/851980/ | 23:01 |
| SpamapS | juju.errors.JujuError: No session file specified | 23:01 |
| hazmat | SpamapS, hmm. they should all have session files specified if the env was started with the latest trunk | 23:02 |
| SpamapS | JUJU_ZOOKEEPER=localhost:2181 python -m juju.agents.machine -n --logfile=/var/log/juju/machine-agent.log | 23:03 |
| SpamapS | --pidfile=/var/run/juju/machine-agent.pid', 'JUJU_ZOOKEEPER=localhost:2181 python | 23:03 |
| SpamapS | bootstrap seems to not use the upstart yet? | 23:04 |
| hazmat | SpamapS, quite possible, its a large set of changes that just landed on trunk.. i'll do some additional qa testing now | 23:06 |
| SpamapS | ./juju/providers/common/tests/data/cloud_init_branch_trunk:runcmd: [sudo apt-get install -y python-txzookeeper, sudo mkdir -p /usr/lib/juju, | 23:06 |
| SpamapS | DOH | 23:06 |
| SpamapS | missed some apt-get's | 23:06 |
| hazmat | SpamapS, you want the cake or egg treatment ;-) | 23:07 |
| hazmat | jk | 23:07 |
| SpamapS | hazmat: the egg goes on my face.. and the cake.. well | 23:08 |
| SpamapS | THE CAKE IS A LIE | 23:08 |
| hazmat | but so much tastier | 23:08 |
| SpamapS | oh | 23:11 |
| SpamapS | hahaha | 23:11 |
| SpamapS | hazmat: stand down | 23:11 |
| SpamapS | my local version of juju was the distro version | 23:11 |
| SpamapS | *DUH* | 23:11 |
| hazmat | cool, i do remember testing that earlier (killing machine agent), it looks ok locally.. still not 100% sure about the restart capability | 23:15 |
| SpamapS | hazmat: but there are branches that will solve that in flight? | 23:23 |
| hazmat | SpamapS, yes | 23:24 |
| hazmat | heading out to check out a user group meeting, g'night | 23:24 |
| SpamapS | The system is going down for reboot NOW! | 23:25 |
| * SpamapS crosses fingers | 23:25 | |
| SpamapS | hazmat: have fun | 23:25 |
| SpamapS | initially it looks to have worked quite nicely | 23:26 |
| SpamapS | Heh.. it helps that we reach runlevel 2 at 7 seconds. | 23:26 |
| SpamapS | 2012-02-21 23:25:19,466:2636(0xb73896c0):ZOO_INFO@zookeeper_close@2304: Closing zookeeper sessionId=0x135a235ffb30001 to [127.0.0.1:2181] | 23:27 |
| SpamapS | 2012-02-21 23:25:47,188:576(0xb74ed6c0):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.3 | 23:28 |
| SpamapS | looks like it worked fine for machine/provisioning agent | 23:28 |
| SpamapS | sweet... and they can be restarted with 'service juju-machine-agent restart' | 23:30 |
| SpamapS | I just noticed | 23:52 |
| SpamapS | am I not being asked to verify ssh keys now?! | 23:52 |
| SpamapS | is that landed? | 23:52 |
| SpamapS | because if it is.. woot! | 23:52 |
| SpamapS | if I"m just dumb and fixed my .ssh/config to not be asked.. then ignore moe | 23:52 |
| SpamapS | ignore me too.. but really ignore moe.. that jaerk | 23:52 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!