devinheitmueller | Anybody have any recommended best practices for Ubuntu partitioning for maximum reliability? This is for an “appliance” where very little ever changes on disk, and I want to minimize the risk of the system not booting due to FS corruption when power cycled. | 00:20 |
---|---|---|
devinheitmueller | Back in the days where I rolled my own Linux distros, I would use RO partitions for everything except var, but that doesn’t work very well in off-the-shelf Linux distros. | 00:20 |
cmaloney | Hey devinheitmueller | 00:20 |
devinheitmueller | Hey craig! | 00:21 |
cmaloney | Not sure what the best practices are for RO partitions | 00:21 |
cmaloney | iirc root is pretty safe as long as /tmp is not marked RO | 00:21 |
cmaloney | or using tmpfs | 00:21 |
devinheitmueller | I would assume /var could also be problematic because of logs, etc. | 00:21 |
cmaloney | Absolutely | 00:22 |
devinheitmueller | That said, I would love to just have /tmp, /var, /, and then mark root read-only, but I worry that Ubuntu won’t really be equipped to run in such an environment. | 00:22 |
cmaloney | Well, /home should also be a separate partition | 00:23 |
devinheitmueller | Yeah, but in reality /home will likely never change. | 00:23 |
devinheitmueller | I’m more worried about Ubuntu’s system processes mucking around with the disk and expecting everything to be mounted RW. | 00:23 |
cmaloney | There's still .cache and such | 00:23 |
devinheitmueller | This thing really screams for something based on LFS or OpenWRT, but I don’t really have any say in that at this point. They wanted to leverage an off-the-shelf Ubuntu system as the base OS for their appliance. | 00:24 |
cmaloney | I thought there was a specification for what partitions could be mounted read only at some point | 00:25 |
cmaloney | but it's escaping my noggin' | 00:25 |
devinheitmueller | Perhaps there is. I really don’t claim to have much expertise in Ubuntu. | 00:25 |
devinheitmueller | I’m happy to say that it generally “just works”, so I spend much less time nowadays digging under the hood. | 00:26 |
cmaloney | Well, it was some satandard that iirc Ubuntu didn't follow for a while. | 00:26 |
cmaloney | and it wasn't until recently (2010 or so) that it was | 00:26 |
devinheitmueller | Well, there’s LSB, but yeah many distros don’t follow it very tightly. | 00:26 |
cmaloney | of course Google is showing me folks that have hardware or NFS issues. :) | 00:26 |
devinheitmueller | Yeah, I started with something like “Ubuntu partitioning avoid corruption” and it didn’t return quite what I was interested in. :-) | 00:29 |
cmaloney | Yeah, I'm not sure at this point. | 00:29 |
devinheitmueller | Fair enough. This is where I consider my resident source of Ubuntu expertise, so figured no harm in asking. | 00:30 |
cmaloney | smoser might know more if he's around | 00:30 |
cmaloney | or you might want to check with ubuntu-arm as that is embedded Ubuntu | 00:30 |
devinheitmueller | Not a bad idea. | 00:31 |
cmaloney | Might at least get you in the general direction | 00:31 |
devinheitmueller | I’ll take a closer look and see what they recommend for Ubuntu Server. It’s likely that is tailored more for reliability than convenience. | 00:31 |
cmaloney | yeah, not sure Ubuntu server is going to get you any better answers. | 00:32 |
cmaloney | But worth a shot | 00:32 |
devinheitmueller | yup. | 00:32 |
devinheitmueller | I’m not suggesting we run Ubuntu server, just that that particular crowd might have a mindset more in line with my goals. | 00:32 |
cmaloney | Oh definitely | 00:33 |
devinheitmueller | (i.e. minimize risk of system not booting, configure to automatically fix issues at boot, FS layout which minimizes boot time, etc) | 00:33 |
devinheitmueller | On the flip side, my Tivo runs Embedded Linux and takes 15-20 minutes to boot, so perhaps I should just adjust my expectations. :-) | 00:34 |
cmaloney | heh | 00:34 |
cmaloney | Well, that's partly why ext4fs was devised. :) | 00:34 |
devinheitmueller | Of course, file systems continue to improve. | 00:34 |
devinheitmueller | Ted Tso gave a really good talk last year at NYLUG on the state of Linux filesystems. | 00:35 |
jrwren | devinheitmueller: do you have enough disk space to do it all on an ubuntu cloudimg? | 00:53 |
devinheitmueller | In this case it’s a standalone piece of equipment, not internet connected. | 00:53 |
jrwren | that is ok. | 00:53 |
jrwren | you can ship the cloudimg on your standalone piece. | 00:53 |
devinheitmueller | Hmmm.... | 00:54 |
devinheitmueller | Maybe. Admittedly I don’t really know much about Ubuntu cloudimgs. Will have to add that to the list of things to look up. | 00:54 |
jrwren | so you can have root and everything be RO, then untar cloudimg on each boot | 00:54 |
jrwren | its a 200MB system image, nice and tiny. | 00:54 |
jrwren | what is the application? | 00:54 |
devinheitmueller | It’s a piece of fitness equipment. | 00:55 |
devinheitmueller | Hence the app lets you see stats, watch tv, control the treadmill, etc. | 00:56 |
devinheitmueller | cmaloney: look, it’s that “smoser” guy you were talking about: http://ubuntu-smoser.blogspot.com/2013/02/using-ubuntu-cloud-images-without-cloud.html | 00:57 |
bookiebot | http://goo.gl/nHtgrm - smoser's thoughts: Using Ubuntu cloud-images without a cloud | 00:57 |
jrwren | sounds like a fun challenge. | 00:57 |
jrwren | i'd probably do what you are doing and just gut ubuntu | 00:57 |
devinheitmueller | Yeah, it’s your typical embedded Linux appliance where almost nothing ever changes but somebody can yank the power cord at any time. | 00:57 |
devinheitmueller | Always hard finding the right balance - rolling your own micro-distro (or gutting a distro like Ubuntu) gives you ultimate control, but then you have to maintain it and can’t rely on the upstream for patches. | 00:59 |
jrwren | i like gutting | 01:00 |
jrwren | because i get all the advantage of the vast packages, and I can reasonably maintain it in the form of a patch | 01:00 |
devinheitmueller | The customer is not as technically astute as you are. :-) | 01:00 |
jrwren | they shouldn't need to be. | 01:00 |
devinheitmueller | In reality it’s likley they won’t do anything about this. The only reason I started thinking about it is the dev system I was using had it’s ext4 rootfs lose it’s mind, leaving hundreds of critical items in /lost+found. | 01:01 |
devinheitmueller | Oh, and it didn’t boot. Always nice. | 01:01 |
jrwren | XFS ftw. | 01:02 |
devinheitmueller | XFS feels surprisingly like overkill - it’s an embedded system with a single disk and 300MB of software. It’s not a multi-terabyte storage array. | 01:03 |
devinheitmueller | :-) | 01:03 |
devinheitmueller | In other words, I shouldn’t have to use XFS to feel confident that my system is going to successfully boot. | 01:04 |
jrwren | um... xfs isnt that different from ext | 01:06 |
jrwren | it has nicer fsck properties | 01:06 |
krondor | you could add ext4 checksums to help combat corruption. It's a little newer though (in terms of FS timescales) so ymmv I haven't really played with it. | 01:15 |
krondor | https://ext4.wiki.kernel.org/index.php/Ext4_Metadata_Checksums | 01:16 |
bookiebot | http://goo.gl/tmyAS3 - Ext4 Metadata Checksums - Ext4 | 01:16 |
cmaloney | morning | 12:17 |
brousch__ | indeed | 12:18 |
smoser | well, above, devinheitmueller is gone now, but i'd probably start with ubuntu core. its what ubuntu core is for. | 13:02 |
smoser | and go from there. | 13:02 |
smoser | the read-only stuff, can be made to work. i have worked on 'overlayroot' which mounts root read-only and writes delta via overlayfs. | 13:03 |
smoser | its functional. i think that probalby sint exactly what he was looking for, but using it will tell you nicely where things fall over (by seeing new files in the overlay location). | 13:04 |
smoser | kirkland wrote a blog post on overlayroot at http://blog.dustinkirkland.com/2012/08/introducing-overlayroot-overlayfs.html | 13:04 |
bookiebot | http://goo.gl/AFWu3I - From the Canyon Edge: Introducing overlayroot -- overlayfs + dmcrypt! | 13:04 |
smoser | its built into our images. | 13:04 |
rick_h_ | morning and TGIF | 13:13 |
cmaloney | Yeah, no kidding | 13:20 |
jrwren | ut oh, someone posted this GIANT jpg link to reddit -https://upload.wikimedia.org/wikipedia/commons/7/78/30_Doradus%2C_Tarantula_Nebula.jpg | 13:49 |
bookiebot | http://goo.gl/BVS6wO - image/jpeg | 13:50 |
jrwren | greg-g's data bill is going to quintuple | 13:50 |
brousch__ | It loads so slowly! | 13:53 |
jrwren | cuz its HUGE | 14:06 |
jrwren | and maybe greg-g has it throttled. | 14:06 |
greg-g | jrwren: you think reddit, the 61st popular site by Alexa, will have a traffic impact on us, the 4th popular site? | 14:41 |
brousch__ | Is that a dare? | 14:42 |
* brousch__ send it to 4chan | 14:42 | |
greg-g | when was that image posted? | 14:42 |
greg-g | http://i.imgur.com/8LXvhEI.png | 14:42 |
bookiebot | http://goo.gl/R1RXlG - image/png | 14:42 |
greg-g | that's the past week | 14:42 |
greg-g | actually, that's just css/js, that blip was us deploying a new version | 14:43 |
greg-g | here comes upload.wikimedia.org traffic: | 14:44 |
krondor | reddit punches above its traffic weight due to the burstiness (yes I say that's a word) of top posts | 14:44 |
jrwren | greg-g: my thought was that the file is so large that it would take fewer hits to impact you. | 14:44 |
krondor | though I doubt it would be a problem :) | 14:44 |
greg-g | last week of upload traffic: http://i.imgur.com/3UfqB7E.png | 14:44 |
bookiebot | http://goo.gl/YlPCRU - image/png | 14:44 |
jrwren | at the same time, it could easily be served via CDN | 14:44 |
jrwren | do you use a CDN? | 14:44 |
greg-g | jrwren: cache | 14:44 |
greg-g | we are our own cdn | 14:44 |
jrwren | what cache? | 14:45 |
greg-g | varnish boxen | 14:45 |
jrwren | centralized or distributed? | 14:45 |
greg-g | we have 3 caching locations | 14:45 |
greg-g | SF, Virginia, and Amsterdam | 14:45 |
jrwren | i see. | 14:45 |
greg-g | soon Dallas (that'll be a full fledged DC, with caching obviously) | 14:46 |
jrwren | and you don't care about data quantity hitting them? | 14:46 |
greg-g | no, they're caches :) | 14:46 |
jrwren | or you'd not notice? | 14:46 |
greg-g | not that image from reddit | 14:46 |
cmaloney | I love that my CPU fan starts going into overdrive when I load that image. | 14:46 |
greg-g | see the last imgur link from me | 14:46 |
jrwren | still, they have a data rate limit of the aggregate of their interfaces, right? | 14:46 |
jrwren | yes, I see it, its always the same. | 14:46 |
jrwren | nice consistent graph. | 14:46 |
jrwren | so, I get that this reddit link doesn't impact you AT ALL. | 14:47 |
greg-g | here's the last day for upload caches (images): http://i.imgur.com/hkIFpYK.png | 14:47 |
bookiebot | http://goo.gl/nOXKPc - image/png | 14:47 |
jrwren | still, I wonder what it would take. | 14:47 |
greg-g | a lot | 14:47 |
greg-g | :) | 14:47 |
jrwren | what is your daily max aggregate cache traffic rate? | 14:47 |
greg-g | I don't think I should publicly say our utilization % | 14:47 |
jrwren | ok | 14:47 |
jrwren | that is probably a good idea. | 14:48 |
greg-g | which is why I didn't include the numbers on the left of the graph ;) | 14:48 |
jrwren | understood. | 14:48 |
greg-g | we are a #4 website afterall, people like to make names for themselves ;) | 14:48 |
bookiebot | closed - importer doesn't handle spaces - https://github.com/bookieio/Bookie/issues/4 | 14:48 |
greg-g | heh | 14:48 |
cmaloney | http://theroaring420s.bandcamp.com/album/the-roaring-420s-what-is-psych?t=5 | 14:48 |
bookiebot | http://goo.gl/VkGCYJ - The Roaring 420s - What Is Psych? | The Roaring 420s | 14:48 |
jrwren | greg-g: I know! and you don't even use DDoS protection! | 14:48 |
greg-g | bookiebot: not all #666s are bookie bugs | 14:48 |
cmaloney | I love some of the band names of stoner bands. | 14:49 |
greg-g | jrwren: nope, we're 100% in house, except for one or two third-party monitoring (in caes our monitors go down, heh) | 14:49 |
cmaloney | I think that's more secure that way | 14:50 |
jrwren | you can do 100% in house ddos proection :p | 14:50 |
jrwren | when I said you don't use it, I didn't mean cloud ddos b.s. like cloudflair, I meant an Arbor Networks mitigation appliance :p | 14:50 |
cmaloney | Since you're not relying on mirrors that might try to change thing son Wikipedia | 14:50 |
greg-g | jrwren: oh, I'm not 100% sure if we don't, but I'm 90% sure we don't have one of those | 14:50 |
cmaloney | jrwren: Come Monday will that be a Juju deployment? :) | 14:50 |
jrwren | cmaloney: nah. juju should probably have that kind of protection too ;] | 14:51 |
cmaloney | hah | 14:51 |
greg-g | jrwren: whenever we have had spikes in traffic for weird reasons (either bad actors or good intention'd people's bots gone crazy) I've never heard of any automatic anything | 14:52 |
greg-g | really, raw traffic won't hurt us, it's intelligent queries to specific things | 14:52 |
rick_h_ | anyone want to buy an x230 that you can't use the microphone jack on? | 14:53 |
cmaloney | Has wikipedia ever been ddosed? | 14:53 |
rick_h_ | (or the built in mic) | 14:53 |
jrwren | rick_h_: oh no! is that what you tweeted about? | 14:53 |
cmaloney | rick_h_: so it's official then? XPS in da house? | 14:53 |
greg-g | cmaloney: successfully? not that I know of | 14:53 |
krondor | cmaloney: oh I'm sure, probably like weekly just not hard enough to count | 14:53 |
rick_h_ | jrwren: no, I tweeted about juju core $@#!$@$ me over and cauing me pain and agony | 14:53 |
greg-g | cmaloney: we've seen people try | 14:53 |
greg-g | :) | 14:53 |
cmaloney | greg-g: Oh no doubt | 14:53 |
jrwren | ah, poor jujucore. they should have hired me :p | 14:53 |
rick_h_ | jrwren: but my new laptop should be here tomorrow which is required because my x230 has sound issues and I can't use it for meeting | 14:53 |
cmaloney | rick_h_: I'd hold out on selling | 14:54 |
cmaloney | Make sure you're 100% good with parting | 14:54 |
rick_h_ | cmaloney: it's going to get tough for the wife to see me around with a desktop and 3 laptops | 14:54 |
rick_h_ | cmaloney: I've not booted it in a week | 14:54 |
cmaloney | Just put it on the closet for a while | 14:54 |
cmaloney | let it pass | 14:54 |
jrwren | old laptops make good cloud nodes. tell her its part of your home cloud. | 14:54 |
greg-g | so, when I first started, some random dude threatened ddos'ing us on IRC, I mentioned it to our networking lead, her first reaction was laughing and just walking away. From then on I haven't worried ;) | 14:54 |
cmaloney | and then when you don't want to go into the closet to retrieve the x230 then it's time to make peace. | 14:54 |
rick_h_ | jrwren: yea, did think about that for a maas controller letting me have 3 nucs to operate | 14:55 |
rick_h_ | might still do that if no one's interested | 14:55 |
jrwren | greg-g: understood, still, unless you can handle 300Gbps, I wouldn't get too cocky. | 14:55 |
greg-g | jrwren: no, we can't do 300gbps, that I'll say :) | 14:55 |
jrwren | greg-g: then if the right people wanted to take you down, they could. | 14:55 |
greg-g | jrwren: I mean, you can look up our ASN stuff, too | 14:56 |
greg-g | jrwren: totally | 14:56 |
jrwren | oh yeah! | 14:56 |
krondor | jrwren: well, they probably have agreements with their providers to help with that mitigation upstream too | 14:56 |
krondor | still that interface will be a delay and no doubt cause downtime with that type of hit | 14:56 |
greg-g | jrwren: to help: http://wikimediafoundation.org/wiki/Peering | 14:57 |
bookiebot | http://goo.gl/TvEAZr - Peering - Wikimedia Foundation | 14:57 |
cmaloney | This reminds me of the talk that we had at work in 2003 right before the power grid toook a hit that knocked out MI and Ohio. :) | 14:57 |
jrwren | oh man, that is so cool to visit a site adn it says, you are visiting from... "..." and its an ipv6 addy | 14:57 |
cmaloney | http://en.wikipedia.org/wiki/Northeast_America_blackout_of_2003 | 14:58 |
bookiebot | http://goo.gl/1UUPVo - Northeast America blackout of 2003 - Wikipedia, the free encyclopedia | 14:58 |
greg-g | cmaloney: :) | 14:58 |
krondor | cmaloney: cascading failure scenarios are a thing :) ask Blackberry, I think they blamed that for at least 8 of their outages | 14:58 |
jrwren | cmaloney: i remember it well, my dad had just bought 2 quarts of ice cream, and I got to help eat it. | 14:58 |
greg-g | jrwren: ours? | 14:58 |
cmaloney | I remember driving home to Belleville and being unable to call JoDee on the cellphone | 14:59 |
krondor | that reminds me I need to yell at comcast again about ipv6, not that it will help | 14:59 |
greg-g | https://www.peeringdb.com/private/participant_view.php?id=1365 gives you numbers (guest/guest login) | 14:59 |
bookiebot | http://goo.gl/jzcSYc - PeeringDB Login | 14:59 |
cmaloney | Though the phone system at work was working | 14:59 |
krondor | i was reasonably impressed how long it took the cell towers to start dying, and in more populated areas they just kept filling the generators so they didn't | 14:59 |
jrwren | greg-g: nah, http://bgp.he.net/AS33668 | 15:00 |
bookiebot | http://goo.gl/twBRwD - 403 Forbidden | 15:00 |
cmaloney | it's also the time when joDee looked at the phone (which didn't ring because the power was cut), picked it up, and surprised both her and her friend that was calling. | 15:00 |
cmaloney | krondor: yeah, but they were slammed. | 15:00 |
jrwren | greg-g: i was looking up: http://bgp.he.net/AS14907 to see you | 15:00 |
bookiebot | http://goo.gl/t5UX4H - 403 Forbidden | 15:00 |
jrwren | huh, i wonder why it says 403. I wonder if they filter based on UA | 15:01 |
brousch__ | cmaloney: Don't drag all of Michigan into your little blackout. We were fine over here. | 15:01 |
greg-g | http://bgp.he.net/AS14907 | 15:01 |
bookiebot | http://goo.gl/t5UX4H - 403 Forbidden | 15:01 |
greg-g | yeah | 15:01 |
krondor | brousch__: i was fine once I drove 30 mins to grand blanc too... mmm delicious AC and gas for my car | 15:02 |
jrwren | lol, yup, curl -A 'Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0' and I can hit that page. | 15:02 |
cmaloney | Sad thing is we were waiting for a blackout so we could do some stargazing | 15:02 |
jrwren | whose bot is that? | 15:02 |
cmaloney | that's when we realized the fucking Wayne County fair was happening a few blocks away | 15:02 |
cmaloney | jrwren: mrgoodcat's bot | 15:02 |
cmaloney | so we had floodlights and hte sound of tractor pulls all night | 15:03 |
cmaloney | fuckers. | 15:03 |
jrwren | alright, I'm AFK for at least the day, time for real vacation. | 15:03 |
cmaloney | jrwren: Have fun! | 15:03 |
greg-g | getting real sick of this barking dog next door | 15:34 |
greg-g | too bad I'm on calls all morning and can't go to the coffee shop | 15:34 |
greg-g | or back to that bar... man they had a good rueben | 15:34 |
cmaloney | Mmm, reuben | 15:44 |
brousch__ | Reubens would be awesome if they weren't covered in decomposing cabbage | 15:46 |
greg-g | brousch__: I'm german. I love that stuff. | 15:49 |
greg-g | we make our own, actually | 15:49 |
cmaloney | Nothing wrong with decomposing cabbage | 16:22 |
StoneTable | rick_h_: Speedy response to that bug. Thanks! :D | 16:48 |
rick_h_ | StoneTable: oh hey, how goes? | 16:48 |
rick_h_ | StoneTable: heh, sorry I floated by the FF note at first | 16:48 |
StoneTable | Pretty good, thanks. You? | 16:48 |
StoneTable | No worries! | 16:49 |
rick_h_ | party party | 16:49 |
rick_h_ | StoneTable: what's got you poking at the GUI? | 16:50 |
rick_h_ | doing anything fun? | 16:50 |
StoneTable | Interview prep. Sent my resume in last week, and setting up a time to talk to someone over there. | 16:51 |
rick_h_ | oh cool, who you talking to? | 16:51 |
StoneTable | Antonio Rosales | 16:51 |
rick_h_ | oh awesome, great guy | 16:51 |
StoneTable | Good to hear. I'm looking forward to that chat. | 16:52 |
rick_h_ | cool, let me know if you've got any questions | 16:53 |
StoneTable | And the juju/Charms stuff is pretty slick, so I'm really excited to play with that more. I wish I'd had that 8 or so years ago. | 16:53 |
StoneTable | I sure will, thanks! | 16:53 |
greg-g | cmaloney: wow, a 16EUR/$800 USD cd? | 19:36 |
greg-g | cmaloney: re the stoner rock band you linked before | 19:36 |
cmaloney | greg-g: Eh? | 19:51 |
greg-g | that 420 whatever band's cd was 16 Euros | 19:51 |
cmaloney | If I hit "buy it now" I get 21.24 | 19:52 |
cmaloney | 21.66 rather | 19:52 |
cmaloney | Granted I don't think four tracks is 21.66. :) | 19:52 |
greg-g | sorry, it was a bad "OMG, USD is so low compared to EUR" joke | 19:52 |
cmaloney | Yes, yes it was. ;) | 19:52 |
cmaloney | Though I've seen folks post albums that were ridiculous in price. | 19:52 |
cmaloney | like $8 a track or some shit. | 19:53 |
cmaloney | Actually I think the prime offender that I was thinking of lowered his prices on his music after I told him he was... ahem... high. :) | 19:57 |
greg-g | :) | 19:59 |
greg-g | was it also stoner rock? | 19:59 |
greg-g | cmaloney: btw, I'm liking http://juddmadden.bandcamp.com/album/everything-in-waves, it's CC BY-NC, you should use it for OMC | 20:05 |
bookiebot | http://goo.gl/Gz89Dh - Judd Madden | 20:05 |
cmaloney | greg-g: It's on my list. :) | 20:37 |
greg-g | it'd be a long episode. He seems to like 8+ minute songs | 20:38 |
cmaloney | Yeah, he doesn't write short songs. | 20:38 |
cmaloney | IIRC it's instrumental | 20:39 |
cmaloney | so perfect for instrumetalcast. | 20:39 |
greg-g | yeppers | 20:41 |
=== greg-g_ is now known as greg-g | ||
cmaloney | Oh, btw: the person who said that the Aereo case might have an impact on cloud broadcasting for personal use: | 21:01 |
cmaloney | http://arstechnica.com/tech-policy/2014/06/in-aereos-wake-fox-targets-dishs-tv-streaming-service/ | 21:01 |
bookiebot | http://goo.gl/BbOJD7 - In Aereo’s wake, Fox targets Dish’s TV streaming service | Ars Technica | 21:02 |
cmaloney | We're going to need some more fires, because there aren't enough to handle the "dying in" load. | 21:02 |
greg-g | bad article, but interesting: http://www.zdnet.com/blog/facebook/facebook-passwords-are-not-case-sensitive-update/3612 | 22:40 |
bookiebot | http://goo.gl/mk6YT - Facebook passwords are not case sensitive (update) | ZDNet | 22:40 |
greg-g | basically, FB accepts 3 forms of your password: | 22:41 |
greg-g | 1) original password as you typed it (duh) | 22:41 |
greg-g | 2) original password with a capitalized first letter, in case your mobile device does that annoying thing | 22:41 |
greg-g | 3) your password with the case flipped on all letters, in case you have caps lock on | 22:41 |
greg-g | horrible horrible horrible reporting from ZDnet, of course, cuz I still have no idea why they're still around | 22:42 |
gamerchick02 | wow yeah i don't know how they've stayed in business either | 22:51 |
gamerchick02 | and hello from windows 8 *ducks and covers* | 22:52 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!