[00:20] Anybody have any recommended best practices for Ubuntu partitioning for maximum reliability? This is for an “appliance” where very little ever changes on disk, and I want to minimize the risk of the system not booting due to FS corruption when power cycled. [00:20] Back in the days where I rolled my own Linux distros, I would use RO partitions for everything except var, but that doesn’t work very well in off-the-shelf Linux distros. [00:20] Hey devinheitmueller [00:21] Hey craig! [00:21] Not sure what the best practices are for RO partitions [00:21] iirc root is pretty safe as long as /tmp is not marked RO [00:21] or using tmpfs [00:21] I would assume /var could also be problematic because of logs, etc. [00:22] Absolutely [00:22] That said, I would love to just have /tmp, /var, /, and then mark root read-only, but I worry that Ubuntu won’t really be equipped to run in such an environment. [00:23] Well, /home should also be a separate partition [00:23] Yeah, but in reality /home will likely never change. [00:23] I’m more worried about Ubuntu’s system processes mucking around with the disk and expecting everything to be mounted RW. [00:23] There's still .cache and such [00:24] This thing really screams for something based on LFS or OpenWRT, but I don’t really have any say in that at this point. They wanted to leverage an off-the-shelf Ubuntu system as the base OS for their appliance. [00:25] I thought there was a specification for what partitions could be mounted read only at some point [00:25] but it's escaping my noggin' [00:25] Perhaps there is. I really don’t claim to have much expertise in Ubuntu. [00:26] I’m happy to say that it generally “just works”, so I spend much less time nowadays digging under the hood. [00:26] Well, it was some satandard that iirc Ubuntu didn't follow for a while. [00:26] and it wasn't until recently (2010 or so) that it was [00:26] Well, there’s LSB, but yeah many distros don’t follow it very tightly. [00:26] of course Google is showing me folks that have hardware or NFS issues. :) [00:29] Yeah, I started with something like “Ubuntu partitioning avoid corruption” and it didn’t return quite what I was interested in. :-) [00:29] Yeah, I'm not sure at this point. [00:30] Fair enough. This is where I consider my resident source of Ubuntu expertise, so figured no harm in asking. [00:30] smoser might know more if he's around [00:30] or you might want to check with ubuntu-arm as that is embedded Ubuntu [00:31] Not a bad idea. [00:31] Might at least get you in the general direction [00:31] I’ll take a closer look and see what they recommend for Ubuntu Server. It’s likely that is tailored more for reliability than convenience. [00:32] yeah, not sure Ubuntu server is going to get you any better answers. [00:32] But worth a shot [00:32] yup. [00:32] I’m not suggesting we run Ubuntu server, just that that particular crowd might have a mindset more in line with my goals. [00:33] Oh definitely [00:33] (i.e. minimize risk of system not booting, configure to automatically fix issues at boot, FS layout which minimizes boot time, etc) [00:34] On the flip side, my Tivo runs Embedded Linux and takes 15-20 minutes to boot, so perhaps I should just adjust my expectations. :-) [00:34] heh [00:34] Well, that's partly why ext4fs was devised. :) [00:34] Of course, file systems continue to improve. [00:35] Ted Tso gave a really good talk last year at NYLUG on the state of Linux filesystems. [00:53] devinheitmueller: do you have enough disk space to do it all on an ubuntu cloudimg? [00:53] In this case it’s a standalone piece of equipment, not internet connected. [00:53] that is ok. [00:53] you can ship the cloudimg on your standalone piece. [00:54] Hmmm.... [00:54] Maybe. Admittedly I don’t really know much about Ubuntu cloudimgs. Will have to add that to the list of things to look up. [00:54] so you can have root and everything be RO, then untar cloudimg on each boot [00:54] its a 200MB system image, nice and tiny. [00:54] what is the application? [00:55] It’s a piece of fitness equipment. [00:56] Hence the app lets you see stats, watch tv, control the treadmill, etc. [00:57] cmaloney: look, it’s that “smoser” guy you were talking about: http://ubuntu-smoser.blogspot.com/2013/02/using-ubuntu-cloud-images-without-cloud.html [00:57] http://goo.gl/nHtgrm - smoser's thoughts: Using Ubuntu cloud-images without a cloud [00:57] sounds like a fun challenge. [00:57] i'd probably do what you are doing and just gut ubuntu [00:57] Yeah, it’s your typical embedded Linux appliance where almost nothing ever changes but somebody can yank the power cord at any time. [00:59] Always hard finding the right balance - rolling your own micro-distro (or gutting a distro like Ubuntu) gives you ultimate control, but then you have to maintain it and can’t rely on the upstream for patches. [01:00] i like gutting [01:00] because i get all the advantage of the vast packages, and I can reasonably maintain it in the form of a patch [01:00] The customer is not as technically astute as you are. :-) [01:00] they shouldn't need to be. [01:01] In reality it’s likley they won’t do anything about this. The only reason I started thinking about it is the dev system I was using had it’s ext4 rootfs lose it’s mind, leaving hundreds of critical items in /lost+found. [01:01] Oh, and it didn’t boot. Always nice. [01:02] XFS ftw. [01:03] XFS feels surprisingly like overkill - it’s an embedded system with a single disk and 300MB of software. It’s not a multi-terabyte storage array. [01:03] :-) [01:04] In other words, I shouldn’t have to use XFS to feel confident that my system is going to successfully boot. [01:06] um... xfs isnt that different from ext [01:06] it has nicer fsck properties [01:15] you could add ext4 checksums to help combat corruption. It's a little newer though (in terms of FS timescales) so ymmv I haven't really played with it. [01:16] https://ext4.wiki.kernel.org/index.php/Ext4_Metadata_Checksums [01:16] http://goo.gl/tmyAS3 - Ext4 Metadata Checksums - Ext4 [12:17] morning [12:18] indeed [13:02] well, above, devinheitmueller is gone now, but i'd probably start with ubuntu core. its what ubuntu core is for. [13:02] and go from there. [13:03] the read-only stuff, can be made to work. i have worked on 'overlayroot' which mounts root read-only and writes delta via overlayfs. [13:04] its functional. i think that probalby sint exactly what he was looking for, but using it will tell you nicely where things fall over (by seeing new files in the overlay location). [13:04] kirkland wrote a blog post on overlayroot at http://blog.dustinkirkland.com/2012/08/introducing-overlayroot-overlayfs.html [13:04] http://goo.gl/AFWu3I - From the Canyon Edge: Introducing overlayroot -- overlayfs + dmcrypt! [13:04] its built into our images. [13:13] morning and TGIF [13:20] Yeah, no kidding [13:49] ut oh, someone posted this GIANT jpg link to reddit -https://upload.wikimedia.org/wikipedia/commons/7/78/30_Doradus%2C_Tarantula_Nebula.jpg [13:50] http://goo.gl/BVS6wO - image/jpeg [13:50] greg-g's data bill is going to quintuple [13:53] It loads so slowly! [14:06] cuz its HUGE [14:06] and maybe greg-g has it throttled. [14:41] jrwren: you think reddit, the 61st popular site by Alexa, will have a traffic impact on us, the 4th popular site? [14:42] Is that a dare? [14:42] * brousch__ send it to 4chan [14:42] when was that image posted? [14:42] http://i.imgur.com/8LXvhEI.png [14:42] http://goo.gl/R1RXlG - image/png [14:42] that's the past week [14:43] actually, that's just css/js, that blip was us deploying a new version [14:44] here comes upload.wikimedia.org traffic: [14:44] reddit punches above its traffic weight due to the burstiness (yes I say that's a word) of top posts [14:44] greg-g: my thought was that the file is so large that it would take fewer hits to impact you. [14:44] though I doubt it would be a problem :) [14:44] last week of upload traffic: http://i.imgur.com/3UfqB7E.png [14:44] http://goo.gl/YlPCRU - image/png [14:44] at the same time, it could easily be served via CDN [14:44] do you use a CDN? [14:44] jrwren: cache [14:44] we are our own cdn [14:45] what cache? [14:45] varnish boxen [14:45] centralized or distributed? [14:45] we have 3 caching locations [14:45] SF, Virginia, and Amsterdam [14:45] i see. [14:46] soon Dallas (that'll be a full fledged DC, with caching obviously) [14:46] and you don't care about data quantity hitting them? [14:46] no, they're caches :) [14:46] or you'd not notice? [14:46] not that image from reddit [14:46] I love that my CPU fan starts going into overdrive when I load that image. [14:46] see the last imgur link from me [14:46] still, they have a data rate limit of the aggregate of their interfaces, right? [14:46] yes, I see it, its always the same. [14:46] nice consistent graph. [14:47] so, I get that this reddit link doesn't impact you AT ALL. [14:47] here's the last day for upload caches (images): http://i.imgur.com/hkIFpYK.png [14:47] http://goo.gl/nOXKPc - image/png [14:47] still, I wonder what it would take. [14:47] a lot [14:47] :) [14:47] what is your daily max aggregate cache traffic rate? [14:47] I don't think I should publicly say our utilization % [14:47] ok [14:48] that is probably a good idea. [14:48] which is why I didn't include the numbers on the left of the graph ;) [14:48] understood. [14:48] we are a #4 website afterall, people like to make names for themselves ;) [14:48] closed - importer doesn't handle spaces - https://github.com/bookieio/Bookie/issues/4 [14:48] heh [14:48] http://theroaring420s.bandcamp.com/album/the-roaring-420s-what-is-psych?t=5 [14:48] http://goo.gl/VkGCYJ - The Roaring 420s - What Is Psych? | The Roaring 420s [14:48] greg-g: I know! and you don't even use DDoS protection! [14:48] bookiebot: not all #666s are bookie bugs [14:49] I love some of the band names of stoner bands. [14:49] jrwren: nope, we're 100% in house, except for one or two third-party monitoring (in caes our monitors go down, heh) [14:50] I think that's more secure that way [14:50] you can do 100% in house ddos proection :p [14:50] when I said you don't use it, I didn't mean cloud ddos b.s. like cloudflair, I meant an Arbor Networks mitigation appliance :p [14:50] Since you're not relying on mirrors that might try to change thing son Wikipedia [14:50] jrwren: oh, I'm not 100% sure if we don't, but I'm 90% sure we don't have one of those [14:50] jrwren: Come Monday will that be a Juju deployment? :) [14:51] cmaloney: nah. juju should probably have that kind of protection too ;] [14:51] hah [14:52] jrwren: whenever we have had spikes in traffic for weird reasons (either bad actors or good intention'd people's bots gone crazy) I've never heard of any automatic anything [14:52] really, raw traffic won't hurt us, it's intelligent queries to specific things [14:53] anyone want to buy an x230 that you can't use the microphone jack on? [14:53] Has wikipedia ever been ddosed? [14:53] (or the built in mic) [14:53] rick_h_: oh no! is that what you tweeted about? [14:53] rick_h_: so it's official then? XPS in da house? [14:53] cmaloney: successfully? not that I know of [14:53] cmaloney: oh I'm sure, probably like weekly just not hard enough to count [14:53] jrwren: no, I tweeted about juju core $@#!$@$ me over and cauing me pain and agony [14:53] cmaloney: we've seen people try [14:53] :) [14:53] greg-g: Oh no doubt [14:53] ah, poor jujucore. they should have hired me :p [14:53] jrwren: but my new laptop should be here tomorrow which is required because my x230 has sound issues and I can't use it for meeting [14:54] rick_h_: I'd hold out on selling [14:54] Make sure you're 100% good with parting [14:54] cmaloney: it's going to get tough for the wife to see me around with a desktop and 3 laptops [14:54] cmaloney: I've not booted it in a week [14:54] Just put it on the closet for a while [14:54] let it pass [14:54] old laptops make good cloud nodes. tell her its part of your home cloud. [14:54] so, when I first started, some random dude threatened ddos'ing us on IRC, I mentioned it to our networking lead, her first reaction was laughing and just walking away. From then on I haven't worried ;) [14:54] and then when you don't want to go into the closet to retrieve the x230 then it's time to make peace. [14:55] jrwren: yea, did think about that for a maas controller letting me have 3 nucs to operate [14:55] might still do that if no one's interested [14:55] greg-g: understood, still, unless you can handle 300Gbps, I wouldn't get too cocky. [14:55] jrwren: no, we can't do 300gbps, that I'll say :) [14:55] greg-g: then if the right people wanted to take you down, they could. [14:56] jrwren: I mean, you can look up our ASN stuff, too [14:56] jrwren: totally [14:56] oh yeah! [14:56] jrwren: well, they probably have agreements with their providers to help with that mitigation upstream too [14:56] still that interface will be a delay and no doubt cause downtime with that type of hit [14:57] jrwren: to help: http://wikimediafoundation.org/wiki/Peering [14:57] http://goo.gl/TvEAZr - Peering - Wikimedia Foundation [14:57] This reminds me of the talk that we had at work in 2003 right before the power grid toook a hit that knocked out MI and Ohio. :) [14:57] oh man, that is so cool to visit a site adn it says, you are visiting from... "..." and its an ipv6 addy [14:58] http://en.wikipedia.org/wiki/Northeast_America_blackout_of_2003 [14:58] http://goo.gl/1UUPVo - Northeast America blackout of 2003 - Wikipedia, the free encyclopedia [14:58] cmaloney: :) [14:58] cmaloney: cascading failure scenarios are a thing :) ask Blackberry, I think they blamed that for at least 8 of their outages [14:58] cmaloney: i remember it well, my dad had just bought 2 quarts of ice cream, and I got to help eat it. [14:58] jrwren: ours? [14:59] I remember driving home to Belleville and being unable to call JoDee on the cellphone [14:59] that reminds me I need to yell at comcast again about ipv6, not that it will help [14:59] https://www.peeringdb.com/private/participant_view.php?id=1365 gives you numbers (guest/guest login) [14:59] http://goo.gl/jzcSYc - PeeringDB Login [14:59] Though the phone system at work was working [14:59] i was reasonably impressed how long it took the cell towers to start dying, and in more populated areas they just kept filling the generators so they didn't [15:00] greg-g: nah, http://bgp.he.net/AS33668 [15:00] http://goo.gl/twBRwD - 403 Forbidden [15:00] it's also the time when joDee looked at the phone (which didn't ring because the power was cut), picked it up, and surprised both her and her friend that was calling. [15:00] krondor: yeah, but they were slammed. [15:00] greg-g: i was looking up: http://bgp.he.net/AS14907 to see you [15:00] http://goo.gl/t5UX4H - 403 Forbidden [15:01] huh, i wonder why it says 403. I wonder if they filter based on UA [15:01] cmaloney: Don't drag all of Michigan into your little blackout. We were fine over here. [15:01] http://bgp.he.net/AS14907 [15:01] http://goo.gl/t5UX4H - 403 Forbidden [15:01] yeah [15:02] brousch__: i was fine once I drove 30 mins to grand blanc too... mmm delicious AC and gas for my car [15:02] lol, yup, curl -A 'Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0' and I can hit that page. [15:02] Sad thing is we were waiting for a blackout so we could do some stargazing [15:02] whose bot is that? [15:02] that's when we realized the fucking Wayne County fair was happening a few blocks away [15:02] jrwren: mrgoodcat's bot [15:03] so we had floodlights and hte sound of tractor pulls all night [15:03] fuckers. [15:03] alright, I'm AFK for at least the day, time for real vacation. [15:03] jrwren: Have fun! [15:34] getting real sick of this barking dog next door [15:34] too bad I'm on calls all morning and can't go to the coffee shop [15:34] or back to that bar... man they had a good rueben [15:44] Mmm, reuben [15:46] Reubens would be awesome if they weren't covered in decomposing cabbage [15:49] brousch__: I'm german. I love that stuff. [15:49] we make our own, actually [16:22] Nothing wrong with decomposing cabbage [16:48] rick_h_: Speedy response to that bug. Thanks! :D [16:48] StoneTable: oh hey, how goes? [16:48] StoneTable: heh, sorry I floated by the FF note at first [16:48] Pretty good, thanks. You? [16:49] No worries! [16:49] party party [16:50] StoneTable: what's got you poking at the GUI? [16:50] doing anything fun? [16:51] Interview prep. Sent my resume in last week, and setting up a time to talk to someone over there. [16:51] oh cool, who you talking to? [16:51] Antonio Rosales [16:51] oh awesome, great guy [16:52] Good to hear. I'm looking forward to that chat. [16:53] cool, let me know if you've got any questions [16:53] And the juju/Charms stuff is pretty slick, so I'm really excited to play with that more. I wish I'd had that 8 or so years ago. [16:53] I sure will, thanks! [19:36] cmaloney: wow, a 16EUR/$800 USD cd? [19:36] cmaloney: re the stoner rock band you linked before [19:51] greg-g: Eh? [19:51] that 420 whatever band's cd was 16 Euros [19:52] If I hit "buy it now" I get 21.24 [19:52] 21.66 rather [19:52] Granted I don't think four tracks is 21.66. :) [19:52] sorry, it was a bad "OMG, USD is so low compared to EUR" joke [19:52] Yes, yes it was. ;) [19:52] Though I've seen folks post albums that were ridiculous in price. [19:53] like $8 a track or some shit. [19:57] Actually I think the prime offender that I was thinking of lowered his prices on his music after I told him he was... ahem... high. :) [19:59] :) [19:59] was it also stoner rock? [20:05] cmaloney: btw, I'm liking http://juddmadden.bandcamp.com/album/everything-in-waves, it's CC BY-NC, you should use it for OMC [20:05] http://goo.gl/Gz89Dh - Judd Madden [20:37] greg-g: It's on my list. :) [20:38] it'd be a long episode. He seems to like 8+ minute songs [20:38] Yeah, he doesn't write short songs. [20:39] IIRC it's instrumental [20:39] so perfect for instrumetalcast. [20:41] yeppers === greg-g_ is now known as greg-g [21:01] Oh, btw: the person who said that the Aereo case might have an impact on cloud broadcasting for personal use: [21:01] http://arstechnica.com/tech-policy/2014/06/in-aereos-wake-fox-targets-dishs-tv-streaming-service/ [21:02] http://goo.gl/BbOJD7 - In Aereo’s wake, Fox targets Dish’s TV streaming service | Ars Technica [21:02] We're going to need some more fires, because there aren't enough to handle the "dying in" load. [22:40] bad article, but interesting: http://www.zdnet.com/blog/facebook/facebook-passwords-are-not-case-sensitive-update/3612 [22:40] http://goo.gl/mk6YT - Facebook passwords are not case sensitive (update) | ZDNet [22:41] basically, FB accepts 3 forms of your password: [22:41] 1) original password as you typed it (duh) [22:41] 2) original password with a capitalized first letter, in case your mobile device does that annoying thing [22:41] 3) your password with the case flipped on all letters, in case you have caps lock on [22:42] horrible horrible horrible reporting from ZDnet, of course, cuz I still have no idea why they're still around [22:51] wow yeah i don't know how they've stayed in business either [22:52] and hello from windows 8 *ducks and covers*