[00:20] <devinheitmueller> Anybody have any recommended best practices for Ubuntu partitioning for maximum reliability?  This is for an “appliance” where very little ever changes on disk, and I want to minimize the risk of the system not booting due to FS corruption when power cycled.
[00:20] <devinheitmueller> Back in the days where I rolled my own Linux distros, I would use RO partitions for everything except var, but that doesn’t work very well in off-the-shelf Linux distros.
[00:20] <cmaloney> Hey devinheitmueller
[00:21] <devinheitmueller> Hey craig!
[00:21] <cmaloney> Not sure what the best practices are for RO partitions
[00:21] <cmaloney> iirc root is pretty safe as long as /tmp is not marked RO
[00:21] <cmaloney> or using tmpfs
[00:21] <devinheitmueller> I would assume /var could also be problematic because of logs, etc.
[00:22] <cmaloney> Absolutely
[00:22] <devinheitmueller> That said, I would love to just have /tmp, /var, /, and then mark root read-only, but I worry that Ubuntu won’t really be equipped to run in such an environment.
[00:23] <cmaloney>  Well, /home should also be a separate partition
[00:23] <devinheitmueller> Yeah, but in reality /home will likely never change.
[00:23] <devinheitmueller> I’m more worried about Ubuntu’s system processes mucking around with the disk and expecting everything to be mounted RW.
[00:23] <cmaloney> There's still .cache and such
[00:24] <devinheitmueller> This thing really screams for something based on LFS or OpenWRT, but I don’t really have any say in that at this point.  They wanted to leverage an off-the-shelf Ubuntu system as the base OS for their appliance.
[00:25] <cmaloney> I thought there was a specification for what partitions could be mounted read only at some point
[00:25] <cmaloney> but it's escaping my noggin'
[00:25] <devinheitmueller> Perhaps there is.  I really don’t claim to have much expertise in Ubuntu.
[00:26] <devinheitmueller> I’m happy to say that it generally “just works”, so I spend much less time nowadays digging under the hood.
[00:26] <cmaloney> Well, it was some satandard that iirc Ubuntu didn't follow for a while.
[00:26] <cmaloney> and it wasn't until recently (2010 or so) that it was
[00:26] <devinheitmueller> Well, there’s LSB, but yeah many distros don’t follow it very tightly.
[00:26] <cmaloney> of course Google is showing me folks that have hardware or NFS issues. :)
[00:29] <devinheitmueller> Yeah, I started with something like “Ubuntu partitioning avoid corruption” and it didn’t return quite what I was interested in.  :-)
[00:29] <cmaloney> Yeah, I'm not sure at this point.
[00:30] <devinheitmueller> Fair enough.  This is where I consider my resident source of Ubuntu expertise, so figured no harm in asking.
[00:30] <cmaloney> smoser might know more if he's around
[00:30] <cmaloney> or you might want to check with ubuntu-arm as that is embedded Ubuntu
[00:31] <devinheitmueller> Not a bad idea.
[00:31] <cmaloney> Might at least get you in the general direction
[00:31] <devinheitmueller> I’ll take a closer look and see what they recommend for Ubuntu Server.  It’s likely that is tailored more for reliability than convenience.
[00:32] <cmaloney> yeah, not sure Ubuntu server is going to get you any better answers.
[00:32] <cmaloney> But worth a shot
[00:32] <devinheitmueller> yup.
[00:32] <devinheitmueller> I’m not suggesting we run Ubuntu server, just that that particular crowd might have a mindset more in line with my goals.
[00:33] <cmaloney> Oh definitely
[00:33] <devinheitmueller> (i.e. minimize risk of system not booting, configure to automatically fix issues at boot, FS layout which minimizes boot time, etc)
[00:34] <devinheitmueller> On the flip side, my Tivo runs Embedded Linux and takes 15-20 minutes to boot, so perhaps I should just adjust my expectations.  :-)
[00:34] <cmaloney> heh
[00:34] <cmaloney> Well, that's partly why ext4fs was devised. :)
[00:34] <devinheitmueller> Of course, file systems continue to improve.
[00:35] <devinheitmueller> Ted Tso gave a really good talk last year at NYLUG on the state of Linux filesystems.
[00:53] <jrwren> devinheitmueller: do you have enough disk space to do it all on an ubuntu cloudimg?
[00:53] <devinheitmueller> In this case it’s a standalone piece of equipment, not internet connected.
[00:53] <jrwren> that is ok.
[00:53] <jrwren> you can ship the cloudimg on your standalone piece.
[00:54] <devinheitmueller> Hmmm....
[00:54] <devinheitmueller> Maybe.  Admittedly I don’t really know much about Ubuntu cloudimgs.  Will have to add that to the list of things to look up.
[00:54] <jrwren> so you can have root and everything be RO, then untar cloudimg on each boot
[00:54] <jrwren> its a 200MB system image, nice and tiny.
[00:54] <jrwren> what is the application?
[00:55] <devinheitmueller> It’s a piece of fitness equipment.
[00:56] <devinheitmueller> Hence the app lets you see stats, watch tv, control the treadmill, etc.
[00:57] <devinheitmueller> cmaloney: look, it’s that “smoser” guy you were talking about:  http://ubuntu-smoser.blogspot.com/2013/02/using-ubuntu-cloud-images-without-cloud.html
[00:57] <bookiebot> http://goo.gl/nHtgrm - smoser's thoughts: Using Ubuntu cloud-images without a cloud
[00:57] <jrwren> sounds like a fun challenge.
[00:57] <jrwren> i'd probably do what you are doing and just gut ubuntu
[00:57] <devinheitmueller> Yeah, it’s your typical embedded Linux appliance where almost nothing ever changes but somebody can yank the power cord at any time.
[00:59] <devinheitmueller> Always hard finding the right balance - rolling your own micro-distro (or gutting a distro like Ubuntu) gives you ultimate control, but then you have to maintain it and can’t rely on the upstream for patches.
[01:00] <jrwren> i like gutting
[01:00] <jrwren> because i get all the advantage of the vast packages, and I can reasonably maintain it in the form of a patch
[01:00] <devinheitmueller> The customer is not as technically astute as you are.  :-)
[01:00] <jrwren> they shouldn't need to be.
[01:01] <devinheitmueller> In reality it’s likley they won’t do anything about this.  The only reason I started thinking about it is the dev system I was using had it’s ext4 rootfs lose it’s mind, leaving hundreds of critical items in /lost+found.
[01:01] <devinheitmueller> Oh, and it didn’t boot.  Always nice.
[01:02] <jrwren> XFS ftw.
[01:03] <devinheitmueller> XFS feels surprisingly like overkill - it’s an embedded system with a single disk and 300MB of software.  It’s not a multi-terabyte storage array.
[01:03] <devinheitmueller> :-)
[01:04] <devinheitmueller> In other words, I shouldn’t have to use XFS to feel confident that my system is going to successfully boot.
[01:06] <jrwren> um... xfs isnt that different from ext
[01:06] <jrwren> it has nicer fsck properties
[01:15] <krondor> you could add ext4 checksums to help combat corruption.  It's a little newer though (in terms of FS timescales) so ymmv I haven't really played with it.
[01:16] <krondor> https://ext4.wiki.kernel.org/index.php/Ext4_Metadata_Checksums
[01:16] <bookiebot> http://goo.gl/tmyAS3 - Ext4 Metadata Checksums - Ext4
[12:17] <cmaloney> morning
[12:18] <brousch__> indeed
[13:02] <smoser> well, above, devinheitmueller is gone now, but i'd probably start with ubuntu core. its what ubuntu core is for.
[13:02] <smoser> and go from there.
[13:03] <smoser> the read-only stuff, can be made to work. i have worked on 'overlayroot' which mounts root read-only and writes delta via overlayfs.
[13:04] <smoser> its functional. i think that probalby sint exactly what he was looking for, but using it will tell you nicely where things fall over (by seeing new files in the overlay location).
[13:04] <smoser> kirkland wrote a blog post on overlayroot at http://blog.dustinkirkland.com/2012/08/introducing-overlayroot-overlayfs.html
[13:04] <bookiebot> http://goo.gl/AFWu3I - From the Canyon Edge: Introducing overlayroot -- overlayfs + dmcrypt!
[13:04] <smoser> its built into our images.
[13:13] <rick_h_> morning and TGIF
[13:20] <cmaloney> Yeah, no kidding
[13:49] <jrwren> ut oh, someone posted this GIANT jpg link to reddit -https://upload.wikimedia.org/wikipedia/commons/7/78/30_Doradus%2C_Tarantula_Nebula.jpg
[13:50] <bookiebot> http://goo.gl/BVS6wO - image/jpeg
[13:50] <jrwren> greg-g's data bill is going to quintuple
[13:53] <brousch__> It loads so slowly!
[14:06] <jrwren> cuz its HUGE
[14:06] <jrwren> and maybe greg-g has it throttled.
[14:41] <greg-g> jrwren: you think reddit, the 61st popular site by Alexa, will have a traffic impact on us, the 4th popular site?
[14:42] <brousch__> Is that a dare?
[14:42]  * brousch__ send it to 4chan
[14:42] <greg-g> when was that image posted?
[14:42] <greg-g> http://i.imgur.com/8LXvhEI.png
[14:42] <bookiebot> http://goo.gl/R1RXlG - image/png
[14:42] <greg-g> that's the past week
[14:43] <greg-g> actually, that's just css/js, that blip was us deploying a new version
[14:44] <greg-g> here comes upload.wikimedia.org traffic:
[14:44] <krondor> reddit punches above its traffic weight due to the burstiness (yes I say that's a word) of top posts
[14:44] <jrwren> greg-g: my thought was that the file is so large that it would take fewer hits to impact you.
[14:44] <krondor> though I doubt it would be a problem :)
[14:44] <greg-g> last week of upload traffic: http://i.imgur.com/3UfqB7E.png
[14:44] <bookiebot> http://goo.gl/YlPCRU - image/png
[14:44] <jrwren> at the same time, it could easily be served via CDN
[14:44] <jrwren> do you use a CDN?
[14:44] <greg-g> jrwren: cache
[14:44] <greg-g> we are our own cdn
[14:45] <jrwren> what cache?
[14:45] <greg-g> varnish boxen
[14:45] <jrwren> centralized or distributed?
[14:45] <greg-g> we have 3 caching locations
[14:45] <greg-g> SF, Virginia, and Amsterdam
[14:45] <jrwren> i see.
[14:46] <greg-g> soon Dallas (that'll be a full fledged DC, with caching obviously)
[14:46] <jrwren> and you don't care about data quantity hitting them?
[14:46] <greg-g> no, they're caches :)
[14:46] <jrwren> or you'd not notice?
[14:46] <greg-g> not that image from reddit
[14:46] <cmaloney> I love that my CPU fan starts going into overdrive when I load that image.
[14:46] <greg-g> see the last imgur link from me
[14:46] <jrwren> still, they have a data rate limit of the aggregate of their interfaces, right?
[14:46] <jrwren> yes, I see it, its always the same.
[14:46] <jrwren> nice consistent graph.
[14:47] <jrwren> so, I get that this reddit link doesn't impact you AT ALL.
[14:47] <greg-g> here's the last day for upload caches (images): http://i.imgur.com/hkIFpYK.png
[14:47] <bookiebot> http://goo.gl/nOXKPc - image/png
[14:47] <jrwren> still, I wonder what it would take.
[14:47] <greg-g> a lot
[14:47] <greg-g> :)
[14:47] <jrwren> what is your daily max aggregate cache traffic rate?
[14:47] <greg-g> I don't think I should publicly say our utilization %
[14:47] <jrwren> ok
[14:48] <jrwren> that is probably a good idea.
[14:48] <greg-g> which is why I didn't include the numbers on the left of the graph ;)
[14:48] <jrwren> understood.
[14:48] <greg-g> we are a #4 website afterall, people like to make names for themselves ;)
[14:48] <bookiebot> closed - importer doesn't handle spaces - https://github.com/bookieio/Bookie/issues/4
[14:48] <greg-g> heh
[14:48] <cmaloney> http://theroaring420s.bandcamp.com/album/the-roaring-420s-what-is-psych?t=5
[14:48] <bookiebot> http://goo.gl/VkGCYJ - The Roaring 420s - What Is Psych? | The Roaring 420s
[14:48] <jrwren> greg-g: I know! and you don't even use DDoS protection!
[14:48] <greg-g> bookiebot: not all #666s are bookie bugs
[14:49] <cmaloney> I love some of the band names of stoner bands.
[14:49] <greg-g> jrwren: nope, we're 100% in house, except for one or two third-party monitoring (in caes our monitors go down, heh)
[14:50] <cmaloney> I think that's more secure that way
[14:50] <jrwren> you can do 100% in house ddos proection :p
[14:50] <jrwren> when I said you don't use it, I didn't mean cloud ddos b.s. like cloudflair, I meant an Arbor Networks mitigation appliance :p
[14:50] <cmaloney> Since you're not relying on mirrors that might try to change thing son Wikipedia
[14:50] <greg-g> jrwren: oh, I'm not 100% sure if we don't, but I'm 90% sure we don't have one of those
[14:50] <cmaloney> jrwren: Come Monday will that be a Juju deployment? :)
[14:51] <jrwren> cmaloney: nah. juju should probably have that kind of protection too ;]
[14:51] <cmaloney> hah
[14:52] <greg-g> jrwren: whenever we have had spikes in traffic for weird reasons (either bad actors or good intention'd people's bots gone crazy) I've never heard of any automatic anything
[14:52] <greg-g> really, raw traffic won't hurt us, it's intelligent queries to specific things
[14:53] <rick_h_> anyone want to buy an x230 that you can't use the microphone jack on?
[14:53] <cmaloney> Has wikipedia ever been ddosed?
[14:53] <rick_h_> (or the built in mic)
[14:53] <jrwren> rick_h_: oh no! is that what you tweeted about?
[14:53] <cmaloney> rick_h_: so it's official then? XPS in da house?
[14:53] <greg-g> cmaloney: successfully? not that I know of
[14:53] <krondor> cmaloney:  oh I'm sure, probably like weekly just not hard enough to count
[14:53] <rick_h_> jrwren: no, I tweeted about juju core $@#!$@$ me over and cauing me pain and agony
[14:53] <greg-g> cmaloney: we've seen people try
[14:53] <greg-g> :)
[14:53] <cmaloney> greg-g: Oh no doubt
[14:53] <jrwren> ah, poor jujucore. they should have hired me :p
[14:53] <rick_h_> jrwren: but my new laptop should be here tomorrow which is required because my x230 has sound issues and I can't use it for meeting
[14:54] <cmaloney> rick_h_: I'd hold out on selling
[14:54] <cmaloney> Make sure you're 100% good with parting
[14:54] <rick_h_> cmaloney: it's going to get tough for the wife to see me around with a desktop and 3 laptops
[14:54] <rick_h_> cmaloney: I've not booted it in a week
[14:54] <cmaloney> Just put it on the closet for a while
[14:54] <cmaloney> let it pass
[14:54] <jrwren> old laptops make good cloud nodes. tell her its part of your home cloud.
[14:54] <greg-g> so, when I first started, some random dude threatened ddos'ing us on IRC, I mentioned it to our networking lead, her first reaction was laughing and just walking away. From then on I haven't worried ;)
[14:54] <cmaloney> and then when you don't want to go into the closet to retrieve the x230 then it's time to make peace.
[14:55] <rick_h_> jrwren: yea, did think about that for a maas controller letting me have 3 nucs to operate
[14:55] <rick_h_> might still do that if no one's interested
[14:55] <jrwren> greg-g: understood, still, unless you can handle 300Gbps, I wouldn't get too cocky.
[14:55] <greg-g> jrwren: no, we can't do 300gbps, that I'll say :)
[14:55] <jrwren> greg-g: then if the right people wanted to take you down, they could.
[14:56] <greg-g> jrwren: I mean, you can look up our ASN stuff, too
[14:56] <greg-g> jrwren: totally
[14:56] <jrwren> oh yeah!
[14:56] <krondor> jrwren:  well, they probably have agreements with their providers to help with that mitigation upstream too
[14:56] <krondor> still that interface will be a delay and no doubt cause downtime with that type of hit
[14:57] <greg-g> jrwren: to help: http://wikimediafoundation.org/wiki/Peering
[14:57] <bookiebot> http://goo.gl/TvEAZr - Peering - Wikimedia Foundation
[14:57] <cmaloney> This reminds me of the talk that we had at work in 2003 right before the power grid toook a hit that knocked out MI and Ohio. :)
[14:57] <jrwren> oh man, that is so cool to visit a site adn it says, you are visiting from... "..." and its an ipv6 addy
[14:58] <cmaloney> http://en.wikipedia.org/wiki/Northeast_America_blackout_of_2003
[14:58] <bookiebot> http://goo.gl/1UUPVo - Northeast America blackout of 2003 - Wikipedia, the free encyclopedia
[14:58] <greg-g> cmaloney: :)
[14:58] <krondor> cmaloney: cascading failure scenarios are a thing :) ask Blackberry, I think they blamed that for at least 8 of their outages
[14:58] <jrwren> cmaloney: i remember it well, my dad had just bought 2 quarts of ice cream, and I got to help eat it.
[14:58] <greg-g> jrwren: ours?
[14:59] <cmaloney> I remember driving home to Belleville and being unable to call JoDee on the cellphone
[14:59] <krondor> that reminds me I need to yell at comcast again about ipv6, not that it will help
[14:59] <greg-g> https://www.peeringdb.com/private/participant_view.php?id=1365 gives you numbers (guest/guest login)
[14:59] <bookiebot> http://goo.gl/jzcSYc - PeeringDB Login
[14:59] <cmaloney> Though the phone system at work was working
[14:59] <krondor> i was reasonably impressed how long it took the cell towers to start dying, and in more populated areas they just kept filling the generators so they didn't
[15:00] <jrwren> greg-g: nah, http://bgp.he.net/AS33668
[15:00] <bookiebot> http://goo.gl/twBRwD - 403 Forbidden
[15:00] <cmaloney> it's also the time when joDee looked at the phone (which didn't ring because the power was cut), picked it up, and surprised both her and her friend that was calling.
[15:00] <cmaloney> krondor: yeah, but they were slammed.
[15:00] <jrwren> greg-g: i was looking up: http://bgp.he.net/AS14907  to see you
[15:00] <bookiebot> http://goo.gl/t5UX4H - 403 Forbidden
[15:01] <jrwren> huh, i wonder why it says 403. I wonder if they filter based on UA
[15:01] <brousch__> cmaloney: Don't drag all of Michigan into your little blackout. We were fine over here.
[15:01] <greg-g> http://bgp.he.net/AS14907
[15:01] <bookiebot> http://goo.gl/t5UX4H - 403 Forbidden
[15:01] <greg-g> yeah
[15:02] <krondor> brousch__: i was fine once I drove 30 mins to grand blanc too... mmm delicious AC and gas for my car
[15:02] <jrwren> lol, yup, curl -A 'Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0'   and I can hit that page.
[15:02] <cmaloney> Sad thing is we were waiting for a blackout so we could do some stargazing
[15:02] <jrwren> whose bot is that?
[15:02] <cmaloney> that's when we realized the fucking Wayne County fair was happening a few blocks away
[15:02] <cmaloney> jrwren: mrgoodcat's bot
[15:03] <cmaloney> so we had floodlights and hte sound of tractor pulls all night
[15:03] <cmaloney> fuckers.
[15:03] <jrwren> alright, I'm AFK for at least the day, time for real vacation.
[15:03] <cmaloney> jrwren: Have fun!
[15:34] <greg-g> getting real sick of this barking dog next door
[15:34] <greg-g> too bad I'm on calls all morning and can't go to the coffee shop
[15:34] <greg-g> or back to that bar... man they had a good rueben
[15:44] <cmaloney> Mmm, reuben
[15:46] <brousch__> Reubens would be awesome if they weren't covered in decomposing cabbage
[15:49] <greg-g> brousch__: I'm german. I love that stuff.
[15:49] <greg-g> we make our own, actually
[16:22] <cmaloney> Nothing wrong with decomposing cabbage
[16:48] <StoneTable> rick_h_: Speedy response to that bug. Thanks! :D
[16:48] <rick_h_> StoneTable: oh hey, how goes?
[16:48] <rick_h_> StoneTable: heh, sorry I floated by the FF note at first
[16:48] <StoneTable> Pretty good, thanks. You?
[16:49] <StoneTable> No worries!
[16:49] <rick_h_> party party
[16:50] <rick_h_> StoneTable: what's got you poking at the GUI?
[16:50] <rick_h_> doing anything fun?
[16:51] <StoneTable> Interview prep. Sent my resume in last week, and setting up a time to talk to someone over there.
[16:51] <rick_h_> oh cool, who you talking to?
[16:51] <StoneTable> Antonio Rosales
[16:51] <rick_h_> oh awesome, great guy
[16:52] <StoneTable> Good to hear. I'm looking forward to that chat.
[16:53] <rick_h_> cool, let me know if you've got any questions
[16:53] <StoneTable> And the juju/Charms stuff is pretty slick, so I'm really excited to play with that more. I wish I'd had that 8 or so years ago.
[16:53] <StoneTable> I sure will, thanks!
[19:36] <greg-g> cmaloney: wow, a 16EUR/$800 USD cd?
[19:36] <greg-g> cmaloney: re the stoner rock band you linked before
[19:51] <cmaloney> greg-g: Eh?
[19:51] <greg-g> that 420 whatever band's cd was 16 Euros
[19:52] <cmaloney> If I hit "buy it now" I get 21.24
[19:52] <cmaloney> 21.66 rather
[19:52] <cmaloney> Granted I don't think four tracks is 21.66. :)
[19:52] <greg-g> sorry, it was a bad "OMG, USD is so low compared to EUR" joke
[19:52] <cmaloney> Yes, yes it was. ;)
[19:52] <cmaloney> Though I've seen folks post albums that were ridiculous in price.
[19:53] <cmaloney> like $8 a track or some shit.
[19:57] <cmaloney> Actually I think the prime offender that I was thinking of lowered his prices on his music after I told him he was... ahem... high. :)
[19:59] <greg-g> :)
[19:59] <greg-g> was it also stoner rock?
[20:05] <greg-g> cmaloney: btw, I'm liking http://juddmadden.bandcamp.com/album/everything-in-waves, it's CC BY-NC, you should use it for OMC
[20:05] <bookiebot> http://goo.gl/Gz89Dh - Judd Madden
[20:37] <cmaloney> greg-g: It's on my list. :)
[20:38] <greg-g> it'd be a long episode. He seems to like 8+ minute songs
[20:38] <cmaloney> Yeah, he doesn't write short songs.
[20:39] <cmaloney> IIRC it's instrumental
[20:39] <cmaloney> so perfect for instrumetalcast.
[20:41] <greg-g> yeppers
[21:01] <cmaloney> Oh, btw: the person who said that the Aereo case might have an impact on cloud broadcasting for personal use:
[21:01] <cmaloney> http://arstechnica.com/tech-policy/2014/06/in-aereos-wake-fox-targets-dishs-tv-streaming-service/
[21:02] <bookiebot> http://goo.gl/BbOJD7 - In Aereo’s wake, Fox targets Dish’s TV streaming service | Ars Technica
[21:02] <cmaloney> We're going to need some more fires, because there aren't enough to handle the "dying in" load.
[22:40] <greg-g> bad article, but interesting: http://www.zdnet.com/blog/facebook/facebook-passwords-are-not-case-sensitive-update/3612
[22:40] <bookiebot> http://goo.gl/mk6YT - Facebook passwords are not case sensitive (update) | ZDNet
[22:41] <greg-g> basically, FB accepts 3 forms of your password:
[22:41] <greg-g> 1) original password as you typed it (duh)
[22:41] <greg-g> 2) original password with a capitalized first letter, in case your mobile device does that annoying thing
[22:41] <greg-g> 3) your password with the case flipped on all letters, in case you have caps lock on
[22:42] <greg-g> horrible horrible horrible reporting from ZDnet, of course, cuz I still have no idea why they're still around
[22:51] <gamerchick02> wow yeah i don't know how they've stayed in business either
[22:52] <gamerchick02> and hello from windows 8 *ducks and covers*