/srv/irclogs.ubuntu.com/2014/06/27/#ubuntu-us-mi.txt

devinheitmuellerAnybody have any recommended best practices for Ubuntu partitioning for maximum reliability?  This is for an “appliance” where very little ever changes on disk, and I want to minimize the risk of the system not booting due to FS corruption when power cycled.00:20
devinheitmuellerBack in the days where I rolled my own Linux distros, I would use RO partitions for everything except var, but that doesn’t work very well in off-the-shelf Linux distros.00:20
cmaloneyHey devinheitmueller00:20
devinheitmuellerHey craig!00:21
cmaloneyNot sure what the best practices are for RO partitions00:21
cmaloneyiirc root is pretty safe as long as /tmp is not marked RO00:21
cmaloneyor using tmpfs00:21
devinheitmuellerI would assume /var could also be problematic because of logs, etc.00:21
cmaloneyAbsolutely00:22
devinheitmuellerThat said, I would love to just have /tmp, /var, /, and then mark root read-only, but I worry that Ubuntu won’t really be equipped to run in such an environment.00:22
cmaloney Well, /home should also be a separate partition00:23
devinheitmuellerYeah, but in reality /home will likely never change.00:23
devinheitmuellerI’m more worried about Ubuntu’s system processes mucking around with the disk and expecting everything to be mounted RW.00:23
cmaloneyThere's still .cache and such00:23
devinheitmuellerThis thing really screams for something based on LFS or OpenWRT, but I don’t really have any say in that at this point.  They wanted to leverage an off-the-shelf Ubuntu system as the base OS for their appliance.00:24
cmaloneyI thought there was a specification for what partitions could be mounted read only at some point00:25
cmaloneybut it's escaping my noggin'00:25
devinheitmuellerPerhaps there is.  I really don’t claim to have much expertise in Ubuntu.00:25
devinheitmuellerI’m happy to say that it generally “just works”, so I spend much less time nowadays digging under the hood.00:26
cmaloneyWell, it was some satandard that iirc Ubuntu didn't follow for a while.00:26
cmaloneyand it wasn't until recently (2010 or so) that it was00:26
devinheitmuellerWell, there’s LSB, but yeah many distros don’t follow it very tightly.00:26
cmaloneyof course Google is showing me folks that have hardware or NFS issues. :)00:26
devinheitmuellerYeah, I started with something like “Ubuntu partitioning avoid corruption” and it didn’t return quite what I was interested in.  :-)00:29
cmaloneyYeah, I'm not sure at this point.00:29
devinheitmuellerFair enough.  This is where I consider my resident source of Ubuntu expertise, so figured no harm in asking.00:30
cmaloneysmoser might know more if he's around00:30
cmaloneyor you might want to check with ubuntu-arm as that is embedded Ubuntu00:30
devinheitmuellerNot a bad idea.00:31
cmaloneyMight at least get you in the general direction00:31
devinheitmuellerI’ll take a closer look and see what they recommend for Ubuntu Server.  It’s likely that is tailored more for reliability than convenience.00:31
cmaloneyyeah, not sure Ubuntu server is going to get you any better answers.00:32
cmaloneyBut worth a shot00:32
devinheitmuelleryup.00:32
devinheitmuellerI’m not suggesting we run Ubuntu server, just that that particular crowd might have a mindset more in line with my goals.00:32
cmaloneyOh definitely00:33
devinheitmueller(i.e. minimize risk of system not booting, configure to automatically fix issues at boot, FS layout which minimizes boot time, etc)00:33
devinheitmuellerOn the flip side, my Tivo runs Embedded Linux and takes 15-20 minutes to boot, so perhaps I should just adjust my expectations.  :-)00:34
cmaloneyheh00:34
cmaloneyWell, that's partly why ext4fs was devised. :)00:34
devinheitmuellerOf course, file systems continue to improve.00:34
devinheitmuellerTed Tso gave a really good talk last year at NYLUG on the state of Linux filesystems.00:35
jrwrendevinheitmueller: do you have enough disk space to do it all on an ubuntu cloudimg?00:53
devinheitmuellerIn this case it’s a standalone piece of equipment, not internet connected.00:53
jrwrenthat is ok.00:53
jrwrenyou can ship the cloudimg on your standalone piece.00:53
devinheitmuellerHmmm....00:54
devinheitmuellerMaybe.  Admittedly I don’t really know much about Ubuntu cloudimgs.  Will have to add that to the list of things to look up.00:54
jrwrenso you can have root and everything be RO, then untar cloudimg on each boot00:54
jrwrenits a 200MB system image, nice and tiny.00:54
jrwrenwhat is the application?00:54
devinheitmuellerIt’s a piece of fitness equipment.00:55
devinheitmuellerHence the app lets you see stats, watch tv, control the treadmill, etc.00:56
devinheitmuellercmaloney: look, it’s that “smoser” guy you were talking about:  http://ubuntu-smoser.blogspot.com/2013/02/using-ubuntu-cloud-images-without-cloud.html00:57
bookiebothttp://goo.gl/nHtgrm - smoser's thoughts: Using Ubuntu cloud-images without a cloud00:57
jrwrensounds like a fun challenge.00:57
jrwreni'd probably do what you are doing and just gut ubuntu00:57
devinheitmuellerYeah, it’s your typical embedded Linux appliance where almost nothing ever changes but somebody can yank the power cord at any time.00:57
devinheitmuellerAlways hard finding the right balance - rolling your own micro-distro (or gutting a distro like Ubuntu) gives you ultimate control, but then you have to maintain it and can’t rely on the upstream for patches.00:59
jrwreni like gutting01:00
jrwrenbecause i get all the advantage of the vast packages, and I can reasonably maintain it in the form of a patch01:00
devinheitmuellerThe customer is not as technically astute as you are.  :-)01:00
jrwrenthey shouldn't need to be.01:00
devinheitmuellerIn reality it’s likley they won’t do anything about this.  The only reason I started thinking about it is the dev system I was using had it’s ext4 rootfs lose it’s mind, leaving hundreds of critical items in /lost+found.01:01
devinheitmuellerOh, and it didn’t boot.  Always nice.01:01
jrwrenXFS ftw.01:02
devinheitmuellerXFS feels surprisingly like overkill - it’s an embedded system with a single disk and 300MB of software.  It’s not a multi-terabyte storage array.01:03
devinheitmueller:-)01:03
devinheitmuellerIn other words, I shouldn’t have to use XFS to feel confident that my system is going to successfully boot.01:04
jrwrenum... xfs isnt that different from ext01:06
jrwrenit has nicer fsck properties01:06
krondoryou could add ext4 checksums to help combat corruption.  It's a little newer though (in terms of FS timescales) so ymmv I haven't really played with it.01:15
krondorhttps://ext4.wiki.kernel.org/index.php/Ext4_Metadata_Checksums01:16
bookiebothttp://goo.gl/tmyAS3 - Ext4 Metadata Checksums - Ext401:16
cmaloneymorning12:17
brousch__indeed12:18
smoserwell, above, devinheitmueller is gone now, but i'd probably start with ubuntu core. its what ubuntu core is for.13:02
smoserand go from there.13:02
smoserthe read-only stuff, can be made to work. i have worked on 'overlayroot' which mounts root read-only and writes delta via overlayfs.13:03
smoserits functional. i think that probalby sint exactly what he was looking for, but using it will tell you nicely where things fall over (by seeing new files in the overlay location).13:04
smoserkirkland wrote a blog post on overlayroot at http://blog.dustinkirkland.com/2012/08/introducing-overlayroot-overlayfs.html13:04
bookiebothttp://goo.gl/AFWu3I - From the Canyon Edge: Introducing overlayroot -- overlayfs + dmcrypt!13:04
smoserits built into our images.13:04
rick_h_morning and TGIF13:13
cmaloneyYeah, no kidding13:20
jrwrenut oh, someone posted this GIANT jpg link to reddit -https://upload.wikimedia.org/wikipedia/commons/7/78/30_Doradus%2C_Tarantula_Nebula.jpg13:49
bookiebothttp://goo.gl/BVS6wO - image/jpeg13:50
jrwrengreg-g's data bill is going to quintuple13:50
brousch__It loads so slowly!13:53
jrwrencuz its HUGE14:06
jrwrenand maybe greg-g has it throttled.14:06
greg-gjrwren: you think reddit, the 61st popular site by Alexa, will have a traffic impact on us, the 4th popular site?14:41
brousch__Is that a dare?14:42
* brousch__ send it to 4chan14:42
greg-gwhen was that image posted?14:42
greg-ghttp://i.imgur.com/8LXvhEI.png14:42
bookiebothttp://goo.gl/R1RXlG - image/png14:42
greg-gthat's the past week14:42
greg-gactually, that's just css/js, that blip was us deploying a new version14:43
greg-ghere comes upload.wikimedia.org traffic:14:44
krondorreddit punches above its traffic weight due to the burstiness (yes I say that's a word) of top posts14:44
jrwrengreg-g: my thought was that the file is so large that it would take fewer hits to impact you.14:44
krondorthough I doubt it would be a problem :)14:44
greg-glast week of upload traffic: http://i.imgur.com/3UfqB7E.png14:44
bookiebothttp://goo.gl/YlPCRU - image/png14:44
jrwrenat the same time, it could easily be served via CDN14:44
jrwrendo you use a CDN?14:44
greg-gjrwren: cache14:44
greg-gwe are our own cdn14:44
jrwrenwhat cache?14:45
greg-gvarnish boxen14:45
jrwrencentralized or distributed?14:45
greg-gwe have 3 caching locations14:45
greg-gSF, Virginia, and Amsterdam14:45
jrwreni see.14:45
greg-gsoon Dallas (that'll be a full fledged DC, with caching obviously)14:46
jrwrenand you don't care about data quantity hitting them?14:46
greg-gno, they're caches :)14:46
jrwrenor you'd not notice?14:46
greg-gnot that image from reddit14:46
cmaloneyI love that my CPU fan starts going into overdrive when I load that image.14:46
greg-gsee the last imgur link from me14:46
jrwrenstill, they have a data rate limit of the aggregate of their interfaces, right?14:46
jrwrenyes, I see it, its always the same.14:46
jrwrennice consistent graph.14:46
jrwrenso, I get that this reddit link doesn't impact you AT ALL.14:47
greg-ghere's the last day for upload caches (images): http://i.imgur.com/hkIFpYK.png14:47
bookiebothttp://goo.gl/nOXKPc - image/png14:47
jrwrenstill, I wonder what it would take.14:47
greg-ga lot14:47
greg-g:)14:47
jrwrenwhat is your daily max aggregate cache traffic rate?14:47
greg-gI don't think I should publicly say our utilization %14:47
jrwrenok14:47
jrwrenthat is probably a good idea.14:48
greg-gwhich is why I didn't include the numbers on the left of the graph ;)14:48
jrwrenunderstood.14:48
greg-gwe are a #4 website afterall, people like to make names for themselves ;)14:48
bookiebotclosed - importer doesn't handle spaces - https://github.com/bookieio/Bookie/issues/414:48
greg-gheh14:48
cmaloneyhttp://theroaring420s.bandcamp.com/album/the-roaring-420s-what-is-psych?t=514:48
bookiebothttp://goo.gl/VkGCYJ - The Roaring 420s - What Is Psych? | The Roaring 420s14:48
jrwrengreg-g: I know! and you don't even use DDoS protection!14:48
greg-gbookiebot: not all #666s are bookie bugs14:48
cmaloneyI love some of the band names of stoner bands.14:49
greg-gjrwren: nope, we're 100% in house, except for one or two third-party monitoring (in caes our monitors go down, heh)14:49
cmaloneyI think that's more secure that way14:50
jrwrenyou can do 100% in house ddos proection :p14:50
jrwrenwhen I said you don't use it, I didn't mean cloud ddos b.s. like cloudflair, I meant an Arbor Networks mitigation appliance :p14:50
cmaloneySince you're not relying on mirrors that might try to change thing son Wikipedia14:50
greg-gjrwren: oh, I'm not 100% sure if we don't, but I'm 90% sure we don't have one of those14:50
cmaloneyjrwren: Come Monday will that be a Juju deployment? :)14:50
jrwrencmaloney: nah. juju should probably have that kind of protection too ;]14:51
cmaloneyhah14:51
greg-gjrwren: whenever we have had spikes in traffic for weird reasons (either bad actors or good intention'd people's bots gone crazy) I've never heard of any automatic anything14:52
greg-greally, raw traffic won't hurt us, it's intelligent queries to specific things14:52
rick_h_anyone want to buy an x230 that you can't use the microphone jack on?14:53
cmaloneyHas wikipedia ever been ddosed?14:53
rick_h_(or the built in mic)14:53
jrwrenrick_h_: oh no! is that what you tweeted about?14:53
cmaloneyrick_h_: so it's official then? XPS in da house?14:53
greg-gcmaloney: successfully? not that I know of14:53
krondorcmaloney:  oh I'm sure, probably like weekly just not hard enough to count14:53
rick_h_jrwren: no, I tweeted about juju core $@#!$@$ me over and cauing me pain and agony14:53
greg-gcmaloney: we've seen people try14:53
greg-g:)14:53
cmaloneygreg-g: Oh no doubt14:53
jrwrenah, poor jujucore. they should have hired me :p14:53
rick_h_jrwren: but my new laptop should be here tomorrow which is required because my x230 has sound issues and I can't use it for meeting14:53
cmaloneyrick_h_: I'd hold out on selling14:54
cmaloneyMake sure you're 100% good with parting14:54
rick_h_cmaloney: it's going to get tough for the wife to see me around with a desktop and 3 laptops14:54
rick_h_cmaloney: I've not booted it in a week14:54
cmaloneyJust put it on the closet for a while14:54
cmaloneylet it pass14:54
jrwrenold laptops make good cloud nodes. tell her its part of your home cloud.14:54
greg-gso, when I first started, some random dude threatened ddos'ing us on IRC, I mentioned it to our networking lead, her first reaction was laughing and just walking away. From then on I haven't worried ;)14:54
cmaloneyand then when you don't want to go into the closet to retrieve the x230 then it's time to make peace.14:54
rick_h_jrwren: yea, did think about that for a maas controller letting me have 3 nucs to operate14:55
rick_h_might still do that if no one's interested14:55
jrwrengreg-g: understood, still, unless you can handle 300Gbps, I wouldn't get too cocky.14:55
greg-gjrwren: no, we can't do 300gbps, that I'll say :)14:55
jrwrengreg-g: then if the right people wanted to take you down, they could.14:55
greg-gjrwren: I mean, you can look up our ASN stuff, too14:56
greg-gjrwren: totally14:56
jrwrenoh yeah!14:56
krondorjrwren:  well, they probably have agreements with their providers to help with that mitigation upstream too14:56
krondorstill that interface will be a delay and no doubt cause downtime with that type of hit14:56
greg-gjrwren: to help: http://wikimediafoundation.org/wiki/Peering14:57
bookiebothttp://goo.gl/TvEAZr - Peering - Wikimedia Foundation14:57
cmaloneyThis reminds me of the talk that we had at work in 2003 right before the power grid toook a hit that knocked out MI and Ohio. :)14:57
jrwrenoh man, that is so cool to visit a site adn it says, you are visiting from... "..." and its an ipv6 addy14:57
cmaloneyhttp://en.wikipedia.org/wiki/Northeast_America_blackout_of_200314:58
bookiebothttp://goo.gl/1UUPVo - Northeast America blackout of 2003 - Wikipedia, the free encyclopedia14:58
greg-gcmaloney: :)14:58
krondorcmaloney: cascading failure scenarios are a thing :) ask Blackberry, I think they blamed that for at least 8 of their outages14:58
jrwrencmaloney: i remember it well, my dad had just bought 2 quarts of ice cream, and I got to help eat it.14:58
greg-gjrwren: ours?14:58
cmaloneyI remember driving home to Belleville and being unable to call JoDee on the cellphone14:59
krondorthat reminds me I need to yell at comcast again about ipv6, not that it will help14:59
greg-ghttps://www.peeringdb.com/private/participant_view.php?id=1365 gives you numbers (guest/guest login)14:59
bookiebothttp://goo.gl/jzcSYc - PeeringDB Login14:59
cmaloneyThough the phone system at work was working14:59
krondori was reasonably impressed how long it took the cell towers to start dying, and in more populated areas they just kept filling the generators so they didn't14:59
jrwrengreg-g: nah, http://bgp.he.net/AS3366815:00
bookiebothttp://goo.gl/twBRwD - 403 Forbidden15:00
cmaloneyit's also the time when joDee looked at the phone (which didn't ring because the power was cut), picked it up, and surprised both her and her friend that was calling.15:00
cmaloneykrondor: yeah, but they were slammed.15:00
jrwrengreg-g: i was looking up: http://bgp.he.net/AS14907  to see you15:00
bookiebothttp://goo.gl/t5UX4H - 403 Forbidden15:00
jrwrenhuh, i wonder why it says 403. I wonder if they filter based on UA15:01
brousch__cmaloney: Don't drag all of Michigan into your little blackout. We were fine over here.15:01
greg-ghttp://bgp.he.net/AS1490715:01
bookiebothttp://goo.gl/t5UX4H - 403 Forbidden15:01
greg-gyeah15:01
krondorbrousch__: i was fine once I drove 30 mins to grand blanc too... mmm delicious AC and gas for my car15:02
jrwrenlol, yup, curl -A 'Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0'   and I can hit that page.15:02
cmaloneySad thing is we were waiting for a blackout so we could do some stargazing15:02
jrwrenwhose bot is that?15:02
cmaloneythat's when we realized the fucking Wayne County fair was happening a few blocks away15:02
cmaloneyjrwren: mrgoodcat's bot15:02
cmaloneyso we had floodlights and hte sound of tractor pulls all night15:03
cmaloneyfuckers.15:03
jrwrenalright, I'm AFK for at least the day, time for real vacation.15:03
cmaloneyjrwren: Have fun!15:03
greg-ggetting real sick of this barking dog next door15:34
greg-gtoo bad I'm on calls all morning and can't go to the coffee shop15:34
greg-gor back to that bar... man they had a good rueben15:34
cmaloneyMmm, reuben15:44
brousch__Reubens would be awesome if they weren't covered in decomposing cabbage15:46
greg-gbrousch__: I'm german. I love that stuff.15:49
greg-gwe make our own, actually15:49
cmaloneyNothing wrong with decomposing cabbage16:22
StoneTablerick_h_: Speedy response to that bug. Thanks! :D16:48
rick_h_StoneTable: oh hey, how goes?16:48
rick_h_StoneTable: heh, sorry I floated by the FF note at first16:48
StoneTablePretty good, thanks. You?16:48
StoneTableNo worries!16:49
rick_h_party party16:49
rick_h_StoneTable: what's got you poking at the GUI?16:50
rick_h_doing anything fun?16:50
StoneTableInterview prep. Sent my resume in last week, and setting up a time to talk to someone over there.16:51
rick_h_oh cool, who you talking to?16:51
StoneTableAntonio Rosales16:51
rick_h_oh awesome, great guy16:51
StoneTableGood to hear. I'm looking forward to that chat.16:52
rick_h_cool, let me know if you've got any questions16:53
StoneTableAnd the juju/Charms stuff is pretty slick, so I'm really excited to play with that more. I wish I'd had that 8 or so years ago.16:53
StoneTableI sure will, thanks!16:53
greg-gcmaloney: wow, a 16EUR/$800 USD cd?19:36
greg-gcmaloney: re the stoner rock band you linked before19:36
cmaloneygreg-g: Eh?19:51
greg-gthat 420 whatever band's cd was 16 Euros19:51
cmaloneyIf I hit "buy it now" I get 21.2419:52
cmaloney21.66 rather19:52
cmaloneyGranted I don't think four tracks is 21.66. :)19:52
greg-gsorry, it was a bad "OMG, USD is so low compared to EUR" joke19:52
cmaloneyYes, yes it was. ;)19:52
cmaloneyThough I've seen folks post albums that were ridiculous in price.19:52
cmaloneylike $8 a track or some shit.19:53
cmaloneyActually I think the prime offender that I was thinking of lowered his prices on his music after I told him he was... ahem... high. :)19:57
greg-g:)19:59
greg-gwas it also stoner rock?19:59
greg-gcmaloney: btw, I'm liking http://juddmadden.bandcamp.com/album/everything-in-waves, it's CC BY-NC, you should use it for OMC20:05
bookiebothttp://goo.gl/Gz89Dh - Judd Madden20:05
cmaloneygreg-g: It's on my list. :)20:37
greg-git'd be a long episode. He seems to like 8+ minute songs20:38
cmaloneyYeah, he doesn't write short songs.20:38
cmaloneyIIRC it's instrumental20:39
cmaloneyso perfect for instrumetalcast.20:39
greg-gyeppers20:41
=== greg-g_ is now known as greg-g
cmaloneyOh, btw: the person who said that the Aereo case might have an impact on cloud broadcasting for personal use:21:01
cmaloneyhttp://arstechnica.com/tech-policy/2014/06/in-aereos-wake-fox-targets-dishs-tv-streaming-service/21:01
bookiebothttp://goo.gl/BbOJD7 - In Aereo’s wake, Fox targets Dish’s TV streaming service | Ars Technica21:02
cmaloneyWe're going to need some more fires, because there aren't enough to handle the "dying in" load.21:02
greg-gbad article, but interesting: http://www.zdnet.com/blog/facebook/facebook-passwords-are-not-case-sensitive-update/361222:40
bookiebothttp://goo.gl/mk6YT - Facebook passwords are not case sensitive (update) | ZDNet22:40
greg-gbasically, FB accepts 3 forms of your password:22:41
greg-g1) original password as you typed it (duh)22:41
greg-g2) original password with a capitalized first letter, in case your mobile device does that annoying thing22:41
greg-g3) your password with the case flipped on all letters, in case you have caps lock on22:41
greg-ghorrible horrible horrible reporting from ZDnet, of course, cuz I still have no idea why they're still around22:42
gamerchick02wow yeah i don't know how they've stayed in business either22:51
gamerchick02and hello from windows 8 *ducks and covers*22:52

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!