n0p_damn, just got bit by a combination of `unattended-upgrades`, a prod server missing a pin, and http://www.ubuntu.com/usn/usn-2881-1/07:03
n0p_unattended-upgrades just restarted a prod mysql instance ;_;07:03
=== n0p_ is now known as n0p
n0p(takes 20 min to restart a mysql server with our load)07:05
cscheib_n0p: ouch11:59
cmaloneyn0p: Ugh13:37
rick_h_good news is it's back up? :)13:50
jrwrenwow, 20min sql server restarts?!? wtf? what would cause that? Locks?13:52
n0pjrwren: even on a clean shutdown, it still needs to look at every db/table for some strange reason14:35
n0pmind you we have thousands of DBs each with >hundred tables14:36
jrwrenwith many TPS?14:37
jrwrenzomg, I just asked for TPS report.14:37
n0pNR reports ~5k/queries per second, not sure how many are writes14:39
jrwrenstill, that is enough to make slowness understandable.14:42
jrwrensounds like a fun problem to solve! :)14:42
n0pit's pretty smooth right now as long as we move traffic *before* a db restart14:42
n0phonestly, the system fails gracefully, it's just annoying to shuffle in the middle of the night14:43
rick_h_that good eh?16:48
cmaloneyIt's fun to play with production systems that are akin to magic boxes that nobody wants to offend.16:51
cmaloneyLest they incur wrath and stop performing.16:52
rick_h_yea, the whole 'production is not like anything else' is the 'someone please kill me' of the world16:52
rick_h_which jrwren knows even we deal with :P16:52
cmaloneyWell, we're using a platform which folks don't quite understand16:53
rick_h_always helpful16:53
cmaloneywhich apparently has a hang-up when you create two IDs that match16:53
cmaloneyand we're not sure how to clear out data in there16:53
cmaloneyfor an eventual "go-live"16:54
cmaloneywhich from my perspective is rather broken if the answer isn't "truncate that mo-fo"16:54
cmaloneySo I have no less than three different ways to denote that this record is indeed "test"16:55
cmaloneybecause adding a new field is easier than bending a system to our will.16:55
cmaloneyand previous system used integers, so appending "test" does bad things.16:56
cmaloneyprepending rather16:56
jrwrenwe are dealing with it right now exactly... even though staging is almost exactly like production... it still doesn't have teh load that production has. We almost need something to put load on staging similar to the load production has.16:56
cmaloneyAnd they wonder why I listen to angry music and drink coffee16:56
cmaloneybecause I can't drink on the job.16:56
jrwrenso really... its almost always true, production is not like anything else.16:56
rick_h_jrwren: yea, interesting there though. We could 'simulate' load by just mirroring apache logs over to production with a diff charm watching that16:57
rick_h_jrwren: that'd be interesting to try out sometime16:57
rick_h_near real-time duplication with a url rewrite rule and curl16:57
jrwrenrick_h_: something to watch the logs and create the same requests?  yeah, that would be awesome.17:00
jrwrenrick_h_: gets a little more tricky with http bodies for POST/PUT requests17:00
rick_h_jrwren: true enough17:00
rick_h_jrwren: just an interesting replay idea, you're right that it's not that simple17:01
jrwrenrick_h_: would make a SWEET cross model charm17:01
rick_h_jrwren: :)17:01
mrgoodcatwhat time is chc tonight?17:07
rick_h_mrgoodcat: 7pm17:07
mrgoodcatthink i might make it tonight17:08
mrgoodcati hope17:08
mrgoodcathaven't been in months17:08
rick_h_ /join #snappy17:12
mrgoodcatat least its just a leading space. i frequently type bash commands into my irc client17:13
ColonelPanic001I hate when I do sql in the terminal18:06
ColonelPanic001after having done db stuff for a while18:06
ColonelPanic001instead of ls I start typing "select" and just stop and feel bad18:06
jrwrenselect * from files;18:07
ColonelPanic001where the_one_i_need = 1;18:09
jrwreni don't pipe ls to grep very often18:10
mrgoodcati do18:40
mrgoodcatanybody know how you would disable usb except for keyboards?18:41
n0ps/keyboards/keyboard/ + superglue :-P19:07
n0pmrgoodcat: https://github.com/dkopecek/usbguard ? no recommondation, just googled19:08
n0pfound via query: https://www.google.com/search?q=whitelist%20usb%20devices%20with%20dbus19:10
cmaloneymrgoodcat: I'm curious what you're attempting? :)19:34
cmaloneyOne theory: Make it so the USB key can't be removed once inserted19:35
cmaloneyeg: some physical clamp. :)19:36
mrgoodcatcmaloney: http://grizzhacks.com19:38
jrwrenmrgoodcat: modern windows kernels have group policy which can do it. are you asking on linux?19:38
mrgoodcatone project idea was to make a door lock that uses yubikey as the key19:39
mrgoodcatbut the problem is that USB isn't really somethign we want to expose to the outside world19:39
mrgoodcatjrwren: yea would likely run on a raspberry pi zero19:39
jrwrenmrgoodcat: remove all usb kernel modules except hsb_hid19:39
jrwrenthat should get you to KB/Mouse only19:39
mrgoodcati guess that's reasonable19:39
mrgoodcati was thinking there was probably a way to do it with modprobe but removing the drivers sounds simpler19:40
cmaloneyNot sure that would prevent an attack though19:42
cmaloneyjust reduce the overall footprint19:42
mrgoodcatright because you still have to prevent problems that can be caused with the keyboard19:43
mrgoodcatcatching signals is the obvious thing19:43
mrgoodcatbut you have to make sure there's no way to shell out19:43
jrwrenyou could write some dbus module to bind first inserted kb and ignore all other usb ids19:44
mrgoodcatrun the program under a user with no shell is also obvious19:44
mrgoodcatthe other idea was to make the brain an arduino and just implement only the functionality we need19:46
mrgoodcatgreatly reduced attack surface19:46
mrgoodcatalso greatly improved lifespan on the battery backup in case of power failure19:48
cmaloneyYeah, that's a better approach19:49
cmaloneyhave the Arduino communicate with the RPi19:49
cmaloneyThat way the Arduino presents a flattened surface and communicates in a secure way with the RPi19:49
mrgoodcatwell it would elminate the rpi19:51
mrgoodcatonce you have an arduino the rpi doesn't have a need19:51
cmaloneyAu contraire20:03
cmaloneythe RPi serves the internet connectivity need. ;)20:03
cmaloneybecause having your door lock as a hacking target is the goal. ;)20:04
jrwreni never grocked the whole paring arduino and rpi20:04
jrwreni'd always use one20:05
jrwrenif I need inet, i'd use an rpi and no arduino20:05
mrgoodcatcmaloney: ha that does appear to be the goal20:06
cmaloneyToday is when I wish I wasn't afraid of clowns so I could have joined the circus.20:15
cmaloneyjrwren: Arduino is great for analog data entry20:15
cmaloneyand certain shields work better on ARduino20:15
jrwrenisn't rpi GPIO good for analog?20:15
cmaloneyso it can do the data collection and send the input to the RPi proper20:16
jrwrenah, so part availability, that makes sense20:16
mrgoodcatand the rpi is easier to actually process the data on20:25
mrgoodcatfor this particular application though i think the arduino will be sufficient20:26
cmaloneyBut it makes it less interesting. :)20:35
mrgoodcati'd argue that it makes it more interesting20:35
jrwreni can imagine processing data on an arduino being very limited20:46
mrgoodcateh well obviously you're memory limited20:52
mrgoodcatbut a yubikey isn't exactly a ton of data20:52
mrgoodcatand you have to write everything in c afaik20:52
mrgoodcatand you want to watch out for external dependencies in libraries and whatnot20:53
mrgoodcatthe real problem is data storage though. arduino doesn't have persistent storage for programs afaik20:53
jrwrenslow clock is what I was thinking.20:59
mrgoodcatoh like it would process too slowly21:00
mrgoodcatI don't see that being a problem with aes (the encryption used by the yubikey)21:02
cmaloneyAES is supposed to be gentle on the CPU21:19
cmaloneyat least it shows up a lot in hardware-based crypto21:19
jrwrenfor some definition of gentle, but aren't a lot of arduinos like 1Mhz?21:20
jrwren1Mhz 8bit at that, so to do AES with it its gonna take many clocks to do 256bit math21:21
mrgoodcat128 bit key21:21
mrgoodcatfor yubikey21:21
mrgoodcat32 bytes of encrypted data21:22
mrgoodcatthe crypto++ library requires 16 cycles per byte plus 1041 to set up the key21:22
mrgoodcatso 512 cycles for the decryption21:23
mrgoodcat1553 total including key setup21:23
mrgoodcatobviously that's on a 64 bit system though so it's easier to do the math21:23
mrgoodcatas long as the decryption time is under a quarter of a second though i doubt it would be bothersome21:24
cmaloneyNot sure if yubikey is 128bit21:41
cmaloneyThere's another library that will handle 128 - 256bit21:42
mrgoodcatyubikey is 12823:05
mrgoodcatalready got that cloned down to play with heh23:05
mrgoodcatfun fact of the day: sending a string to an arduino to be decrypted and reading the result back is in fact _not_ the fastest way to decrypt aes encrypted data23:06
mrgoodcatit's actually not as slow as you might think though23:07
gamerchick02cmaloney, guess who just got a pebble time round? :-D23:48
gamerchick02and crap he's not here. oh well.23:49
gamerchick02rick_h_ do you rock a pebble?23:51

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!