| n0p_ | damn, just got bit by a combination of `unattended-upgrades`, a prod server missing a pin, and http://www.ubuntu.com/usn/usn-2881-1/ | 07:03 |
|---|---|---|
| n0p_ | unattended-upgrades just restarted a prod mysql instance ;_; | 07:03 |
| === n0p_ is now known as n0p | ||
| n0p | (takes 20 min to restart a mysql server with our load) | 07:05 |
| cscheib_ | n0p: ouch | 11:59 |
| cmaloney | n0p: Ugh | 13:37 |
| rick_h_ | good news is it's back up? :) | 13:50 |
| jrwren | wow, 20min sql server restarts?!? wtf? what would cause that? Locks? | 13:52 |
| n0p | jrwren: even on a clean shutdown, it still needs to look at every db/table for some strange reason | 14:35 |
| n0p | mind you we have thousands of DBs each with >hundred tables | 14:36 |
| jrwren | with many TPS? | 14:37 |
| jrwren | zomg, I just asked for TPS report. | 14:37 |
| n0p | NR reports ~5k/queries per second, not sure how many are writes | 14:39 |
| jrwren | still, that is enough to make slowness understandable. | 14:42 |
| jrwren | sounds like a fun problem to solve! :) | 14:42 |
| n0p | it's pretty smooth right now as long as we move traffic *before* a db restart | 14:42 |
| n0p | honestly, the system fails gracefully, it's just annoying to shuffle in the middle of the night | 14:43 |
| cmaloney | blergh | 16:47 |
| rick_h_ | that good eh? | 16:48 |
| cmaloney | It's fun to play with production systems that are akin to magic boxes that nobody wants to offend. | 16:51 |
| cmaloney | Lest they incur wrath and stop performing. | 16:52 |
| rick_h_ | hah | 16:52 |
| rick_h_ | yea, the whole 'production is not like anything else' is the 'someone please kill me' of the world | 16:52 |
| rick_h_ | which jrwren knows even we deal with :P | 16:52 |
| cmaloney | Well, we're using a platform which folks don't quite understand | 16:53 |
| rick_h_ | always helpful | 16:53 |
| cmaloney | which apparently has a hang-up when you create two IDs that match | 16:53 |
| cmaloney | and we're not sure how to clear out data in there | 16:53 |
| cmaloney | for an eventual "go-live" | 16:54 |
| cmaloney | which from my perspective is rather broken if the answer isn't "truncate that mo-fo" | 16:54 |
| cmaloney | So I have no less than three different ways to denote that this record is indeed "test" | 16:55 |
| cmaloney | because adding a new field is easier than bending a system to our will. | 16:55 |
| rick_h_ | hah | 16:55 |
| cmaloney | and previous system used integers, so appending "test" does bad things. | 16:56 |
| cmaloney | prepending rather | 16:56 |
| jrwren | we are dealing with it right now exactly... even though staging is almost exactly like production... it still doesn't have teh load that production has. We almost need something to put load on staging similar to the load production has. | 16:56 |
| cmaloney | And they wonder why I listen to angry music and drink coffee | 16:56 |
| cmaloney | because I can't drink on the job. | 16:56 |
| jrwren | so really... its almost always true, production is not like anything else. | 16:56 |
| rick_h_ | jrwren: yea, interesting there though. We could 'simulate' load by just mirroring apache logs over to production with a diff charm watching that | 16:57 |
| rick_h_ | jrwren: that'd be interesting to try out sometime | 16:57 |
| rick_h_ | near real-time duplication with a url rewrite rule and curl | 16:57 |
| jrwren | rick_h_: something to watch the logs and create the same requests? yeah, that would be awesome. | 17:00 |
| jrwren | rick_h_: gets a little more tricky with http bodies for POST/PUT requests | 17:00 |
| rick_h_ | jrwren: true enough | 17:00 |
| rick_h_ | jrwren: just an interesting replay idea, you're right that it's not that simple | 17:01 |
| jrwren | rick_h_: would make a SWEET cross model charm | 17:01 |
| rick_h_ | jrwren: :) | 17:01 |
| cmaloney | https://plus.google.com/+CarlosSanchezMusic4Life/posts/J26drRFDfob | 17:02 |
| mrgoodcat | what time is chc tonight? | 17:07 |
| rick_h_ | mrgoodcat: 7pm | 17:07 |
| mrgoodcat | thanks | 17:08 |
| rick_h_ | np | 17:08 |
| mrgoodcat | think i might make it tonight | 17:08 |
| mrgoodcat | i hope | 17:08 |
| mrgoodcat | haven't been in months | 17:08 |
| rick_h_ | party | 17:08 |
| cmaloney | Sweet | 17:10 |
| rick_h_ | /join #snappy | 17:12 |
| rick_h_ | doh | 17:12 |
| cmaloney | heh | 17:12 |
| mrgoodcat | at least its just a leading space. i frequently type bash commands into my irc client | 17:13 |
| ColonelPanic001 | I hate when I do sql in the terminal | 18:06 |
| ColonelPanic001 | after having done db stuff for a while | 18:06 |
| ColonelPanic001 | instead of ls I start typing "select" and just stop and feel bad | 18:06 |
| jrwren | select * from files; | 18:07 |
| ColonelPanic001 | where the_one_i_need = 1; | 18:09 |
| jrwren | i don't pipe ls to grep very often | 18:10 |
| mrgoodcat | i do | 18:40 |
| mrgoodcat | anybody know how you would disable usb except for keyboards? | 18:41 |
| n0p | s/keyboards/keyboard/ + superglue :-P | 19:07 |
| n0p | mrgoodcat: https://github.com/dkopecek/usbguard ? no recommondation, just googled | 19:08 |
| n0p | found via query: https://www.google.com/search?q=whitelist%20usb%20devices%20with%20dbus | 19:10 |
| cmaloney | mrgoodcat: I'm curious what you're attempting? :) | 19:34 |
| cmaloney | One theory: Make it so the USB key can't be removed once inserted | 19:35 |
| cmaloney | eg: some physical clamp. :) | 19:36 |
| mrgoodcat | cmaloney: http://grizzhacks.com | 19:38 |
| jrwren | mrgoodcat: modern windows kernels have group policy which can do it. are you asking on linux? | 19:38 |
| mrgoodcat | one project idea was to make a door lock that uses yubikey as the key | 19:39 |
| mrgoodcat | but the problem is that USB isn't really somethign we want to expose to the outside world | 19:39 |
| mrgoodcat | jrwren: yea would likely run on a raspberry pi zero | 19:39 |
| jrwren | mrgoodcat: remove all usb kernel modules except hsb_hid | 19:39 |
| jrwren | *usb_hid | 19:39 |
| jrwren | that should get you to KB/Mouse only | 19:39 |
| mrgoodcat | i guess that's reasonable | 19:39 |
| mrgoodcat | i was thinking there was probably a way to do it with modprobe but removing the drivers sounds simpler | 19:40 |
| cmaloney | Not sure that would prevent an attack though | 19:42 |
| cmaloney | just reduce the overall footprint | 19:42 |
| mrgoodcat | right because you still have to prevent problems that can be caused with the keyboard | 19:43 |
| mrgoodcat | catching signals is the obvious thing | 19:43 |
| mrgoodcat | but you have to make sure there's no way to shell out | 19:43 |
| jrwren | you could write some dbus module to bind first inserted kb and ignore all other usb ids | 19:44 |
| mrgoodcat | run the program under a user with no shell is also obvious | 19:44 |
| mrgoodcat | the other idea was to make the brain an arduino and just implement only the functionality we need | 19:46 |
| mrgoodcat | greatly reduced attack surface | 19:46 |
| mrgoodcat | also greatly improved lifespan on the battery backup in case of power failure | 19:48 |
| cmaloney | Yeah, that's a better approach | 19:49 |
| cmaloney | have the Arduino communicate with the RPi | 19:49 |
| cmaloney | That way the Arduino presents a flattened surface and communicates in a secure way with the RPi | 19:49 |
| mrgoodcat | well it would elminate the rpi | 19:51 |
| mrgoodcat | once you have an arduino the rpi doesn't have a need | 19:51 |
| cmaloney | Au contraire | 20:03 |
| cmaloney | the RPi serves the internet connectivity need. ;) | 20:03 |
| cmaloney | because having your door lock as a hacking target is the goal. ;) | 20:04 |
| jrwren | i never grocked the whole paring arduino and rpi | 20:04 |
| jrwren | i'd always use one | 20:05 |
| jrwren | if I need inet, i'd use an rpi and no arduino | 20:05 |
| mrgoodcat | cmaloney: ha that does appear to be the goal | 20:06 |
| cmaloney | Today is when I wish I wasn't afraid of clowns so I could have joined the circus. | 20:15 |
| cmaloney | jrwren: Arduino is great for analog data entry | 20:15 |
| cmaloney | and certain shields work better on ARduino | 20:15 |
| jrwren | isn't rpi GPIO good for analog? | 20:15 |
| cmaloney | so it can do the data collection and send the input to the RPi proper | 20:16 |
| jrwren | ah, so part availability, that makes sense | 20:16 |
| mrgoodcat | and the rpi is easier to actually process the data on | 20:25 |
| mrgoodcat | for this particular application though i think the arduino will be sufficient | 20:26 |
| cmaloney | But it makes it less interesting. :) | 20:35 |
| mrgoodcat | i'd argue that it makes it more interesting | 20:35 |
| jrwren | i can imagine processing data on an arduino being very limited | 20:46 |
| mrgoodcat | eh well obviously you're memory limited | 20:52 |
| mrgoodcat | but a yubikey isn't exactly a ton of data | 20:52 |
| mrgoodcat | and you have to write everything in c afaik | 20:52 |
| mrgoodcat | and you want to watch out for external dependencies in libraries and whatnot | 20:53 |
| mrgoodcat | the real problem is data storage though. arduino doesn't have persistent storage for programs afaik | 20:53 |
| jrwren | slow clock is what I was thinking. | 20:59 |
| mrgoodcat | oh like it would process too slowly | 21:00 |
| mrgoodcat | I don't see that being a problem with aes (the encryption used by the yubikey) | 21:02 |
| cmaloney | AES is supposed to be gentle on the CPU | 21:19 |
| cmaloney | at least it shows up a lot in hardware-based crypto | 21:19 |
| jrwren | for some definition of gentle, but aren't a lot of arduinos like 1Mhz? | 21:20 |
| jrwren | 1Mhz 8bit at that, so to do AES with it its gonna take many clocks to do 256bit math | 21:21 |
| mrgoodcat | 128 bit key | 21:21 |
| mrgoodcat | for yubikey | 21:21 |
| mrgoodcat | 32 bytes of encrypted data | 21:22 |
| mrgoodcat | the crypto++ library requires 16 cycles per byte plus 1041 to set up the key | 21:22 |
| mrgoodcat | so 512 cycles for the decryption | 21:23 |
| mrgoodcat | 1553 total including key setup | 21:23 |
| mrgoodcat | obviously that's on a 64 bit system though so it's easier to do the math | 21:23 |
| mrgoodcat | as long as the decryption time is under a quarter of a second though i doubt it would be bothersome | 21:24 |
| cmaloney | https://github.com/DavyLandman/AESLib | 21:40 |
| cmaloney | Not sure if yubikey is 128bit | 21:41 |
| cmaloney | There's another library that will handle 128 - 256bit | 21:42 |
| cmaloney | http://forum.arduino.cc/index.php?topic=88890.0 | 21:42 |
| jrwren | cool | 22:00 |
| mrgoodcat | yubikey is 128 | 23:05 |
| mrgoodcat | already got that cloned down to play with heh | 23:05 |
| mrgoodcat | fun fact of the day: sending a string to an arduino to be decrypted and reading the result back is in fact _not_ the fastest way to decrypt aes encrypted data | 23:06 |
| mrgoodcat | it's actually not as slow as you might think though | 23:07 |
| gamerchick02 | cmaloney, guess who just got a pebble time round? :-D | 23:48 |
| gamerchick02 | and crap he's not here. oh well. | 23:49 |
| gamerchick02 | rick_h_ do you rock a pebble? | 23:51 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!