| sakhi | moonin | 06:17 |
|---|---|---|
| superfly | ahoy sakhi | 06:30 |
| Kilos | morning superfly and other fellas | 07:01 |
| Kilos | Maaz, coffee on | 07:02 |
| * Maaz puts the kettle on | 07:02 | |
| Maaz | Coffee's ready for Kilos! | 07:06 |
| Kilos | Maaz, ty | 07:06 |
| Maaz | You're Welcome I'm sure | 07:06 |
| sakhi | Maaz: tea on | 07:18 |
| Maaz | sakhi: Sorry... | 07:18 |
| Kilos | lol hiya sakhi | 07:18 |
| sakhi | :) only coffee | 07:19 |
| Kilos | yip | 07:19 |
| === smorar_ is now known as smorar | ||
| Kilos | hi smorar | 07:59 |
| superfly | hi Kilos | 08:22 |
| inetpro | good morning | 08:31 |
| Kilos | lo inetpro | 08:31 |
| superfly | tumbleweed: ping | 10:12 |
| tumbleweed | pong | 10:13 |
| superfly | are you a debian developer? | 10:14 |
| superfly | I recall something of the sort | 10:14 |
| tumbleweed | as of pretty recently, yes | 10:15 |
| superfly | congrats :-) | 10:15 |
| tumbleweed | thanks :) | 10:15 |
| superfly | also, I might be asking you for some help in the near future, just to forewarn you ;-) | 10:15 |
| Symmetria | heh | 11:08 |
| Symmetria | so after a hell of a lot of analysis | 11:08 |
| Symmetria | we figured out what was killing the mirror.ac.za server | 11:09 |
| Symmetria | (though we are still getting a new server in) | 11:09 |
| linuxboy | and, what was it? | 11:09 |
| Symmetria | the box was dying the moment we had a ton of rsyncs running to sync from external points, divided the syncs so that the we trigger the syncs to kick off every hour for a different mirror, so the sync processes are now divided over the day to 24 different times and where we have more than 24 individual syncs or something has to sync more than once we just put 2 or 3 rsyncs kicking off at the same time | 11:10 |
| Symmetria | and the box has been stable since then | 11:10 |
| Symmetria | we were picking up a lot of wierd kernel panic errors specifically to do with rsync | 11:10 |
| Symmetria | very strange | 11:10 |
| linuxboy | why would you update all your mirrors at the same tiem? | 11:11 |
| froztbyte | why not? | 11:12 |
| linuxboy | load | 11:13 |
| linuxboy | bandwidth | 11:13 |
| linuxboy | CPU | 11:13 |
| linuxboy | IO | 11:13 |
| froztbyte | and if you have enough of all those? | 11:16 |
| Symmetria | linuxbox originally it was about scheduling the updates to all happen after hours | 11:25 |
| Symmetria | legacy configs | 11:25 |
| tumbleweed | Symmetria: why run our mirrors with a scheduling engine that runs 3 syncs at a time | 11:25 |
| Symmetria | we switched it to update any time because we have the bandwidth now to do it whenever | 11:25 |
| tumbleweed | (given our bandwidth, 3 at a time makes sense, but with yours, 1/2 is probably more sane) | 11:25 |
| tumbleweed | that way they also lock, so if for some reason (i.e. network issues) one day's syncs run over, they won't be stomped upon | 11:26 |
| Symmetria | yeah we're looking at that as well, though I think a lot of these issues will be solved by the new system thats been ordered as well | 11:26 |
| tumbleweed | Symmetria: btw which ftpd do you use? | 11:27 |
| Symmetria | the new mirror.ac.za client facing system will allow us to move the current box into a backend sync system and segregate the two | 11:27 |
| Symmetria | tumble, pure-ftpd | 11:27 |
| tumbleweed | vsftpd makes our kernel use an insane amount of memory when it's under heavy load | 11:27 |
| Symmetria | tumbleweed heh, did you see the specs of the new client facing server we have ordered? | 11:27 |
| tumbleweed | yeah that makes sense (split) if you have shared storage | 11:27 |
| * tumbleweed can imagine, and doesn't want to :) | 11:27 | |
| Symmetria | that new server is... mind blowing :) | 11:28 |
| tumbleweed | also, if you have some free rackspace and don't mind some bandwidth use, debian is looking for a security.debian.org in ZA (but it'd have to be a box admined by them) | 11:28 |
| Symmetria | heh its got 2 x 6 core 3.3ghz xeon cpus in it (so 12 cores total), 64gig of 1333mhz error correcting ram, high speed 600gig SAS disks in mirror config for the base operating system, 5 x PCI-E 16x slots for the Perc 5e external SAS system controllers to drive the disk arrays | 11:29 |
| tumbleweed | err not in ZA, but in africa in genereal... | 11:29 |
| Symmetria | we have the rack space, we'd require access to the box but they could admin it (we don't allow boxes on the backbone that we have zero access to for obvious reasons) | 11:29 |
| Symmetria | you can tell them to email me | 11:29 |
| Symmetria | heh right now I'm trying to free up space on diskspace3 by moving stuff to the new SAN, but *SNORE* the amount of data I'm moving is taking a long long time | 11:31 |
| tumbleweed | well, I can bounce you the request. I assume they'd be looking for hardware donations too (but those can come from elsewhere...) | 11:31 |
| Symmetria | moved debian onto the new disk array which freed up 500gig but 500gig will last days the way some of the other mirrors are growing | 11:31 |
| Symmetria | hardware is not a problem, I have several servers we could probably throw at it | 11:31 |
| Symmetria | dependant on how much disk space they are looking for | 11:31 |
| * Symmetria watches as a 7.5 terabyte single mirror copies from one san to the other | 11:32 | |
| Symmetria | heh, its interesting, the mirrors which provide us the most grief are never the open source stuff | 11:32 |
| Symmetria | its the scientific data mirrors | 11:32 |
| Symmetria | (in this case the human genome project mirror, which is *INSANELY* huge) | 11:33 |
| Symmetria | heh and when they do an update release, you're looking at 300+gig churn (on an almost daily basis) | 11:33 |
| Symmetria | gbdb/mm9/multiz30way/phastCons30wayPlacental.wib | 11:34 |
| Symmetria | 1914580468 100% 60.45MB/s 0:00:30 (xfer#5134, to-check=206126/231289) | 11:34 |
| tumbleweed | the open source stuff has had a while to learn how to be mirrored effectively | 11:34 |
| Symmetria | heh look at the size of those files | 11:34 |
| Symmetria | and they are all that size, well, some of them are bigger | 11:34 |
| Symmetria | file sizes ranging between 2 and 6 gigs a file | 11:34 |
| Symmetria | 7.9T /diskspace3/hg/ | 11:35 |
| Symmetria | heh | 11:35 |
| Symmetria | gawd damn | 11:35 |
| tumbleweed | drubin: weechat keeps building sucessfully. I'm tempted to leave the team :) | 11:48 |
| Symmetria | tumbleweed btw | 11:48 |
| Symmetria | if you ever want huge storage space | 11:48 |
| Symmetria | for cheap (cheap by the standards of highly reliable storage) | 11:49 |
| Symmetria | the dell MD1000 arrays are things of beauty | 11:49 |
| Symmetria | heh, 28 terabyte active array on eSAS for like, 80 grand | 11:49 |
| Symmetria | and they are *FAST* | 11:49 |
| tumbleweed | Symmetria: LEG doesn't have that kind of cash :/ (we make do with software raid, and partial mirroring) | 11:54 |
| Symmetria | *nod* we spend the kinda money as do on mirror because its becoming more and more a critical service for academic data sets | 11:55 |
| Symmetria | it used to be a lot about saving bandwidth on the opensource stuff, and we will obviously continue to do that and treat it with the same priority as everything on the mirror | 11:56 |
| Symmetria | but the academic data sets are becoming more and more critical, and getting bigger and bigger | 11:56 |
| tumbleweed | it's actually a pity that there aren't decent raid solutions in the 10-20k range | 11:56 |
| Symmetria | tumbleweed just one piece of advice | 11:57 |
| tumbleweed | open source is running into the same issues. Debian is having to deal with academic datasets appearing in packages | 11:57 |
| Symmetria | no matter what anyone tells you, stay the hell away from anything iSCSI based | 11:57 |
| Symmetria | we've thrown test after test at it, as have a lot of other people I know | 11:58 |
| Symmetria | and the performance BLOWS | 11:58 |
| tumbleweed | hah | 11:58 |
| Symmetria | even with hardware offload iSCSI controllers (and I had some pretty damn expensive iSCSI hardware offloads) | 11:58 |
| tumbleweed | ever played with ATAoE? it looks interesting but not that much hardware for it | 11:58 |
| Symmetria | heh, we were testing with hardware offload cards that cost 10 grand a piece an the performance still blew | 11:58 |
| Symmetria | tumbleweed, heh, havent played with it, I generally stick to the philosophy that ethernet was not meant as a disk carrying technology :P | 11:59 |
| Symmetria | and people should stop trying to be cheap :P cause it doesnt work | 11:59 |
| tumbleweed | sounds reasonable :) | 11:59 |
| Symmetria | you wanna throw shit like that over a network, use fiber channel | 11:59 |
| Symmetria | it was what it was designed for | 11:59 |
| Symmetria | LOL, interestingly enough, while working on our DWDM units the other day, I discovered you can actually turn up 10G fiber channel lambdas | 12:00 |
| Symmetria | directly on the DWDM units | 12:00 |
| Symmetria | only a matter of time before I start offering that one to the clients | 12:00 |
| drubin | Maaz: weather for cape town | 15:02 |
| Maaz | drubin: In Cape Town, South Africa at 4:00 PM SAST on February 08, 2011: 24°C; Humidity: 61%; Wind: South at 41 km/h; Conditions: Partly Cloudy; Sunrise/set: 6:13 AM SAST/7:45 PM SAST; Moonrise/set: 10:58 AM SAST/10:09 PM SAST | 15:02 |
| Morganvd | Kilos!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! | 16:11 |
| Morganvd | HI | 16:12 |
| Symmetria | heh | 16:16 |
| Symmetria | wow, vendors can be real dodgy | 16:16 |
| Symmetria | we did some testing on a hardware device, produced a report, the vendor saw the report, phones me, and begs me not to show that report to any other south african isp | 16:17 |
| Kilos | hiya Morganvd sorry i am on p3 with no spound and trying to fix my 80g on p4 | 16:17 |
| Morganvd | lol | 16:17 |
| Symmetria | Im like, errr, sorry, we're an open book, you dont like me disclosing what we find, dont ask me to test | 16:17 |
| Morganvd | did you sign a non disclosure? | 16:17 |
| Symmetria | Morganvd hell no, if a vendor asks me to evaluate his product, we're transparent | 16:18 |
| Symmetria | anything I find in testing I will disclose | 16:18 |
| Symmetria | and if they dont like that, I wont test their hardware and I wont buy their hardware | 16:18 |
| Morganvd | well then you in your fullest of rights | 16:18 |
| Morganvd | Symmetria: if you have a few min later | 16:19 |
| Morganvd | i would like to ask your advice about something | 16:19 |
| Symmetria | ask away | 16:24 |
| Symmetria | I got a second now | 16:24 |
| Symmetria | just trying to plan a lab test and see what results I EXPECT to get bfore I start | 16:24 |
| linuxboy | Symmetria: URL to test results plz | 17:07 |
| Symmetria | heh linuxboy not publishign them on the web, but am prepared to discuss what we found the device couldnt do when asked by an ISP or anyone else | 17:11 |
| Symmetria | *shrug* very basically, we wanted to terminate pseudowires on svi's (switched virtual interfaces), the device doesnt seem to be able to do that, and unless we screwed up in our lab tests and they have some other way to do it, that makes the device unusable in the network | 17:12 |
| Owkkuri | superfly: http://www.omgubuntu.co.uk/2011/02/android-apps-coming-to-non-android-phones-maybe-even-ubuntu/ | 18:23 |
| Symmetria | *HRM* | 19:49 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!