| jamespage | coreycb: hey - noted a few bumps to debhelper compat level 10 from debian - that's going to be awkward to backport without a backport of debhelper >= 10 first | 00:39 |
|---|---|---|
| jamespage | there is on in backports - we might be able to use that | 00:40 |
| jamespage | bradm: are you charm deployed? or just using the packages? | 00:40 |
| bradm | jamespage: charm deployed | 00:41 |
| jamespage | bradm: ok so two options on each service | 00:41 |
| jamespage | bradm: you can set "openstack-origin" to the new UCA pocket, this will perform a parallel upgrade on all units of the service | 00:41 |
| jamespage | or you can toggle action-managed-upgrade to True via config; after that setting the config also needs an execution of the openstack-upgrade action on each unit | 00:42 |
| jamespage | intent is that you can do things unit by unit | 00:42 |
| bradm | jamespage: that could be fun with things that require a db schema change, doing it unit by unit | 00:44 |
| jamespage | bradm: the lead unit will take care of the db migration so you should do that one first | 00:44 |
| bradm | jamespage: are there any recommendations about service orders to do the upgrades in? | 00:45 |
| bradm | I guess keystone first, similar to how the charm upgrades are done | 00:45 |
| jamespage | yep | 00:46 |
| bradm | ok, and this is liberty -> mitaka, so it seems I need to upgrade ceph to jewel first too | 00:46 |
| bradm | jamespage: there's no rollback is there? not expecting to need it, but have to ask the question | 00:49 |
| jamespage | bradm: no | 00:54 |
| bradm | jamespage: excellent, good to know. | 00:55 |
| jamespage | bradm: actually ceph does things in a different way - if you change the source, it will perform a managed upgrade across the units | 00:55 |
| bradm | jamespage: is any of this documented anywhere? all the cloud archive page says about it is to 'upgrade the packags' | 00:56 |
| bradm | packages. | 00:56 |
| ReedK0 | is anyone here good at repairing grub? | 00:57 |
| jamespage | bradm: in the charm readme's; there is a pending task to write a 'how to upgrade' section for the charm deployment guide | 01:05 |
| jamespage | bradm: https://docs.openstack.org/charm-deployment-guide/latest/ | 01:06 |
| bradm | jamespage: aha, cool. | 01:08 |
| drab | Epx998: a war? | 01:49 |
| Epx998 | work around | 01:54 |
| Epx998 | drab: ubuntu doesnt like to an interface not followed with a 0 during network installs | 01:55 |
| drab | I see | 01:55 |
| drab | anyway, I had that problem on some supermicro servers with intel quad nics | 01:55 |
| Epx998 | yeah | 01:55 |
| Epx998 | exactly what we are seeing, onboard nics 1gb, plus offboard intel x550 10gb | 01:56 |
| drab | war for me was to disable the [un]predictable interaface naming so that stuff was once again called eth0 etc, then it worked just fine passing the interface name to select via preseed/kernel param | 01:56 |
| Epx998 | we want to use 10gb out the gate, but d-i netboot does not like | 01:56 |
| Epx998 | i suggested that to the guys trying to provision, they said it was too much work. so i creasted a new tftp entry where i specified eth4, the installer sees it, then tries eth0 regardless and falls on its face | 01:57 |
| drab | how is one kernel parameter too much work? all you need is biosdevsomethingiforgot=0 | 01:58 |
| drab | anyway | 01:58 |
| drab | that's all I've got | 01:59 |
| drab | altho if you already have eth0 and 4 that doesn't sound like your problem | 01:59 |
| drab | sarnold: fwiw someone suggested a good way to do it with nginx using "geo" | 02:02 |
| drab | http://nginx.org/en/docs/http/ngx_http_geo_module.html | 02:02 |
| drab | so I can use geo to create a "rolling" variable and if the ip matches whichever sets of clients I wanna rollout I just catch them there and then use map to use a different pool | 02:04 |
| drab | http://prabu-lk.blogspot.com/2017/03/select-different-backend-pools-based-on.html | 02:04 |
| sarnold | drab: heh, neither one -quite- describes what's going on.. | 02:06 |
| sarnold | drab: still it looks like it can do the job | 02:06 |
| drab | I guess I just have to get over this idea that nginx is mostly a web server thingie... and tbh most of the verbiage on the site talks about http | 02:07 |
| drab | which makes me somewhat uneasy about using it for generic tcp protocols | 02:07 |
| drab | also this nginx plus thing isn't helping... half of the times I'm confused if it's payware feature or not... | 02:07 |
| === an3k is now known as an3k^ | ||
| === an3k^ is now known as an3k | ||
| jamespage | coreycb: I think we're going to need to bump lescina to xenial | 04:03 |
| jamespage | coreycb: the lack of support for conditional deps is created patching requirements which are unrelated to the uca | 04:03 |
| lordievader | Good morning | 07:13 |
| SmokinGrunts | morn' lordievader | 07:46 |
| lordievader | Hey SmokinGrunts | 07:47 |
| lordievader | How are you doing? | 07:47 |
| SmokinGrunts | pretty good. I got vibes from fielding a successful ##networking issue, and so I feel good | 07:47 |
| * SmokinGrunts toots the ole ego horn | 07:48 | |
| SmokinGrunts | and you? | 07:48 |
| === SmokinGrunts is now known as SG_Sleeps | ||
| === JanC is now known as Guest4348 | ||
| === JanC_ is now known as JanC | ||
| lordievader | Doing good here. | 08:21 |
| lordievader | Is ##networking an interesting channel? | 08:21 |
| adac | Hi there! Where does iptables-persistent save it's rules? How can I clean them up? | 08:52 |
| lordievader | adac: According to [1] in `/etc/iptables/rules.v{4,6}`. [1] http://www.microhowto.info/howto/make_the_configuration_of_iptables_persistent_on_debian.html#idp21024 | 08:54 |
| adac | lordievader, ok thanks! So then I can simply delete it's content to reset it? I currently do run this script to clean now all up (reset the iptables firewall) https://gist.github.com/anonymous/cc01da7ccd09e292fb44e468e656163e when I run "iptables -L" then I have no more output. But when I print iptables-save -c I still get this output: | 08:59 |
| adac | https://gist.github.com/anonymous/fe479f70643d0578a40c9f7d1adb8194 | 08:59 |
| adac | I'm wondering if my iptables now is really cleaned fully or not? :) | 08:59 |
| lordievader | No you don't want to blindly delete the content. | 09:00 |
| lordievader | The commands you posted do flush the entire iptables. | 09:01 |
| lordievader | It seems you (or some program) made CATTLE_* chains. | 09:01 |
| lordievader | `sudo iptables -vnL` should show these. | 09:02 |
| adac | lordievader, yes that was my intention to completely flush it. This CATTLE_* Chains come from rancher docker container who creates them. I was wondering why these are not deleted with my script I posted before? | 09:05 |
| adac | Maybe they are immediately re-generated | 09:05 |
| adac | by rancher | 09:05 |
| adac | I try to shut down rancher and the flush all again | 09:05 |
| adac | iptables -vnL does also not show them. No clue why actually | 09:06 |
| lordievader | `iptables -X` probably does not remove them if there are still rules in them. | 09:07 |
| lordievader | Hmm, if `iptables -vnL` does not show them `iptables-save` should not either. | 09:07 |
| adac | lordievader, exactly what i also tought it should no be shown anymore after I executed this script. I now stopped the docker process (so no container runs anymore and therefore no iptables rules can be created by them) and then just re-run my flush script | 09:09 |
| adac | and with iptables-save -c they are still shown :D | 09:09 |
| adac | how's tat possible? | 09:10 |
| lordievader | What happens when you explicitly delete the chains? | 09:10 |
| lordievader | Also, could you pastebin the output of `iptables -vnL`? | 09:10 |
| adac | http://lubos.rendek.org/remove-all-iptables-prerouting-nat-rules/ | 09:10 |
| adac | this is iptables -vnL after stopping docker and re-run the flush script: https://gist.github.com/anonymous/44cde2821e55d9fd2379ca71d270e3e7 | 09:11 |
| adac | and this at the same time is the output of iptables-save -c | 09:12 |
| adac | https://gist.github.com/anonymous/9f11adf6d862e3e926e0b5dd03846b96 | 09:12 |
| lordievader | Interesting | 09:15 |
| lordievader | Oh well, you can always just edit the rules. | 09:15 |
| lordievader | The saved file I mean. | 09:15 |
| adac | lordievader, https://serverfault.com/a/200642 I think this finally flushed it :D | 09:16 |
| lordievader | Yeah, that does the same but then through awk ;) | 09:17 |
| adac | hehehe | 09:17 |
| adac | great I finally have a reset and can now try again to set this up properly! | 09:17 |
| adac | Thanks for your support! | 09:17 |
| lordievader | No problem. | 09:18 |
| Jenshae | Hi all. joelio you know how you like ZFS just a little bit, right? | 11:02 |
| joelio | hey | 11:08 |
| joelio | a little yes :) | 11:08 |
| Jenshae | I did a Ubuntu Server FakeRaid on my assistant's machine to get them familiar with the OS but I am not satisfied with the boot time. I have found a little 20GB SSD in one of the machines here and reclaimed it by re-installing their main drive. However, it being so small, I want to know how to safely store applications on a ZFS pool. | 11:08 |
| Jenshae | Which would mean moving /usr /var and /etc I am guessing? | 11:08 |
| joelio | You can use it as a cache drive | 11:08 |
| joelio | (if you've done ZFS) | 11:09 |
| Jenshae | Does one of those hold the bulk and the rest are just links or a executables? | 11:09 |
| joelio | I wouldn't use fakeraid btw, mdadm | 11:09 |
| joelio | mdadm all the way if you're doing raid | 11:09 |
| Jenshae | Sec | 11:09 |
| Jenshae | So, the SSD is being installed as a lone drive. | 11:09 |
| joelio | are you using ZFS for boot? | 11:10 |
| Jenshae | Then I am going to ZFS attach the other three drives that were previously used in the FakeRaid | 11:10 |
| Jenshae | ZFS for boot? Me? :D | 11:10 |
| joelio | right, so ZFS is irrelevant here | 11:10 |
| Jenshae | SSD = / (boot) and ZFS drives = /home + (/etc /var /usr ???) | 11:11 |
| joelio | I'd ditch the fakeraid, use an alternate iso (server) to reinstall... use software raid | 11:11 |
| Jenshae | The FakeRaid is ditched. | 11:11 |
| joelio | you could potentially use the ssd, but if it dies you've lost /boot and it's not redundant | 11:12 |
| joelio | what I'd do is... | 11:12 |
| joelio | use mdadm to make a boot and small root on the drives (so it's raided) | 11:12 |
| Jenshae | It is just a desktop for their learning purposes. | 11:12 |
| joelio | and then use the remaining space on the drives to create zvols which you can use in a raidz | 11:13 |
| joelio | then use the ssd drive to act as a cache layer for zfs pools | 11:13 |
| Jenshae | You can RAID a 20GB SSD with 3x 160GB? I am hoping to make the desktop I slap on snappier and faster to load. | 11:13 |
| joelio | it sounds messy, but without doing full root ZFS that's the easiest route and will give you redundancy | 11:13 |
| joelio | no, don't use ssd in the raid | 11:13 |
| joelio | they're asymetrical in size and performance | 11:14 |
| joelio | *asymmetrical | 11:14 |
| joelio | you can use ssd as a cache layer in ZFS (or btrfs) | 11:14 |
| joelio | or if you don't want to use ZFS, bcache or EnhanceIO etc | 11:14 |
| joelio | that's what I'd do anyway :) | 11:14 |
| * Jenshae looks around the maze dazed | 11:15 | |
| joelio | also bear in mind it's not just about being I/O bound, could be CPU or whatever too | 11:15 |
| joelio | does that make sense? | 11:15 |
| lordievader | If it is running systemd, `systemd-analyze blame` may help. | 11:15 |
| Jenshae | Okay, let's start with step one. I plug in the 3x 160GB, fire up the USB with the ISO and then create a mdadm. Then I plug in the SSD and install the OS with just /boot going on the SSD? | 11:17 |
| Jenshae | I think the machine's bios is auto pushing for UEFI. | 11:17 |
| Jenshae | How does that fit in? | 11:17 |
| Jenshae | The mdadm is 3x160GB or it is 3x20GB and the Zpool goes over 3x140GB? | 11:18 |
| joelio | I'd not allocate the full 160GB to mdadm, but say 20G of it, leaving 140G on each disk for zvols | 11:19 |
| Jenshae | Okay so second part. | 11:20 |
| joelio | leave the ssd for a ZFS cache pool, add that after you've installed and created the ZFS pools | 11:20 |
| Jenshae | How do I get the mdadm to write from RAM to the hard drives? Can the SSD be plugged in straight away with the 160GBs from the start? | 11:20 |
| joelio | this is quite a bit of faff tbh, are you sure you want to go this route btw? :D | 11:20 |
| joelio | just making sure you know first and I'm not sending you down a rabbit hole | 11:21 |
| joelio | Jenshae: otherwise do a full disk mdadm raid setup | 11:22 |
| Jenshae | The way I did it at home was, I installed everything onto one 500GB drive. Then I put /.steam onto a zpool made up of 3x500GB SSHDs and my games' loading time improved. | 11:22 |
| joelio | and use the SSD cache with EnchanceIO or bcache on the md device perhaps? | 11:22 |
| joelio | what do you mean about mdadm wiriting to ram? | 11:23 |
| Jenshae | I want to give him the fastest desktop possible and then give him storage space in a pool after that. | 11:23 |
| joelio | that doesn't make sense :) | 11:23 |
| joelio | mdadm is the raid admin tool | 11:23 |
| Jenshae | When the USB is running, it is storing temp data into RAM partitions, right? | 11:23 |
| joelio | USB? | 11:23 |
| Jenshae | Including how things are partitions when installing. | 11:23 |
| Jenshae | The ISO / live USB. | 11:23 |
| joelio | USB is just for install media, ignore that | 11:24 |
| joelio | plus, don't use live - use the alternate iso | 11:24 |
| joelio | the text based / ncurses one | 11:24 |
| Jenshae | I need to use the USB to create the raid to then install into the raid, no? | 11:24 |
| joelio | like I said, create a server/alternate install image, whether you boot from CD/DVD/USB etc is irrelevant :) | 11:25 |
| Jenshae | Okay, I think I am starting to see the light. | 11:25 |
| joelio | https://help.ubuntu.com/community/Installation/SoftwareRAID | 11:25 |
| joelio | here you go :) | 11:25 |
| Jenshae | ... now where is the desktop environment installed? Does it have a specific partition? | 11:26 |
| joelio | it's all over the FS | 11:26 |
| joelio | /usr /var /etc | 11:26 |
| ikonia | /etc can't be a seperate file system | 11:26 |
| joelio | I'm not sure that's what was being asked | 11:27 |
| ikonia | thats fine, I'm just clarifying | 11:27 |
| joelio | plus is is techncailly possible btw root=/blah etc=/blah :) | 11:28 |
| joelio | well within reason | 11:28 |
| joelio | not that you should mind :) | 11:28 |
| ikonia | it's not possible | 11:29 |
| Jenshae | Can specific folders within /var and /usr be mapped to the zpool? | 11:33 |
| Jenshae | Like say GIMP is installed and is rather big, I want it to go into the zpool instead of the mdadm raid. | 11:34 |
| joelio | When you've made the ZFS, rsync the contents and update fstab | 11:37 |
| joelio | ikonia: https://power-of-linux.blogspot.co.uk/2010/03/booting-with-etc-in-separate-partition.html | 11:39 |
| joelio | as mentioned, not advisable | 11:39 |
| joelio | anything is techincally possible here, just how much effort you want to put in | 11:39 |
| ikonia | joelio: thats a bit of a cheat though isn't it | 11:41 |
| ikonia | it's not really possible - it's someone cheating | 11:41 |
| joelio | except it completely is possible as... they did it | 11:41 |
| joelio | it's not cheating, it's programming | 11:41 |
| Jenshae | Missing a third party for the golden apple story. | 11:43 |
| Jenshae | Thank you, This will be interesting and a bit surprising for them. :D | 11:50 |
| cpaelzer | coreycb: jamespage: fyi bug 1682102 - no more need to drop seccomp when backporting for UCA-queens | 12:01 |
| ubottu | bug 1682102 in libseccomp (Ubuntu Xenial) "libseccomp should support GA and HWE kernels" [High,Fix committed] https://launchpad.net/bugs/1682102 | 12:01 |
| Jenshae | ^.^ Nice one cpaelzer. Do you guys have a #social-room? | 12:08 |
| cpaelzer | if "you guys" is server folks then I think you are here :-) | 12:10 |
| cpaelzer | IMHO no reason to separate too much unless it overloads the channel | 12:10 |
| Jenshae | Okay. You do all seem rather quiet, had all the conversations? Like joelio and ikonia having a difference of opinion there. I doubt everyone here shares "dank memes" or plays / makes a game together. It seems very disassociated from each other. Drop a bit of work on launchpad with a message, pick up another bit, etc. | 12:14 |
| Jenshae | Not sure that comes across right. This channel seems like it is the "office", where is the "team building"? | 12:15 |
| cpaelzer | oh I see - that "social" - I guess we are all work addicts :-/ | 12:15 |
| cpaelzer | there will be a chan for that but I don't know oO | 12:16 |
| * Jenshae thinks I have stumbled upon a cave of abused dwarves and elves that need to be dragged out to have fun in the sun. :P | 12:19 | |
| cpaelzer | hehe | 12:20 |
| cpaelzer | white skin will become the trend again | 12:20 |
| joelio | got told this channel wasn't for fun a few years back :D | 12:21 |
| Jenshae | What about a #ubuntu-server server on Discord or would Discord's spying be too invasive? | 12:22 |
| joelio | oh sorry that was #ubuntu :) | 12:22 |
| Jenshae | Discord is handy because you can make a channel where only certain people can post things, that means a smaller group can link in their favourite or video guides they have made to inform new users without having all the "dank meme" videos dropped in there. The same applies for images / diagrams and text or links, naturally. | 12:23 |
| Jenshae | Then it does have voice also, for those who struggle to explain a new concept in text, whilst thinking aloud. | 12:24 |
| Jenshae | It can all split down in a tree from the main "ubuntu-server" | 12:25 |
| Jenshae | Finding X2GO very handy for RDPing into the archive server. | 12:28 |
| joelio | perhaps, although personally prefer IRC.. Slack and Gittr and just not IRC (they have their own shiny features, but still not IRC.. although can interface with them) | 12:40 |
| joelio | also used https://www.nomachine.com/ a bit (not open source though, but works well) | 12:41 |
| joelio | (re RDP/VNC stuff) | 12:41 |
| Jenshae | I like IRC because you can IRSSI into it, use it on an old mobile and the bandwidth / hardware usage is very small. I wonder if anyone is making a fancier client for it to display links the way Slack and Discord do? | 13:00 |
| jonfatino | Ubuntu needs to add http boot for filesystem.squashfs in casper init script | 14:42 |
| jonfatino | Debian already has this and no one these days wants to live boot from a stupid nfs server | 14:42 |
| jonfatino | https://forum.kde.org/viewtopic.php?f=309&t=136596 | 14:42 |
| jonfatino | Here is a modified caster script that supports http - https://pastebin.com/raw/V6W39XJu | 14:42 |
| drab | ikonia: sdeziel fwiw I'm settling for ldirectord + fwmark , at least as my first choice to test, have not gone through a setup yet | 14:55 |
| drab | but after figuring out how you'd do it with haproxy and nginx it's all "too shiny" and new for my taste | 14:55 |
| drab | and actually not as flexible once I learned about fwmark | 14:55 |
| drab | fwmark + ipset seems a really kickass solution since I can basically re-route entire sets of clients on the fly no restarts required with just one simple command (ipset add xxx xxx) | 14:56 |
| sdeziel | drab: good to hear that you found some possible solution | 14:56 |
| ikonia | drab: well done for making a ca;; | 14:56 |
| ikonia | call | 14:56 |
| drab | i'll setup a test host today with a few containers, ldirectord on the host and we'll see what happens, but on a whiteboard it looks sane and simple | 14:56 |
| drab | without fwmark probably haproxy or nginx would have won | 14:57 |
| ikonia | well done you | 14:59 |
| drab | yeah well, bouncing around ideas in chan was as usual very helpful | 14:59 |
| drab | beats the rubber duck :) | 14:59 |
| blackboxsw | Jenshae: just mention board games and you'll get a few of us chatting ;) | 15:30 |
| blackboxsw | Even the more "dwarvy" of us | 15:30 |
| Jenshae | blackboxsw: Go room on KGS server? :P (Go feels a bit hollow to me now that AI beat humans) | 16:30 |
| Jenshae | Have a good evening o7 | 16:31 |
| drab | in other, maybe less fun news, anybody here happens to be familiar with SAS, expanders, backplanes and raid cards? even tho I get the gist, I'm having a bit of a problem working through the nomenclature and being clear about what's what. | 16:31 |
| joelio | I've dealt with SES once or twice yea :) | 16:31 |
| joelio | drab: ^ | 16:31 |
| sdeziel | jonfatino: this would be best put on Launchpad, maybe a merge proposal to the relevant package | 16:32 |
| drab | joelio: for a starter, expanders, backplanes, raid cards and HBAs are all different things, correct? sometimes it seems that backplanes are also expanders, but that's the first thing i'm not clear about | 16:35 |
| drab | also are SAS SFF-8087 always 4-lane connectors? meaning a backplane with 2 connectors would at most only accommodate 8 drives? | 16:35 |
| drab | or if it accommodates more you'd have oversubscription? | 16:35 |
| drab | or would that be a case where an expander would come into play? and still have oversubscription tho | 16:36 |
| joelio | no idea about the specific SAS questions... but HBA's are hardware cards, which generally have a SAS connectior, which connects an enclosure to a given backplane | 16:39 |
| joelio | that can feed a single backplane with reduced bandwidht, or the enclousre can (generally) be carved up so you can have multiple SAS connections, to increase throughput (but reduce number of attached disks) | 16:40 |
| joelio | SES is generally the enclosure type | 16:41 |
| joelio | some tools exist for doing drive id notifications etc | 16:41 |
| joelio | ledctl etc | 16:41 |
| drab | any reason you keep saying SES? first time I thought it was just a typo for SAS, but now I'm doubting that | 16:41 |
| joelio | they're different things | 16:41 |
| joelio | https://en.wikipedia.org/wiki/SCSI_Enclosure_Services | 16:42 |
| joelio | SES ^^ | 16:42 |
| drab | oh, I see | 16:42 |
| joelio | https://en.wikipedia.org/wiki/SES-2_Enclosure_Management more currently | 16:42 |
| joelio | https://en.wikipedia.org/wiki/Serial_Attached_SCSI | 16:42 |
| joelio | SAS ^^ | 16:42 |
| drab | not sure this applies here then, these are all internal drives, but ifrst time I hear about SES and only quickly glanced at that wikipedia page | 16:43 |
| drab | thanks for sharing something new | 16:43 |
| joelio | oh, you mentioned enclosure, so.. :) | 16:43 |
| joelio | that's SES in my book | 16:43 |
| joelio | or expander sorry | 16:43 |
| drab | oh, I did? :) | 16:43 |
| drab | right, expander, thought that was a diff thing | 16:43 |
| drab | it seems to be a card you can add to multiplex sort of way more drives than the controller can natively drive | 16:44 |
| drab | but again, still trying to figure that out | 16:44 |
| drab | not sure that defintiion is correct | 16:44 |
| drab | also it seems the case backplanes cam have built-in expanders | 16:44 |
| joelio | HBA is the card that does the SAS connectivity (Like an LSI 9201 for example ) | 16:45 |
| joelio | https://www.scan.co.uk/products/16-port-lsi-sas-9201-16i-6gb-s-sasplussata-to-pci-express-host-bus-adapter-4x-internal-mini-sas-upto | 16:45 |
| joelio | that's internally attached SAS | 16:45 |
| joelio | but you can get external SAS that feed a SES | 16:45 |
| joelio | and slice and dice depending on how much performance you want | 16:46 |
| drab | see, on that card, how do you get 512 non raid devices? if it has 4 SF-8087 ports, which it does say, and each one can get 4 drives, that's 16, which it also mentiones | 16:46 |
| drab | where's the 512 coming from? | 16:46 |
| joelio | it's serially attached | 16:46 |
| joelio | so one port may have 48+ dries | 16:47 |
| joelio | *drives | 16:47 |
| joelio | but you can chain etc | 16:47 |
| joelio | operative word being serial ;) | 16:47 |
| joelio | so 512 in this case will be a limitation in the spec | 16:48 |
| joelio | I really doubt you'd address 512 drives from a single card like that | 16:48 |
| joelio | you could... I guess......... | 16:48 |
| joelio | there's not logical difference between an internal and external SAS port too, they're the same thing jjst in different places | 16:49 |
| joelio | hopefully makes some sense (probably not explained the best!) | 16:49 |
| drab | joelio: how would you physically serially attach even 48 drives? | 17:02 |
| drab | everything I'm reading is basically saying "if you need more than 8 drives use an expander or multiple HBAs" | 17:02 |
| drab | ok, I think I'm getting some place... | 17:10 |
| === CodeMouse92 is now known as CodeMouse92__ | ||
| drab | so it seems that: | 17:11 |
| drab | 1) an HBA or raid car connects usually through PCI to the mobo and has one or more SF8087 ports on, each of which can drive up to 4 drives | 17:11 |
| drab | 2) an expander can be connected to the HBA or raid controller, each port on the expander (again SF8087) can drive up to 4 drives. this allows you to address > 8 drives (unless somehow you get a pricey HBA with more than 2 ports, still it seems there aren't many with more than 8, so for 48 bays you still need an expander) | 17:12 |
| drab | 3) some backplanes have built in expanders with one or more SF8087 ports on it (and chipsets) connecting to one or more ports on the HBA | 17:13 |
| drab | depending on the speed of the expander, and the number of ports going to the HBA, you can end up with oversubscription | 17:14 |
| drab | here's a decent picture I found: http://img.my.csdn.net/uploads/201203/5/0_13309440980Tj9.gif | 17:14 |
| dirtycajunrice | thats not ture... | 17:14 |
| drab | I think that's it and relatively clear unless I missed something | 17:14 |
| dirtycajunrice | i just bought an H700 | 17:14 |
| drab | ok, great, what did I get wrong? | 17:14 |
| dirtycajunrice | dell PERC H700 has 2 mini sas and can be daisy chained up to 255 drives | 17:15 |
| dirtycajunrice | It cost me ~80USD on ebay | 17:15 |
| dirtycajunrice | drab, the lingo for HBA/Raid Controllers sucks. | 17:15 |
| drab | yeah that I figured :) | 17:16 |
| dirtycajunrice | if i didnt work at a datacenter i wouldnt understand either. | 17:16 |
| dirtycajunrice | But how many drives are you trying to control. | 17:16 |
| drab | dirtycajunrice: how do you physically daisy chain drives to 2 mini sas? | 17:16 |
| drab | isn't each SF8087 cable coming out with 4 sas cables? | 17:16 |
| drab | ie 8 drives | 17:16 |
| dirtycajunrice | so the way it works (normally) is that the 2 mini sas cables go to a backplane in the 2 IN ports . That backplane handles more connections and normally has OUT sas connections for daisy chaining | 17:17 |
| dirtycajunrice | if you want an example look at a Dell R510 12 Bay. or a Dell R730XD | 17:18 |
| drab | ok, right, see my 3), like I said it seems some backplanes have built in expanders | 17:18 |
| drab | so you do have your HBA going to an expander, it's just built into the backplane | 17:18 |
| qman__ | yeah, your example isn't wrong, it just isn't commonly built that way | 17:18 |
| qman__ | usually the expander is built into the backplane | 17:19 |
| drab | right | 17:19 |
| dirtycajunrice | yes what qman__ said | 17:19 |
| dirtycajunrice | almost ALWAYS | 17:19 |
| dirtycajunrice | are you against buying a rackmount ? | 17:19 |
| drab | that's ok, still, there is an expander in the mix, it's not straight HBA to disks | 17:19 |
| drab | no I'm not, but I'm trying to get my terminology and design straight before I buy anything | 17:19 |
| dirtycajunrice | drab, no thats not possible. the technology isnt designed that way | 17:19 |
| drab | I dislike purchasing "black boxes" | 17:19 |
| dirtycajunrice | if you are about to purchase, do you want a wonderful site? | 17:20 |
| drab | ebay.com ? :P | 17:20 |
| dirtycajunrice | trust me. i did white boxes.... IPMI is the way of the world | 17:20 |
| dirtycajunrice | drab, https://labgopher.com/ | 17:20 |
| dirtycajunrice | it scrapes ebay, and gives you only helpful information | 17:20 |
| qman__ | I usually ebay dell or supermicro | 17:20 |
| drab | qman__: yeah, I'm looking at a bunch of supermicros actually | 17:21 |
| dirtycajunrice | qman__, me as well. Using this site i got 2 620s, a 510, and a 420 all for under 1200 bucks | 17:21 |
| drab | x9s, the 10s seem still too expensive | 17:21 |
| qman__ | nice | 17:21 |
| dirtycajunrice | dell = hp > supermicro | 17:21 |
| dirtycajunrice | this is coming from an enterprise background. | 17:21 |
| dirtycajunrice | nothing beats iLO/iDRAC | 17:21 |
| qman__ | the problem with iDRAC in particular is that it has to have the full license | 17:22 |
| drab | I'll take that with a bag of salt if you don't mind... I've just about found any opinion and its opposite in a few days of googling... which isn't new, that's been about true for any tech I've ever looked at | 17:22 |
| qman__ | I'm not familiar with how HP's stuff works | 17:22 |
| qman__ | supermicro's stuff isn't sublicensed like that, so you get what you get | 17:22 |
| drab | and of course all opinions coming from ppl with X years of experience :) | 17:22 |
| dirtycajunrice | im personally a dell guy... but qman__ idrac express does not require a licence... | 17:23 |
| dirtycajunrice | drab, that is true. "Mileage may vary" is the key phrase | 17:23 |
| drab | righyt | 17:23 |
| qman__ | different license levels have different feature sets, and the cheapest license level's feature set is pretty lame | 17:23 |
| qman__ | at least with many models | 17:23 |
| dirtycajunrice | all it needs is console and snmp | 17:23 |
| qman__ | i.e. no console | 17:23 |
| dirtycajunrice | all the rest is just lagniappe | 17:24 |
| qman__ | supermicro IPMI, on the other hand, is just one product, they don't have different licesnes or feature sets | 17:24 |
| dirtycajunrice | its the AMD to Intel :P | 17:25 |
| dirtycajunrice | but that comes with the AMD bugs. a mileage may vary situation again | 17:25 |
| qman__ | iDRAC has plenty of bugs too | 17:25 |
| drab | dirtycajunrice: labgopher is really neat, thanks for sharing | 17:26 |
| dirtycajunrice | the only bug that affects me is the browser one. but IPMI has browser compatibility with literally every vendor i have tested | 17:26 |
| dirtycajunrice | HP doesnt like firefox... Dell doesnt like chrome.... | 17:26 |
| dirtycajunrice | etc etc | 17:26 |
| dirtycajunrice | oh and drab if you DO decide to go dell, most of the servers that are sold have idrac enterprise licence already added since they came from a working environment | 17:28 |
| dirtycajunrice | (idrac licences cant be migrated. they are bound to the machine they are installed on) | 17:28 |
| JanC | until someone decides you have to pay for a yearly license? ;) | 17:31 |
| drab | I'm sorry I started this :P | 17:32 |
| drab | but thanks for clarifying/confirming, I think I get what's what now | 17:32 |
| dirtycajunrice | haha its ok | 17:39 |
| dirtycajunrice | thats the point of IRC | 17:39 |
| dirtycajunrice | talk | 17:39 |
| dirtycajunrice | argue | 17:39 |
| dirtycajunrice | be pedantic | 17:39 |
| dirtycajunrice | its fun :P | 17:39 |
| === jancoow_ is now known as jancoow | ||
| coreycb | jamespage: is there a job that uploads cloud-archive-utils? we need a bionic version. https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/tools/+packages | 18:46 |
| jonfatino | Anyone know why I can't echo something twice in bash but only once. | 19:12 |
| jonfatino | DATE=date echo $date echo $date | 19:13 |
| jonfatino | it only echos date once :-( | 19:13 |
| TJ- | jonfatino: "DATE=date; echo $DATE $DATE" ? or do you mean "DATE=date; echo $($DATE)" ? | 19:18 |
| jonfatino | weird bash bug | 19:19 |
| jonfatino | PASSWORD=$(date +%s|sha256sum|base64|head -c 32) | 19:19 |
| jonfatino | echo $PASSWORD | 19:19 |
| jonfatino | echo $PASSWORD | 19:19 |
| jonfatino | Fixed it | 19:19 |
| nacc | jonfatino: there is a bash channel which is probably a better place to ask in the future | 19:21 |
| drab | uhm , not sure what you're running but DATE=`date` ; echo $DATE ; echo $DATE worked just fine for me | 19:21 |
| dirtycajunrice | drab, | 19:21 |
| dirtycajunrice | dont you dare | 19:21 |
| dirtycajunrice | backtick | 19:21 |
| dirtycajunrice | ever again | 19:21 |
| dirtycajunrice | XD | 19:21 |
| drab | it's the end of the season, I've heard backticks are trendy again | 19:22 |
| dirtycajunrice | rofl | 19:22 |
| sdeziel | doing "VAR=value command" makes VAR available to that command only | 19:30 |
| jamespage | coreycb: its part of the build recipe hooked up to the branch - you can add bionic and request a rebuild | 19:58 |
| coreycb | jamespage: ok i'll look for that, thx | 20:02 |
| drab | dirtycajunrice: | 21:13 |
| dirtycajunrice | drab, | 21:14 |
| drab | whups, I was gonna say, about that labgopher, there seem to be a ton of G6, I think you said you have exp with HPs... aren't G6s too old? | 21:14 |
| drab | we can't afford latest, but afaik we're at G9s, so G6s is 3 gens ago | 21:14 |
| dirtycajunrice | They are old. But old is relative to what you are doing | 21:14 |
| drab | right | 21:14 |
| dirtycajunrice | for example | 21:14 |
| dirtycajunrice | my ESX 6.5 hosts are 620s for dell | 21:15 |
| dirtycajunrice | its the oldest you can go for esxi | 21:15 |
| dirtycajunrice | but my iSCSI server is a 510 | 21:15 |
| dirtycajunrice | because why not? cheaper and more bays | 21:15 |
| dirtycajunrice | and has no requirement to be super new | 21:15 |
| drab | I don't know HP, but in the case of SM for example, X8s are still kind of popular, but you can get X9s and they have a completely diff design mobo wise allowing much faster access to PCI-E (and therefore faster disk access with a PCI SAS HBA) | 21:15 |
| dirtycajunrice | so let the server's job dictate the cost | 21:15 |
| dirtycajunrice | drab, what is the goal of the server(s) you are looking to buy | 21:16 |
| drab | so it's not really worth to buy X8s when you can get X9s for about the same price | 21:16 |
| drab | NAS + VM host | 21:16 |
| dirtycajunrice | so 2 servers? | 21:16 |
| drab | 24bays, doing nfs for homedirs and samba | 21:16 |
| dirtycajunrice | wait wait backup | 21:16 |
| dirtycajunrice | how many servers | 21:17 |
| drab | one if possible, 2 diff zfs pools, hoping to put something like a E5-25xx in it, 8 cores | 21:17 |
| dirtycajunrice | ok | 21:17 |
| dirtycajunrice | so 1 server | 21:17 |
| dirtycajunrice | 24 2.5in bays? | 21:17 |
| dirtycajunrice | because 3.5 in is in DAE territory | 21:17 |
| drab | from what I can see, 3.5, 2.5 seems too expensive | 21:17 |
| drab | what's DAE? direct attach something? | 21:17 |
| drab | DAS? | 21:17 |
| dirtycajunrice | Directly attached Expansion | 21:18 |
| drab | ok, different than a DAS? | 21:18 |
| dirtycajunrice | a DAE is a shelf you attach to expand a DAS | 21:18 |
| drab | ah, ok | 21:18 |
| dirtycajunrice | but you are multiusing your server so its foggy :P | 21:18 |
| dirtycajunrice | lemme look | 21:18 |
| dirtycajunrice | gimme 10 | 21:18 |
| drab | yeah, well, we don't have the money (it's a charity) to get multiple machines (unless it makes sense to get 2 cheaper ones, but it's often not the case) | 21:19 |
| drab | besides, the older the more generally power hungry | 21:19 |
| dirtycajunrice | i mean... | 21:19 |
| drab | not to mention that if you wanna hold on a few spare parts, like PSUs, you need twice as much | 21:19 |
| dirtycajunrice | to be honest | 21:19 |
| dirtycajunrice | 24 bays is not cheap as 1 server | 21:19 |
| dirtycajunrice | but can absolutely be affordable as 2 | 21:19 |
| dirtycajunrice | are you using enterprise drives or consumer drives | 21:20 |
| drab | true, they could prolly do with 12. the thing is, I'm here, I may not be able to volunteer for them in the future so I'm trying to put in something that will last them 5-10 years, cavia adding some drives if their archives grow (they do a lot of media stuff for history projects) | 21:21 |
| drab | dirtycajunrice: enterprisey I'm hoping, maybe WD reds | 21:21 |
| dirtycajunrice | WD Reds are consumer drives | 21:22 |
| dirtycajunrice | enterprise drives are literally HP or dell signed drives from the manufacturer | 21:22 |
| drab | eeer, ok, fine, NAS drive then? | 21:22 |
| dirtycajunrice | with special Firmware | 21:22 |
| dirtycajunrice | it matters for if enterprise servers will read them | 21:22 |
| dirtycajunrice | sec | 21:22 |
| drab | I see | 21:22 |
| drab | well then no, no enterprise drives | 21:22 |
| dirtycajunrice | whats your budget? | 21:23 |
| Epx998 | finally being asked to migrate off ub12.... | 21:23 |
| dirtycajunrice | (i dont user enterprise drives either. i have 12 8TB toshiba x300s lol | 21:24 |
| drab | dirtycajunrice: about 1K including disks and will need at least 128GB to run all VMs | 21:25 |
| dirtycajunrice | drab, https://www.ebay.com/itm/DELL-POWEREDGE-R510-12-BAYS-2x-QUAD-CORE-L5520-2-26GHz-24GB-NO-HDD-NO-RAIL/132224991815?hash=item1ec9394247:g:nb0AAOSwJtdZ-g5g | 21:26 |
| dirtycajunrice | thats the server | 21:26 |
| dirtycajunrice | you can get caddys for like 30 bucks | 21:26 |
| dirtycajunrice | ram is a mother right now tho | 21:27 |
| dirtycajunrice | but thats even a problem in consumer | 21:27 |
| dirtycajunrice | the market is artificially inflated | 21:27 |
| drab | thanks | 21:28 |
| drab | Epx998: I just finished that 2 weeks ago | 21:44 |
| drab | now got a few 14 that I'm getting rid of and moving to containers | 21:44 |
| drab | they actually had some ub11 too going around... | 21:44 |
| drab | different question I guess I'm still confused about regarding HBAs | 21:45 |
| Epx998 | from before | 21:45 |
| drab | LSI SAS 9211-4i PCI Express to 6Gb/s SAS HBA <-- this guy has one SF8087 port splitting to 4 lanes | 21:45 |
| drab | I'm understanding that each port can do 6Gbs, ie that's not comulating for all the ports at once, but that's the first thing I have doubts about | 21:46 |
| drab | "through four internal 6Gb/s ports" so it definitely seems it's 6Gps per port, however that SF cable is going at once to the backplane... does that mean it's the same as 24Gpbs to the backplane? which then the drives would share? | 21:48 |
| drab | ok, I think the SAS configuration table explains that, full duplex SAS is 4.8GBps, so about 400MB/s per drive | 22:02 |
| sarnold | drab: my home machine uses an sas expander; I think either sas port on the HBA can drive any of the drives | 22:20 |
| sarnold | drab: so that's roughly eight sata-lanes of performance, and I've got nine drives plugged into the thing; the lights all seem to blink simultaneously though so it feels more than good enough at the job, haha | 22:21 |
| sdeziel | 8 SAS drives hooked to a "home" machine surely is good enough ;) | 22:22 |
| sarnold | sdeziel: I blame my friend who talked me into 3-way mirrors | 22:23 |
| sdeziel | sarnold: I thought that friends only recommend mirror for ZFS :P | 22:23 |
| sarnold | sdeziel: two-way mirrors? your friends must not care for your data much :) | 22:24 |
| drab | eer, a 3-way mirror sounds like a logical impossiblity... how's a mirror 3 way? :) | 22:25 |
| sarnold | drab: easy: zpool create pool mirror sda sdb sdc mirror sdd sde sdf mirror sdg sdh sdi | 22:25 |
| sarnold | tada! | 22:26 |
| sarnold | three 3-way mirrors! :) | 22:26 |
| drab | oh, a mirror with 3 vdevs, I see, ok | 22:26 |
| sarnold | vdev with 3 disks :) | 22:26 |
| drab | eer, that one | 22:27 |
| * drab runs 6 disks in raidz2 | 22:27 | |
| drab | seems good enough with 2 disks possible failure and more available space, no? | 22:27 |
| drab | on a 3disks vdev you get the same 2 drive failures, but the capacity of only one | 22:28 |
| drab | unless I'm missing something again | 22:28 |
| sarnold | yeah you've got a pretty good sweet spot there | 22:28 |
| sarnold | but I'd expect roughly nine times 100MB/s for bulk reads from three 3-way mirrors, vs roughly 100MB/s bulk reads from 6-disk raidz2 | 22:29 |
| sdeziel | I've read somewhere that it was a pain to grow a RAIDZ(2) setup | 22:30 |
| sarnold | (I don't think I can actually get my queue depths deep enough to get that kind of throughput though) | 22:30 |
| sdeziel | trying to find the link to that | 22:30 |
| drab | ah, that's new. would love to read that | 22:30 |
| sarnold | sdeziel: it definitely is; the next logical step is to add another six disks in a new vdev, and of course then all the writes would go to the new vdev until they're about the same capacity... | 22:30 |
| sdeziel | http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ | 22:31 |
| drab | that said, the plan was (given the 12 bays), to just add another vdev with the next batch of 6 hds | 22:31 |
| sarnold | but I could easily add just another three disks to mine .. and suffer the same write-problem :) | 22:31 |
| sdeziel | sarnold: yeah, that was my recollection from this article ^ | 22:31 |
| drab | ah, I learned something, I guess I misunderstood how vdevs where added together, I was still thinking raid10 | 22:36 |
| sarnold | drab: that's probably the right intuition | 22:37 |
| drab | I'm unsure about the maintenance windows tho, once a vdev with a 2xdisks mirror has a failure you are one disk away from losing everything | 22:39 |
| drab | so yeah, it seems to me if you're gonna be running mirrors then it has to be 3-disks per mirror | 22:39 |
| drab | and that gets kind of expensive | 22:39 |
| sarnold | yes it does | 22:40 |
| sarnold | which is why a 6-disk raidz2 feels like a very nice sweet spot | 22:40 |
| drab | speaking of ZFS, the other thing I had misunderstood (no wonder..) is how the ZIL is supposed to work | 22:40 |
| drab | I thought writes would go to it, ie behave like a write cache | 22:40 |
| drab | btu that's only for synchronous writes afaiu | 22:41 |
| sarnold | there's several interacting concepts here | 22:42 |
| drab | async writes end up in mem and never touch the ZIL/SLOG | 22:42 |
| sarnold | all pools have ZIL, you can use a SLOG to put the ZIL on super-fast storage | 22:43 |
| drab | right | 22:43 |
| drab | basically before going with ZFS I had looked at... I now forget the name... for linux | 22:43 |
| drab | where you basically end up with something like a SHDD | 22:43 |
| sarnold | bcachefs? | 22:43 |
| drab | putting a couple of SSDs in front of a bunch of disks | 22:43 |
| drab | ah yes, that's the one | 22:43 |
| drab | I thought ZFS with SLOG on a diff disk would work like that, but that's not the case | 22:44 |
| drab | and it seems in a sense less performant than a setup with bcachefs | 22:44 |
| drab | because in terms of returning to the app, with bcachefs you only have to wait for writes to be done in the SSD | 22:44 |
| sarnold | indeed, writes to the main drives still get flushed within a few seconds as writes to the slog, but the application is allowed to cobntinue once the write to the slog is complete | 22:44 |
| drab | mmmh, but that seems true only for synchronous writes | 22:45 |
| sarnold | I found that far fewer operations go through the slog than I expected. My intution suggested that atomic operations like mkdir would go through slog but I _never_ saw the slog write counters increment no matter what workload I tried :) | 22:45 |
| drab | https://github.com/zfsonlinux/zfs/issues/1012 | 22:46 |
| drab | https://www.ixsystems.com/blog/o-slog-not-slog-best-configure-zfs-intent-log/ | 22:47 |
| drab | Use case: If your use case involves synchronous writes, utilizing a SLOG for your ZIL will provide benefit. Database applications, NFS environments, particularly for virtualization, as well as backups are known use cases with heavy synchronous writes. | 22:48 |
| drab | which goes to the point that for async writes the whole ZIL/SLOG seems unhelpful | 22:48 |
| drab | from that second link forcing all writes to be sync seems to be a matter of security of not losing data, but to me seems to actually make a diff in terms of performance as control to the app will be returned as soon as data is written to the SLOG | 22:49 |
| drab | I guess I'll hvae to test that | 22:49 |
| sarnold | drab: fwiw i'm quite happy to leave the defaults at the defaults | 22:50 |
| sarnold | drab: and even though I've got a partition of my nvme set aside for slog, one of these reboots I'm going to disable it and just use the whole nvme for l2arc instead | 22:50 |
| drab | yeah, I gave a share pof the NVME to slog right now and the rest of l2arc, but I'm fundamentally bugged by this default behavior | 22:51 |
| drab | I would basically except, like in the case of bcachefs, to basically see nvme-like speeds for all writes | 22:52 |
| drab | with "long term" storage being a sort of deferred write, ie from nvme device to HDDs | 22:52 |
| drab | so app -> mirror nvme -> raidz2 on 6 drives | 22:53 |
| drab | basically use the NVME as a cheaper version of those super expensive battery backed ram, zeusram or whatever, forgot what it's called | 22:54 |
| keithzg | Hmm has anyone ever had Windows users experiencing DNS cache corruption or such while connected to an OpenVPN server? Trying to debug some users' sporadic issues (the VPN server is running Ubuntu 16.04 of course) and it's narrowing down to a point that simultaneously very specific yet extremely mysterious. | 22:58 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
