| Goop | I have two drives--one that is slow and costs my money to use/pull from, then I have a fast drive, is free to use, but is much smaller. Is there software that I could use to automatically allocate data and predict where to put it? | 00:18 |
|---|---|---|
| sarnold | none that I know of, but zfs may come close, if the fast drive is fast enough to use for l2arc | 00:21 |
| Goop | The thing is, I will be hosting a website with more data than what is on a VPS, the extra data will be pulled/pushed off-site (hence, there's a data transfer cost). | 00:24 |
| sarnold | Goop: hrm, if your data usage matches many other sites, maybe something like squid, nginx, or varnish would fit well | 00:28 |
| rbasak | Goop: bcache? | 00:30 |
| === boshhead_ is now known as boshhead | ||
| lordievader | Good morning | 06:04 |
| === win0err_ is now known as win0err | ||
| === win0err_ is now known as win0err | ||
| AtuM | Hi! | 08:13 |
| AtuM | I have a question regarding networkmanager.. can nmcli also return a PSK for a wifi connection, or should I parse the /etc/NetworkManager/system-connections/ files instead? | 08:14 |
| cpaelzer | anyone around for a little netplan question - I assume I hit something trvial atm - I have a config that works well when "netplan apply" - but after boot it is not applied | 08:29 |
| AtuM | nmcli c s WIFI -s still hides the psk key.. | 08:29 |
| cpaelzer | I wonder if I missed an enable/disable somewhere | 08:29 |
| cpaelzer | TL;DR - after boot no device is up; then "sudo netplan apply" and all is good | 08:31 |
| cpaelzer | what am I missing | 08:31 |
| AtuM | found it! "nmcli -s c s WIFI" | 08:31 |
| AtuM | shows secrets :D | 08:31 |
| cpaelzer | FYI: none of these solutions is matching my case https://askubuntu.com/questions/1019146/netplan-does-not-apply-at-startup | 08:38 |
| blackflow | so... the inevitable hard to debug problems of complex, unneded, additional layers..... how unexpected. :) | 08:50 |
| cpaelzer | :-) | 08:52 |
| cpaelzer | the networkd files even look normal, so it seems to be more a networkd than netplan issue in this case | 08:53 |
| blackflow | cpaelzer: of course, you checked the journal/syslog? | 08:53 |
| blackflow | networkd files look normal before or aftre netplan apply? as far as I gather, netplan has a systemd generator that's fired up on book and produces transient networkd config. | 08:54 |
| cpaelzer | it considers the link not managed | 08:55 |
| cpaelzer | that is what networkctl and journal are telling me | 08:55 |
| cpaelzer | they are the same before and after netplan apply | 08:55 |
| cpaelzer | hanging around in /run/systemd/network | 08:55 |
| cpaelzer | I seem to miss the piece of the puzzle that makes networkd consider those configs | 08:56 |
| cpaelzer | on boot it seems not to happen and "netplan apply" lets it happen, yet I don't know yet what it is | 08:56 |
| blackflow | cpaelzer: what I'd try next is take the network config produced by netplan and make it permanent in /etc, and temporarily disable netplan. If the net still fails on boot, that will clearly show the problem being networkd. | 08:59 |
| blackflow | you know like a race condition or something with interface (re)naming or whatev. | 09:00 |
| cpaelzer | yeah | 09:00 |
| cpaelzer | I thought the config is on permanent in the file in /run/systemd/network | 09:01 |
| * cpaelzer wonders if those are generated as well | 09:01 | |
| cpaelzer | let me check modification date after a reboot | 09:01 |
| blackflow | cpaelzer: no, /run is tmpfs | 09:01 |
| cpaelzer | true | 09:01 |
| cpaelzer | so it might create those after networkd started | 09:01 |
| cpaelzer | and apply might simply restart networking which picks them up | 09:01 |
| cpaelzer | let me just restart networkd after a boot | 09:01 |
| cpaelzer | copied the files to /etc/systemd/network - devices are still down and unmanaged after reboot | 09:18 |
| blackflow | cpaelzer: I dived into the whole thing a bit and from what I read, it appears netplan is doing it all wrong. systemd generators should NOT, according to documentation and devs, be used for network config. | 09:18 |
| cpaelzer | I was about to prep a bug anyway | 09:18 |
| blackflow | cpaelzer: so in your case it means networkd has a problem with that link and/or config? | 09:20 |
| cpaelzer | seems that way, btu then why is it working after netplan apply | 09:21 |
| cpaelzer | since the .link/-network files do not change | 09:21 |
| cpaelzer | same md5sums in /run/systemd/network/* before and after | 09:21 |
| blackflow | cpaelzer: looking at the netplan source, the apply command pretty much only re-runs the generator and restarts networkd. When you tested networkd config only, did you disable netplan? | 09:25 |
| cpaelzer | blackflow: actually what would you disable to do so, the generator? | 09:34 |
| cpaelzer | netplan-wpa@service should not matter I thouht but I can disable that | 09:34 |
| cpaelzer | and I restarted systemd-networkd.service on my own | 09:35 |
| cpaelzer | before | 09:35 |
| cpaelzer | as I assumed this is what the apply would do | 09:35 |
| blackflow | cpaelzer: mv /etc/netplan/01-netcfg.yaml /root/ | 09:36 |
| blackflow | netplan won't do anything if there's no config | 09:36 |
| cpaelzer | I can do that in addition | 09:36 |
| blackflow | I thought you did. That would test the networkd config without potential interference by netplan, and at least show where the problem is, networkd (if the problem persists) or netplan (if it doesn't. | 09:37 |
| cpaelzer | will retry with that in addition | 09:37 |
| cpaelzer | blackflow: even with that extended disabling it is not coming up managing the interfaces | 09:44 |
| cpaelzer | but I agree that from the code the apply shouldn't du much more than restarting after laying out the files | 09:46 |
| cpaelzer | reading that section again | 09:46 |
| cpaelzer | it does a set of replug/rebinds | 09:47 |
| blackflow | I think at this point the problem is no more in netplan, but in networkd, if plain networkd config won't fix this. | 09:48 |
| cpaelzer | Could be a race | 09:48 |
| cpaelzer | like devices completely there before it can handle events | 09:49 |
| cpaelzer | and "netplan apply" will replug devices | 09:49 |
| cpaelzer | which makes it realize "hey there is something" | 09:49 |
| cpaelzer | I'll try to isolat this replkug done by netplan | 09:49 |
| blackflow | but if you disabled the netplan config, and the plain networkd config didn't work, then it's not neplan. | 09:49 |
| cpaelzer | and do it manually | 09:49 |
| cpaelzer | blackflow: I agree, yet I still want/need to debug further | 09:49 |
| cpaelzer | and finding what magic "netplan apply" does to fix it mgiht help to find the root cause | 09:50 |
| blackflow | there's no magic there, the apply command code is pretty clear. it restarts the network after re-applying the generator which only re-creates the networkd config file | 09:56 |
| blackflow | so, what I'd try is run that sequence with netplan disabled (no config in /etc/netplan/). create an explicit networkd config (eg. copy the one netplan creates), reboot. If the network is not configured, restart systemd-networkd. If the network is not configured, I'd then look into the source of netplan's systemd generator. | 09:57 |
| cpaelzer | I already look in the apply source and this tries to confuse me I think | 09:59 |
| cpaelzer | this clearly has no networkManager | 09:59 |
| cpaelzer | ls: cannot access '/run/NetworkManager': No such file or directory | 10:00 |
| cpaelzer | python3 -c 'import glob; print(bool(glob.glob("/run/NetworkManager/system-connections/netplan-*")))' | 10:00 |
| cpaelzer | False | 10:00 |
| cpaelzer | but | 10:00 |
| cpaelzer | on debug without anything in /etc/netplan/ | 10:00 |
| cpaelzer | oh no | 10:00 |
| cpaelzer | forget the last 5 lines | 10:00 |
| cpaelzer | it says "no" NM configuration | 10:00 |
| cpaelzer | I was wondering if it was misdetecting | 10:00 |
| blackflow | yeah. It tries one or the other. | 10:01 |
| cpaelzer | and without a config it says no to both | 10:01 |
| cpaelzer | DEBUG:no netplan generated networkd configuration exists | 10:01 |
| cpaelzer | DEBUG:no netplan generated NM configuration exists | 10:01 |
| cpaelzer | but | 10:01 |
| cpaelzer | then it still does something | 10:01 |
| cpaelzer | replugging the device | 10:01 |
| cpaelzer | DEBUG:replug enp0s3: unbinding virtio0 from /sys/bus/virtio/drivers/virtio_net | 10:01 |
| cpaelzer | DEBUG:replug enp0s3: rebinding virtio0 to /sys/bus/virtio/drivers/virtio_net | 10:01 |
| cpaelzer | which is odd | 10:01 |
| cpaelzer | and might be the trigger for networkd to handle it | 10:01 |
| cpaelzer | will go on after lunch | 10:01 |
| cpaelzer | thanks already for the discussion blackflow | 10:01 |
| blackflow | I ran a test with a test server here, disabled netplan and had an explicit networkd config, the network didn't come up post boot. | 10:02 |
| blackflow | so there's some kind of mess going on there for sure. now I'm intrigued... | 10:02 |
| cpaelzer | really | 10:02 |
| cpaelzer | interesting | 10:02 |
| * cpaelzer taking lunch | 10:02 | |
| blackflow | bon appetit. | 10:03 |
| blackflow | Jun 7 11:28:49 dev networkd-dispatcher[1746]: WARNING: systemd-networkd is not running, output will be incomplete. | 10:03 |
| blackflow | oh that's right. systemd-networkd.service is NOT enabled by default on bionic. | 10:04 |
| blackflow | status says "enabled-runtime". now I enabled it explicitly, and lets see what reboot does... | 10:04 |
| blackflow | cpaelzer: yup. That was it. systemd-networkd.service is NOT enabled by default in Bionic. netplan's generator starts it at runtime. By having an explicit networkd config (one in /etc/systemd/network/foo.network), and explicitly enabling systemd-networkd.service, it works fine without netplan. | 10:08 |
| blackflow | So, to control that variable, I'd do the same for your case. If network works fine wihtout netplan that way, then you know where the problem is: in netplan, and posssibly a race as you suggested. | 10:09 |
| blackflow | Dunno why netplan is doing it that way. Clearly systemd does not expect or want generators to mess with network config. | 10:10 |
| blackflow | https://github.com/systemd/systemd/issues/4066 | 10:11 |
| blackflow | yup, the generator does this: https://github.com/CanonicalLtd/netplan/blob/master/src/networkd.c#L541 | 10:15 |
| blackflow | So uh... can we agree that netplan is unnecessary reinvention of the wheel that at the same time goes against systemd specification and causes more trouble than it fixes? :) I can't help it, I _have_ to complain about it. | 10:17 |
| cpaelzer | blackflow: in my case systemd-networkd.service was default enabled | 10:57 |
| cpaelzer | but thanks for cross checking your csae | 10:57 |
| cpaelzer | and I'm not going to stop you from complaining, free speech and all that :-) | 10:58 |
| cpaelzer | I like what it solves, and I like that it is completely optional - so you can ignore/avoid it if you want | 10:58 |
| cpaelzer | That replug event is it | 11:05 |
| cpaelzer | I was able to confirm that this is what makes it work | 11:05 |
| cpaelzer | so it is both | 11:05 |
| cpaelzer | networkd issue to ignore it and either networkd/netplans issue to somehow ensure this is working | 11:05 |
| blackflow | cpaelzer: minor detail, just to be 100% sure we're talking about the same thing, when you say systemd-networkd.service was enabled, was it "enabled-runtime" or just "enabled"? | 11:09 |
| cpaelzer | blackflow: it was just enabled | 11:10 |
| cpaelzer | like | 11:10 |
| cpaelzer | loaded (/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled) | 11:10 |
| cpaelzer | and active (running) | 11:10 |
| blackflow | and with that, and no netplan config in /etc/netplan/, the network doesn't come up on boot, if you have explicit networkd confg in /etc/systemd/network/foo.network ? | 11:11 |
| cpaelzer | blackflow: exactly | 11:12 |
| cpaelzer | it seems it misses the event of the device appearing | 11:12 |
| cpaelzer | unbind/bind on the driver makes it work | 11:12 |
| blackflow | and it does if you restart systemd-networkd.service post boot? | 11:12 |
| blackflow | ah I se | 11:12 |
| blackflow | *see | 11:12 |
| cpaelzer | If I restart systemd-networkd.service nothing changes (It might reload the config, but has still no event) | 11:12 |
| blackflow | then it probably isn't a netplan issue at all in this case. | 11:13 |
| cpaelzer | yeah, that is what I think | 11:13 |
| cpaelzer | for me it seems to depend on the bus, I can trigger it with PCIe KVM guests but not with PCI | 11:14 |
| cpaelzer | I have reported bug 1775566 about it | 11:15 |
| ubottu | bug 1775566 in systemd (Ubuntu) "networkd not applying config - missing events?" [Undecided,New] https://launchpad.net/bugs/1775566 | 11:15 |
| cpaelzer | xnox: is that what you would look into - or who would I most likely talk to? ^^ | 11:15 |
| blackflow | cpaelzer: reading the bug report, which Ubuntu/systemd is this? I see the "renamed from eth0" message and afaik it's dropped from recent systemd (the one in bionic afaik)? | 11:22 |
| blackflow | ah bionic.... weird... | 11:22 |
| cpaelzer | yep bionic | 11:22 |
| blackflow | maybe I'm wrong about that message. I do recall reading about a systemd issue about it being spammy and thus removed or something. | 11:24 |
| cpaelzer | my networkd link files ahve rename rules | 11:25 |
| cpaelzer | maybe because of that | 11:26 |
| cpaelzer | but then we just reaized my rules are ignored, so maybe not | 11:26 |
| blackflow | cpaelzer: hey, running these tests on my test server exposed a hardware fault (aftre several reboots), heh heh heh. Filing a hardware issue ticket with the hosting company. | 11:42 |
| cpaelzer | yay | 11:45 |
| cpaelzer | I think my case is a oversimplified virtual guest | 11:45 |
| cpaelzer | I'm adding proper busses now | 11:45 |
| cpaelzer | not sure how exactly that would cause it, but it could break some of the kernel events usually missing | 11:45 |
| cpaelzer | so I need to be sure | 11:45 |
| blackflow | cpaelzer: fwiw, I have a bionic installation in a qemu-kvm VM, with virtio, and this for network config only host-side: -net nic,model=virtio | 12:16 |
| blackflow | cpaelzer: and no problem in the guest. | 12:16 |
| cpaelzer | blackflow: so far I can only reproduce it with -M pc-q35-bionic but not with pc-i440fx-bionic | 12:22 |
| cpaelzer | I have asked apw to look from the kernel POV if there is anything different in terms of device events in that case | 12:22 |
| apw | oh machine type specific, and the differnece between those two is ... | 12:36 |
| blackflow | *drum roll* | 12:39 |
| === success is now known as Guest53657 | ||
| blackflow | Anyone knows how to figure out "predictable" NIC names? I'm installing Ubuntu via debootstrap on some servers, via Debian jessie, which is configured for eth* . I need to figure out what the NIC name will be. | 14:14 |
| blackflow | on one such instance I ran with net.ifnames=0 in order to boot into Bionic proper. But I still have no clue which one is it, looking at udevadm info /sys/class/net/eth0 . | 14:15 |
| blackflow | (and btw, yes, I've seen this https://github.com/systemd/systemd/blob/master/src/udev/udev-builtin-net_id.c#L20 -- and I assumed it'd be enp3s0 (which is also ID_NET_NAME_PATH given by udevadm), but it apparently isn't... ) | 14:17 |
| blackflow | udevadm info as seen by Bionic https://dpaste.de/Sbuo | 14:17 |
| rbasak | blackflow: one way to do that is to determine in advance what the MAC address will be, then arrange to either rename the NIC according to the MAC or use MAC-based naming | 14:18 |
| blackflow | ID_NET_NAME_MAC=enx001e67ad6d8f according to udevadm info, I suppose (and yes, that's the MAC of the NIC) | 14:19 |
| blackflow | but uhh... no, no mac based names pls. what I'd like to figure out is how to determine which of the several name schemes (see the dpaste above) will be used? | 14:20 |
| rbasak | Use https://git.launchpad.net/ubuntu/+source/systemd/tree/src/udev/udev-builtin-net_id.c?h=applied/ubuntu/bionic-devel rather than upstream, since that will show you the actual code in use in Bionic, including patches. | 14:20 |
| rbasak | (the git importer is still experimental though that particular one should be current at least for now I think) | 14:20 |
| blackflow | rbasak: good idea, and I see it, but I still don't know which of all those rules will be chosen. | 14:21 |
| blackflow | I got eno1, enp3s0 and enx001e67ad6d8f as candidates. | 14:21 |
| rbasak | https://lists.ubuntu.com/archives/ubuntu-devel/2015-May/038761.html might have some information. I'm not sure if it's still current though. | 14:22 |
| blackflow | rbasak: oh one more thing, and I think that's where the problem is.... is systemd present in initramfs in Bionic? | 14:22 |
| rbasak | I'd expect initramfs to be pre-init, but I don't actually know. | 14:23 |
| blackflow | The situation I have here is initramfs set up with Dropbear for remote unlocking of LUKS'ed ZFS root pool. with net.ifnames=0 in kernel cmdline, all is good, I rely on eth0. Without it, guessed enp3s0 is "unknown device". | 14:23 |
| rbasak | Can't dropbear listen on all interfaces? Or is it that you're trying to configure the interface first? | 14:24 |
| blackflow | rbasak: it's not just that. the default initramfs scripts require explicit config, NIC name included, even with DHCP. | 14:25 |
| blackflow | I _could_ whip up a custom initramfs, but eh... | 14:25 |
| rbasak | Your dropbear LUKS unlock isn't a custom initramfs? :) | 14:25 |
| blackflow | rbasak: it is on Gentoo (without systemd), not on Ubuntus. | 14:26 |
| rbasak | FWIW, I wrote https://github.com/basak/netkeyscript to solve this problem, but that relies on a trusted network | 14:26 |
| blackflow | incredibly, Ubuntu's initramfs supports encrypted root and dropbear just fine. :) as long as ethX is used. | 14:26 |
| rbasak | Looks like it's hardcoded to eth0 | 14:26 |
| rbasak | I wanted to make it work on all interfaces but never got round to doing that | 14:27 |
| blackflow | rbasak: I got a solution that fetches the key over https using a self signed cert and its own (exclusive) CA but it's still in testing | 14:28 |
| blackflow | neway, I'm not sure where my current problem is actually... I am guessing there's systemd in initramfs and NIC renaming happens according to those rules, but expected enp3s0 is just not there. | 14:28 |
| rbasak | You might have an issue with limited entropy there | 14:28 |
| blackflow | how do you reckon, if eth0 works just fine? | 14:29 |
| rbasak | I wouldn't be surprised to learn that interface renaming only happens after pivot | 14:29 |
| blackflow | rbasak: ran some tests with a KVM VM here. ens3 (virtio net, it's very much predictable to be ens3). and it works. | 14:32 |
| blackflow | So it appears to me that I simply expect the wrong name. enp3s0 ain't in (on the actual server). | 14:32 |
| blackflow | I am __really__ trying to give systemd a chance, but any attempt at doing so, I get spat in the face by it....... | 14:34 |
| rbasak | Why not just use net.ifnames=0 if you know you'll only have one NIC? | 14:46 |
| blackflow | well that's the thing, there will be more than one, and I wanted to "get with the program" and use systemd as it was designed... but I guess that's impossibru. | 14:48 |
| rbasak | What would you do without systemd? | 14:49 |
| blackflow | I'd enjoy my work much more :) | 14:49 |
| blackflow | rbasak: and if that was a serious question, I'd do some udev magick and bind eth devices with MACs so they don't randomly change. | 14:50 |
| blackflow | but I know how to solve this, it's just that I'm really trying to do it the "systemd way" and not be a hater. | 14:50 |
| rbasak | You can still do that I think. | 14:51 |
| rbasak | Add a 70-persistent-net.rules with what you want | 14:52 |
| blackflow | if I'll be forced to go with ethX, then yeah. | 14:53 |
| compdoc | Ive been using the new nic naming for years now. no issues | 14:56 |
| blackflow | rbasak: it's eno1 .... wtf. | 14:58 |
| blackflow | compdoc: yeah but how do you know in advance which name will it be? that's my problem. | 15:06 |
| blackflow | rbasak: the folks in #systemd have clarified what's going on. There's an order of names tried for the link NamePolicy, and onboard is before path, that's why eno1 (onboard) was chosen before enp3s0 (path), on this server. | 15:13 |
| blackflow | which also means it could be forced with a .link file. | 15:14 |
| rbasak | I see. | 15:15 |
| rbasak | Useful to know. Thank you for reporting back. | 15:15 |
| blackflow | yw | 15:15 |
| nacc | rbasak: powersj: just to close the loop from #snappy; they recommend we switch git-ubuntu jenkins to use the snapcraft snap | 16:50 |
| powersj | I can do that | 16:53 |
| powersj | rbasak: objections? | 16:54 |
| nacc | powersj: he may want you to wait until we've figured out the issue we have now, just was an fyi for now | 16:57 |
| powersj | ok | 16:57 |
| rbasak | nacc, powersj: yeah, let's wait until LP switches | 17:15 |
| rbasak | Since right now part of the issue is reproducibility, so I'd rather not get ahead of LP | 17:15 |
| nacc | rbasak: ack | 17:16 |
| cliluw | Does Canonical ever release backwards-incompatible versions into its official Ubuntu repos? | 18:43 |
| cliluw | across the same Ubuntu version | 18:43 |
| sarnold | I seem to recall a mysql update somewhere along the way that had some awkward changes | 18:45 |
| sarnold | and when samba addressed some flaws in ancient SMB, we didn't do the usual backport just fixes but went with new versions, and that introduced new problems.. | 18:46 |
| rbasak | cliluw: the policy is here: https://wiki.ubuntu.com/StableReleaseUpdates | 18:51 |
| rbasak | cliluw: with an exception for security issues. | 18:51 |
| rbasak | Basically we try not to do that, but sometimes we have no choice. | 18:51 |
| === beisner-sick is now known as beisner | ||
| smb | sarnold, I am NOT old! :-P | 19:28 |
| sarnold | smb: my apologies :D | 19:29 |
| nacc | cliluw: do you have a specific question in mind? | 19:30 |
| sarnold | cliluw: oh! a debian advisory about memcached just reminded me that we released an update to disable listening to UDP by default https://people.canonical.com/~ubuntu-security/cve/2018/CVE-2018-1000115.html | 19:42 |
| cliluw | nacc: Just wondering if I should do `apt-get upgrade` or `unattended-upgrades`. | 19:51 |
| sdeziel | cliluw: I'd recommend installing apt-listchanges if you have concerns or want to know more about what's about to be installed/upgraded | 19:52 |
| sdeziel | cliluw: this will display you the update's changelog and allow you to abort/continue the upgrade after reading it | 19:53 |
| nacc | cliluw: i mean, you could do unattended-upgrades for just security updates, e.g., or some other setup | 19:57 |
| DirtyCajun | anyone with tgtd experience that can help me with tuning? my tgtd is using all 16 threads and causing serious iowait | 19:58 |
| DirtyCajun | my tgtd instance is causing high io on a 16-core server. SSDs, 10Gbps nics and its seriously bottlenecking because of iowait... any thoughts? | 20:42 |
| nacc | DirtyCajun: is it actively serving out targets/ | 20:48 |
| DirtyCajun | Yes | 20:49 |
| DirtyCajun | targets are up and functioning | 20:49 |
| DirtyCajun | initiators are connected and reading/writing | 20:50 |
| DirtyCajun | (connected to esxi host) | 20:50 |
| nacc | DirtyCajun: i mean, is the load higher than you expect? | 20:50 |
| DirtyCajun | i mean... significantly... its a 16 core box and its pegged at 16 cores but no actual utilization | 20:50 |
| DirtyCajun | https://i.imgur.com/xJLnmjp.png | 20:51 |
| DirtyCajun | https://i.imgur.com/dIc4IfC.png | 20:52 |
| nacc | DirtyCajun: have you read things like http://bencane.com/2012/08/troubleshooting-high-io-wait-in-linux/ ? | 20:54 |
| DirtyCajun | haha yes | 20:55 |
| DirtyCajun | this is explicitly with tgtadm/tgtd | 20:55 |
| nacc | DirtyCajun: ... i mean you have 16 processes doing heavy IO | 20:55 |
| nacc | DirtyCajun: not sure what you expect to see? | 20:56 |
| DirtyCajun | It just doesnt make sense that the iscsi target default implementation would run >#cores threads | 20:56 |
| DirtyCajun | it SEEMS like its doing that much io because there are a crap ton > cores running simultaniously | 20:57 |
| nacc | i mean, 16 is double | 20:59 |
| nacc | not exactly a 'crap ton' imo | 20:59 |
| === uplime is now known as nchambers | ||
| DirtyCajun | there were 47 threads running | 21:05 |
| DirtyCajun | and only 16 cores | 21:05 |
| DirtyCajun | i DID just find that each "Target" runs 16 threads | 21:05 |
| DirtyCajun | so im going to try changing it from 3 targets to 1 target and 3 backing stores | 21:05 |
| DirtyCajun | should drop it down to 16+1 parent thread | 21:06 |
| DirtyCajun | (fingers crossed.) | 21:06 |
| _KaszpiR_ | DirtyCajun what SSD? | 21:18 |
| DirtyCajun | Micron 2TB | 21:18 |
| _KaszpiR_ | nvme? | 21:19 |
| DirtyCajun | no | 21:19 |
| _KaszpiR_ | SATA3? | 21:19 |
| DirtyCajun | yes | 21:19 |
| _KaszpiR_ | install dstat | 21:19 |
| _KaszpiR_ | and then try dstat -tarsm | 21:19 |
| _KaszpiR_ | also what's your sysctl -a|grep -i dirty | 21:21 |
| DirtyCajun | vm.dirty_background_bytes = 0 | 21:22 |
| DirtyCajun | vm.dirty_background_ratio = 10 | 21:22 |
| DirtyCajun | vm.dirty_bytes = 0 | 21:22 |
| DirtyCajun | vm.dirty_expire_centisecs = 3000 | 21:22 |
| DirtyCajun | vm.dirty_ratio = 20 | 21:22 |
| DirtyCajun | vm.dirty_writeback_centisecs = 500 | 21:22 |
| DirtyCajun | vm.dirtytime_expire_seconds = 43200 | 21:22 |
| DirtyCajun | but i thought dirty_Bytes werent relevant if there is no filesystem on the host itself | 21:23 |
| _KaszpiR_ | is that a single ssd deice? | 21:23 |
| _KaszpiR_ | or some mdadm/raid lvm whatever? | 21:23 |
| DirtyCajun | I have 2 LUNS. One is a direct to single 1tb one is a Raid5 mdadm to 3x 2tb | 21:24 |
| _KaszpiR_ | what about /sys/block/*/queue/nr_requests | 21:26 |
| _KaszpiR_ | also you may try to increase max_sectors_kb to match max_hw_sectors_kb | 21:26 |
| DirtyCajun | all at 128 | 21:27 |
| DirtyCajun | the most frustrating part of all this is its ONLY on read | 21:28 |
| DirtyCajun | erp | 21:28 |
| DirtyCajun | only on write | 21:28 |
| DirtyCajun | read is blazing fast | 21:28 |
| _KaszpiR_ | cause read is cached | 21:29 |
| _KaszpiR_ | I'd rather read about those disks how they work | 21:29 |
| DirtyCajun | i mean... i moved 180GB in no time | 21:29 |
| DirtyCajun | (off not on obviously) | 21:29 |
| DirtyCajun | if moving to 1 tgt doesnt work i will | 21:30 |
| DirtyCajun | gotta get everything off the tgt to local storage before i can tho | 21:30 |
| _KaszpiR_ | also did you check how much iops you get there? | 21:30 |
| DirtyCajun | using fio i hit 26.5K read ios | 21:30 |
| DirtyCajun | iops* | 21:30 |
| DirtyCajun | and 20k write | 21:30 |
| _KaszpiR_ | that depends on the scenario | 21:31 |
| _KaszpiR_ | random or req? | 21:31 |
| DirtyCajun | seq | 21:31 |
| _KaszpiR_ | try random | 21:31 |
| DirtyCajun | random was like 4500 | 21:31 |
| _KaszpiR_ | with file size 4G or more | 21:31 |
| DirtyCajun | yep. (i work at a datacenter with san storage) | 21:31 |
| DirtyCajun | so all that was my first tests haha | 21:31 |
| _KaszpiR_ | ah | 21:31 |
| DirtyCajun | but im using enterprise os's at work | 21:31 |
| DirtyCajun | dell / hp / solidfire etc | 21:32 |
| DirtyCajun | so doesnt help at home haha | 21:32 |
| _KaszpiR_ | what micron model is it? | 21:32 |
| DirtyCajun | Micron 1100 MTFDDAK2T0TBN-1AR1ZABYY | 21:32 |
| DirtyCajun | the numbers are sexy on there. | 21:33 |
| _KaszpiR_ | it's slower than 1T evo 850 | 21:34 |
| _KaszpiR_ | in raddom 4k r/w | 21:35 |
| DirtyCajun | well yeah. but it was 14c/GB | 21:35 |
| DirtyCajun | so i cant complain for the price | 21:35 |
| _KaszpiR_ | well, half o the price for 860 pro ;D | 21:35 |
| _KaszpiR_ | exackly | 21:35 |
| nicolas17 | my laptop had a 24GB SanDisk SSD >.> | 21:36 |
| _KaszpiR_ | also make sure ssd device is not overheating over 75C | 21:40 |
| _KaszpiR_ | or 70C, they start to crawl | 21:40 |
| nicolas17 | I wonder at what temperature they start to lose data | 21:41 |
| _KaszpiR_ | DirtyCajun another thought, lower number of threads for tgtd | 21:45 |
| DirtyCajun | So it looks like it makes NUM threads == total host threads per target | 21:45 |
| DirtyCajun | Thats why im moving everything off and going to 1 target. if that fixes it thats fine | 21:46 |
| DirtyCajun | ok. so update | 22:23 |
| DirtyCajun | With it on 1 target its still running 16xNUM LUNS +1 Parent (33) | 22:23 |
| DirtyCajun | but it seems to be managing the connections better | 22:23 |
| DirtyCajun | only issue is when i move a VM from lun1 to lun2 | 22:23 |
| DirtyCajun | it presents the initial symtoms | 22:23 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
