| arkanoid | hello, I'm experiencing weird (wrong) ram usage on a fresh ubuntu 22.04 VM running on hyper-v with dynamic memory enabled | 04:17 |
|---|---|---|
| arkanoid | basically the displayed memory assigned/requested on the hypervisor is wildly different from the one displayed in linux shell (top) | 04:18 |
| oerheks | arkanoid, hard to say where to file a bug report, hyperV or 22.04 ... | 07:53 |
| rbasak | arkanoid: I'm not sure what "dynamic memory" means. Is that memory ballooning? AIUI, to do that you need an agent inside the VM to assist. Otherwise the VM will always use the maximum. Is that what you're trying to do - have I understood this correctly? If so, have you installed an agent inside the VM? | 10:40 |
| === Thumpxr6 is now known as Thumpxr | ||
| === Thumpxr5 is now known as Thumpxr | ||
| arkanoid | rbasak: thanks for the answer. I don't think I have. I checked the official docs and it is not requested. It should "just-work". I've checked the presence of lsmod | grep "hv" and they are there | 15:39 |
| rbasak | arkanoid: OK. Sorry, this is beyond my immediately knowledge then. | 15:42 |
| rbasak | *immediate | 15:42 |
| arkanoid | rbasak: have a look here https://learn.microsoft.com/en-us/windows-server/virtualization/hyper-v/supported-ubuntu-virtual-machines-on-hyper-v | 15:43 |
| arkanoid | line "dynamic memory" | 15:43 |
| arkanoid | rbasak: I've just installed a brand new vanilla 22.04 VM. On first login, htop shows 222MB / 1.85GB (vm configured for dynamic allocation min 512MB max 2048MB). Hyper-V Management shows Assigned Memory 2048 MB, Memory Demand 839Mb | 16:25 |
| arkanoid | *839MB | 16:26 |
| rbasak | Sorry, I don't have time to look any deeper into this. | 16:27 |
| rbasak | Hopefully someone else here will be able to help | 16:27 |
| arkanoid | rbasak: thanks for the feedback | 16:32 |
| Odd_Bloke | arkanoid: I don't know if I can really help much, but what would you expect to see? (i.e. which of those numbers aren't what you expect?) | 16:51 |
| arkanoid | well, memory requirements inside the VM is much much lower than what the hypervisor thinks. In my production ubuntu VM I have htop displaying 1.6GB used+buffer+cache, hyperv on same machine shows "memory demand: 4792MB" which is == the maximum allowed by dynamic memory policy. VM goes all you can eat according to the hypervisor | 16:54 |
| arkanoid | consuming 2.5x times memory on hypervisors is not quite good | 16:55 |
| Odd_Bloke | Right, I think what's happening is that once a particular address has been requested by the VM, the hypervisor allocates it to the VM permanently (because it has no way of knowing if the VM is still using it or not). | 16:58 |
| Odd_Bloke | So over time, the VM will end up requesting every address in its memory, and therefore end up forcing the HV to allocate the full dynamic memory space. | 17:00 |
| arkanoid | Odd_Bloke: it could be, the VM is using postgres interally, not much pressure, but not quite still neither | 17:00 |
| arkanoid | but isn't the dynamic memory + hv_* mods + guest stuff all made to limit this? | 17:00 |
| arkanoid | I mean, is the dynamic memory only meant to go up and never down? dealloc is not handled? | 17:01 |
| Odd_Bloke | I don't really know how it works for Hyper-V specifically, I'm afraid, but I believe this is the function that the agent r'basak described fulfils: it observes the VM's state and communicates to the HV which blocks can be deallocated. | 17:01 |
| arkanoid | do know where I can read about this agent? | 17:02 |
| Odd_Bloke | I believe it would need to be something Hyper-V specific so I don't know where to direct you, I'm afraid. | 17:03 |
| arkanoid | Odd_Bloke: thanks for the feedback | 17:04 |
| minimal | is this not handled by the hv_balloon kernel module? | 17:24 |
| minimal | but then it seems you'd also need to enable "Dynamic Memory" in the Hyper-V settings for the VM | 17:27 |
| arkanoid | minimal: done both since step 1 | 17:33 |
| arkanoid | issue is vm goes all-you-can-eat immediately | 17:34 |
| sdeziel | arkanoid: I have only theoretical knowledge of VM balloon mechanism but what I understood is that the VM starts with the full amount and it's only when the host goes through memory pressure that it asks the VM if it's OK to inflate the balloon (which is AFAIK, just a kernel module/driver, no agent involved) and if the guest's driver says yes, I can give you back this and this pages, then the | 17:42 |
| sdeziel | host get them back. The "returned" pages might not be as much as the host demanded for if the guest is unable to free more pages | 17:42 |
| minimal | the hv_balloon module does appear to provide a "pages_ballooned" value inside debugfs to indicate the number of "pages we have given back to host" | 17:45 |
| arkanoid | minimal: if it's working surely it's not working well. 2.5x memory consumption is a lot | 17:48 |
| arkanoid | I'm trying right now with linux-azure kernel/metapackage | 17:49 |
| Odd_Bloke | arkanoid: Is your host under memory pressure? If not, then it sounds like you shouldn't expect any pages to be returned to the host, so the 2.5x memory consumption would be expected. | 18:22 |
| arkanoid | Odd_Bloke: how do I know? Htop seems locked at 1.6GB usage+cache | 20:00 |
| arkanoid | Linux doesn't show any pressure problem, it shows 2.4GB free | 20:01 |
| arkanoid | now the numbers are slightly different, but here's the copypaste from htop header: MiB Mem : 3905.1 total, 1092.1 free, 1638.3 used, 1174.7 buff/cache | 20:05 |
| arkanoid | free -h output: https://termbin.com/q3f0 | 20:06 |
| arkanoid | hyper-v "Memory demand" is 4792MB | 20:07 |
| Odd_Bloke | If I understood sdeziel correctly, then I don't think you'll see any change in the memory assigned to the guest unless the _host_ experiences memory pressure (and so asks hv_balloon in the guest to give it some pages back). | 20:08 |
| Itay40 | I've been using Ubuntu server 20 ISO and installing nvidia drivers on it - all smooth. | 20:13 |
| Itay40 | Lately I started using ubuntu server 20 cloudimg and it just doesn't work, gpu are not detected properly | 20:13 |
| Itay40 | is there some known large differences between the two installation types or is it my skill issue? | 20:13 |
| Odd_Bloke | How are you installing the drivers? | 20:30 |
| Itay40 | Installing in both servers in a similar way, using cuda runfile and following the docs | 20:33 |
| Odd_Bloke | Does `uname -r` match between the two? | 20:37 |
| Itay40 | checking | 20:38 |
| Itay40 | exact same, both 5.15.0-94-generic | 20:43 |
| Odd_Bloke | Is the `linux-image-generic` package installed on the cloud image server? | 20:45 |
| Itay40 | no, only linux-image-generic-hwe-20.04 | 20:46 |
| Odd_Bloke | Aha, right, on focal. That was my one idea, I'm afraid, I'm not sure what's going on. | 20:50 |
| Itay40 | thanks for the attempt! | 20:51 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!