=== arraybolt3_ is now known as arraybolt3 === arraybolt3_ is now known as arraybolt3 === keithzg_ is now known as keithzg [08:27] is some amd gpu driver required for 6800U laptop cpu, or is it already included in kernel? [09:11] guesswhat: it's in the kernel, but I'm not sure if 5.19 is recent enough [09:11] odc got spikes 100% in google meeting [09:11] chrome with hw accel enabled [09:12] Ouch [09:12] `lspci -k | grep -A 3 -E "(VGA|3D)"` using amdgpu module [09:14] https://www.phoronix.com/news/AMD-Radeon-680M-Linux-Mesa [09:14] shoud be fine === jgee118 is now known as jgee11 [18:22] Hmm, nginx-logs throwing this: live/live.sock failed (13: Permission denied) while connecting to upstream, client ... live.sock is 777 which is what I have on another system which works. Ubuntu 22.04, new system. Perms seem ok. Python + gunicorn + nginx. Any sugggestions? [18:22] Spent hours on this already, running out of ideas [18:29] Hi there! it is strange my htop and iotop command on the server are loadning really really slow. What can cause that actually? [18:30] I was checking the read/write and disk io, but all seems normal in this regard [18:30] while the server has some decent load, it still has a lot of free resources [18:30] How can I properly debug this? [18:40] adac: what else is slow? [18:41] maybe they're trying to read off a remote disk/share of some sorts also? [18:42] foo: `namei -l /path/to/live/live.sock` would help so would seeing your nginx config [18:42] effendy[m], actually "ls" and all this normal commands, "pwd" all this is working very fast [18:47] effendy[m], actually I run a bare metal kubernetes cluster on this server (3 servers actually). Within this cluster I do some smb mounts with this guy: https://github.com/kubernetes-csi/csi-driver-smb but other then that, and this never made any issues so far on other clusters I have, I do not read from remote disks or so [18:49] It might be related to that though, but this is just a supposition. [18:49] If there was any way you could disable it temporarily, it would interesting to see if it reacts in the same way. [18:53] effendy[m], I will try and uninstall it bit later. Just have to leave now for some time. Will inform you a bit later! Thanks for the hint! [18:53] :) [18:55] effendy[m], inform only if you like to know if it worked of course :-) [18:56] adac: I do, of course! [18:56] perfect! :-) [19:15] sdeziel: thank you. https://bpa.st/54FQ - it's all stock [19:15] foo: please use `namei -l` as it report on the ownership/perms [19:16] foo: please also paste the relevant nginx/conf.d and nginx/sites-enabled files [19:24] sdeziel: err, sorry, I was going to say... that didn't seem too helpful. pebkac [19:24] sdeziel: thank you! https://bpa.st/R77Q [19:24] sdeziel: note, groups www-data returns www-data : www-data dev === blackro0t is now known as blackroot [19:25] sdeziel: I'm really curious, I've done this set up many times and not sure what I'm missing. [19:25] foo: you likely need to edit nginx's systemd unit to add `SupplementaryGroups=dev` to it. `sudo systemctl edit nginx` [19:26] foo: cause I don't think that nginx will load the supplementaty groups when it switch UID/GID from root:root to www-data:www-data [19:29] sdeziel: thanks, testing now. I think I want to uncomment out the service from sudo systemctl edit nginx and then add SupplementaryGroups=dev to it, *googles [19:30] foo: no, you just need `[Service]\nSupplementaryGroups=dev\n` [19:31] sdeziel: ahh, thank you. It looks like the other test is: chown -R dev:www-data /home/dev/myapp [19:40] foo: hmm, I tested my SupplementalGroups= theory and it's wrong apparently, you don't need this for nginx to pick up the supplementary groups. Restarting nginx was all that was needed, no need for edits [19:40] sdeziel: huh, thank you for testing. I appreciate your thoroughness [20:06] foo: I whipped up a quick PoC trying to mimic your setup and I don't have the permission denied problem you have :/ [20:19] sdeziel: huh. [20:19] sdeziel: FWIW, this was a brand new Ubuntu 22.04 set up. Stock everything. I pasted the service conf, nginx conf... I can't think of what else is here that's nuanced. /home/dev is a dev user, but I doubt that matters. [20:24] foo: I setup a fresh 22.04 container, put nginx-light and created a tiny python3 service using your myapp-api.service unit... [20:26] sdeziel: I really really really appreciate you taking the time to do that. I wonder what else I can have on here that could be causing that [20:39] foo: are you using ACLs? [20:42] sdeziel: negative [20:44] foo: maybe try enabling debug error logging for your `location /`? `error_log /var/log/nginx/myapp.err debug;` [20:48] Oh, not a bad idea - can give that a shot. [21:33] sdeziel: fwiw debug logging is for the whole log for a server block not just a location [21:33] it's also noisy as sin (that looks like nginx) [21:34] teward: https://nginx.org/en/docs/ngx_core_module.html#error_log mentions `location` as a valid context [21:34] but yeah, it's quite noisy [21:34] sdeziel: agreed, it's VERY noisy, i'm a little behind thoug hand am still doing scrollback to figure out what exactly is going on [21:34] and no matter how I break the access to the Unix socket, the only error I can get is the permission denied... [21:35] it won't tell if it cannot connect due to not having "execute right" on something or the path... [21:35] welcome to how C relays access perms or not [21:35] you either 'have permission' or 'don't [21:36] that's not nginx that's just how C works : [21:36] C / filesystem notices [21:36] *yawns* i need more coffee [21:36] let me play some catchup [21:40] foo: i would NOT have things live in your home directory [21:41] nginx runs as www-data by default in Ubuntu. per your namei, your socket is at /home/dev/myapp/live/live.sock. All home directories default to 750 which PREVENTS non-owner and non-group access to the directory at the /home/dev/ level. [21:42] www-data does not have the ability to read or traverse into /home/dev/ [21:42] you have two choices: [21:42] (1) give 'other users' traverse/read permissions to the base directory: (UNSAFE) - sudo chmod o+rx /home/dev [21:43] (2) place your socket somewhere else such as `/var/run` or `/run` or in `/tmp` with proper 666 permissions (read/write). [21:43] third choice: ACLs [21:44] (3) explicitly give ACL permissions to the www-data user for your home directory (also still unsafe): sudo apt install acl && sudo setfacl -m u:www-data:rx /home/dev/ [21:44] (you still have to install ACL to make that work) [21:45] i don't think the issue lies in your *python* app but instead actually your intermediate directories between / and the actual full location of your socket. [21:45] sdeziel: ^^ [21:45] debugging logs won't help it'll just say permission denied [21:46] but such things should be taking after programs / applications that *make* sockets readable/writable by other people/users (a-la PHP, etc.) rather than run everything in home directory [22:22] teward: that's the thing, `www-data` is a member of the `dev` group [22:22] teward: and yeah, I am pretty sure the problem isn't the Python app [22:25] foo: I agree with teward's suggestion of putting the socket under `/run`. You can even tell systemd to create `/run/myapp-api` (`RuntimeDirectory=`) to contain the socket file === arraybolt3_ is now known as arraybolt3 [22:31] foo: strace'ing the nginx child handling the request (the one getting a perm denied) might be useful [22:31] probably more useful than that debug log [22:36] sdeziel: i would not trust www-data being a member of the grou pas far as i can throw it [22:36] whether the system says so or not [22:36] it's the *least* reliable mechanism to validate access [22:36] (runtimedirectory=... is the better approach by *far*)