[08:27] <guesswhat> is some amd gpu driver required for 6800U laptop cpu, or is it already included in kernel?
[09:11] <odc> guesswhat: it's in the kernel, but I'm not sure if 5.19 is recent enough 
[09:11] <guesswhat> odc got spikes 100% in google meeting
[09:11] <guesswhat> chrome with hw accel enabled
[09:12] <odc> Ouch
[09:12] <guesswhat> `lspci -k | grep -A 3 -E "(VGA|3D)"` using amdgpu module
[09:14] <guesswhat> https://www.phoronix.com/news/AMD-Radeon-680M-Linux-Mesa
[09:14] <guesswhat> shoud be fine
[18:22] <foo> Hmm, nginx-logs throwing this: live/live.sock failed (13: Permission denied) while connecting to upstream, client ... live.sock is 777 which is what I have on another system which works. Ubuntu 22.04, new system. Perms seem ok. Python + gunicorn + nginx. Any sugggestions? 
[18:22] <foo> Spent hours on this already, running out of ideas
[18:29] <adac> Hi there! it is strange my htop and iotop command on the server are loadning really really slow. What can cause that actually?
[18:30] <adac> I was checking the read/write and disk io, but all seems normal in this regard
[18:30] <adac> while the server has some decent load, it still has a lot of free resources
[18:30] <adac> How can I properly debug this?
[18:40] <effendy[m]> adac: what else is slow?
[18:41] <effendy[m]> maybe they're trying to read off a remote disk/share of some sorts also?
[18:42] <sdeziel> foo: `namei -l /path/to/live/live.sock` would help so would seeing your nginx config
[18:42] <adac> effendy[m], actually "ls" and all this normal commands, "pwd" all this is working very fast
[18:47] <adac> effendy[m], actually I run a bare metal kubernetes cluster on this server (3 servers actually). Within this cluster I do some smb mounts with this guy: https://github.com/kubernetes-csi/csi-driver-smb but other then that, and this never made any issues so far on other clusters I have, I do not read from remote disks or so
[18:49] <effendy[m]> It might be related to that though, but this is just a supposition.
[18:49] <effendy[m]> If there was any way you could disable it temporarily, it would interesting to see if it reacts in the same way.
[18:53] <adac> effendy[m], I will try and uninstall it bit later. Just have to leave now for some time. Will inform you a bit later! Thanks for the hint!
[18:53] <effendy[m]> :)
[18:55] <adac> effendy[m], inform only if you like to know if it worked of course :-)
[18:56] <effendy[m]> adac: I do, of course!
[18:56] <adac> perfect! :-)
[19:15] <foo> sdeziel: thank you. https://bpa.st/54FQ - it's all stock
[19:15] <sdeziel> foo: please use `namei -l` as it report on the ownership/perms
[19:16] <sdeziel> foo: please also paste the relevant nginx/conf.d and nginx/sites-enabled files
[19:24] <foo> sdeziel: err, sorry, I was going to say... that didn't seem too helpful. pebkac
[19:24] <foo> sdeziel: thank you! https://bpa.st/R77Q
[19:24] <foo> sdeziel: note, groups www-data returns www-data : www-data dev
[19:25] <foo> sdeziel: I'm really curious, I've done this set up many times and not sure what I'm missing. 
[19:25] <sdeziel> foo: you likely need to edit nginx's systemd unit to add `SupplementaryGroups=dev` to it. `sudo systemctl edit nginx`
[19:26] <sdeziel> foo: cause I don't think that nginx will load the supplementaty groups when it switch UID/GID from root:root to www-data:www-data
[19:29] <foo> sdeziel: thanks, testing now. I think I want to uncomment out the service from sudo systemctl edit nginx and then add SupplementaryGroups=dev to it, *googles
[19:30] <sdeziel> foo: no, you just need `[Service]\nSupplementaryGroups=dev\n`
[19:31] <foo> sdeziel: ahh, thank you. It looks like the other test is: chown -R dev:www-data /home/dev/myapp
[19:40] <sdeziel> foo: hmm, I tested my SupplementalGroups= theory and it's wrong apparently, you don't need this for nginx to pick up the supplementary groups. Restarting nginx was all that was needed, no need for edits
[19:40] <foo> sdeziel: huh, thank you for testing. I appreciate your thoroughness 
[20:06] <sdeziel> foo: I whipped up a quick PoC trying to mimic your setup and I don't have the permission denied problem you have :/
[20:19] <foo> sdeziel: huh. 
[20:19] <foo> sdeziel: FWIW, this was a brand new Ubuntu 22.04 set up. Stock everything. I pasted the service conf, nginx conf... I can't think of what else is here that's nuanced. /home/dev is a dev user, but I doubt that matters. 
[20:24] <sdeziel> foo: I setup a fresh 22.04 container, put nginx-light and created a tiny python3 service using your myapp-api.service unit...
[20:26] <foo> sdeziel: I really really really appreciate you taking the time to do that. I wonder what else I can have on here that could be causing that
[20:39] <sdeziel> foo: are you using ACLs?
[20:42] <foo> sdeziel: negative 
[20:44] <sdeziel> foo: maybe try enabling debug error logging for your `location /`? `error_log /var/log/nginx/myapp.err debug;`
[20:48] <foo> Oh, not a bad idea - can give that a shot. 
[21:33] <teward> sdeziel: fwiw debug logging is for the whole log for a server block not just a location
[21:33] <teward> it's also noisy as sin (that looks like nginx)
[21:34] <sdeziel> teward: https://nginx.org/en/docs/ngx_core_module.html#error_log mentions `location` as a valid context
[21:34] <sdeziel> but yeah, it's quite noisy
[21:34] <teward> sdeziel: agreed, it's VERY noisy, i'm a little behind thoug hand am still doing scrollback to figure out what exactly is going on
[21:34] <sdeziel> and no matter how I break the access to the Unix socket, the only error I can get is the permission denied...
[21:35] <sdeziel> it won't tell if it cannot connect due to not having "execute right" on something or the path...
[21:35] <teward> welcome to how C relays access perms or not
[21:35] <teward> you either 'have permission' or 'don't
[21:36] <teward> that's not nginx that's just how C works :
[21:36] <teward> C / filesystem notices
[21:36] <teward> *yawns* i need more coffee
[21:36] <teward> let me play some catchup
[21:40] <teward> foo: i would NOT have things live in your home directory
[21:41] <teward> nginx runs as www-data by default in Ubuntu.  per your namei, your socket is at /home/dev/myapp/live/live.sock.  All home directories default to 750 which PREVENTS non-owner and non-group access to the directory at the /home/dev/ level.
[21:42] <teward> www-data does not have the ability to read or traverse into /home/dev/
[21:42] <teward> you have two choices:
[21:42] <teward> (1) give 'other users' traverse/read permissions to the base directory: (UNSAFE) - sudo chmod o+rx /home/dev
[21:43] <teward> (2) place your socket somewhere else such as `/var/run` or `/run` or in `/tmp` with proper 666 permissions (read/write).
[21:43] <teward> third choice: ACLs
[21:44] <teward> (3) explicitly give ACL permissions to the www-data user for your home directory (also still unsafe): sudo apt install acl && sudo setfacl -m u:www-data:rx /home/dev/
[21:44] <teward> (you still have to install ACL to make that work)
[21:45] <teward> i don't think the issue lies in your *python* app but instead actually your intermediate directories between / and the actual full location of your socket.
[21:45] <teward> sdeziel: ^^
[21:45] <teward> debugging logs won't help it'll just say permission denied
[21:46] <teward> but such things should be taking after programs / applications that *make* sockets readable/writable by other people/users (a-la PHP, etc.) rather than run everything in home directory
[22:22] <sdeziel> teward: that's the thing, `www-data` is a member of the `dev` group
[22:22] <sdeziel> teward: and yeah, I am pretty sure the problem isn't the Python app
[22:25] <sdeziel> foo: I agree with teward's suggestion of putting the socket under `/run`. You can even tell systemd to create `/run/myapp-api` (`RuntimeDirectory=`) to contain the socket file
[22:31] <sdeziel> foo: strace'ing the nginx child handling the request (the one getting a perm denied) might be useful
[22:31] <sdeziel> probably more useful than that debug log
[22:36] <teward> sdeziel: i would not trust www-data being a member of the grou pas far as i can throw it
[22:36] <teward> whether the system says so or not
[22:36] <teward> it's the *least* reliable mechanism to validate access
[22:36] <teward> (runtimedirectory=... is the better approach by *far*)