| DEvil0000 | so nobody here can help me with this kernel issue? | 07:41 |
|---|---|---|
| apw | DEvil0000, "this" ? | 09:27 |
| ogra_ | apw, i have a user running CI tests in qemu/kvm for ubuntu on top of a Centos box ... his VMs fill up /run/udev/data with tons of cgroup entries over time (ls output is at https://bugs.launchpad.net/snapd/+bug/1687507/+attachment/4870568/+files/run_udev_data.log) ...https://bugs.launchpad.net/snapd/+bug/1687507 any idea ? (he runs a hwe kernel in the VMs) | 10:22 |
| ubot5` | Ubuntu bug 1687507 in snapd "Memory leak (/tmp file system filling up)" [Undecided,New] | 10:22 |
| apw | ogra_, so he is saying that within the VM we are dropping those failes in /run, or are they in his host ? | 10:28 |
| apw | ogra_, but if they are /run/udev/data files i would assume they are related to systemd in whichever ? | 10:28 |
| ogra_ | apw, they are inside the VM ... he runs his VMs with 200MB ram each and after a week or so they all run out of ram | 10:29 |
| apw | ogra_, ok so then our systemd seems most likely given the names | 10:29 |
| ogra_ | i would expect udev to only react to device creastion here | 10:29 |
| apw | ogra_, which you do for every snap install, you make a new loop | 10:30 |
| ogra_ | well, thats not loop devices there | 10:30 |
| apw | no it is session things, perhaps one for every run of a snap command | 10:31 |
| ogra_ | zyga_, ^^^ ? | 10:32 |
| zyga_ | ogra_: looking | 10:32 |
| apw | ogra_, but they look like regular files, and they arn't in a directory the kernel makes up out of the ether, they look like actual disk blocks | 10:32 |
| ogra_ | well ... i see bits like: -rw-r--r-- 1 root root 31 May 1 06:08 +cgroup:kmalloc-1024(30177:apt-daily.service | 10:32 |
| zyga_ | ogra_: I saw this bug | 10:32 |
| ogra_ | thats definitely not from snapd | 10:32 |
| zyga_ | ogra_: perhaps the real memory is hidden somewhere else that we don't see | 10:32 |
| zyga_ | aha | 10:32 |
| zyga_ | do you have a log of what is in the slab? | 10:33 |
| apw | ogra_, no but that doesn't make it a kernel thing. that is a systemd unit which ran | 10:33 |
| apw | zyga_, these are actual files in a directory on a ramfs | 10:33 |
| ogra_ | zyga_, we only have the ls output of the udev db atm | 10:33 |
| apw | the memory is being using storing them | 10:33 |
| apw | i would suggest whatever is making the sessions is triggering udev to leak an internal file | 10:33 |
| zyga_ | apw: aha, | 10:34 |
| zyga_ | apw: is that an actual ramfs or did you mean tmpfs? | 10:34 |
| ogra_ | its a tmpfs ... | 10:35 |
| ogra_ | usually /run has 10% of your ram | 10:35 |
| apw | zyga_, tmpfs prolly, as it is /run | 10:35 |
| apw | so according to the other logs in teh forum, i think there is a session being logged in syslog for every execution of his snap commands | 10:36 |
| zyga_ | apw: what do you mean by session specifically? (I'm not fully aware of how various "sessions" are managed by linux) | 10:37 |
| ogra_ | Apr 28 09:36:50 ci-comp11-dut systemd[1]: Started Session 1817 of user root. | 10:37 |
| apw | zyga_, that is userspace shit :) | 10:37 |
| ogra_ | yxou mean that line i guess | 10:37 |
| apw | ogra_, yeah, that is what it looks like to me | 10:38 |
| apw | perhaps that is an install of a snap, dunno, he is implying he is doing repeated installs | 10:38 |
| ogra_ | yeah, no wonder if he does CI | 10:39 |
| ogra_ | (though he should simply kill the VM and use a fresh one to do it right :) ) | 10:39 |
| apw | ogra_, he could, but i would open a systemd task on that bug at least, as it looks to be leaking, or if it is not they might be able to tell us better waht the hell those files represent | 10:40 |
| apw | ogra_, and what the reporter is doing wrong (this is systemd after all) | 10:41 |
| apw | "you didn't want to do it like _that_" | 10:41 |
| zyga_ | do you guys think there's an actual snapd bug somewhere? | 10:41 |
| ogra_ | zyga_, unlikely given the type of files created there | 10:41 |
| apw | zyga_, without knowing what the session represents ... you might ask for one of the file content as well | 10:41 |
| LocutusOfBorg | [12:42:15] <LocutusOfBorg> hello cjwatson wrt the git/ssh repo cloning issue, I finally found it | 10:43 |
| LocutusOfBorg | [12:42:16] <LocutusOfBorg> http://kernel.ubuntu.com/git/cking/stress-ng.git/ | 10:43 |
| LocutusOfBorg | [12:42:31] <LocutusOfBorg> this one is git only clonable (ok not really a launchpad host) | 10:43 |
| LocutusOfBorg | cking, ^^ :) | 10:43 |
| LocutusOfBorg | do you mind having https too? | 10:43 |
| apw | LocutusOfBorg, what does that mean, ie which protocols don't work | 10:43 |
| LocutusOfBorg | git clone https:// | 10:43 |
| * apw wonders how that works ... hmmm | 10:43 | |
| LocutusOfBorg | if you look at the page, it is not even shown as "clonable" | 10:44 |
| LocutusOfBorg | probably some configuration needs an enable | 10:44 |
| apw | that host is somewhat of a legacy mess ... no idea if it even can do https | 10:44 |
| apw | i suspect it does not at all ... | 10:45 |
| cking | I guess that's the reason | 10:45 |
| apw | cking, would it make more sense to just mirror that into LP and be happy ? | 10:45 |
| LocutusOfBorg | :( | 10:45 |
| LocutusOfBorg | we have git protocol blocked here at work | 10:45 |
| apw | cking, we could even add it to the primary mirroring list if you care to keep both working | 10:45 |
| infinity | LocutusOfBorg: Tell the people who run your work network to stop being terrible? | 10:45 |
| cking | apw, I'd prefer LP to mirror the original git repo if thats possible | 10:46 |
| apw | infinity, but but they need to record everything you do, oh except https:// cause you can't do anything dodgy with that | 10:46 |
| apw | cking, i guess it depends where in the namespace it goes to | 10:46 |
| infinity | cking: Wasn't there a goal to eventually get *all* the master repos off of wani, though? | 10:47 |
| LocutusOfBorg | infinity, it is the customer's network :) unfortunately security team has a blacklist everything opinion | 10:47 |
| cking | infinity, yeah, I'm being lazy | 10:47 |
| apw | LocutusOfBorg, letting https:// out basically means they don't stop anything | 10:47 |
| cking | I have it mirrored at https://github.com/ColinIanKing/stress-ng too | 10:47 |
| apw | cking, oh ... heh | 10:48 |
| LocutusOfBorg | apw, well, somebody was doing reverse ssh over git port :) | 10:48 |
| LocutusOfBorg | they even blocked ssh on 443 | 10:48 |
| apw | LocutusOfBorg, and they are now doing it over https no doubt | 10:48 |
| cking | i guess you could download https://github.com/ColinIanKing/stress-ng/archive/master.zip | 10:48 |
| infinity | LocutusOfBorg: You can't "block ssh on 443". I mean, you can block the pre-ssh handshake, but you can't stop people from setting up ssl tunnels and then going to town. | 10:48 |
| apw | cking, he can clone from github if it is a mirror | 10:49 |
| infinity | LocutusOfBorg: So, indeed, as Andy says, if you allow 443/https, you allow everything. Just in a way you can't filter/inspect. | 10:49 |
| LocutusOfBorg | infinity, yes, of course, they just made things harder | 10:49 |
| LocutusOfBorg | anyhow git over ssh works nicely, and I don't want to hack things on a build server | 10:49 |
| apw | "harder" for someone who was reverse tunnelling ssh over the git port, is not going to be actually hard | 10:49 |
| cking | apw, I keep a mirror of all my repos on github because folks in .cn sometimes can't access my repos any other way | 10:50 |
| apw | cking, greak, then LocutusOfBorg should mirror from | 10:50 |
| apw | github for now, and we can think about where that repo should move to | 10:50 |
| apw | at our leisure | 10:50 |
| cking | apw, the "at our leisure" generally means never | 10:50 |
| apw | cking, no we have cards now | 10:51 |
| cking | apw, OK, I'll slap it on the card deck | 10:51 |
| apw | cking, so ... who owns that, is it cking or is it c-k-t or something else | 10:51 |
| cking | apw, tis me | 10:51 |
| apw | then i cannot mirror _to_ your LP, but if you switched master to lp:~cking i could mirror it back to kernel, and you could add the symlink there | 10:51 |
| apw | cking, as an aside does github auto-mirror for you ? | 10:52 |
| cking | apw, sounds like a plan | 10:52 |
| cking | cking, I generally push to my repo and mirror one each push | 10:53 |
| LocutusOfBorg | thanks! | 10:53 |
| LocutusOfBorg | cking, I'm pretty sure there is some automatic push feature on github | 10:53 |
| infinity | cking: I'd argue that if you're going to the trouble of moving it, you might want it owned by a stress-ng-hackers team or something instead of cking. Future proof it a bit. | 10:53 |
| cking | i suspect there is | 10:53 |
| LocutusOfBorg | but I never found how does it work | 10:53 |
| cking | infinity, yeah, sounds like a good idea | 10:53 |
| apw | infinity, in which case we could add the mirror-bot | 10:54 |
| apw | to that team until we get the fine-grained permissions we are meant to have | 10:54 |
| apw | cking, i assume you have an LP project already anyhow | 10:54 |
| cking | https://launchpad.net/stress-ng | 10:55 |
| LocutusOfBorg | or maybe just | 10:55 |
| LocutusOfBorg | add an hook | 10:55 |
| LocutusOfBorg | BTW the network is mostly windows only, I'm the only one with such connection issues, and I don't want to complain to /dev/null network admins, corporate networks can be time consuming | 10:57 |
| LocutusOfBorg | (I do tethering and live happy for now) | 10:57 |
| apw | LocutusOfBorg, well it sounds like you have a work-around, use the github mirror as that is https: already | 10:58 |
| LocutusOfBorg | yes, thanks! I forwarded this helpful message to my colleague | 10:59 |
| cking | LocutusOfBorg, and let me know if you find any stress-ng issues too | 11:02 |
| LocutusOfBorg | cking, it is an awesome tool! we use it to stress our custom yocto BSP :) | 11:03 |
| cking | LocutusOfBorg, thanks, that's good to know. can you star it on the git hub page :-) | 11:04 |
| LocutusOfBorg | done :) | 11:04 |
| LocutusOfBorg | I didn't even star my projects | 11:04 |
| cking | \o/ | 11:04 |
| LocutusOfBorg | I think I had some feature requests to do for this project, something I feel "hey that would be nice" but I forgot them | 11:05 |
| cking | LocutusOfBorg, well, I'm very open to new feature requests and/or patches too :-) | 11:05 |
| LocutusOfBorg | I probably remember the need to test ram overloading and some cpu micro (e.g. stress test only one single core) or something like that | 11:07 |
| LocutusOfBorg | it happened 1.5 year ago, too much time for my brain to remember | 11:07 |
| LocutusOfBorg | :) | 11:07 |
| cking | ah, stress-ng has progressed a lot since then | 11:07 |
| LocutusOfBorg | sure, I would like to give it a new try :) | 11:07 |
| LocutusOfBorg | flashbench, this is a tool we had to add because stress-ng was not testing flashes :) | 11:11 |
| DEvil0000 | 10:10:48 - DEvil0000: I have a issue with my 14.04 and hope to find some help here. Description: https://hastebin.com/raw/fareguzego | 11:35 |
| DEvil0000 | 10:11:10 - DEvil0000: in short: core dumps of 32Bit processes on a 64Bit OS are not correct somehow. so its about gdb, kernel, ubuntu 14.04/16.04 and packages | 11:35 |
| DEvil0000 | @apw | 11:37 |
| apw | DEvil0000, hmmm so you are saying 32bit core dumps differ depending whether they are dumped by the lts-backport 4.4 or the native 4.4 from a later release ? | 11:39 |
| apw | that is somewhat unexpected as the only real difference is normally the compiler used to build it | 11:43 |
| apw | DEvil0000, you should file a bug against the kernel and add a task for gdb as well i suspect, and drop the number in here | 11:44 |
| DEvil0000 | apw: yes, this is what I see. resulting in incorrect/broken stack trace in gdb later when dumped with the backport kernel | 12:03 |
| apw | DEvil0000, that is most odd as those kernels are essentially the same other than where they are built | 12:03 |
| DEvil0000 | apw: when I install the native 4.4 packages on my 14.04 it works as expected | 12:03 |
| apw | that makes little to no sense in reality | 12:04 |
| apw | but, it is happening, so please file a bug against the kernel with all that detail and how you reproduced it | 12:04 |
| apw | i assume a simple 32bit proggy which does abort will exhibit this behaviour | 12:04 |
| DEvil0000 | apw: not sure - this seams to be quite likely for my processes with about 30 threads but less likely with less threads. hard to find a good example process | 12:05 |
| DEvil0000 | apw: main thread dumps seam to be ok most of the time | 12:07 |
| apw | hmmm | 12:07 |
| apw | it is not 100% reproducible. that is ever odder | 12:08 |
| DEvil0000 | is it save to use the native 4.4 kernel packages instead of the backport ones? | 12:08 |
| DEvil0000 | is maybe the patch level different in some way? | 12:09 |
| DEvil0000 | or maybe a lib needed to compile the kernel? | 12:09 |
| apw | they are identical as far as i know from a patches applied perspective (on x86 at least) | 12:10 |
| apw | and they do not use external libraries | 12:10 |
| apw | safe is a relative term, you won't get updates if you do tha | 12:10 |
| apw | taht | 12:10 |
| DEvil0000 | I have found some reports from others via google having the same issues. | 12:12 |
| DEvil0000 | sometimes with gcore gdb in the title. some for debian. | 12:12 |
| DEvil0000 | updates are not a issue in this case. we have our own pool which is a less updated ubuntu pool clone | 12:12 |
| DEvil0000 | (but yes I tested it on plain and full updated 14.04) | 12:16 |
| DEvil0000 | maybe it is somehow related to this: http://www.bigeng.io/recovering-redis-data-with-gdb/ | 12:18 |
| DEvil0000 | looks similar | 12:18 |
| DEvil0000 | @apw: so you tell me using the native 4.4 should be no issue. why don't you use the same packages for both versions then? maybe even from a common package pool? | 12:25 |
| apw | i didn't actually say there would be no issues but regardless | 12:25 |
| apw | the thing is not compiled using tools you have so no out of tree drivers can be relied on for one | 12:26 |
| apw | that is why we build a native version of the kernel for the series it is run in | 12:26 |
| apw | as the differences between the two are minor, at best, i have no evidence you are not just getting lucky and the problem will come back | 12:27 |
| apw | DEvil0000, nope there are no code differences between the two, /me checks config | 12:29 |
| apw | DEvil0000, and the only thing there is how stack-crashing is detected (due to compiler limitations) but that should not be visible in userspace either | 12:30 |
| === daniel is now known as Guest47234 | ||
| === Guest47234 is now known as Odd_Bloke | ||
| === JanC is now known as Guest15747 | ||
| === JanC_ is now known as JanC | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!