/srv/irclogs.ubuntu.com/2014/04/29/#ubuntu-kernel.txt

fallacyQuestion.  The nouveau driver in Ubuntu 14.04 kernel (3.11.0-20) doesn't work on my system but works fine in the upstream kernel from a later version (3.14.1-031401).  Should I report that as a bug for Ubuntu 14.04?00:58
fallacyoops, wrong info.  It's broken in 3.13.0-24 (which is Ubuntu 14.04).  Works in 3.11.0-20 (from Ubuntu 13.10) and in upstream Ubuntu build 3.14.1-03140100:59
fallacyHaving those upstream packages are really useful for debugging01:03
=== sz0 is now known as sz0`
=== sz0` is now known as sz0
=== sz0 is now known as sz0`
=== sz0` is now known as sz0
nilujeI run nbd-client from initramfs (ubuntu 14.04) to create the device /dev/nbd0, on which the rootfs is mounted. When I shutdown or reboot, I get a kernel panic because some I/O are done and the device is not accessible (http://pastebin.com/P4qWXUFC). I guess nbd-client is killed too early by init. Using systemd, the solution is described here07:15
nilujehttp://www.freedesktop.org/wiki/Software/systemd/RootStorageDaemons/, and renaming nbd-client to @nbd-client (or specifing the option -m in nbd-client 3.8, (https://github.com/yoe/nbd/blob/master/nbd-client.c#L532) would probably solve the problem. Any idea how I should solve the issue?07:15
infinityniluje: See /run/sendsigs.omit.d07:18
infinityniluje: Any PIDs in there will be bypassed by the "kill the world" bit on shutdown.07:18
nilujeinfinity: I'm giving a look07:19
nilujeinfinity: thanks a *LOT*07:26
nilujeit works fine, no more kernel panic07:26
nilujehowever, when the process is killed?07:26
nilujethe connection doesn't seem to be cleanly closed, I guess the nbd-client didn't send a shutdown(2) to the nbd-server07:27
infinityniluje: So, I'm not sure how best you'd mangle that.  S90{halt,reboot} in rc[06].d will fail if the root disappears.07:35
infinityniluje: S60umountroot should at least be making sure the disappearing client doesn't cause data loss.  Assuming nbd has a concept of read-only mounts.07:36
nilujeinfinity: I'm looking into /etc/rc6.d/S60umountroot and I don't really get what I could do here. Could you tell me a bit more?07:42
infinityniluje: Well, nothing there, really.  Just pointint out that that should be remounting your root readonly (if nbd supports such a thing).07:43
infinityniluje: There's a fundamental chicken/egg issue where we can *never* actually umount root completely, or the reboot/halt would fail, cause you'd no longer have binaries to run.07:43
nilujehm right07:44
nilujeinfinity: I tried a mount -t ext4 -o remount,ro /dev/nbd0 / yesterday and it works well, the file system is readonly, can't write anything on it07:45
infinityniluje: Right, and shutdown should be doing that for you, it's just never going to exit the nbd client.07:46
infinityniluje: There are certainly ways we could do this more cleverly, given a bunch of re-engineering (like copying everything we need to run shutdown(8) into a tmpfs, umounting /, and then rebooting), but that's a fair bit of fiddling to get right and has never really been an issue worth caring about, since settling a read-only FS should guarantee that an unclean shudown doesn't really matter.07:48
nilujeinfinity: there's something I don't understand07:50
nilujeYou made me understand why the rootfs can't be unmounted07:50
nilujeWhen I put the nbd-client PID in /run/sendsigs.omit.d, the nbd-client process will not be killed upon reboot07:51
nilujein /etc/rc6.d/S60umountroot, I see a [...] mount    $MOUNT_FORCE_OPT -n -o remount,ro -t dummytype dummydev [...]07:51
nilujewhich, if I understand well, will flush what needs to be flushed and assure me the filesystem will be "clean" once the reboot will occure (and the fs will be remounted rw)07:52
nilujeam I wrong somewhere?07:52
infinityRemounted ro, you mean.07:54
nilujeinfinity: remounted rw07:55
infinityniluje: The whole point of that line is to remount ro.07:55
nilujeI mean, the fs is first remounted ro, then when the system restarts, it is remounted rw07:55
infinityniluje: Oh, sure.  Yes.07:55
infinityniluje: rw when it comes back, obviously.07:55
nilujeok07:55
nilujeand the "ro" part is to flush pending writes, right?07:56
nilujeinfinity: what I don't understand is, if the process isn't killed, why can't I reconnect on nbd-server upon reboot?08:01
infinityThat, I don't know.  I've never used nbd.08:01
nilujethinking about it08:02
nilujeit seems pretty normal, I guess the network it shutdown too08:02
nilujeis*08:02
nilujeI just added some logging in the initramfs to run ps just before launching nbd-client. What I do is: start the machine, put nbd-client PID in /run/sendsigs.omit.d/, reboot. Then, if the process hasn't been killed, shouldn't I see it listed? (it is not)08:07
infinityniluje: How would the process be listed after a reboot?08:09
nilujeinfinity: I'm lost :-(08:15
nilujeI don't understand when the process is killed, when listed in /run/sendsigs.omit.d08:16
infinityniluje: It's not.  The machine reboots.08:25
infinityniluje: Which, of course, kills everything.  But not with a signal.  It just dies.  Cause the machine is dead. :P08:25
nilujeinfinity: ok08:28
niluje:p08:28
nilujeso I don't understand why you can't unmount the rootfs08:28
jk-heya ikepanhc, do you know if Alex is around?08:30
ikepanhcjk-: hi Jeremy :D08:30
ikepanhcjk-: give me a sec08:30
jk-ikepanhc: thanks! :)08:30
alexhungjk-, ping08:31
jk-hi alexhung, just had a few fwts questions; is there a better channel to use?08:32
infinityniluje: Because if you unmount it, how can you call "halt" or "reboot"?08:32
infinityniluje: Or the shell script that is S90reboot...08:32
nilujehm right :p08:33
nilujeinfinity: is there anywhere a ressource explaining step by step what happens during a reboot?08:35
infinityniluje: I doubt it.  It's mostly documented in code.08:35
nilujeok08:35
nilujewouldn't it be possible to mount a tmpfs in which I'd get something executed that'd kill the process at the very last moment?08:36
infinityniluje: I hinted above at that being the very complicated solution to trying to not need root to reboot.08:37
infinityniluje: But it's overkill for literally every use case I've ever run into.08:37
infinityniluje: If nbd-server is exploding when a client disappears, that's a bug, IMO, and should be fixed, not worked around.08:37
infinityniluje: The maintainer of nbd (wouter) happens to be a very nice fellow.  You might ask him about your issues.08:40
infinityniluje: I will continue to maintain the opinion that if the server can't handle a client disappearing on reboot and trying to reconnect later with a new instance, it's fundamentally broken. :P08:40
apwyeah what happens when you lose power or the network for reasons outside your control, you end up in the same mess08:41
nilujeyup08:41
nilujeinfinity: I'm going to dig a bit more before contacting the guys behind nbd.08:42
nilujethanks a lot for your time08:42
infinityniluje: Sure.  He happens to be a decent dude with access to lovely chocolate.08:42
infinityHe tried to kill me with said chocolate last time I saw him.08:42
infinityEvil Belgians.08:42
niluje:D08:43
smbThats killing someone softly... :)08:43
nilujejust wondering, are you a ubuntu committer, a kernel committer, or something?08:43
infinitysmb: That's wouter's style.08:43
infinityniluje: Ubuntu and Debian developer, and random hacker on several upstream projects.  Also, just a guy who can't sleep (but I'm going to go try).08:44
nilujearen't you in london? at 9am?!08:44
infinityniluje: I'm in Calgary.  It's 2:44am.08:44
apwheh ... very few people are in london08:44
smbapw, Beside all the inhabitants08:45
infinityapw: Hey, to be fair, I'm in London more than some people who claim to live there.08:45
apwtrue, but as you arn't, the irc contingent is severely reduced08:45
nilujeok :-) thanks a lot again08:45
apwfallacy (not), yes file a bug against it indeed08:47
=== apw changed the topic of #ubuntu-kernel to: Home: https://wiki.ubuntu.com/Kernel/ || Ubuntu Kernel Team Meeting - Tues April 29th, 2014 - 17:00 UTC || If you have a question just ask, and do wait around for an answer! If the question is should I file a bug for something, likely you can assume yes.
smbapw, Bored of repeating that answer you are?08:50
smb:)08:50
apwheh08:51
mlankhorst;D08:52
smbmlankhorst, So about my whining part #1. bug 1298517 has screenshots about it. ;)08:57
ubot2Launchpad bug 1298517 in mesa (Ubuntu) "Rendering issues in unity with xserver-modeset" [High,Triaged] https://launchpad.net/bugs/129851708:57
mlankhorstmodesetting hates you :P08:58
smbWell, its mostly nearly working. It's just that unity developers obviously don't test in VMs and if they do, they don't use terminals. ;-P09:00
mlankhorstllvmpipe bugs have a low chance of getting fixed, though..09:01
smbUnless "he" stumbles over them trying to present a Ubuntu desktop running in THE CLOUD™09:02
amitk:)09:03
smbamitk, Oh look who is there. Hello stranger. ;)09:04
amitksmb: that should go on the quotes page, too bad I don't have access anymore ;)09:05
smbamitk, Not again. :-P09:05
amitksmb: for the record, I'm always lurking around this channel..09:08
amitkstill use ubuntu core, none of the desktop stuff though09:09
smbamitk, Yeah, and I could tell if I cared to look at the list of people. But I consider you under the towel unless you say something. :)09:09
=== sz0 is now known as sz0`
=== sz0` is now known as sz0
=== sz0 is now known as sz0`
=== sz0` is now known as sz0
apwsmb, i osmetimes wonder why i even have that thing enabled (the list)10:43
smbapw, You got a point there10:45
* apw idly wonders if it can be turned off in his interface .... hmm10:46
=== deffrag__ is now known as deffrag_
apwoh i hate that, spend 3 hours debugging a test suite failure "you kernel is broken" to ... of course ... find the test is broken14:09
mlankhorstsolution: spend less time on testing. ;-)14:16
sconklinarges, fwiw that network namespace cleanup bug has been repro'd by someone else on 3.13.0-24 (upstream bug). I think we have an easy way to reproduce it, working on setting that up now.14:21
argescool14:21
sconklinalso interesting is that it's also been seen on bare metal14:26
diwicapw, that's what we call 'quality time'14:32
apwmlankhorst, you are right .... rm -rf tests ... better14:35
stgrabersconklin: is that the bug where the refcount appears to fail and the kernel thinks there's still something holding eth0 in an otherwise unused ns?14:36
sconklinstgraber: https://bugzilla.kernel.org/show_bug.cgi?id=6519114:36
ubot2bugzilla.kernel.org bug 65191 in Netfilter/Iptables "BUG in nf_nat_cleanup_conntrack" [Normal,New]14:36
stgraberah, I haven't seen that one in a while :)14:37
sconklinwe see it when we scale up and create a bunch of containers, then scale down and remove all of them14:38
jsalisbury**14:50
jsalisbury** Ubuntu Kernel Team Meeting - Today @ 17:00 UTC - #ubuntu-meeting14:50
jsalisbury**14:50
sconklinstgraber: are there any multiple-container scaling tests in the LXC tests?14:55
stgrabersconklin: we have lxc-test-concurrent that tests multi-threaded LXC but the LXC tests are typically simple busybox so they don't do much15:01
sconklinok. Thanks. I'll see whether we have anything that can run outside of our environment to stress this15:03
argessconklin: in your testcase you'll also need to do something that is generating conntrack entries (and allowing them to timeout or get cleaned up)15:07
sconklinyeah, it's not trivial, and I'm not terribly optimistic15:08
BenCapw: I’m working on fixing powerpc64-emb right now. What’s my timeframe on getting it into the next upload?15:22
bjfBenC, you now have 3 weeks :-)15:23
BenCbjf: Not sure if that’s good or bad :)15:24
bjfBenC, you missed the friday cutoff. i'll be uploading new trusty bits today ... if you are talking utopic, that's a different story15:25
BenCbjf: utopic15:25
bjfBenC, ok, then apw and/or rtg can tell you that15:25
BenCI still keep reading that as micro-topic15:25
apwBenC, well we want to get upload soon, but if you have bits to fix power so we don't have to rip rip, that would be welcome15:39
apwBenC, how long do you think you'll be15:39
BenCapw: End of today15:39
apwas tim is away we should be ok :)15:42
stgrabersconklin: hey, so I just reproduced the conntrack panic here on metal with current trusty15:59
stgraber(accidently, I was doing stress testing for another unrelated bug ;))15:59
argesstgraber: whoa. what test case did you use?15:59
stgraberarges: running a -j9 kernel build + one unprivileged container running chrome with a youtube video + another container with 500 sub-containers and around 30 with sub-sub-containers16:00
sconklinstgraber, arges - I'm looking at commit 0eba801b64cc8284d9024c7ece30415a2b981a72 which is in upstream but not stable, could be related - I have not tested upstream tip yet16:00
stgraberso I must have been having around 600 netns on that box, each with around 150 ipv6 routing table entries and 20-30 of those with iptables and ip6tables nat entries16:01
argesI could build a test kernel for stgraber if he can easily reproduce. sconklin is there a launchpad bug for this?16:03
sconklinarges: no16:03
stgraberarges: I'm not sure about "easily" yet but I can try again once that kernel is actually done building and I don't mind my work machine panicing on me :)16:04
argessconklin: ok file a bug please : ). stgraber when you're ready probably be good to first try http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/current/ and then we can try patch guessing16:07
sconklinarges: fwiw the mainline build v3.15-rc2-trusty should have this patch in it16:13
argessconklin: yup.16:13
BenCapw: Successfully linked. Still need to boot test it16:14
apwBenC, sounds hopeful16:17
sconklinarges: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/131427416:36
ubot2Launchpad bug 1314274 in linux (Ubuntu) "BUG in nf_nat_cleanup_conntrack" [Undecided,New]16:36
xnoxexellent usage of buildds =) i like it16:48
jsalisbury##16:54
jsalisbury## Kernel team meeting in 5 minutes16:54
jsalisbury##16:54
jsalisbury##17:00
jsalisbury## Meeting starting now - #ubuntu-meeting17:00
jsalisbury##      agenda: https://wiki.ubuntu.com/KernelTeam/Meeting17:00
jsalisbury##17:00
=== jsalisbury changed the topic of #ubuntu-kernel to: Home: https://wiki.ubuntu.com/Kernel/ || Ubuntu Kernel Team Meeting - Tues May 6th, 2014 - 17:00 UTC || If you have a question just ask, and do wait around for an answer! If the question is should I file a bug for something, likely you can assume yes.
sconklinarges: that bug also happens with the mainline v3.15-rc2-trusty build, I put that in the bug.20:48
sconklinthat means that the patch we talked about doesn't fix it, although that one may need to be in -stable also20:48
sconklinkamal: ^^20:48

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!