hallynapw: thanks - that worked!00:04
manjoanyone know if ports.ubuntu is having issues 00:08
manjolooks like some of the ports mirrors might be down... oh well 00:11
hallynapw: the patch is not quite right as it should grab another reference to the namespace, but it doesn't seem to blow up and does fix the bug, so i'll send a proper patch tongiht - thanks again00:17
hallynor does it not matter?00:24
hallynare we guaranteed that the cgns will stick aroudn for the duration of the mount due to task->mntns ?00:25
hallynthat actually may be the case bu ti need to think it through00:25
manjoapw, shit! 00:44
manjo(initramfs) echo "81780AD9-068C-4A80-A795-56856973B8F9" | tr '[:upper:]' '[:lowe00:44
manjoapw, so tr trick won't work in initramfs environemt 00:45
manjoapw, (initramfs) echo "81780AD9-068C-4A80-A795-56856973B8F9" | awk '{print tolower($000:52
manjothat works00:52
manjoapw, revised patch attached to the bug with real system boot test results 01:17
jsalisburylamont, I posted the last bunch of test kernels in a directory tree.  Location and how they are organized posted to the bug.02:41
lamontjsalisbury: awesome02:45
lamontI'll smash through the bisect tomorrow02:45
tjaaltondo you prefer an oldskool pull request or a merge request via lp? (for xenial)07:27
=== shuduo is now known as shuduo-afk
apwtjaalton, old skool pull request is prolly easier for people as that is what they are used to09:12
tjaaltonok, I'll do that then09:14
tjaaltonapw: enjoy! :)11:19
apwmanjo, i've uploaded an alternative fix (to avoid adding the first awk dependency) to ppa:apw/ubuntu/initramfs-tools-test, if you could test that for me then i can get it uploaded for real later today11:22
apwtjaalton, that is highly doubtful :/11:22
apwtjaalton, pththththth11:28
apwhallyn, yo ... did you bottom out that cgroups fix?  i don't see an email12:18
HerrAmeiseanyone install dist-upgrade 4.2.0-30 this morning and have trouble booting?14:27
HerrAmeisei had to revert back to 4.2.0-27 in order to get it working14:27
apwbjf, kamal ^14:28
apwHerrAmeise, what was your symtoms14:28
apwyou are cirtinaly the first report I have seen of issues with that update14:28
HerrAmeiseUnity would not start up14:29
HerrAmeiseso it went through the normal boot sequence14:29
HerrAmeiseand then just hung forever14:29
HerrAmeiseI couldn't even open a terminal14:30
apwHerrAmeise, ugg, sounds bad.  could you file a bug against the kernle "ubuntu-bug linux" from your working kernel14:30
HerrAmeiseso I had to go to grub and boot with a different kernel version14:30
apwand put whatever details you have in it, and tell us the bug# here14:30
HerrAmeiseyup no problem14:30
apwit is also worth looking in syslog to see if there was anything reported when it broke14:30
HerrAmeiseyep i can do that one sec14:30
apwi am assuming you receieved the kernel via a simple update (update manager or apt-get dist-upgrade sort of thing)14:31
HerrAmeiseyea i did apt-get dist-upgrade14:32
HerrAmeisedidn't build it myself or anything crazy14:32
apwand its hard to not get all the pieces you need as a complete set when its done that way14:32
apw(so that eliminates one possibility)14:33
apwonce you have a bug, i'll paste some initial kernels to test to try and figure out which of the three updates is at fault14:33
bjfapw, news to me. but -30 only has been in -updates for less than 24 hrs.14:33
apwyep, that indeed14:34
HerrAmeiseapw: ok, I am also running this on a VM at work if that is relevant14:34
HerrAmeiseVMware Workstation 1014:34
apwHerrAmeise, VMs would be a common test subject but mostly in KVM14:35
apwHerrAmeise, got a bug # yet ?14:42
stgraberapw: did our uploads fir your kernel adt problem?15:06
apwstgraber, yes ... final one just went green in the last 15m15:07
apw(for xenial)15:07
apwstgraber, and we're seeing ppc64el lxc ADT failures on trusty regardless of kernel by the looks of it, is this expected ?15:19
apwstgraber, and if not i'll file you anohter bug :)15:19
HerrAmeiseapw: is there any other way to report bugs other than through Apport?15:20
HerrAmeiseit's really a PITA15:21
apwin theory you can ask apport to file the info to a blob you can move to anohter machine and submit form there15:21
HerrAmeiseand its not just an application crash15:22
HerrAmeisebtw i upgraded the kernel to 4.2.0-30 on my 32-bit Ubuntu VM and the same thing happened15:25
HerrAmeiseso definitely able to replicate the error15:25
HerrAmeisefirst one was 64-bit15:25
apwit is as likely a vmware realted issue as anything15:25
HerrAmeisei'll try natively when i get home tonight15:28
apwHerrAmeise, the first debugging steps are to try -28 and -29 to see which of 28,29,30 are the first broken one15:29
apwbinary packages for those are in the librarian ^15:30
HerrAmeiseoops sorry wrong window15:38
stgraberapw: the latest ppc64el failure on trusty indicates a DC network failure15:41
stgraberunable to reach the gpg network and cloud-images.ubuntu.com15:41
apwstgraber, i'll ask for them again and see if it goes away then15:42
stgraberthe rest of your results look good so I'd say your kernel is fine, it's just the test runner having some network difficulties15:42
stgraberI know that IS changed the squid proxy IP recently, could be that the ppc64el VMs don't have the right firewall rule or something15:43
stgraberif it fails again, we'll involve pitti15:43
apwstgraber, ack, will let you know15:43
hallynapw: sorry, no, had some technical difficulties.  will try to send it out this morning15:55
apwhallyn, the pending fixes which break you are time sensitive, so i would like to get to a place where i have a plan to add something or rip something and upload at the latest tommorrow15:56
hallynapw: should i send a patch first upstream or first to ubuntu-kernel@?15:57
apwhallyn, either is fine, if you are confident in the fix i cna apply it while upstream grinds on it15:58
apwwere still in a reaosnably felxible period so we can rip it and replace it if upstream has a better idea15:58
apwas i assume my other option is to rip the other thing you applied which causes the issue15:59
apwthe cgn i think it was15:59
hallynapw: http://paste.ubuntu.com/15181046/   <- does that mean anything to you?15:59
hallyn(it kinda blocks me when the server hosting all my work keeps hanging with crap like that - from 3.13 to 3.16 and now on 4.2)16:00
hallynif it migth be hw then i'll just $%)(*$%)($ switch16:00
apwi wonder if that could be the fuse cve i was just reviewing16:01
sforsheehallyn: do you hit the WARN_ON at the end of the cgroup_mount after your change?16:01
hallynwhich warn_on?16:02
sforsheehallyn: was also thinking it might make sense to make kernfs to use sget_userns with init_user_ns always, haven't had time to really think it through yet though16:02
sforsheethe one in the if (pinned_sb) block16:02
hallynsforshee: i don't think so16:03
apwsforshee, you know fuse well, is fuse_fill_write_pages() use on the write or read path ?16:03
apwit ought to be write, but hey, nothing is clear in this world16:04
hallynsforshee: we cannot have cgroup mount use init_user_ns always, if you end up doing the comparison for the sb.16:04
sforsheeapw: write it would seem, invoked by fuse's write_iter callback16:04
apwsforshee, thanks, not that cve then hallyn 16:05
apwsforshee, that stack trace might be something you grok better than i: http://paste.ubuntu.com/15181046/16:05
hallynapw: trying to decie whether to halt my world to have the people hosting the server check the hw for 10 hours16:05
sforsheeapw: looks to me like the kernel is hung waiting for userspace to respond16:06
apwhallyn, if it is always that, it look to my shallow knowledge of fuse that it is waiting on a userspace provider, and isn't interruptible16:06
apwsforshee, ok you see the same as me16:06
hallynok, thx :)16:06
apwhallyn, so i'd not be keen to blame h/w but whatever crap is mounted on fuse16:06
apwis that lxcfs :)16:06
hallynhm, could be.16:07
apwsforshee, do we really hand things to uspace and wait in an uninterruptible way for it to respond, that sounds mad to me16:07
hallynin that case the only thing i can think is that the kernel builds make oom happen killing it (bc nothing else was execising fuse),16:07
hallynexcept i've got 42g ram16:07
apwhallyn, then you'd have an oom in there16:08
apwhallyn, and it looks to start right there, with something not it before16:09
sforsheehallyn: going back to cgfs ... prior to my changes it was going to reuse an existing superblock. Now it still wants to but sget refuses because it's a different userns. Is that right?16:10
sforsheeapw: I think fuse does wait in an uninterruptible way to respond. But there is some kind of abort connection sysfs node to break those waits.16:15
sforsheehallyn: it does seem to me that it will possible to hit that WARN_ON(new_sb) if using kernfs_mount_ns. Does that represent some real problem?16:32
sforsheehallyn: also I'm not sure what you were getting at wrt using init_user_ns always16:33
sforsheein effect that's what's happening before we have s_user_ns. But like I said I need to think it through some more.16:33
hallynsforshee: just to  make sure, you see why i need something like it right?16:38
hallynlooking at fs/sysfs/mount.c, i think i just need to grab/release the ns ((i suppose sb release could be done lazily)16:39
sforsheehallyn: I think so. Previously you ended up reusing a superblock, but now sget returns EBUSY because you're in a different userns.16:39
sforsheeand by passing a namespace you force the kernfs test function to not match the old superblock16:40
sforsheebut there seems to be something inherent to this code that expects to reuse the superblock in some cases16:40
hallynis there?  i was wondering that but didn't see it,16:40
sforsheejust look at pinned_sb16:40
hallynis there a simple way we could re-use it?16:41
hallynwithout hardcoding cgroupfs in sget_userns16:41
sforsheewell, that's where the though of doing sget_userns(..., &init_user_ns) in kernfs_mount_ns came from16:42
sforsheeor I was also considering whether maybe that check only makes sense for device-backed mounts16:42
hallynyeah it doesnt make sense for i.e. sysfs or proc, right?  you can't get another userns's mount there16:44
hallyni wonder whether this will quietly break containers doing 'mount -t devpts' without -o newinstance16:45
hallynwell we pass file_system_type in, so i guess we *could* filter on cgroupfs;  how would we tell whether it's blockdev-backed in sget?16:46
sforsheeyou can't mount devpts from !init_user_ns without newinstance it appears16:47
hallynoh, good16:47
sforsheethere's a flag in the fs type that says whether or not it's device backed16:48
cristian_cjsalisbury: hi16:49
hallynso if (type->flags & FS_REQUIRES_DEV && user_ns != old->s_user_ns) ?16:49
lamontjsalisbury: finally read what you set up for me.  you have outdone expectations, thanks.16:49
sforsheehallyn: yeah, something like that. Still thinking though.16:50
lamontI'll smash through those sometime before the end of lunch today, expect an update in something like 2-4 hours.  I'm assuming that our final kernel is top-of-branch, with the identified commit reverted?16:50
sforsheehallyn: so essentially s_user_ns is used for 2 things. First is translating ids for the backing store, which doesn't apply to psuedo filesystems.16:53
sforsheethe second is for privileges towards the superblock. For cgroups/sysfs do we really want root in the userns to have privileges towards the superblock?16:54
sforsheethough if they aren't they can't remount16:55
hallynsforshee, well in the case of cgfs cgroup_mount() guards remount, but i'm not sure about proc and sysfs16:58
hallyni would assume so16:58
hallynelse that would have been an issue already since we allow mount on those17:00
hallynsforshee: it seems we canassume it's safe if it's not dev-backed and it already has FS_USERNS_MOUNT17:01
sforsheehallyn: assume what's safe? Skipping the checkin in sget_userns?17:02
hallynsforshee: yes17:03
sforsheehallyn: I think at minimum it's probalby okay as a interim fix while we decide what the best fix is17:05
xnoxbjf, what's the minimal amount of maas functionality is required to run kernel-maas testing for s390x?17:10
xnoxas far as can tell that enablement is currently staggnating, and i'm wonder if that can be expedited somehow.17:11
bjfxnox, right now i only deal with bare-metal via maas 17:13
xnoxbjf, and it needs to boot to ssh right?17:14
bjfxnox, yes17:14
xnoxbjf, and you do use the powercycle functions in maas right? e.g. poweroff/on/reboot?17:15
bjfxnox, yup17:15
xnoxbjf, ok, cool.17:15
manjoapw, I had trouble installing from the PPA due to dependcy issues .. posted it to the bug 17:17
manjoapw, https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/154812017:17
ubot5Launchpad bug 1548120 in initramfs-tools (Ubuntu) "[xenial][initramfs-tools] support uppercase and lowercase uuids" [High,In progress]17:17
apwmanjo, those deps are right17:19
apwmanjo, they are internal deps making sure the bits are at the same version from the same source package17:19
manjoapw, hmm I have xenial-propose enabled .. 17:21
manjoapw, should I disable that 1st ? 17:22
apwmanjo, nope shouldn't matter17:22
apwmanjo, did you add the ppa or download the .debs ?17:22
manjoadded your repo17:22
manjousing apt-add-repo17:23
apw    initramfs-tools-bin_0.122ubuntu5~rc1_amd64.deb (81.7 KiB)17:23
apw    initramfs-tools-core_0.122ubuntu5~rc1_all.deb (116.8 KiB)17:23
apw    initramfs-tools_0.122ubuntu5~rc1_all.deb (84.0 KiB)17:24
apwwell that PPA has all three of those in there17:24
apwso your deps should be found from the PPA17:24
manjo  Installed: (none)17:24
manjo  Candidate: 0.122ubuntu5~rc117:24
manjo  Version table:17:24
manjo     0.122ubuntu5~rc1 50017:24
manjo        500 http://ppa.launchpad.net/apw/initramfs-tools-test/ubuntu xenial/main arm64 Packages17:24
manjo Candidate: 0.122ubuntu5~rc117:24
manjo  Version table:17:24
manjo     0.122ubuntu5~rc1 50017:24
manjo        500 http://ppa.launchpad.net/apw/initramfs-tools-test/ubuntu xenial/main arm64 Packages17:24
manjoapw, ah -bin comes from ports 17:25
apwOH its bloody arm6417:25
manjo  Candidate: 0.122ubuntu417:25
manjo  Version table:17:25
manjo *** 0.122ubuntu4 50017:25
manjo        500 http://ports.ubuntu.com/ubuntu-ports xenial-proposed/main arm64 Packages17:25
apwhang on17:25
manjook ☺ 17:25
manjoapw, welcome to my world 17:25
apwmanjo, ok in about 10m there will a ~rc2 in there built for arm64 as well17:29
manjocool 17:30
manjowill post results to the bug 17:30
=== zequence_ is now known as zequence
sforsheehallyn: are you already preparing a patch for the cgfs issue or should I go ahead and make one?17:47
hallynsforshee: I'm looking at the code, but i'm not sure i'm doing it right17:48
hallyn(adding an extra helper fn which tries to get the logic right of whether we need to check the user_ns)17:48
sforsheesounds more complicated than what I was thinking ...17:49
hallynlemme finish tihs up and pastebin it and you can tell me i'm wrong :)17:49
sforsheesounds good17:49
hallynsforshee: http://paste.ubuntu.com/15181923/ ?17:51
hallynseems to be buliding anyway17:54
sforsheehallyn: I think that should work, but the FS_USERNS_MOUNT seems unnecessary17:55
hallynsforshee: so the idea of that is that any virtual fs which we cannot currently mount in a userns will be restricted to your own userns...17:56
hallynagreed not sure if it's needed, and it may be reckless...  might break things...17:56
hallynbut i wasn't certain that it woudl be safe otherwise17:56
hallynno shouldn't break things - it just allows what you had designed to work the way you menat it to for those filesystems :)17:56
sforsheebut if you can't mount it in a user_ns then both namespaces are always init_user_ns anyway17:56
hallynsforshee: oh, youdon't relax the need for FS_USERNS_MOUNT, right17:57
hallynso you're right, shouldn't be necessary so we don't need the helper really17:57
hallynsforshee: so i'll just build and test http://paste.ubuntu.com/15181976/17:58
hallynif you can do a clenaer version of that then even better17:58
sforsheehallyn: or even http://paste.ubuntu.com/15181975/17:58
hallyndo you mind sending that in then? :)18:00
sforsheeyou want to test it first?18:00
hallynis building my version enough or do you want me to do your verbatim version?18:01
sforsheehallyn: I'm probably going to build my version either way before I send it, so I can just give you that version to test18:03
sforsheeI hate to send patches without build testing, typos can be easy to overlook18:03
hallynbuilds taking forever18:10
kamalHerrAmeise, apw, bjf: So far, I'm unable reproduce Herr's issue.  In a 15.10 amd64 qemu vm, I can successfully boot linux-image-4.2.0-{27,28,29,30}-generic with no apparent trouble.18:22
bjfkamal, thanks for looking at that18:24
hallynsforshee...  and still building.  if you get a build, so long as you can 'lxc launch cxenial x1' and x1 boots (gets >3 processes and gets an ip address) then it's good18:35
sforsheehallyn: mine's still building too18:37
lamontjsalisbury: back to you.18:45
lamontand thanks muchly for making it less painful18:45
jsalisburylamont, thanks, I'll take a look18:45
lamontit's also good to have my gut confirmed. :D18:46
jsalisburylamont, I'll build a test kernel with that commit reverted, if it fixes the bug, I'll get in touch with the patch author18:47
lamontack.  scream when18:49
sforsheehallyn: http://people.canonical.com/~sforshee/cgns/, updating my vm to test it now18:52
lamontjsalisbury: fwiw, http://global.shuttle.com/products/productsDetail?productId=148018:53
hallynsforshee: success!18:56
hallyni think18:57
hallynyeah.  dunno why there is a 'lxc' cgroup there, but ...18:57
sforsheehallyn: seems to work for me too19:00
sforsheeI'll send it19:00
sforsheeapw, hallyn: patch sent19:06
tjaaltonumm yeah disregard the pull request, I'll send a new one19:08
hallynsforshee: awesome, thx19:11
cristian_cjsalisbury: hi, again19:30
=== philipballew is now known as Guest44851
=== RAOF_ is now known as RAOF

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!