=== harrisj_ is now known as harrisj [05:14] good morning [05:30] zyga: I ran into a weird bug when mixing the content interface with layouts: https://bugs.launchpad.net/snapd/+bug/1831010 [05:30] Bug #1831010: Snapd fails to upgrade package that uses layouts to remap a content plug mount [05:30] with the mount namespace in place, I can't upgrade the package [05:31] Hey [05:31] morning [05:31] jamesh: can you please try with ... [05:32] https://github.com/snapcore/snapd/pull/6891 [05:32] PR #6891: many: make per-snap mount namespace MS_SHARED [05:35] I'll give it a go. [05:35] Thanks! [05:36] I’ll be in the office soon [05:48] back now [05:48] jamesh: so, I'm not sure that branch will help [05:48] jamesh: but surprisingly it did help with another test that documented a bug [05:48] jamesh: I'm not 100% sure when the kenels returns EBUSY on mount ops, perhaps it was due to the fact than an app was running? [05:48] zyga: I haven't tested it yet, but it wasn't obvious it would [05:49] jamesh: if this is a blocker for you I can take the bug report and jump into the kernel to see what is really going on [05:49] zyga: my guess is that it is trying to unmount the layout mount without first doing the submounts [05:49] jamesh: but we almost always do that, it never complains [05:50] jamesh: we always MS_DETACH (aka --lazy) so that should work [05:50] jamesh: I think there is something else that must be at play [05:50] jamesh: I know you are busy with lots of stuff and that we owe you a ton of reviews but if you could also review https://github.com/snapcore/snapd/pull/6891 I would love to get some feedback -- it's a tricky PR that involves user mount namespaces [05:51] PR #6891: many: make per-snap mount namespace MS_SHARED [05:52] PR snapd#6934 opened: tests: fix repack_snapd_snap_with_deb_content [05:52] mborzecki: I cherry picked a bugfix from mvo so that it can land faster ^ [05:53] zyga: ah, that one [06:01] zyga: any ideas about sid here? https://github.com/snapcore/snapd/pull/6927#partial-pull-merging if that's something in the distro, i'd expect master to fail too [06:01] PR #6927: release: 2.39.1 [06:01] zyga: I simplified things a bit to reduce the number of mounts (i.e. not using gtk-common-themes), and was able to reproduce the problem: https://bugs.launchpad.net/snapd/+bug/1831010/comments/2 [06:01] Bug #1831010: Snapd fails to upgrade package that uses layouts to remap a content plug mount [06:02] jamesh: thanks! [06:02] zyga: I think it is the last duplicated mount not present in the snapd fstab that is confusing things [06:02] curious [06:02] thanks! [06:03] jamesh: it might be this bug https://bugs.launchpad.net/snapd/+bug/1828357 [06:03] Bug #1828357: snap-update-ns incorrectly sorts mount profile entries [06:03] but then again, maybe not, I'll check [06:03] mborzecki: looking [06:04] mborzecki: do you mean: [06:04] dpkg-source: error: can't build with source format '3.0 (quilt)': non-native package version does not contain a revision [06:04] dpkg-source: info: using options from snapd/debian/source/options: --include-removal [06:04] dpkg-buildpackage: error: dpkg-source -b . subprocess returned exit status 255 [06:04] zyga: yup [06:04] feels like new debian feature [06:05] zyga: right, but why it doesn't fail on master [06:05] ah, good question :) [06:05] dunno [06:06] zyga: btw. are you off today? [06:06] mborzecki: no no :) [06:06] today I will be busy [06:43] PR snapd#6934 closed: tests: fix repack_snapd_snap_with_deb_content === pstolowski|afk is now known as pstolowski [06:58] mornings [06:59] morning :) [07:25] pstolowski: hey [07:25] hey marcustomlinson and mborzecki! [08:58] PR snapd#6935 opened: cmd/snap: support snap debug timings --startup=.. and measure loadState time [09:00] zyga: unlikely to be new - that error was first added in dpkg commit acc1f37933 in 2013 (with slightly different text at that point) [09:07] zyga,mborzecki: It's a bug in the PR - top line of packaging/debian-sid/changelog should say 2.39.1-1 rather than 2.39.1 (compare with the following changelog stanza in that same file) [09:07] cjwatson: thanks! [09:08] np, took me a little while to spot it [09:10] zyga: that means the changelog in release/2.39 branch is busted too [09:10] zyga: the one for sid [09:15] zyga: well, the travis job on the branch failed :P [09:17] PR snapd#6936 opened: packaging/debian-sid: fix changelog, add revision [09:19] zyga: pstolowski: ^^ [10:11] PR snapd#6855 closed: overlord: implement store switch remodeling [10:17] re; sorry for missing half of today, I was baby-sitting upstairs due to unexpected errand for my wife; I'll handle paperwork for half day off [10:17] back to work now [10:18] cjwatson: thanks for the analysis, I totally missed that [10:18] jamesh: hey, any luck with the extra patch? [10:24] zyga: sorry. Got side tracked before setting up a test system. [10:24] no worries [10:24] I'll see if I can debug that today [10:24] zyga: I suspect it won't make a difference: https://paste.ubuntu.com/p/GV4bQW8XKy/ [10:24] zyga: I did manage to simplify the test snap down to only using layouts, so it is self contained [10:24] wow [10:24] jamesh: what's the kernel version? [10:25] zyga: this was on an 18.10 box with 4.18.0-15-generic [10:26] I'll grab my other system to see if the same happens with 19.04 [10:28] pstolowski: hey [10:28] pstolowski: do you have a moment for a quick HO? [10:29] zyga: same result on 5.0.0-11-generic [10:29] zyga: hey, sure [10:29] (that system needs a reboot) [10:31] zyga: https://launchpadlibrarian.net/425983579/snapcraft.yaml <- that's the simpler version exhibiting the bug [10:53] pstolowski: pushed to https://github.com/snapcore/snapd/pull/6932 [10:53] PR #6932: tests: run unit tests without networking [10:55] PR snapcraft#2575 closed: catkin spread tests: use kinetic instead of indigo [10:56] zyga: thanks [11:00] journald on opensuse is acting up, i've restarted a spread run that failed on opensuse again and again the log the test is looking for isn't there [11:31] zyga: conflicts in #6932 [11:31] PR #6932: tests: run unit tests without networking [11:32] mborzecki: aha [11:33] * Chipaca wanders off in search of sustenance [11:34] * zyga hands Chipaca a copy of the GNU manifesto ;) [11:34] * Chipaca is suspicious [11:35] Chipaca: to start the fire ;D [11:36] * Chipaca puts on his FSFLA tee and scowls [11:37] mborzecki: resolved === ricab is now known as ricab|lunch [11:42] PR snapd#6936 closed: packaging/debian-sid: fix changelog, add revision [11:43] pushed the changelog fix to #6927 [11:43] PR #6927: release: 2.39.1 [11:48] more unit test failures... https://www.irccloud.com/pastebin/uqQwKN1R/ [11:49] that is curious... [11:58] pstolowski: heh, that test fails *again* [11:58] debugging [11:58] but apparently the unshare -n trick is not enough [11:58] man, that's fum [11:58] mborzecki, hey [11:59] mborzecki, there is an issue on i386, it is going out of memory [11:59] mborzecki, https://paste.ubuntu.com/p/xbqvTQSp6Y/ [11:59] zyga: fails again after your fix? [11:59] yesterday I was researching about it wit mvo [12:00] pstolowski: it passes in unshare -n tests, fails in "osc build" [12:00] mborzecki, but then I spent time doing tests and could reproduce it on beta and candidate [12:01] zyga: maybe loopback is down there after all? [12:02] mborzecki: no, that causes lots of tests to fail [12:02] mborzecki: I'd like to get "osc shell" or something like that to play [12:02] let's see [12:04] cachio: hm looks like something failed while loading apparmor profiles? [12:04] mborzecki, yes [12:04] apparmor_parser: Unable to replace "snap.test-snapd-tools.fail". Out of memory [12:05] mborzecki, kernel: vmalloc: allocation failure: 68497 bytes [12:06] I tested that with kernel on stable and works well [12:06] but kernel on candidate and beta are failing at some point [12:11] cachio: can you look at aa-status? [12:12] zyga, https://paste.ubuntu.com/p/RBPTNpXz4Z/ [12:14] cachio: perhaps after running each test we should remove loaded apparmor profiles [12:15] zyga, I could try that [12:15] zyga, it is weird becuase it doesn't fail with pc-kernel=stable [12:15] cachio: and this is with which kernel? [12:16] this is with kernel on beta or candidate [12:16] works well with kernel on stable [12:16] cachio: do we know what is the difference between those? [12:16] zyga, no [12:16] I left tests running [12:16] cachio: perhaps we can ask the kernel team for a summary? [12:16] pstolowski: so ... [12:16] and I just got the results [12:17] failure in osc build, with extra debugging https://www.irccloud.com/pastebin/xy59XKOr/ [12:17] cachio: +1 [12:17] cachio: curious, have you seen https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1830502? jjohansen is looking at that [12:17] Bug #1830502: apparmor uses excessive memory leading to oom kill [12:18] jdstrand, nice [12:18] jdstrand, that could explains that [12:19] zyga: hmmm that error, looks like temporary/permanent/retryable mess again [12:19] mborzecki: yeah, we just worked on that with pstolowski [12:19] cachio: it may or it may not, but there are debugging details that jjohansen laid out that might be helpful [12:19] !!! https://launchpadlibrarian.net/425906431/snapcraft.yaml [12:19] jdstrand: that's one long layout [12:20] mborzecki: this is supposed to run with this applied https://github.com/snapcore/snapd/pull/6932/commits/c3b1296f1124158fd7e0987e9bd0196b20b3634d [12:20] PR #6932: tests: run unit tests without networking [12:20] zyga: heh, yes. I think I'm going to put something in the review-tools for an abusive use of layouts. I'm not sure what that number should be, but clearly it is less than what is in that yaml :) [12:20] jdstrand, sure [12:20] jdstrand: at most a dozen [12:20] we can expand if we have to [12:21] zyga: well, so now it connection refused? :) [12:21] ack [12:21] mborzecki: yeah [12:21] different error [12:21] uhm [12:23] pstolowski: ah, I see what the bug is now [12:23] https://www.irccloud.com/pastebin/62CfKEFd/ [12:29] zyga: uhmmm.. horror continues [12:29] yeahj [12:29] fixed locally, let's see [12:29] I'll push that to the offline tests PR [12:29] idk maybe our approach is wrong [12:30] mborzecki: perhaps [12:30] mborzecki: we'd have to get some data about how the errors look like [12:30] put that into a table [12:30] then train an AI against someone saying if this is good or bad [12:30] and then if you have a CUDA card we'd retry correctly ;) [12:30] pstolowski: tests passed! [12:30] \o/ [12:31] cachio: did you get a chance to look into updating opensuse to 15.1? [12:32] mborzecki, no, but I can do it today [12:32] mborzecki: but in all seriousness, I think we could be better if we had a good data set and wrote proper algorithm [12:32] cachio: 15.1 is out, hope they published the images too [12:33] mborzecki, nice, I'll check that and create the image if it is available [12:33] mborzecki, thanks for hte heads up [12:33] https://build.opensuse.org/project/show/system:snappy [12:33] darn OSC is *insanely quick* [12:33] I wrote "osc commit" [12:33] moved to another display [12:33] reloaded a tab [12:33] and it's building already [12:35] mborzecki, cachio: FYI, I'm discontinuing builds for openSUSE Leap 42.2 and 42.3 as they don't provide golang-go >= 1.9 [12:35] zyga, ok, so I'll clean up them when I create the PR for opensuse 15.1 [12:36] thanks [12:36] zyga, thanks to you [12:37] build complete [12:38] zyga: sounds fair [12:40] * zyga looks at mount bug from jamesh now === ricab|lunch is now known as ricab [12:58] PR snapd#6931 closed: tests: force removal to prevent restore fails when directory doesn't exist on lp-1801955 test [13:00] cachio: hey, mvo is out and not sure if you need it, but I saw core20 float into the store queue and I've approved it, but it needs to be released to a channel [13:00] zyga: perhaps you're interested in ^ [13:00] jdstrand: thanks [13:00] jdstrand: I think we want it in edge only, we'll discuss that [13:01] I figured and would've put it there, but didn't have the perms [13:27] eh, btrfs corrputed [13:27] haha, rancid butter :) [13:30] zyga: clearly production ready fs [13:30] zyga: can you restore from a snapshot at least or sth? [13:31] yeah, managed to mount a ro copy [13:31] I'll just add a HDD, reformat and move over [13:31] using ext4 was my best tumbleweed choice [13:31] btrfs is cool but no thanks :/ [13:37] rsyncing... [14:18] jdstrand: 20190528-2014UTC click-reviewers-tools version now in production 💪 [14:18] (nothing broke after the big rename-refactor) [14:18] TooLmaN: exciting! :) [14:18] yikes [14:18] TooLmaN: sorry, nm :) [14:18] roadmr: exciting! :) [14:18] (tab fail) [14:18] if you want to get rid of the click-review symlink, I can change the command we invoke [14:18] jdstrand: hehe yes tabs can be tricky [14:19] roadmr: it helps if I type the first letter correctly :) [14:19] hehe a clear case of autocowreck [14:19] roadmr: I would like to get rid of the click-review symlink, but it is far from urgent. maybe let's sit on 20190528-2014UTC for a little while and then revisit? [14:22] jdstrand: sure thing; yes, I think I can change the executable name on my side and if things look stable in a few weeks we can kill the symlink [14:24] roadmr: sounds like a plan [14:27] mborzecki: wanna see something curious [14:27] zyga: hm? [14:30] zyga, mborzecki opensuse-15.1 nor available on glcoud yes [14:30] yet [14:30] I'll see if I could upgrade from 15.0 [14:30] and start using that [14:32] mborzecki: https://bugs.launchpad.net/snapd/+bug/1831010 [14:32] Bug #1831010: Snapd fails to upgrade package that uses layouts to bind mount parent directory of another mount point [14:32] mborzecki: last two comments [14:33] this seems to show that our assumptions that we can detach anything are wrong [14:33] if we need to investigate detach algorithms that untangle mounts then it will be a fun thing to fix [14:37] cachio: can you ask some suse people about this? [14:37] cachio: I found this https://build.opensuse.org/package/show/Cloud:Images:Leap_15.1/openSUSE-Leap-15.1-GCE-Guest [14:38] cachio: https://download.opensuse.org/repositories/Cloud:/Images:/Leap_15.1/images/ [14:38] there are images here [14:38] can you please try those? [14:40] zyga: at least it's reproducible here too [14:42] mborzecki: some ideas here [14:42] mborzecki: we can construct an in memory tree from mountinfo mount-id / parend-id pairs [14:42] and then walk that [14:43] to unmount leaves before inner nodes [14:43] I'm still puzzled by propagation of detach mounts [14:43] perhaps what is really breaking is the propagation of the unmount conflicting with itself? [15:44] nmcli critical error trying to enable wifi interface on ubuntu core (18.04.2) on raspberry pi [15:44] sudo network-manager.nmcli r wifi on gives error: (process:3062): nmcli-CRITICAL **: Error: Could not create NMClient object: Could not connect: Permission denied. [15:45] can anyone suggest what I might do to get wifi working with ubuntu core on a raspberry pi [15:53] tim-jone_: i thought that was handled via netplan [15:53] tim-jone_: but i'm not too certain [15:53] ogra: help? [15:53] following instructions here: https://gist.github.com/cyan198/6ecdb86267498a51587b4337e47e10f6 [15:54] tim-jone_: one thing I see you do that I don't see in that gist is 'sudo' [15:55] tim-jone_: maybe just skip that first part; it might not be disabled at all [15:55] have been using sudo [15:55] Chipaca: his pasted command did include sudo [15:55] cyphermox: exactly [15:55] the gist does not use sudo [15:55] yes that error is with sudo [15:55] tim-jone_: and what happens without sudo? [15:56] same error [15:56] I think you just want to move on to 'network-manager.nmcli d wifi list' [15:56] tim-jone_: what's the output of 'snap connections network-manager'? [15:57] Interface Plug Slot Notes [15:57] dbus network-manager:wpa - - [15:57] firewall-control network-manager:firewall-control :firewall-control - [15:57] modem-manager network-manager:modem-manager :modem-manager - [15:57] network network-manager:network :network - [15:57] network-manager - network-manager:service - [15:57] network-manager network-manager:nmcli - - [15:57] network-setup-observe network-manager:network-setup-observe :network-setup-observe - [15:57] ppp network-manager:ppp :ppp - [15:57] (is there a better way to paste output or should I use a pastebin link or similar in IRC?) [15:57] tim-jone_: paste.ubuntu.com ftw [15:58] tim-jone_: do all nmcli commands end with the same error? [15:59] cyohermox network-manager.nmcli d wifi list gives the same error (with or without sudo) [15:59] mmkay [15:59] so some kind of real permission issue; Chipaca will def know more about it than me if it has to do with the connections [15:59] tim-jone_: can you pastebin 'snap logs network-manager'? [16:00] sudo network-manager.nmcli gives version OK, but try any objects (radio etc) it fails [16:01] yeah, there's something wrong permissions-wise, but the connections seem a'ight -- which, unless i'm missing something, means there's a bug in the snap [16:01] output of snap log network manager : https://paste.ubuntu.com/p/QPwwTx68gv/\ [16:01] but that's snap in rather prominent use so somehting's up [16:01] tim-jone_: that seems ok [16:02] tim-jone_: it seems ogra is away, could you open a topic in forum.snapcraft.io (in the 'devices' category)? [16:02] sorry, 'device' [16:04] OK - will do - but reading the log I pasted I see that despite network manager not working the wlan0 interface is listed via ifconfig which it was not before [16:15] @Chipaca: have opened a topic in device category as you suggested :-) === pstolowski is now known as pstolowski|afk [16:34] woo, my pr is green, time to take a break [16:37] Chipaca, what makes you think i know how to use NM on core ? ... none of my images ever used it ;) [16:37] * ogra just uses a proper netplan.yaml and the network-setup-control interface from a configuration snap [16:45] ogra: is this gist wrong then? : https://gist.github.com/cyan198/6ecdb86267498a51587b4337e47e10f6 (pity its the first hit on google if so) [16:46] BTW did configure /etc/netplan/50-cloud-init.yaml manually and the wifi interface is up and running [16:46] is there some other documentation I should be reading? [16:55] cachio, jdstrand, zyga: that is a different bug [17:15] jjohansen, thanks [17:16] zyga, , I could reproduce the error using the core from stable and the kernel from beta [17:54] what is a common cause for "Reported install problem for xxx" in handlers? [17:55] PR snapd#6927 closed: release: 2.39.1 [17:55] jjohansen: thank you for confirming that [17:55] May 30 12:39:53 localhost snapd[702]: handlers.go:414: Reported install problem for "snapd" as 05c989c0-82d8-11e9-9fdc-fa163e6cac46 OOPSID [17:56] ogra: I just assume you know everything [17:57] * Chipaca vanishes in a cloud of EOD [18:04] PR # closed: core-build#11, core-build#22, core-build#26, core-build#37, core-build#38 [18:05] PR # opened: core-build#11, core-build#22, core-build#26, core-build#37, core-build#38 [18:24] PR snapcraft#2578 opened: keyrings: update ROS signing key [20:43] Hey folks, how can I remove a snap without wasting time on snapshots? [20:44] (I'm removing it because all the (ephemeral) data ended up corrupted; I really don't want a snapshot.) [20:45] there either is or will be a --purge option to snap remove [20:45] not sure if that has landed or not [20:45] There isn't where I am, unfortunately. [20:46] And it looks like because I've started the remove operation, I'm stuck with the snapshotting regardless. [20:46] you might try refreshing to the edge channel of the core snap [20:46] if you use `snap changes` you can see what id of the remove task is, and then abort that task with `snap abort $id` [20:48] Aha, that got me to the point where I could run `snap set system snapshots.automatic.retention=no` before removing again. [20:48] Thanks for the help, ijohnson! [20:48] also fwiw the --purge option is at least in the edge channel too [20:48] glad to help [21:36] PR snapd#6937 opened: cmd/snap-update-ns: detach unused mount points [21:37] jdstrand: ^ [21:37] jamesh: ^ [21:37] running final spread locally before opening [21:40] * zyga EODs