/srv/irclogs.ubuntu.com/2019/05/30/#snappy.txt

=== harrisj_ is now known as harrisj
zygagood morning05:14
jameshzyga: I ran into a weird bug when mixing the content interface with layouts: https://bugs.launchpad.net/snapd/+bug/183101005:30
mupBug #1831010: Snapd fails to upgrade package that uses layouts to remap a content plug mount <snapd:New> <https://launchpad.net/bugs/1831010>05:30
jameshwith the mount namespace in place, I can't upgrade the package05:30
zygaHey05:31
mborzeckimorning05:31
zygajamesh: can you please try with ...05:31
zygahttps://github.com/snapcore/snapd/pull/689105:32
mupPR #6891: many: make per-snap mount namespace MS_SHARED <Bug> <Created by zyga> <https://github.com/snapcore/snapd/pull/6891>05:32
jameshI'll give it a go.05:35
zygaThanks!05:35
zygaI’ll be in the office soon05:36
zygaback now05:48
zygajamesh: so, I'm not sure that branch will help05:48
zygajamesh: but surprisingly it did help with another test that documented a bug05:48
zygajamesh: I'm not 100% sure when the kenels returns EBUSY on mount ops, perhaps it was due to the fact than an app was running?05:48
jameshzyga: I haven't tested it yet, but it wasn't obvious it would05:48
zygajamesh: if this is a blocker for you I can take the bug report and jump into the kernel to see what is really going on05:49
jameshzyga: my guess is that it is trying to unmount the layout mount without first doing the submounts05:49
zygajamesh: but we almost always do that, it never complains05:49
zygajamesh: we always MS_DETACH (aka --lazy) so that should work05:50
zygajamesh: I think there is something else that must be at play05:50
zygajamesh: I know you are busy with lots of stuff and that we owe you a ton of reviews but if you could also review https://github.com/snapcore/snapd/pull/6891 I would love to get some feedback -- it's a tricky PR that involves user mount namespaces05:50
mupPR #6891: many: make per-snap mount namespace MS_SHARED <Bug> <Created by zyga> <https://github.com/snapcore/snapd/pull/6891>05:51
mupPR snapd#6934 opened: tests: fix repack_snapd_snap_with_deb_content <Bug> <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/6934>05:52
zygamborzecki: I cherry picked a bugfix from mvo so that it can land faster ^05:52
mborzeckizyga: ah, that one05:53
mborzeckizyga: any ideas about sid here? https://github.com/snapcore/snapd/pull/6927#partial-pull-merging if that's something in the distro, i'd expect master to fail too06:01
mupPR #6927: release: 2.39.1 <Created by mvo5> <https://github.com/snapcore/snapd/pull/6927>06:01
jameshzyga: I simplified things a bit to reduce the number of mounts (i.e. not using gtk-common-themes), and was able to reproduce the problem: https://bugs.launchpad.net/snapd/+bug/1831010/comments/206:01
mupBug #1831010: Snapd fails to upgrade package that uses layouts to remap a content plug mount <snapd:New> <https://launchpad.net/bugs/1831010>06:01
zygajamesh: thanks!06:02
jameshzyga: I think it is the last duplicated mount not present in the snapd fstab that is confusing things06:02
zygacurious06:02
zygathanks!06:02
zygajamesh: it might be this bug https://bugs.launchpad.net/snapd/+bug/182835706:03
mupBug #1828357: snap-update-ns incorrectly sorts mount profile entries <snapd:New for zyga> <https://launchpad.net/bugs/1828357>06:03
zygabut then again, maybe not, I'll check06:03
zygamborzecki: looking06:03
zygamborzecki: do you mean:06:04
zygadpkg-source: error: can't build with source format '3.0 (quilt)': non-native package version does not contain a revision06:04
zygadpkg-source: info: using options from snapd/debian/source/options: --include-removal06:04
zygadpkg-buildpackage: error: dpkg-source -b . subprocess returned exit status 25506:04
mborzeckizyga: yup06:04
zygafeels like new debian feature06:04
mborzeckizyga: right, but why it doesn't fail on master06:05
zygaah, good question :)06:05
zygadunno06:05
mborzeckizyga: btw. are you off today?06:06
zygamborzecki: no no :)06:06
zygatoday I will be busy06:06
mupPR snapd#6934 closed: tests: fix repack_snapd_snap_with_deb_content <Bug> <Simple 😃> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6934>06:43
=== pstolowski|afk is now known as pstolowski
pstolowskimornings06:58
marcustomlinsonmorning :)06:59
mborzeckipstolowski: hey07:25
pstolowskihey marcustomlinson and mborzecki!07:25
mupPR snapd#6935 opened: cmd/snap: support snap debug timings --startup=.. and measure loadState time <Created by stolowski> <https://github.com/snapcore/snapd/pull/6935>08:58
cjwatsonzyga: unlikely to be new - that error was first added in dpkg commit acc1f37933 in 2013 (with slightly different text at that point)09:00
cjwatsonzyga,mborzecki: It's a bug in the PR - top line of packaging/debian-sid/changelog should say 2.39.1-1 rather than 2.39.1 (compare with the following changelog stanza in that same file)09:07
mborzeckicjwatson: thanks!09:07
cjwatsonnp, took me a little while to spot it09:08
mborzeckizyga: that means the changelog in release/2.39 branch is busted too09:10
mborzeckizyga: the one for sid09:10
mborzeckizyga: well, the travis job on the branch failed :P09:15
mupPR snapd#6936 opened: packaging/debian-sid: fix changelog, add revision <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6936>09:17
mborzeckizyga: pstolowski: ^^09:19
mupPR snapd#6855 closed: overlord: implement store switch remodeling  <Remodel :train:> <Created by pedronis> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/6855>10:11
zygare; sorry for missing half of today, I was baby-sitting upstairs due to unexpected errand for my wife; I'll handle paperwork for half day off10:17
zygaback to work now10:17
zygacjwatson: thanks for the analysis, I totally missed that10:18
zygajamesh: hey, any luck with the extra patch?10:18
jameshzyga: sorry.  Got side tracked before setting up a test system.10:24
zygano worries10:24
zygaI'll see if I can debug that today10:24
jameshzyga: I suspect it won't make a difference: https://paste.ubuntu.com/p/GV4bQW8XKy/10:24
jameshzyga: I did manage to simplify the test snap down to only using layouts, so it is self contained10:24
zygawow10:24
zygajamesh: what's the kernel version?10:24
jameshzyga: this was on an 18.10 box with 4.18.0-15-generic10:25
jameshI'll grab my other system to see if the same happens with 19.0410:26
zygapstolowski: hey10:28
zygapstolowski: do you have a moment for a quick HO?10:28
jameshzyga: same result on 5.0.0-11-generic10:29
pstolowskizyga: hey, sure10:29
jamesh(that system needs a reboot)10:29
jameshzyga: https://launchpadlibrarian.net/425983579/snapcraft.yaml <-  that's the simpler version exhibiting the bug10:31
zygapstolowski: pushed to https://github.com/snapcore/snapd/pull/693210:53
mupPR #6932: tests: run unit tests without networking <Created by zyga> <https://github.com/snapcore/snapd/pull/6932>10:53
mupPR snapcraft#2575 closed: catkin spread tests: use kinetic instead of indigo <Created by kyrofa> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2575>10:55
pstolowskizyga: thanks10:56
mborzeckijournald on opensuse is acting up, i've restarted a spread run that failed on opensuse again and again the log the test is looking for isn't there11:00
mborzeckizyga: conflicts in #693211:31
mupPR #6932: tests: run unit tests without networking <Created by zyga> <https://github.com/snapcore/snapd/pull/6932>11:31
zygamborzecki: aha11:32
* Chipaca wanders off in search of sustenance11:33
* zyga hands Chipaca a copy of the GNU manifesto ;)11:34
* Chipaca is suspicious11:34
zygaChipaca: to start the fire ;D11:35
* Chipaca puts on his FSFLA tee and scowls11:36
zygamborzecki: resolved11:37
=== ricab is now known as ricab|lunch
mupPR snapd#6936 closed: packaging/debian-sid: fix changelog, add revision <Simple 😃> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/6936>11:42
mborzeckipushed the changelog fix to #692711:43
mupPR #6927: release: 2.39.1 <Created by mvo5> <https://github.com/snapcore/snapd/pull/6927>11:43
zygamore unit test failures... https://www.irccloud.com/pastebin/uqQwKN1R/11:48
zygathat is curious...11:49
zygapstolowski: heh, that test fails *again*11:58
zygadebugging11:58
zygabut apparently the unshare -n trick is not enough11:58
zygaman, that's fum11:58
cachiomborzecki, hey11:58
cachiomborzecki, there is an issue on i386, it is going out of memory11:59
cachiomborzecki, https://paste.ubuntu.com/p/xbqvTQSp6Y/11:59
pstolowskizyga: fails again after your fix?11:59
cachioyesterday I was researching about it wit mvo11:59
zygapstolowski: it passes in unshare -n tests, fails in "osc build"12:00
cachiomborzecki, but then I spent time doing tests and could reproduce it on beta and candidate12:00
mborzeckizyga: maybe loopback is down there after all?12:01
zygamborzecki: no, that causes lots of tests to fail12:02
zygamborzecki: I'd like to get "osc shell" or something like that to play12:02
zygalet's see12:02
mborzeckicachio: hm looks like something failed while loading apparmor profiles?12:04
cachiomborzecki, yes12:04
cachioapparmor_parser: Unable to replace "snap.test-snapd-tools.fail".  Out of memory12:04
cachiomborzecki,  kernel: vmalloc: allocation failure: 68497 bytes12:05
cachioI tested that with kernel on stable and works well12:06
cachiobut kernel on candidate and beta are failing at some point12:06
zygacachio: can you look at aa-status?12:11
cachiozyga, https://paste.ubuntu.com/p/RBPTNpXz4Z/12:12
zygacachio: perhaps after running each test we should remove loaded apparmor profiles12:14
cachiozyga, I could try that12:15
cachiozyga, it is weird becuase it doesn't fail with pc-kernel=stable12:15
zygacachio: and this is with which kernel?12:15
cachiothis is with kernel on beta or candidate12:16
cachioworks well with kernel on stable12:16
zygacachio: do we know what is the difference between those?12:16
cachiozyga, no12:16
cachioI left tests running12:16
zygacachio: perhaps we can ask the kernel team for a summary?12:16
zygapstolowski: so ...12:16
cachioand I just got the results12:16
zygafailure in osc build, with extra debugging https://www.irccloud.com/pastebin/xy59XKOr/12:17
zygacachio: +112:17
jdstrandcachio: curious, have you seen https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1830502? jjohansen is looking at that12:17
mupBug #1830502: apparmor uses excessive memory leading to oom kill <apparmor (Ubuntu):New> <https://launchpad.net/bugs/1830502>12:17
cachiojdstrand, nice12:18
cachiojdstrand, that could explains that12:18
mborzeckizyga: hmmm that error, looks like temporary/permanent/retryable mess again12:19
zygamborzecki: yeah, we just worked on that with pstolowski12:19
jdstrandcachio: it may or it may not, but there are debugging details that jjohansen laid out that might be helpful12:19
zyga!!! https://launchpadlibrarian.net/425906431/snapcraft.yaml12:19
zygajdstrand: that's one long layout12:19
zygamborzecki: this is supposed to run with this applied https://github.com/snapcore/snapd/pull/6932/commits/c3b1296f1124158fd7e0987e9bd0196b20b3634d12:20
mupPR #6932: tests: run unit tests without networking <Created by zyga> <https://github.com/snapcore/snapd/pull/6932>12:20
jdstrandzyga: heh, yes. I think I'm going to put something in the review-tools for an abusive use of layouts. I'm not sure what that number should be, but clearly it is less than what is in that yaml :)12:20
cachiojdstrand, sure12:20
zygajdstrand: at most a dozen12:20
zygawe can expand if we have to12:20
mborzeckizyga: well, so now it connection refused? :)12:21
jdstrandack12:21
zygamborzecki: yeah12:21
zygadifferent error12:21
pstolowskiuhm12:21
zygapstolowski: ah, I see what the bug is now12:23
zygahttps://www.irccloud.com/pastebin/62CfKEFd/12:23
pstolowskizyga: uhmmm.. horror continues12:29
zygayeahj12:29
zygafixed locally, let's see12:29
zygaI'll push that to the offline tests PR12:29
mborzeckiidk maybe our approach is wrong12:29
zygamborzecki: perhaps12:30
zygamborzecki: we'd have to get some data about how the errors look like12:30
zygaput that into a table12:30
zygathen train an AI against someone saying if this is good or bad12:30
zygaand then if you have a CUDA card we'd retry correctly ;)12:30
zygapstolowski: tests passed!12:30
pstolowski\o/12:30
mborzeckicachio: did you get a chance to look into updating opensuse to 15.1?12:31
cachiomborzecki, no, but I can do it today12:32
zygamborzecki: but in all seriousness, I think we could be better if we had a good data set and wrote proper algorithm12:32
mborzeckicachio: 15.1 is out, hope they published the images too12:32
cachiomborzecki, nice, I'll check that and create the image if it is available12:33
cachiomborzecki, thanks for hte heads up12:33
zygahttps://build.opensuse.org/project/show/system:snappy12:33
zygadarn OSC is *insanely quick*12:33
zygaI wrote "osc commit"12:33
zygamoved to another display12:33
zygareloaded a tab12:33
zygaand it's building already12:33
zygamborzecki, cachio: FYI, I'm discontinuing builds for openSUSE Leap 42.2 and 42.3 as they don't provide golang-go >= 1.912:35
cachiozyga, ok, so I'll clean up them when I create the PR for opensuse 15.112:35
zygathanks12:36
cachiozyga, thanks to you12:36
zygabuild complete12:37
mborzeckizyga: sounds fair12:38
* zyga looks at mount bug from jamesh now12:40
=== ricab|lunch is now known as ricab
mupPR snapd#6931 closed: tests: force removal to prevent restore fails when directory doesn't exist on lp-1801955 test <Simple 😃> <Created by sergiocazzolato> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6931>12:58
jdstrandcachio: hey, mvo is out and not sure if you need it, but I saw core20 float into the store queue and I've approved it, but it needs to be released to a channel13:00
jdstrandzyga: perhaps you're interested in ^13:00
zygajdstrand: thanks13:00
zygajdstrand: I think we want it in edge only, we'll discuss that13:00
jdstrandI figured and would've put it there, but didn't have the perms13:01
zygaeh, btrfs corrputed13:27
roadmrhaha, rancid butter :)13:27
mborzeckizyga: clearly production ready fs13:30
mborzeckizyga: can you restore from a snapshot at least or sth?13:30
zygayeah, managed to mount a ro copy13:31
zygaI'll just add a HDD, reformat and move over13:31
zygausing ext4 was my best tumbleweed choice13:31
zygabtrfs is cool but no thanks :/13:31
zygarsyncing...13:37
roadmrjdstrand: 20190528-2014UTC click-reviewers-tools version  now in production 💪14:18
roadmr(nothing broke after the big rename-refactor)14:18
jdstrandTooLmaN: exciting! :)14:18
jdstrandyikes14:18
jdstrandTooLmaN: sorry, nm :)14:18
jdstrandroadmr: exciting! :)14:18
jdstrand(tab fail)14:18
roadmrif you want to get rid of the click-review symlink, I can change the command we invoke14:18
roadmrjdstrand: hehe yes tabs can be tricky14:18
jdstrandroadmr: it helps if I type the first letter correctly :)14:19
roadmrhehe a clear case of autocowreck14:19
jdstrandroadmr: I would like to get rid of the click-review symlink, but it is far from urgent. maybe let's sit on 20190528-2014UTC for a little while and then revisit?14:19
roadmrjdstrand: sure thing; yes, I think I can change the executable name on my side and if things look stable in a few weeks we can kill the symlink14:22
jdstrandroadmr: sounds like a plan14:24
zygamborzecki: wanna see something curious14:27
mborzeckizyga: hm?14:27
cachiozyga, mborzecki opensuse-15.1 nor available on glcoud yes14:30
cachioyet14:30
cachioI'll see if I could upgrade from 15.014:30
cachioand start using that14:30
zygamborzecki: https://bugs.launchpad.net/snapd/+bug/183101014:32
mupBug #1831010: Snapd fails to upgrade package that uses layouts to bind mount parent directory of another mount point <snapd:New for zyga> <https://launchpad.net/bugs/1831010>14:32
zygamborzecki: last two comments14:32
zygathis seems to show that our assumptions that we can detach anything are wrong14:33
zygaif we need to investigate detach algorithms that untangle mounts then it will be a fun thing to fix14:33
zygacachio: can you ask some suse people about this?14:37
zygacachio: I found this https://build.opensuse.org/package/show/Cloud:Images:Leap_15.1/openSUSE-Leap-15.1-GCE-Guest14:37
zygacachio: https://download.opensuse.org/repositories/Cloud:/Images:/Leap_15.1/images/14:38
zygathere are images here14:38
zygacan you please try those?14:38
mborzeckizyga: at least it's reproducible here too14:40
zygamborzecki: some ideas here14:42
zygamborzecki: we can construct an in memory tree from mountinfo mount-id / parend-id pairs14:42
zygaand then walk that14:42
zygato unmount leaves before inner nodes14:43
zygaI'm still puzzled by propagation of detach mounts14:43
zygaperhaps what is really breaking is the propagation of the unmount conflicting with itself?14:43
tim-jone_nmcli critical error trying to enable wifi interface on ubuntu core (18.04.2) on raspberry pi15:44
tim-jone_sudo network-manager.nmcli r wifi on gives error: (process:3062): nmcli-CRITICAL **: Error: Could not create NMClient object: Could not connect: Permission denied.15:44
tim-jone_can anyone suggest what I might do to get wifi working with ubuntu core on a raspberry pi15:45
Chipacatim-jone_: i thought that was handled via netplan15:53
Chipacatim-jone_: but i'm not too certain15:53
Chipacaogra: help?15:53
tim-jone_following instructions here: https://gist.github.com/cyan198/6ecdb86267498a51587b4337e47e10f615:53
Chipacatim-jone_: one thing I see you do that I don't see in that gist is 'sudo'15:54
cyphermoxtim-jone_: maybe just skip that first part; it might not be disabled at all15:55
tim-jone_have been using sudo15:55
cyphermoxChipaca: his pasted command did include sudo15:55
Chipacacyphermox: exactly15:55
Chipacathe gist does not use sudo15:55
tim-jone_yes that error is with sudo15:55
Chipacatim-jone_: and what happens without sudo?15:55
tim-jone_same error15:56
cyphermoxI think you just want to move on to 'network-manager.nmcli d wifi list'15:56
Chipacatim-jone_: what's the output of 'snap connections network-manager'?15:56
tim-jone_Interface              Plug                                   Slot                     Notes15:57
tim-jone_dbus                   network-manager:wpa                    -                        -15:57
tim-jone_firewall-control       network-manager:firewall-control       :firewall-control        -15:57
tim-jone_modem-manager          network-manager:modem-manager          :modem-manager           -15:57
tim-jone_network                network-manager:network                :network                 -15:57
tim-jone_network-manager        -                                      network-manager:service  -15:57
tim-jone_network-manager        network-manager:nmcli                  -                        -15:57
tim-jone_network-setup-observe  network-manager:network-setup-observe  :network-setup-observe   -15:57
tim-jone_ppp                    network-manager:ppp                    :ppp                     -15:57
tim-jone_(is there a better way to paste output or should I use a pastebin link or similar in IRC?)15:57
Chipacatim-jone_: paste.ubuntu.com ftw15:57
Chipacatim-jone_: do all nmcli commands end with the same error?15:58
tim-jone_cyohermox network-manager.nmcli d wifi list gives the same error (with or without sudo)15:59
cyphermoxmmkay15:59
cyphermoxso some kind of real permission issue; Chipaca will def know more about it than me if it has to do with the connections15:59
Chipacatim-jone_: can you pastebin 'snap logs network-manager'?15:59
tim-jone_sudo network-manager.nmcli gives version OK, but try any objects (radio etc) it fails16:00
Chipacayeah, there's something wrong permissions-wise, but the connections seem a'ight -- which, unless i'm missing something, means there's a bug in the snap16:01
tim-jone_output of snap log network manager : https://paste.ubuntu.com/p/QPwwTx68gv/\16:01
Chipacabut that's snap in rather prominent use so somehting's up16:01
Chipacatim-jone_: that seems ok16:01
Chipacatim-jone_: it seems ogra is away, could you open a topic in forum.snapcraft.io (in the 'devices' category)?16:02
Chipacasorry, 'device'16:02
tim-jone_OK - will do - but reading the log I pasted I see that despite network manager not working the wlan0 interface is listed via ifconfig which it was not before16:04
tim-jone_@Chipaca: have opened a topic in device category as you suggested :-)16:15
=== pstolowski is now known as pstolowski|afk
Chipacawoo, my pr is green, time to take a break16:34
ograChipaca, what makes you think i know how to use NM on core ? ... none of my images ever used it ;)16:37
* ogra just uses a proper netplan.yaml and the network-setup-control interface from a configuration snap16:37
tim-jone_ogra: is this gist wrong then? : https://gist.github.com/cyan198/6ecdb86267498a51587b4337e47e10f6 (pity its the first hit on google if so)16:45
tim-jone_BTW did configure /etc/netplan/50-cloud-init.yaml manually and the wifi interface is up and running16:46
tim-jone_is there some other documentation I should be reading?16:46
jjohansencachio, jdstrand, zyga: that is a different bug16:55
cachiojjohansen, thanks17:15
cachiozyga, , I could reproduce the error using the core from stable and the kernel from beta17:16
cmatsuokawhat is a common cause for "Reported install problem for xxx" in handlers?17:54
mupPR snapd#6927 closed: release: 2.39.1 <Created by mvo5> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/6927>17:55
zygajjohansen: thank you for confirming that17:55
cmatsuokaMay 30 12:39:53 localhost snapd[702]: handlers.go:414: Reported install problem for "snapd" as 05c989c0-82d8-11e9-9fdc-fa163e6cac46 OOPSID17:55
Chipacaogra: I just assume you know everything17:56
* Chipaca vanishes in a cloud of EOD17:57
mupPR # closed: core-build#11, core-build#22, core-build#26, core-build#37, core-build#3818:04
mupPR # opened: core-build#11, core-build#22, core-build#26, core-build#37, core-build#3818:05
mupPR snapcraft#2578 opened: keyrings: update ROS signing key <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2578>18:24
Odd_BlokeHey folks, how can I remove a snap without wasting time on snapshots?20:43
Odd_Bloke(I'm removing it because all the (ephemeral) data ended up corrupted; I really don't want a snapshot.)20:44
ijohnsonthere either is or will be a --purge option to snap remove20:45
ijohnsonnot sure if that has landed or not20:45
Odd_BlokeThere isn't where I am, unfortunately.20:45
Odd_BlokeAnd it looks like because I've started the remove operation, I'm stuck with the snapshotting regardless.20:46
ijohnsonyou might try refreshing to the edge channel of the core snap20:46
ijohnsonif you use `snap changes` you can see what id of the remove task is, and then abort that task with `snap abort $id`20:46
Odd_BlokeAha, that got me to the point where I could run `snap set system snapshots.automatic.retention=no` before removing again.20:48
Odd_BlokeThanks for the help, ijohnson!20:48
ijohnsonalso fwiw the --purge option is at least in the edge channel too20:48
ijohnsonglad to help20:48
mupPR snapd#6937 opened: cmd/snap-update-ns: detach unused mount points <Created by zyga> <https://github.com/snapcore/snapd/pull/6937>21:36
zygajdstrand: ^21:37
zygajamesh: ^21:37
zygarunning final spread locally before opening21:37
* zyga EODs21:40

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!