=== jaggz| is now known as jaggz === chihchun_afk is now known as chihchun [05:36] PR snapd#3617 opened: Using udev tagging for snap interfaces === sdrobertw is now known as nchip === nchip is now known as sdrobertw [06:10] sergiusens_: kyrofa: hey, any idea what's happening on https://build.snapcraft.io/user/ubuntu/ubuntu-make/59531 ? Local build works (on artful), and the only change with latest successful build on amd64 (https://build.snapcraft.io/user/ubuntu/ubuntu-make/56636) is changing a README.md… [06:10] sergiusens_: kyrofa: as build.snapcraft.io doesn't have a way to retrigger any build, I did a small edit on README.md, but the failure is reproducible: https://build.snapcraft.io/user/ubuntu/ubuntu-make/59842 [06:10] cjwatson: hey, btw, do you know if there is a hidden way to retrigger a build on b.snapcraft.io? ^ [06:27] PR snapd#3618 opened: Merge release/2.26 back into master [06:32] stgraber: hey, I'm looking into https://forum.snapcraft.io/t/snapcraft-cannot-cleanbuild-in-snapped-lxd/1424/ right now and wonder where i can find the lxd snapcraft.yaml and the source of the snap? [06:43] pedronis: unless you think gustavo should have a look as well, I'm keen to merge 3496. sorry for the long delay with the review. [06:43] pedronis: if you could update 3560 once its in, that would be great, then the diff is smaller :) [07:23] mvo: hi, I think it's ok to merge [07:23] * pedronis has breakfast [07:24] PR snapd#3496 closed: cmd/snap-repair: start of Runner, implement first pass of Peek and Fetch [07:24] I'll update the descendants after [07:25] pedronis: thank you! I can also do that I think [07:25] pedronis: but its fine, I will wait for you (enjoy breakfast) - I am in the middle of a different branch right now anyway :) [07:26] mvo: might be better if I do it, I used rebased in at least on of them at some point, fear it might get confusing [07:26] for me [07:26] pedronis: sounds good, thank you [07:43] mvo: done with snapd#3560 , the later ones are based on that one [07:43] PR snapd#3560: cmd/snap-repair: implement most logic to get the next repair to run/retry in a brand sequence [07:54] PR snapd#3616 closed: cmd/snap-repair: implement Runner.Verify and use it to check signatures of repairs from Next === chihchun is now known as chihchun_afk [08:03] good morning [08:03] * zyga-ubuntu waves from very very rainy warsaw [08:14] hey Chipaca [08:14] zyga-ubuntu: \o! [08:15] * zyga-ubuntu is working from the quiet national library today [08:22] zyga-ubuntu: i've got the ideal thing for you then! [08:22] zyga-ubuntu: https://www.youtube.com/watch?v=yf0Amcgxot8&t=79 [08:24] Chipaca: well [08:24] Chipaca: no sound allowed in here :) [08:24] zyga-ubuntu: hey, good morning! [08:25] Chipaca: hey, good morning! [08:25] zyga-ubuntu: https://www.youtube.com/watch?v=g4mHPeMGTJM :-( [08:25] mvo: o/! [08:25] Chipaca: I'll watch it back home [08:25] Chipaca: or at the 2nd library [08:25] * zyga-ubuntu is sad that none of the libraries actually allow you to borrow books anymore [08:25] just read on site [08:26] zyga-ubuntu: whaat :-( [08:26] they should be called more appropriately [08:26] Chipaca: yep [08:26] zyga-ubuntu: that speaks badly of y'awl [08:26] Chipaca: both the national library and the university library [08:26] at least for regular mere non-PHD folks like me [08:26] zyga-ubuntu: wellp, time to work on your phd then [08:26] * Chipaca runs [08:27] yes, just need to find a lottery ticket that lets me not work for 5 years [08:27] then 5 more [08:27] ^_^ [08:28] but yeah, otherwise very much what I want tod o [08:31] zyga-ubuntu: it's really easy, i have a 2-step plan for you [08:31] zyga-ubuntu: 1. be rich; 2. don't be poor [08:31] Chipaca: ah and you forgot one thing [08:31] Chipaca: 3 be young and pretty [08:32] zyga-ubuntu: i counted that under 'rich' [08:32] Chipaca: in the mean time https://github.com/snapcore/snapd/pull/3619 [08:32] PR snapd#3619: interfaces/builtin: reduce duplication and remove cruft in Sanitize{Plug,Slot} [08:32] some gardening [08:32] files changed, 52 [08:32] PR snapd#3619 opened: interfaces/builtin: reduce duplication and remove cruft in Sanitize{Plug,Slot} [08:32] zyga-ubuntu: i'll do that next then [08:32] but lots of red in it :) [08:32] just cruft lifting [08:32] let me go and see if the book I'm interested in is available now [08:39] Chipaca: any news on the profile.d stuff from last night? sorry for nagging [08:39] mvo: no :-( [08:40] Chipaca: can I help in any way, happy to poke at this too, high priority for me currently [08:40] PR snapd#3616 opened: cmd/snap-repair: implement Runner.Verify and use it to check signatures of repairs from Next [08:41] mvo: so... one thing we _could_ do [08:41] mvo: but makes me uncomfortable to think about too much on my own [08:41] mvo: is to make snap tweak the user's bashrc [08:42] mmh [08:42] mvo: it could be either forwards, adding a 'source profile.sh # added by snap' line [08:42] mvo: ir negative, seeking out the broken 'source bash-completion' and commenting it out [08:43] re [08:43] mvo: i like 0 of these -- but it'd get the job done [08:43] on the upside I got the book :) [08:43] and I can work here all day [08:43] so win-win [08:43] (and it's quiet) [08:44] Chipaca: sounds a strange idea (tweaking bashrc); I supose everything else is worse? [08:44] pedronis: more like nothing else works [08:44] that i could think of at least [08:44] but i'm an egg [08:44] because ? [08:45] pedronis: there's a bug in … debian i guess? [08:45] pedronis: there's /etc/profile.d/bash_completion.sh [08:45] but nothing uses it? [08:45] pedronis: that loads the bash completion work [08:46] pedronis: that gets loaded by /etc/profle, along with everything else in profile.d [08:46] pedronis: but then the default .bashrc, from /etc/skel, loads it _again_ [08:46] pedronis: our profile snipped needs to load after bash completion is loaded [08:46] snippet* [08:46] I see bashrc ends up loading: /usr/share/bash-completion/bash_completion [08:47] yes [08:47] so, there's a number of ways that we can make it work [08:47] if we patch bash-completion itself [08:47] Chipaca: step 1: fix debial stable [08:48] Chipaca: ^_^ [08:48] but if we can't do that, then we're stuck [08:48] but at least you can hope to fix that ;) [08:48] zyga-ubuntu: aw bless [08:48] the easiest way to fix bash completion is to make it not load the default completer if there is one already [08:49] Chipaca: one more, this time fully red: https://github.com/snapcore/snapd/pull/3620 [08:49] PR snapd#3620: interfaces/builtin: discard empty Validate{Plug,Slot} [08:50] PR snapd#3620 opened: interfaces/builtin: discard empty Validate{Plug,Slot} [08:51] Chipaca: fwiw it seems to source stuff from /etc/bash_completion.d but as back compat thing [08:51] pedronis: yes [08:52] and then ~/.bash_completion [08:56] mvo: did you see latest comments on https://bugs.launchpad.net/snappy/+bug/1674193 [08:56] Bug #1674193: core snap's configuration hangs on debian | openSUSE | mainline kernel [08:56] mvo: I fail to see how that is possible since that is exactly what was fixed last week [08:57] Chipaca: modifying the users .bashrc makes me uneasy. the other thing is that is tricky is when/how do we do it? iterate over all users? do it in snap run (which is too late)? other? [08:57] zyga-ubuntu: looking [08:57] mvo: thank you [08:57] Chipaca: sorry :/ [08:57] mvo: no, really thank you :-) [08:58] mvo: easiest way to kill something that makes me uneasy to think about doing is by pointing out it wouldn't work :-) [08:58] mvo: unless we are missing something and people start with not-up-to-date debian 9 [08:59] Chipaca: heh :) [08:59] zyga-ubuntu: yeah, its very confusing, I download an image now [09:03] Chipaca: so we need to ship a different bash_completion ? with some bits added? [09:11] zyga-ubuntu: aha, I think I know what is happening there. on undo we will not restart into the old snapd. so I guess what happend is: snapd-2.21 refreshes, restarts into 2.26.14, something fails (configure hook?), things are undone. snapd 2.26.14 keeps running, however there is no 2.26.14 snap anymore. [09:12] mvo: oh [09:12] mvo: did you manage to reproduce it? [09:12] mvo: and didn't we ignore configure hook on core errors? [09:13] zyga-ubuntu: strange enough - I got a configure hang on debina9 though, then it fails with cannot stat /var/lib/snapd/seccomp: no such file or directory [09:13] (errors in the configure hook when running on the core snap) [09:14] zyga-ubuntu: yes, we ignore those I suspect the issue is that it starts from 2.21 but I need to double check [09:14] didrocks: there is not [09:24] zyga-ubuntu: ok, this is confusing - the configure hook (snap-confine to be precise) is waiting for /var/lib/snapd/seccomp/bpf/snap.core.hook.configure.bin to show up. it *does* show up in the host, however strace shows me that snap-confine (the configure hook) is not seeing this. we are not doing any namespace stuff on debian, right? so why would it not see it? [09:25] mvo: we do namespace stuff on debian just the same but I believe we read the bpf parts before that *and* even if I'm wrong /var/lib/snapd is shared with the host [09:25] mvo: there's systemd in your system, right/ [09:27] zyga-ubuntu: let me check, this is a stock skretch install [09:27] zyga-ubuntu: yes, systemd is there [09:28] hmm mhm [09:28] magic [09:28] and why didn't it ever happen in spread? [09:29] zyga-ubuntu: spread uses sid, right ? maybe its rleated to stretch [09:29] zyga-ubuntu: anyway, puzzling [09:29] zyga-ubuntu: what was the nsenter magic again to get inside the env of the core configure hook? [09:29] mvo: I think spread uses sid, yes [09:29] mvo: but spread qemu is stretch [09:30] mvo: nsenter -m/run/snapd/ns/core.mnt [09:30] zyga-ubuntu: /var/lib/snapd is empty in there [09:31] zyga-ubuntu: like completely empty [09:31] zyga-ubuntu: but mount claims /var/lib/snapd is mounted (type ext4 etc) [09:32] mvo: can you pastebin /proc/self/mountinfo [09:32] (from that namespace) [09:32] and from outside [09:32] zyga-ubuntu: mountinfo says /var/lib/snapd//detleted /var/lib/snapd [09:32] and pastebin both to the forum [09:32] oh [09:32] cute!?! (not) [09:32] anyway, let's do this in the forum [09:32] very interesting I must say [09:39] zyga-ubuntu: created the topic now [09:41] * zyga-ubuntu looks [09:42] mvo: did you pure snapd at any time? [09:42] mvo: maybe what happened is that the core.mnt namespace sticks around [09:42] you purge snapd [09:43] and re-install [09:43] and we use a stale core.mnt file [09:43] (hence //deleted) [09:43] do a quick test please [09:43] unmount /run/snapd/ns/core.mnt [09:43] and install core again [09:44] zyga-ubuntu: sure, one sec [09:44] * zyga-ubuntu found the power plug and is happy not to live on a ticking clock anymore [09:44] but whatever it is, my fix for stale mount namespace needs to include //deleted checks [09:50] mvo: any luck with unmounted core.mnt? [09:50] zyga-ubuntu: I was just trying to reproduce it again [09:51] zyga-ubuntu: I did reboot into a different kernel for testing in the meantime [09:51] zyga-ubuntu: hrm, hrm, now I cannot reproduce it anymore, even when I purge snapd [09:52] mvo: replied [09:52] mvo: run conifugre hook, purge, run configure hook [09:55] Chipaca: naming is hard(tm) [09:55] zyga-ubuntu: did this now, not breaking [09:55] (╯°□°）╯︵ ┻━┻ [09:55] mvo: I must say it is interesting how many edges we found to be rough over the months/year [09:55] Chipaca: huh? [09:56] zyga-ubuntu: yes, the re-exec thing from arbitrary starting points makes things complicated [09:56] zyga-ubuntu: that was to pedronis :-) [09:56] ┬─┬ ノ( ゜-゜ノ) sorry [09:57] * zyga-ubuntu wonders how many other fantastic emjoji we'd have if the table could include a teapot or a plant [09:58] zyga-ubuntu: ┬─┬⃰͡ (ᵔᵕᵔ͜ ) [09:58] zyga-ubuntu: it's just too small for your eyes to see [09:59] hahah [09:59] nice :) [09:59] mvo: can you use snapshots in your vm? [09:59] mvo: maybe start with a pristine image, snapshot that [09:59] mvo: install snapd, snapshot that (separately) [09:59] mvo: and try to get into the problem [09:59] mvo: then whatever you have can be inspected forever [09:59] zyga-ubuntu: yeah, too late now, whatever I did I did not snapshot :( [10:00] zyga-ubuntu: and can not get back into currently [10:00] mvo: since now you said purging doesn't make it broken I wonder if that shoots my hypothesis [10:00] zyga-ubuntu: like - now it is working [10:00] zyga-ubuntu: I think your hypothesis is good, maybe it was not a purge, maybe it was something else (an upgrade?) [10:01] zyga-ubuntu: I took a somewhat old VM, apt full-upgraded, dpkg --purge snapd (because it had core already); snap install --edge core; BOOM [10:01] mvo: do we remove /run/snapd/ns on purge? [10:01] if we do then purge is not the thing [10:01] because that thing cannot be stale when it is gone [10:02] zyga-ubuntu: we do umount /run7snapd/ns on purge [10:03] mvo: in that case it must have been something else [10:03] mvo: is dpkg doing some fancy things when it updates packages? any directory changes? [10:04] mvo: did the apt-get --dist-upgrade include snapd? [10:07] mvo: when did we start to do that though? (unmount /run/snapd/ns in purge? ) [10:09] pedronis: I see message related to it on my debian box, I need to check when exactly we started doing that [10:10] zyga-ubuntu: yes, snapd was upgraded 2.21-2, 2.21-2+b1 [10:10] zyga-ubuntu: /me tries to reproduce using upgrade [10:11] mvo: aha, interesting [10:11] mvo: I doubt dpkg would do something that would cause the package update to remove a directory like /var/lib/snapd though [10:11] mvo: but if you can try the 2.21-2 -> 2.21-2+b1 update again [10:11] that would be a useful data point [10:12] I recall that //deleted shows up when the backing store goes away [10:12] I moved my kernel sources to my desktop so I cannot cross reference locally, let me check something online [10:14] zyga-ubuntu: hrm, hrm, upgrade, purge, install, upgrade, no boom, just works [10:19] mvo: http://elixir.free-electrons.com/linux/latest/source/fs/dcache.c#L3363 and http://elixir.free-electrons.com/linux/latest/source/fs/proc_namespace.c#L130 === sergiusens_ is now known as sergiusens [10:34] cjwatson, did anything change with the builders last night ? all (except s390x it seems) my daily builds of the core snap suddenly fail with: [10:34] P: Begin bootstrapping system... [10:34] [2017-07-25 04:17:00] lb_testroot [10:34] E: need root privileges [10:34] P: Begin unmounting filesystems... [10:34] P: Saving caches... [10:34] chroot: cannot change root directory to 'chroot': No such file or directory [10:34] cjwatson, https://launchpad.net/~snappy-dev/+snap/core for details [10:35] ogra_: fix for https://bugs.launchpad.net/launchpad-buildd/+bug/1702656 was (mostly) rolled out [10:35] Bug #1702656: Run snapcraft as non-root (with passwordless sudo) [10:35] cjwatson, ah, thanks [10:35] ogra_: can you just run with sudo? [10:38] cjwatson, is that configured passwordless in the buildd ? [10:39] ogra_: yes [10:40] zyga-ubuntu: thanks, I give up on this for now and have lunch. slightly frustrating [10:40] ogra_: I think just changing "ENV := PROJECT=ubuntu-core ..." to "ENV := sudo PROJECT=ubuntu-core ..." would do the trick [10:40] Maybe with SUDO := sudo and $(SUDO) if you want a bit of extra configurability [10:41] thanks! will add that [10:42] (though it might need more, the check and install targets in the makefile touch the chroot as well) [10:42] mvo: I'll check it myself back home [10:42] mvo: one interesting fact is that snap-confine (which may run in classic snap environment) now needs to run snap-update-ns [10:42] mvo: so if the rabbit hole wasn't deep enough I just brough a new shovel [10:42] (now as in litterally right now on my disk in a new branch) [10:52] Hello all [10:52] niemeyer: o/ [10:53] ogra_: ah, right, yeah [10:53] ogra_: would appreciate confirmation when you get a result, as we ought to do manual upgrades of s390x [10:53] cjwatson, will let you know [10:54] ta [10:54] mvo: i have good news: on fedora 25 the profile.d approach would work [11:08] PR core#51 opened: use sudo to work around LP: #1702656 [11:19] mvo, zyga-ubuntu ^^^^ can i get two acks to actually test-build in the LP buildd [11:19] (travis passes fine but wont help since the LP env is different now) [11:26] yep [11:26] ogra_: +1 [11:26] thx [11:27] ogra_: yes, go for it [11:27] Chipaca: oh, nice. what is the difference? [11:27] thanks ! [11:27] PR core#51 closed: use sudo to work around LP: #1702656 [11:28] * ogra_ triggers test builds and crosses fingers that he catched all occurences [11:28] ogra_: sorry for the disruption, I test-built various things but totally forgot about core being a special snowflake [11:28] PR snapd#3618 closed: Merge release/2.26 back into master [11:28] I think it's long-term better this way [11:29] cjwatson, well there is more in my set of snaps that uses chroots in chroots etc ... but now that i know about it i can adapt ... no prob [11:31] ppisati, ^^^ you might need to look into this too for the kernel snap builds (adding sudo to the commands that actually require root ... like the chroot and debootstrap commands in the Makefile) [11:37] grrr ... forgot to sync Gh to LP before hitting the build button ... [11:43] bah ... needs more sudo it seems [11:43] rm -f config/hooks/* [11:43] rm: cannot remove 'config/hooks/00-uid-gid-fix.chroot_early': Permission denied [11:43] rm: cannot remove 'config/hooks/01-divert-grub-install.chroot_early': Permission denied [11:43] rm: cannot remove 'config/hooks/01-setup_user.chroot': Permission denied [11:43] ... [11:44] funny, i wouldnt have expected that to be root owned [11:44] (given the unprivileged snapcraft did put it in place originally) [11:49] PR core#52 opened: also use sudo for all file operations [11:50] mvo, zyga-ubuntu ^^^^ one more please [11:51] (it should be fine then) [11:51] ogra_: looking [11:51] ogra_: done [11:51] thx [11:51] +1 [11:52] * ogra_ waits for travis ... [11:53] mvo: they don't do bash completion in .bashrc :-) [11:53] mvo: they just have the /etc/profile.d/bash_completion.sh [11:54] Chipaca: More comments on snapd#3614.. let me know if you'd like to talk about those [11:54] PR snapd#3614: daemon, client, cmd/snap: implement "snap status" [11:54] niemeyer: i've seen them come in, yes [11:54] i'm beyond the point of talking about this stuff. I'll just do whatever you say. [11:55] Chipaca: aha, nice [11:55] Chipaca: Okay, so let's keep that in the PR queue until you're ready to talk about it again [11:56] niemeyer: i've been working on this same bit of functionality for over a month, i'm ready to be done with it [11:56] Chipaca: Or somebody else is.. we need someone that is willing to think through the points made so we can have a shared idea of the problem [11:57] PR snapd#3621 opened: cmd/snap-{confine,update-ns}: apply mount proifles using snap-update-ns [11:57] first WIP branch on snap-update-ns/snap-confine and layouts [11:57] ignore it for now, I mainly pushed it to see if tests pass and discuss something with jdstrand later [11:57] Chipaca: In fairness, many of the points I made are exactly the same as a month ago [11:58] i'm going to go have lunch [11:58] Nice.. [11:58] I'm past starving now, so I can spend some time to collect notes and then have a call with everyone [11:59] zyga-ubuntu: Tests are broken on snapd#3620 [11:59] PR snapd#3620: interfaces/builtin: discard empty Validate{Plug,Slot} [12:01] PR core#52 closed: also use sudo for all file operations [12:02] cachio: In #3604, why is it increasing the number of workers? [12:03] mvo: Can you have a look at snapd#3604 when you have a moment, specifically on the #cgo statement on snap-seccomp, to see if it looks okay considering the issues we know about [12:03] PR snapd#3604: tests: enable main suite for opensuse [12:04] niemeyer: oh, thanks [12:11] cachio, fgimenez: random error on interface-cups-control: https://travis-ci.org/snapcore/snapd/builds/257218359?utm_source=github_status&utm_medium=notification [12:11] sorry, interfaces-cups-control [12:11] let me know if you want to collect the long [12:11] *log [12:11] and if not I'll re-trigger this [12:12] "Error - scheduler not responding" [12:13] cjwatson, ... hmm the build succeeds, but the staging step in snapcraft falls over when it tries to make use of the resulting dir ... https://launchpad.net/~snappy-dev/+snap/core/+build/59965/+files/buildlog_snap_ubuntu_xenial_amd64_core_BUILDING.txt.gz [12:13] (since it is all root owned ... ) [12:17] zyga-ubuntu: thanks! indeed, lpr scheduler not working, have you seen it on other archs? [12:19] nope, first time I saw this [12:19] and I suspsect it's not related to the PR [12:19] fgimenez: btw, I noticed that we have a disparity in debian testing [12:19] fgimenez: the qemu image tests debian 9 [12:19] cjwatson, snapcraft itself would have to run the staging step under sudo, i dont think i can do anything on the snap side for this ... :/ [12:19] fgimenez: but the linode machine tracks sid [12:20] fgimenez: we should keep those separate as sid is moving [12:21] cjwatson, couldnt you make the sudo behaviour conditional based on the snap type in snapcraft.yaml so that "type: os" and "type: kernel" simple keep the old behaviour ? [12:21] *simply [12:21] (and run the build as root) [12:31] snapd#3568 has a review.. any takes for the second review? [12:31] PR snapd#3568: snapctl: add new `snapctl internal configure-core` [12:33] niemeyer: I can do that after standup, on my way to find anything edible that doens't run away [12:34] zyga-ubuntu: Thanks! [12:34] zyga-ubuntu: and good luck :) [12:37] ogra_: Hmm. Ugly, but I guess we may have no alternative. [12:37] niemeyer: mvo I've been pinged about https://bugs.launchpad.net/juju/+bug/1705988 (error is -> run hook "configure": cannot change profile for the next exec call: No such file or directory) [12:37] Bug #1705988: snap install --classic juju fails [12:39] ogra_: What about gadget? [12:39] cjwatson, usually doesnt need root [12:39] ogra_: It'd mean we have to start parsing snapcraft.yaml directly in launchpad-buildd, which has previously been unnecessary. [12:39] niemeyer: 'm back from lunch. Less grumpy now. [12:40] niemeyer: let me rephrase what I said: I think you're right on most of the points, and on the points I don't, I haven't been able to convince you otherwise. I'll be implementing your suggestions as I understand them, and we can iterate. [12:40] Chipaca: Thank you! [12:41] cjwatson, yeah ... understood ... os has always been special though ... since you have a full root that includes /dev and all ... not sure if there is any way to do it differently ... [12:41] Chipaca: You'll also see some self-conflict on the comments.. which means I'm also trying to figure that stuff out [12:41] ogra_: I'm inclined to just revert this launchpad-buildd change for now. [12:41] i'm fine with that as well ... :) [12:41] mvo: you already branched 2.27 a while ago right? you will not take a new snapshot? [12:41] ev probably won't be, but such is life ... [12:42] cjwatson, doesnt lp-buildd actually call snapcraft step-wise ? (i.e. not plain snapcraft but each step) [12:42] ogra_: Only in that it calls pull separately. [12:42] cjwatson, then "snapcraft stage" could simply always run as root while all other steps keep the dropped privs [12:43] ah, k ... [12:43] ogra_: Yeah, that's one viable option for the future. I think it requires more testing than I want to put into a "stop the bleeding" change, though. [12:43] yeah [12:43] ogra_: I'll update the bug with some ideas once I've reverted. [12:44] snapcraft snap would likely also fall over on the /dev permissions [12:44] so it'd be more [12:45]