[04:15] morning [05:39] PR snapd#5694 closed: tests/main/layout: cleanup after the test [05:39] mvo: hey [05:41] hey mborzecki [05:41] zyga: can I merge 5307? [05:42] zyga: I would like to squash it, I would love to cherry pick this to 2.35 to fix core18s extrausers [05:54] mvo: could you take a look at https://github.com/snapcore/snapd/pull/5614 ? tha's a small change to the code + some tests [05:54] PR #5614: interfaces: parallel instances support, extend unit tests [05:59] sure [06:00] * mvo looks [06:00] mvo: thanks! [06:26] mborzecki: added some comments, sorry probably not strictly related to your pr but it seems its a good opportunity to improve this area [06:26] mvo: thanks, looking [06:27] PR snapd#5698 closed: osutil: reorg and stub out things to get it building on darwin [06:40] good morning! [06:40] mvo: looking [06:41] zyga: thanks [06:41] mvo: while jamie did not comment on the final look of the branch he didn't -1 the earlier version that wasn't much different [06:41] mvo: so I think yeah but the final call is up to you [06:41] mvo: in case jamie objects after the fact we can always roll out a .1 [06:41] * zyga had a small success by waking up the kids two hours earlier than usual [06:42] zyga: hey [06:42] hey hey [06:45] zyga: ok, I see. hm,hm [06:45] jamie mentioned he was going to look at it [06:45] so it's in his queue [06:47] while fighting spread yesterday (the unexpected failure I saw over and over) I refreshed all of the images on https://spread.zygoon.pl/images/ [06:47] I'm wondering why they grow so much though, 18.10 is 2.5x bigger than 14.04 [06:50] zyga: maybe 1G of it is all 0s [06:50] the images are qcow2, that would not be needed [06:50] and indeed they are not zero as they took a while to send there [06:54] zyga: maybe cached packages were left behind? [06:55] mborzecki: hmm, maybe but I used the same tool for all of those [06:55] I'll jump into those images if I have some time [06:55] I was just curious [07:00] zyga: a trick i use to be able to quick revert back to the clean image is to do a clean install, then use that image a backing file for qcow2 images, then you just have the diff images that you can rebase/commit once they are good enough [07:00] mborzecki: how do you rebase/diff things? [07:00] I never did that myself [07:03] qemu-img create -f qcow2 -b your-clean-install.img img-with-diff.img; do stuff inside, then qemu-img commit to move the changes back to the original image, or qemu-img rebase if you have an image chain [07:03] zyga: real life saver in case you screw up the os in the image at some point :P [07:03] interesting, and are the images aware of each other directly or do I need to tell qemu to use a chain somehow? [07:03] zyga: it's recorded in the image file [07:08] brb [07:09] mvo: pushed the changes to https://github.com/snapcore/snapd/pull/5614 [07:09] PR #5614: interfaces: parallel instances support, extend unit tests === pstolowski|afk is now known as pstolowski [07:10] morning [07:11] pstolowski: hey [07:13] wonder if i could push the parallel-installs-interfaces spread test to 5614 [07:13] re [07:15] got to go out for a while, back in ~1h or so === doko_ is now known as doko [07:59] zyga: jdstrand: Same problem with snap 2.35 and kernel 4.18.3. :-/ [08:00] can you provide the detailed denial please? [08:01] re [08:02] Just trying to get it to spit one out [08:04] mvo: review on 5699 [08:05] zyga: thanks, looking [08:06] zyga: thanks for the review, I will fix the issue(s) [08:07] did you mean the sleep 1h? [08:07] that was the most curious thing to me [08:07] zyga: I replied, no I did not, I hit the wrong key [08:07] zyga: or maybe a freudian slip, my mind wanted to go to sleep for 1h (at least ;) [08:07] haha :-) [08:09] zyga: replied, I will look into alternatives to squid I think, something in a snap [08:11] zyga: These are the only messages from "dmesg -w" while I ran "snap remove lxd" (which doesn't seem to have worked) and "snap remove postman", "snap install postman" which both seemed to work fine, but postman is not available to run. [08:11] https://paste.ubuntu.com/p/Z5C742DfZ8/ [08:11] Gargoyle: STATUS messages are just information, they are not a denial yet. What happens when you install and run the "hello-world" snap? [08:11] mvo: pushed a spread test for parallel instances & interfaces to #5614 [08:11] PR #5614: interfaces: parallel instances support, extend unit tests [08:14] zyga: No errors from snap - seems to have installed fine, but I have no "hello-world" command to run. (Syslog output during the installation: https://paste.ubuntu.com/p/jgDwrkyK4k/) [08:14] how about snap run hello-world [08:15] PR snapd#5700 opened: tests: significantly reduce execution time for managers test [08:16] zyga: Yup. That works! [08:16] ok [08:16] Gargoyle: can you try the snap you wanted to use before [08:17] "snap run postman" works too. [08:17] same for discord. [08:17] so the issue is fixed? [08:18] Oh. No. [08:19] so what is the issue? [08:19] They only work with "snap run discord". I cannot run just "discord" nor do they show in gnome. [08:19] log out and back in [08:19] ok [08:25] No joy. And now not even happy booting back into 4.15 kernel! :/ [08:25] welcome back, so what happened? [08:26] same situation. "snap run discord" works, but as far as the rest of the system is concerned, discord doesn't exist. Same for postman. [08:26] so gnome doesn't see the application, right? [08:27] what does "which discord" tell you? [08:27] correct [08:27] "discord not found" [08:27] and what does "echo $PATH" print? [08:28] . /home/paul/bin:/home/paul/.composer/vendor/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games [08:28] Gargoyle: interesting, are you setting PATH in your startup scripts anywhere? it seems to be influencing the default value that ought to also include /snap/bin/ [08:30] zyga: Only in .zshrc where I have: "export PATH=$HOME/bin:$HOME/.composer/vendor/bin:/usr/local/bin:$PATH" [08:30] ah, you are using zsh [08:31] perhaps that's the reason stuff is not working for you [08:31] yup [08:31] mborzecki: once you are back, do you remember if we had a way to inject environment into zsh; I recall a new systemd feature but I don't know if we ended up using it [08:32] zyga: iirc mvo came up with something that was shell agnostic [08:33] Gargoyle: are you using konsole by any chance? [08:33] Gargoyle: can you please do one more experiment, switch to bash, check of this fixes it; it will helps us scope the issue to just zsh [08:34] then we can go after that separately from any kernel issues [08:34] zyga: environment.d [08:34] mvo: anything I can man search for; I don't remember the details [08:36] updated arch package to 2.35 [08:36] mborzecki: Nope. Default Gnome terminal [08:36] zyga: yup. One sec [08:36] Gargoyle: I switched to zsh now, checking what shows up [08:38] PR snapd#5701 opened: image: add type to seed.yaml to allow sorting in firstboot [08:38] Gargoyle: are you on ubuntu? [08:38] pedronis: high level feedback on -^ would be great. if this looks sensible I will continue this path otherwise derive in firstboot [08:38] hm [08:38] I switched to zsh and I cannot log in [08:38] mborzecki: Yup [08:38] * zyga wonders [08:39] Gargoyle: can you post /etc/zsh/zprofile ? [08:39] No difference in bash. [08:40] * zyga can log in now [08:40] Gargoyle: I suspect you may need to reboot [08:40] mborzecki: Nope. It doesn't exist. /etc/zsh/zprofile is just comments, too. [08:40] depending on how early the profiles are picked up [08:40] Gargoyle: I confirm that there's no /snap/bin in PATH now [08:40] and that will also explain why gnome shell doesn't see any snap apps [08:41] zyga: in zsh on ubuntu? [08:41] I'll reboot my system now and then look at environment.d [08:41] mborzecki: I switched to zsh yes [08:41] zyga: yeah, so there's a problem that ubuntu's zsh setup does not source /etc/profile files (while it does so on arch for instance) [08:41] zyga: fwiw, we added /usr/lib/environment.d/990-snapd.conf [08:42] mvo: I'll rebuild the package and see if it is used [08:42] zyga: iir there's a bug report for this, from ogra maybe? [08:42] thanks! [08:42] mvo: yes, that works! [08:42] I have /snap/bin at the end of path [08:42] and I see snaps in gnome shell [08:43] Gargoyle: I'm running 2.35 from package built from master, reexec won't give you all of that yet [08:43] I'll rebuild the release branch [08:43] zyga: this is in 2.34.2 already so should be ok [08:44] I see [08:44] (sorry, haven't followed the previous discussion so I may miss things) [08:44] Gargoyle: what's the version of the debian package you have, you can check with "apt-cache policy snapd" [08:44] brb, rebooting [08:44] k [08:44] mborzecki, not mine, but i was active in it ... [08:44] zyga: https://forum.snapcraft.io/t/gnome-3-shell-snaps-integration/2678/12 [08:45] bug #1640514 [08:45] Bug #1640514: /snap/bin is not added to the PATH when using zsh [08:45] and along with that bug #1637220 [08:45] Bug #1637220: zsh has no /snap/bin in PATH [08:46] thank you ogra [08:46] I will update those bugs [08:46] well, the fix only helps if people actually went to 18.04 [08:46] so leave the 16.04 tasks open [08:47] ogra: yeah, this does depend on recent enough systemd, right? [08:47] right [08:47] how do I add 16.04 task? [08:47] nominate foir series [08:47] how do I do that [08:48] I never did that before, I may not have rights to do that [08:48] there is a link [08:48] zyga: https://paste.ubuntu.com/p/jGgcYf8DX5/ [08:48] everyone can nominate [08:48] approving the nomination is restricted ... [08:49] what do I need to click on, there's no "nominate" on the page [08:49] Gargoyle: so that seems ok [08:49] Gargoyle: I wonder what's different between our installations [08:49] zyga, done [08:49] ogra: what did you do :) [08:50] (the link right underneath the bug tasks with the little clock icon) [08:50] OK. Purged 4.18 from the system, force re-installed 4.15 and reverted back to "snap core --stable". and my $PATH is back! https://paste.ubuntu.com/p/YSNyg67xP4/ [08:50] next to "Also affects distribution/package" [08:50] aha [08:51] ogra: hmmm, [08:51] ogra: I don't see that icon, I only see it now after you did the nomination [08:51] weird, thats a common function that everyone should have access to [08:52] nomination isnt restricted ... only approving the nomination is [08:52] Only thing I can't do is remove lxd properly (so I can re-install) : https://paste.ubuntu.com/p/Rfr9gnSdQk/ [08:52] Bug #1640514 changed: /snap/bin is not added to the PATH when using zsh Xenial):Confirmed> [08:53] Gargoyle: you may need to stop the containers first but I defer to stgraber [08:53] zyga, also ... i dont think it is invalid in zsh xenial ... the fact that environment handling got improved doesnt fix the fact that zsh's env handling isnt using standards [08:54] yeah but for _this_ bug I think it is invalid now [08:54] zyga: hm so i dropped /etc/profile.d/snapd.sh, and have only /usr/lib/environment.d/990-snapd.conf, and I don't see $SNAP_MOUNT_DIR/bin being added to my PATH [08:54] I've used zsh since day 1, and not had problems. [08:55] mborzecki: I'm rebuilding 2.35 now from the release branch [08:55] I'll know soon :) [08:55] zyga: i'm on 2.35 on arch now ;) [08:55] maybe packaging skew? [08:56] mborzecki: 2.35 still has /etc/profile.d/apps-bin-path.sh [08:56] but also has /usr/lib/environment.d/990-snapd.conf [08:56] zyga: yesh, removed that manually here to check if environment.d works [08:57] mborzecki: interesting, only PATH is set, XDG_DATA_DIRS is not set at all [08:57] zyga: oh, so you do have PATH set in your setup via environment.d? [08:58] mborzecki: I _suspect_ so because I have empty profiles otherwise, let me reboot to check [08:58] zyga, well, if you prefer to put the blame on snapd ... :) (people come back to that bug when they hit the issue in 16.04... ) [08:58] zyga: can you do systemctl --user show-environment | grep PATH ? [08:58] mborzecki: sure, one sec [08:59] fyke% systemctl --user show-environment | grep PATH [08:59] PATH=/home/zyga/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin [08:59] WINDOWPATH=1 [09:01] zyga: heh, you have snap there, I don't :) [09:01] mborzecki: what do you have in /etc/environment.d [09:01] zyga: nothing, the file goes to /usr/lib/environment.d/ by default [09:02] ah sorry [09:02] I meant that one [09:02] what do you have in /usr/lib/environment.d/990-snapd.conf [09:02] did you reboot after installing that update? [09:03] zyga: yup [09:03] PR snapd#5701 closed: image: add type to seed.yaml to allow sorting in firstboot [09:04] Any ideas how I can get rid of my "disabled,broken" lxd snap? https://paste.ubuntu.com/p/G2d9XV5Tx5/ [09:04] zyga: hmm https://bugs.archlinux.org/task/56716 uhh and https://bugs.archlinux.org/task/56683 and https://bugs.archlinux.org/task/47884 looks like PATH is overwritten on arch in /etc/profile [09:04] mmm, can you snap install lxd [09:04] and then remove all the containers [09:05] and then remove it again? [09:05] nope. it thinks its already installed. "snap "lxd" is already installed, see 'snap help refresh'" [09:05] mborzecki: uh [09:05] Gargoyle: and snap remove lxd? [09:05] mborzecki: I'm happy systemd helps to clean up this mes [09:05] mess [09:06] zyga: That's in that pastebin. Complaining about readonly filesystem [09:06] Gargoyle: can you reboot again, I suspect some things are mounted there; you can try to unmount them or just reboot (maybe, no promises) [09:07] I'll give it a whirl, but it;s been broken for the last few reboots already [09:07] ah [09:07] then wait [09:08] check what is mounted [09:09] Maybe I am not seeing the wood for all the trees:- https://paste.ubuntu.com/p/D2j94chnMd/ [09:10] hmm [09:10] I don't see anything [09:10] what is in /var/snap/lxd [09:11] "common" [09:12] and inside? [09:12] it goes all the way along: /var/snap/lxd/common/lxd/storage-pools/default/images/de2ea148defc21d223376e53ab10ff1bccd003d92d5e6c6822398494ad934859 . No sign of a symlink anyhwere [09:12] and you cannot rm -rf that yourself, right? [09:13] nope. Not even as root [09:13] I think you may need to ask stgraber for help [09:13] maybe it's a zfs pool or something [09:13] I don't know exactly [09:13] opk [09:13] I need to jump into some code [09:13] Save me, stgraber. You're my only hope! :P [09:13] mvo: I commented on #5701, I still think we should do it in firstboot code [09:13] mborzecki: about environment.d [09:14] mborzecki: we don't set XDG_DATA_DIRS [09:14] PR #5701: image: add type to seed.yaml to allow sorting in firstboot [09:14] zyga: something fishy on arch, by the time it gets to /etc/profile the PATH does not contain any sign of snap [09:14] mborzecki: I think we should, right/ [09:15] zyga: yeah, i think so [09:15] I'll try to change that [09:16] pedronis: yeah, I noticed and already closed it. thanks for the quick feedback [09:18] PR snapd#5702 opened: snap: add ByType sorting [09:19] zyga: we need a snapd type, one issue is that we need a patch for it, i.e. if snapd is already installed we need to transition it [09:20] zyga: maybe we can get away without [09:20] mvo: reviewed [09:20] zyga: given that core18 is not stable yet and its not instalable anywhere else [09:20] mvo:why would we need a patch? [09:20] zyga: if you have snapd installed already [09:20] mvo: I mean, it's just snap info that would change [09:20] ah [09:20] I see [09:20] yeah [09:20] well [09:20] zyga: but yeah, I would love to add it [09:21] we could do in memory tweak [09:21] because it's clear it's special case [09:21] zyga: aha, nice idea, that would be a nice way out [09:21] and special casing on name or snap id as in other parts is clearly showing its cost [09:21] mvo: yeah, would not break rollback [09:21] so anyway [09:21] +1 but I wanted to voice that [09:21] zyga: ok, I will do it [09:21] * zyga hugs mvo [09:21] * mvo hugs zyga [09:22] let's add a type in one pr [09:22] and then tackle the issues like in-memory migration [09:22] and various uses [09:22] (so let's land this one as is) [09:22] zyga: yeah, I agree [09:23] zyga: I am in the middle of something else (firstboot ordering) but once that is done I will do the cleanup [09:23] * zyga nods [09:23] I'll jump into something too but please drag me back when you are into it [09:23] zyga: thanks, will do [09:24] zyga: can you post your /etc/profile? [09:25] probably vanilla /etc/profile from bionic https://www.irccloud.com/pastebin/cCczVjri/ [09:25] mborzecki: note that we also have /etc/profile.d/ [09:26] zyga: can you grep -n systemd-environment /etc/profile.d/* ? [09:26] empty [09:26] (I mean, I ran: grep -n systemd-environment /etc/profile.d/*) [09:27] unless you meant something more [09:27] zyga: no, that's fine, I have no clue how stuff from systemd-environment-d-generator gets to your session then [09:28] I rebooted [09:28] I think it's set by systemd when it starts the session proceses [09:28] is arch using systemd as a session manager?\ [09:28] brb, getting hungry over essentially no breakfast [09:29] zyga: yeah, the manpages are not too helpful either [09:31] https://www.irccloud.com/pastebin/IzURDL3V/ [09:31] zyga: ^ i suspect this is known? i've seen it some time ago; now failed on fedora [09:32] zyga: travis log - https://api.travis-ci.org/v3/job/418670092/log.txt [09:32] PR snapd#5702 closed: snap: add ByType sorting [09:35] Heyo [09:35] Chipaca: Moin [09:35] niemeyer: hi [09:35] niemeyer: ssh, I'm not here [09:35] Chipaca: Did the notes on the PR make the idea more clear, or do you still think we have an issue in terms of aptch sublevels [09:35] Chipaca: Oh, is that a day off? [09:36] niemeyer: I'm not sure it'll dtrt wrt reverts to a pre-sublevel snapd and back [09:36] Chipaca: I can't read ssh without reading ssh (but I guess you meant ssssssh) - anyway, enjoy your day off [09:36] mvo: i meant shh [09:37] Chipaca: Ah, I see it is, sorry.. I had understood the holidays were next week on, sorry for the force disturbance.. :) [09:37] niemeyer: no worries, I'm the one in the channel [09:37] mvo: I left a comment in 5702 about an open issue [09:38] though I saw you closed it [09:38] pedronis: yeah, I was thinking about zygas comment [09:38] pedronis: but maybe I just do that in a followup [09:38] Chipaca: Well, I'm always on the channel as well :) Anyway, the idea is that pre-sublevel patches are just like post sublevel patches.. the sublevel zero is the one that introduces backwards incompatible changes. So when a revert occurs, nothing is applied.. when the revert is undone to a more recent sublevel, the patches are reapplied [09:38] Bye Chipaca :) [09:38] pedronis: sorry for the flip-flop, I think I will reopen and just do a followup with the snapd type [09:39] Anyway, the point is that I think we're good, unless I'm missing something [09:39] We just need to be careful with non-zero sublevels as they need to be idempotent [09:40] PR snapd#5702 opened: snap: add ByType sorting [09:41] niemeyer: how do we know we reverted, sublevel unaware snapds will not touch patch-sublevel [09:41] pedronis: The behavior across levels is still the same [09:42] pedronis: 6 => 5 will blow up still [09:42] zyga: mvo: fwiw the environment.d generator does not seem to have effect on fedora either [09:42] niemeyer: yes, but we cannot afford that, wasn't it the point of doing this [09:42] pedronis: As will 7 => 6.X [09:43] pedronis: No.. the point is that we can patch as 6.1, and have 6.0 untouched [09:43] pedronis: Hmm.. I see.. 6.0 won't actually mark it back as 6.0, so we won't see 6.1 reapplied.. hmm [09:43] yes [09:43] the issue is the preexisting level 6 snapd out there already [09:44] (sorry I thought we were at 5 not 6 atm) [09:45] current level 6 snapd doesn't know about sublevels [09:46] Yeah.. that part is sort of okay.. the problem is how to detect when 6.1 is reinstalled that it needs to reapply.. :/ [09:46] yes === sparkieg` is now known as sparkiegeek [09:47] pedronis: Only choice I see is special casing 6.0 [09:48] Well, the whole 6 series really.. we'll never know whether a revert to a 6.0 happened [09:49] Oh.. or will we [09:49] or do something like what chipaca suggested detect we are reverting to a pre sublevel core/snapd and reset sublevel before restarting [09:49] not super clear how to do the detecting though [09:50] We might be nastier than that and look at the history.. we have it in the state [09:50] Well, that sounds less nasty actually, in terms of proper isolation [09:51] you mean look at what was the previous revision? [09:52] Right.. we have some good data available [09:54] We have the history, and we have a refresh timestamp [09:56] If the state level is 6, we can easily tell whether the refresh time was touched since the last restart, and if so whether we've been in a sublevel-blind revision recently [09:56] All of that just looking at the state, which we have at hand [09:57] pstolowski: ^ [09:58] The good thing is that we only need to kick that logic when the state level is 6 [09:59] reading [10:03] niemeyer: right.. i understand the problem of reverting back to sublevel-blind 6.0... not yet clear on the details of proposed logic, will need to check the code [10:03] btw, thanks for the review [10:03] (i'm on it atm) [10:07] pedronis: can you take a look at https://github.com/snapcore/snapd/pull/5700 when you have a moment? [10:07] PR #5700: tests: significantly reduce execution time for managers test