jameshmborzecki: hi.  I saw in a bug report that you'd been looking at the GNOME 3.38 Wayland incompatibility.  I've filed https://gitlab.gnome.org/GNOME/mutter/-/issues/1454 upstream to see if we can get things fixed there.  We could presumably do something on the snapd side too though.05:32
mborzeckijamesh: thanks, so the problem is mutter dropped the abstract socket for Xwayland completely?05:33
jameshmborzecki: yes.05:33
mborzeckijamesh: heh, tough luck, since it's in fedora 33 now, and probably other distros that ship with 3.3805:34
jameshlike Ubuntu 20.1005:34
mborzeckioops, school run, back in a bit05:34
jamesh(although we don't default to a Wayland session)05:34
mborzeckijamesh: otoh, at least the yuzu snap that was tried in https://bugzilla.redhat.com/show_bug.cgi?id=1883621 should have picked up wayland socket?06:21
jameshmborzecki: based on the output, it's skipping Qt's Wayland backend and using X1106:23
jameshthe rest of the message show it using the XCB backend, which is X1106:24
mborzeckijamesh: heh, so I tried setting QT_QPA_PLATFORM=wayland and it seems to have no effect06:28
jameshthe snap may not ship the Wayland plugin06:28
mborzeckijamesh: there's this message `Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, xcb.` but it's not clear whether those plugins were found, or whether that's some general list06:29
jameshmborzecki: presumably it has its plugins in $SNAP/usr/lib/x86_64-linux-gnu/qt5/plugins/platforms or something similar06:31
mborzeckijamesh: fwiw /usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/libqwayland-generic.so exists in the snap06:31
jameshmborzecki: this is kind of beside the point though: yes, any of those individual snaps could work if they were ported to Wayland.  No, that wouldn't solve the underlying problem06:34
jameshmborzecki: I'd like to see this fixed upstream, but there's likely to be systems hanging around for a long time with broken X11 access06:35
mborzeckijamesh: hm wodnering whether there's anything in snapd we could do besides printing warning06:36
jameshmborzecki: bind mount host system /tmp/.X11-unix into the sandbox06:36
jameshwhich kind of breaks our "snaps can do anything with they private /tmp" promise06:36
mborzeckiyeah, but maybe it's a legitimate exception06:37
mborzeckiaffected systems are those on recent enough mutter, so maybe not that many yet and we have a bit of time left06:38
jameshPerhaps tie it to the interfaces that grant x11 access06:38
jameshIt's recent enough Mutter _and_ using Wayland06:38
jameshI'm not sure if Fedora is Wayland by default yet06:38
mborzeckiand there's gnome 38.1 coming too06:38
mborzeckiiirc it is, and hopefully f33 doesn't ship until gnome 38.1?06:39
mborzeckireminds me i should update fedora packages to 2.4706:40
mborzeckijamesh: i'll ping zyga when he's around, we can always propose a fix on the snapd side too and then discuss with pedronis/mvo whether it should land06:46
jameshsounds good06:46
mborzeckijamesh: hopefully it's fairly simple tweak in x11/wayland interfaces mount profiles06:47
jameshmborzecki: I wouldn't think any change would be necessary in the wayland interface?06:47
mborzeckijamesh: don't remember which interface assumes the 'handling' of Xwayland06:49
jameshmborzecki: It shouldn't care if it is talking to Xorg on bare metal or Xwayland06:50
jameshIf you're running a snapped Xwayland, you'd need an x11 slot for other snaps to connect to06:51
jamesh(of course, this is where things get complicated: if it isn't the implicit :x11 slot, what directory do you bind mount on the plug side?06:52
mborzeckipstolowski: hey06:58
mborzeckineed to take some paperwork to my accountant, i'll be online in a bit06:59
* dot-tobias says morning07:33
pedroniszyga: hi, is there something I should re-review or review for you?08:27
zygagood morning08:35
zygapedronis not yet, I will finish one or maybe both older branches08:35
zygapedronis they are close but not yet08:35
zygamborzecki, jamesh: hey guys08:35
jameshhi zyga08:35
zygaare you talking about the xwayland change in mutter recently?08:36
zygajamesh: is the bind mounted thing a socket?08:37
zygawe need apparmor permission for the socket original bind path08:37
zygaas bind mounting doesn't change that08:38
jameshzyga: yeah.  Mutter/gnome-shell 3.38 no longer creates the abstract "\0/tmp/.X11-unix/Xn" socket08:38
zygaI read the bug thread in the morning08:38
zyga(I had some calls and mandatory walks)08:39
zygais the abstract socket path a default value of some environment variable?08:39
zyganot abstract08:39
zygacould we set an environment variable and bind mount the non-abstract socket somewhere?08:39
zygaI would prefer not to bind mount it to /tmp08:39
zygathough not strongly, just feels dirty / old design08:40
zygasockets go to run nowadays08:40
mborzeckizyga: hm yeah, good question, maybe we could overwrite the location somehow08:40
zygaworth checking what the libraries do08:40
mborzeckizyga: and if it was run, it should work out of the box without dirty tricks08:40
zygaI have some patches to finish from last night but it's something we could do for the next release for sure08:40
mborzecki(sans some apparmor policy tweaks)08:40
zygamborzecki yeah08:40
jameshzyga: if an app is run with e.g. DISPLAY=:0, it will attempt to connect to "/tmp/.X11-unix/X0" as an abstract namespace and regular socket (not sure which order it tries)08:42
jameshDISPLAY=:1 would try .../X1, etc.08:42
mborzeckijamesh: right, the question is whether we can cantrol the /tmp/.. prefix with some env variable08:43
mborzeckior maybe it's a well known convention and here's no way around it08:43
jameshXorg and Xwayland listen on both by default.  In the new Mutter, they're attempting to start Xwayland lazily: open the sockets, wait for someone to try and connect, then have Xwayland adopt the file descriptors08:44
jameshthey had this implemented with the two sockets, but it was only polling the abstract namespace socket.  This meant clients that connected to the regular socket before Xwayland started would hang08:45
jameshTo fix that, they just got rid of the abstract namespace socket08:45
mborzeckijamesh: if i understood zyga's idea correctly, we could try to bind /tmp/.X11-unix to some location under /run/<snap?>, and if possible set an env variable to redirect the app there, so as to avoid polluting /tmp inside the snap mount ns08:45
jameshThe bug I filed is proposing to revert that and just fix it to poll both sockets08:45
zygamborzecki yes08:46
zygajamesh right, I think that's totally fine as well08:46
jameshI don't think there is any environment variable to redirect it from /tmp: if there was, I'm sure desktops would have started using it long ago08:47
jameshso mounting elsewhere would require putting a symlink in /tmp08:48
pedronismborzecki: hi, are you working on a follow up to #9443 to move key saving?08:49
mborzeckipedronis: yes, that and a followup to #9440 to replace lazy bootloader lookup with ForGadget() when creating observers (though i should open that in a short bit since the code is ready)08:50
pedronismborzecki: thx08:51
pedronispstolowski: hi, are you blocked on me atm?08:53
jameshzyga: here's the code where it tries to open the display: https://gitlab.freedesktop.org/xorg/lib/libxcb/-/blob/master/src/xcb_util.c#L226 -- the socket path is hard coded.08:55
pstolowskipedronis: not really, but thanks for asking! do you intend to look at services PR (would be great), or are you ok if it lands with existing reviews? that will wait for 2.48 branch anyway, so not urgent, and not blocking08:55
pedronispstolowski: I'll try to look at it but not this week08:56
pedronispstolowski: I will try to re-review #9391 though08:56
pstolowskipedronis: i'm currently busy with the recent issues.. i think we need to think about undo of snap remove09:04
pstolowskiand yes, landing #9391 would be nice09:06
mborzeckipstolowski: need reviews for that?09:09
pstolowskimborzecki: it got +1, but is missing review from Samuele at this point. but every review is welcome :)09:11
pedronispstolowski: yes, I saw from the SU notes, that you are looking into some bugs :/09:21
zygajamesh unfortunate09:23
zygajamesh let's see what upstream response is09:23
mborzeckijamesh: thanks for finding that code09:29
jameshmborzecki: there's similar code in libxtrans: I'm not actually sure which one usually gets run for modern apps.  In both cases, the paths are hard coded into the library09:30
ijohnsonmorning folks09:33
ijohnson#9471 and #9472 are both really simple and need 2nd +1s09:33
mborzeckiijohnson: thanks, did 9471, and 72 now too09:36
mborzeckiijohnson: have you heard back from hellsworth about dhcp & nm maybe?09:37
zygaijohnson good morning!09:38
ijohnsonHey zyga and mborzecki09:41
ijohnsonmborzecki thanks! I'm talking with hellsworth later today re nm ran out of time yesterday09:41
mborzeckiijohnson: great, thanks, i suspect we'll need to copy /var/lib/dhcp too, but better to have a confirmation09:42
ijohnsonYeah indeed09:42
mborzeckiijohnson: fwiw it's already listed in writable-paths too09:42
ijohnsonyeah it's easy enough to add to that list later when we know it's needed, because we could need other things for example too09:48
ijohnsonpedronis: hi, do you want to wait until after the sprint to discuss 9458 and exposing a way to build a testkeys trusting snapd snap for CI only?09:54
* zyga reboots09:58
ijohnsonoh zyga I forgot to ask how is your thumb doing!?10:04
zygaijohnson still in one piece10:05
ijohnsonthat is a good thing10:05
zygajust a cut10:05
zygathe nail is the worst part10:05
zygait heals so slowly10:05
ijohnsonyeah that sounds really painful10:05
zygabrand new knife10:05
ijohnsonknife works good though it sounds!10:05
zygahaha, yeah10:05
ijohnsonso I assume the knife is okay too10:05
ijohnsonmy grandpa used to tell this story about this guy who worked on a farm wearing and cut his hand up while wearing gloves and he complained "my hand will grow back but this glove never will"10:06
ijohnsonwell presumably my grandpa still tells the story he's still around after all :-)10:07
zygaijohnson that's a gruesome story :)10:08
ijohnsonhaha yeah10:08
dot-tobiashi ijohnson 👋10:15
ijohnsonhey dot-tobias sorry I saw your message yesterday but didn't get a chance to finalize all the commands to debug your issue, will try to finish that after a little bit this morning10:16
dot-tobiasijohnson no worries, can image you're busy – appreciate any help whenever you have the time 😊10:17
ijohnsonyes of course, we hope to be releasing soon so issues like this are important to work through and solve :-)10:17
mborzeckitime to drive back home10:28
mborzeckiback online in a bit10:29
* zyga-x240 writes some tests10:29
pedronisijohnson: yes10:37
ijohnsonpedronis: ok10:37
pedronisijohnson: I mean, I looked at it, I have opionions, but this week is probably not a good time to discuss them10:52
ijohnsonthat's fine, I assumed as much10:52
ijohnsonzyga: +1 to 761410:54
zygaijohnson I spoke with mvo10:54
zygait should land after core 20 releases10:54
zygato stay on the safe side10:54
zygaI will add a label10:54
ijohnsonzyga: no problem, I guess it will live to see it's first birthday then10:54
zygaI've replied to the dprintf comment as well10:56
zygathere's a bit of function pointer thing coming up later, dprintf is used in v1 mode10:56
zygav2 uses a different implementation with essentially the same idea (add this major:minor thing)10:56
ijohnsonah ok, I wasn't sure why but if that machinery will be changed, just seems that dprintf is more generically useful than just for udev10:56
zygathat's true10:57
zygait's pretty neat actually10:57
zygaoh well, after core20 we can dust this off10:57
zygathank you for the review10:57
zygaI hope I didn't miss anything, this branch survived a few small changes to the original code10:57
ijohnsonyeah it all looks good to me, I look forward to the day we can get rid of the shell script based udev helper entirely :-)10:58
zygame too :)10:58
zygaactually there are some things we could do that would also fix the udev raciness problem10:58
zygabut that's not close yet10:58
ijohnsonmvo: pedronis: can we document resilience.vitality-hint for folks to use generically ? or is there more work that will come to that setting / feature that necessitates not mentioning it to the world yet11:01
mvoijohnson: I think documenting this is a good idea11:02
ijohnsonok, I will try to write something maybe degville can take a look and integrate into the relevant area11:02
* zyga coffee11:08
zygaperfect while spread runs :)11:10
ijohnsondegville: I just made edits directly to the system-options page, can you review my proposed addition? It's a little awkward to phrase/explain the feature in a way that doesn't just say "when you set this setting then we set OOMScoreAdjust to this thing"11:18
degvilleijohnson: yes, of course - thanks for making those updates.11:18
* ijohnson still thinks it's magic that changes to the forum get reflected on the snapcraft.io/docs page immediately11:19
* zyga puts on his canadian pullover11:33
zygathat time of the year11:33
ogra...when the lawn mowers magically turn into leaf blowers ...11:42
zygaat least they don't suck ;)11:56
zygacachio: https://github.com/snapcore/snapd/pull/947511:57
zygaijohnson: if you have a moment ^ that's fairly short and simple11:57
cachiozyga, nice, thanks, I'll take a look11:58
=== pedronis_ is now known as pedronis
pedronispstolowski: I reviewed #939112:13
pstolowskipedronis: ty12:20
zygabrb, see you at standup12:45
* zyga runs more tests and thinks of lunch14:20
zygait's getting late14:20
=== King_InuYasha is now known as Conan_Kudo
=== Conan_Kudo is now known as King_InuYasha
=== King_InuYasha is now known as Conan_Kudo
=== Conan_Kudo is now known as King_InuYasha
zyga-x240a little bit more and things should work15:37
* cachio lunch15:59
vidal72[m]why bind mounting /tmp/.unix-x11 would break "snaps can do anything with they private /tmp" promise?16:16
vidal72[m]they would still able to do anything except removing that dir but why someone would want to do that?16:18
vidal72[m]it's likely that nobody would even notice there is new dir there16:19
zygavidal72[m] I think because /tmp was empty and had no meaningful content16:21
zyganow it does not16:21
vidal72[m]yeah but why it matters?16:22
vidal72[m]from user/developer perspective?16:23
vidal72[m]it's the same like you when add any other subpath elsewhere which didn't existed but exist now16:24
vidal72[m]for app this should be invisible change16:26
zyga-x240vidal72[m]: I think it's not that, it's just that we try to "tame" the system by defining some rules (any rules are good really)16:44
zyga-x240vidal72[m]: I think that by having a swarm of exceptions we revert back to the original moving zoo16:44
zyga-x240personally I would prefer if the X11 socket was configurable16:45
zyga-x240or if the location was more sensible16:45
* zyga-x240 debugs more tests16:48
vidal72[m]but shouldn't the rules be backed by some reasoning? otherwise you end up with rules for the sake of rules which are disconnected for actual needs16:49
zyga-x240vidal72[m]: well, in this specific case, it's clear that /tmp is not a place for sockets16:52
zyga-x240it's not an accident we've been using /run for that for a long while now16:52
zyga-x240I realize X11 predates that16:52
zyga-x240vidal72[m]: just like we are evolving the stack to support containerized apps elsewhere, I don't see why changing the location of the X11 socket, over time, is out of the question16:53
vidal72[m]I think the expectation is x11 will die over time which is why such fundamental changes are unlikely16:56
zyga-x240vidal72[m]: well, I think that X11 will be with my grandchildren16:56
zyga-x240and that changing a socket path _location_ to include additional one is fairly trivial16:57
zyga-x240even if that has to be made across a few libraries16:57
zyga-x240that doesn't mean I think it should be done16:57
zyga-x240but I do think we should not size it as something it is not16:57
vidal72[m]even if it's done then it's not solution for urgent snap issue which occurs right now16:58
zyga-x240I think the current issue is easier to solve by reverting the change16:59
vidal72[m]reverting the change break flatpak16:59
vidal72[m]which is why it was done in first place16:59
zyga-x240I'm not an expert on those libraries but I believe it's possible to handle both locations, is it not?16:59
zyga-x240perhaps jamesh should discuss the details17:00
vidal72[m]and gnome devs are quite close to flatpak17:00
zyga-x240I'm sure all changes are done in good faith17:02
vidal72[m]I think gnome devs weren't aware that it affects snap but if they have to chose to support only one thing then...17:03
zyga-x240I really doubt in the software world there's no way to solve this17:03
zyga-x240I'm more surprised it's a change now17:04
vidal72[m]beside that snap requiring abstract socket is problem on its own and you didn't saw problem with fixing it on snap side back then https://forum.snapcraft.io/t/xorg-abstract-socket-is-mandatory-for-running-snaps/4580/917:04
zyga-x240wasn't it a long-standing location17:04
zyga-x240or did it suddenly break now due to other changes?17:04
zyga-x240vidal72[m]: my opinion there was pure pragmatism, we can still bind mount to /tmp17:05
zyga-x240vidal72[m]: but I really would prefer not to, at least without considering the alternatives or a bigger framework that could track such hacks over time17:05
vidal72[m]it stopped working because gnome mutter disabled support for using xorg abstract socket for xwayland, they weren't aware someone is relying on it17:09
zyga-x240vidal72[m]: right, but was it a problem for flatpak just now or was this always broken in some way?17:10
zyga-x240you said it was done to fix flatpak somehow17:10
vidal72[m]always broken17:10
zyga-x240Guess not that many people run flatpak and wayland then17:11
zyga-x240perhaps by 2022 wayland is complete enough to switch over, I would really like that17:11
zyga-x240for 2020 the usability issues were too grave, IIRC17:11
ograwell, your prob is Xwayland, not wayland after all 😉17:12
vidal72[m]yes it;s xwayland + on-dmand launch gnome mutter feature which I think was experimental until recently17:12
ogra(i'm pretty sure native wayland apps will just run fine ... its just that there are so few still 🙂 )17:13
vidal72[m]it was long unfixed until gnome core dev hitted it :)17:13
zyga-x240very unusual ;-) (just kidding)17:13
zyga-x240I mean, it's just a bug17:13
zyga-x240people with the right experience will look at it17:14
zyga-x240from snapd point of view, we can always add hacks17:14
zyga-x240my entire point is that I'd love to have structure to those17:14
ograjames and daniel are already deeply in it it seems17:14
zyga-x240I would love some tea with milk17:14
zyga-x240I think so, the problem is in good hands17:14
vidal72[m]I would love to see this "hack" in snap as my system has x11 abstract socket disabled for increasing security17:18
zyga-x240vidal72[m]: perhaps I'm mistaken but I believe wayland is pretty much insecure until the entire transition is complete, including input, which is still years away17:19
zyga-x240so while good, the security situation is not terribly improved yet17:19
ograpeople should just use mir ....17:20
* zyga-x240 makes small progress on improving snapd test suite17:21
vidal72[m]zyga-x240: I don't use wayland at all, I mean I disabled xorg abstract socket17:21
zyga-x240vidal72[m]: oh, I misunderstood then, how is that better?17:21
vidal72[m]because restricting access to unix socket (which is just a file) is much easier that restricting access to abstract socket which needs new network namespace17:23
zyga-x240vidal72[m]: I see, that's true indeed17:23
zyga-x240in snapd, with apparmor, we can restrict both, I believe, but that applies to snap apps17:24
cachiozyga-x240, when you have some time, could you please take another look to #942517:54
jdstrandzyga-x240: we can do both provided we have the af_unix patch. without that, only named sockets (not abstract or anonymous)17:55
zyga-x240I see17:56
zyga-x240cachio: ^18:01
zyga-x240cachio: could you please look at https://github.com/snapcore/snapd/pull/947818:05
jdstrandzyga-x240: thanks for choosing 14 instead of 60 days for forcing a refresh. please also see https://forum.snapcraft.io/t/wip-refresh-app-awareness/10736/4418:07
* jdstrand meant to comment on that sooner18:07
zyga-x240jdstrand: looking18:07
zyga-x240thank you, replied!18:09
zyga-x240jdstrand: I think that the best outcome is that this gathers interest of app developers and we can consolidate snapctl APIs into popular snaps, such as chrome, so that they can provide a more complete experience18:10
jdstrandzyga-x240: thanks! iirc, isn't the current feature no longer experimental in 2.47? does it make sense to mention that there as well? (since stable is still 2.46)?18:11
zyga-x240jdstrand: it will be experimental for at least one more release18:11
zyga-x240jdstrand: (after this one)18:11
zyga-x240jdstrand: we need --ignore-running switch to snap refresh, among others18:11
jdstrandzyga-x240: yes. snaps ideally would be involved because they know best what should happen and how to respond18:11
zyga-x240jdstrand: we need some selinux changes for the monitoring being on by default18:11
zyga-x240jdstrand: my thoughts exactly18:11
zyga-x240jdstrand: so it's not something we will enable yet but I hope that it can be done soon, now that core20 seemingly is close to release, and more review time is available18:12
* jdstrand nods18:12
jdstrandI mean, there is always going to be cases where the snap publisher isn't the upstream and they don't really have the capacity to integrate with snapctl, etc as fully as perhaps is desired, so notifications will always be helpful (and a nice fallback for those snaps that do try to do something, but are buggy/etc)18:14
zyga-x240jdstrand: I was hoping that with the help of the desktop team18:14
zyga-x240jdstrand: we could have an action on the notification18:14
* jdstrand nods18:15
zyga-x240jdstrand: and interacting with the action would send X11/wayland message asking the application to close gracefully18:15
zyga-x240jdstrand: (which would be done without any privilege escalation, locally in the session)18:15
zyga-x240jdstrand: but that has no guarantee to close the app, simply to ask it18:15
jdstrandthat could be nice and would be a great improvement18:15
zyga-x240I had a look at that and had a depression18:15
zyga-x240two decades ago I bought a windows petzold book on how winapi looks like18:16
zyga-x240and that had such message in first lesson18:16
zyga-x240X11 does not18:16
zyga-x240so ;)18:16
zyga-x240well, I'm sure the desktop team is more knowledgeable than I am18:16
zyga-x240and that something can be arranged18:16
zyga-x240(i was especially fond of recalling how windows coalesces messages that invalidate each other, before those are delivered to the application)18:17
zyga-x240cachio: one more but not priority: https://github.com/snapcore/snapd/pull/947918:18
zyga-x240and this is really early code18:18
jdstrandI suspect you'd need shell help there rather than just taking it on. X11 should give you everything you'd need to do something (perhaps drastic), but I suspect there would be obstacles doing that on wayland. perhaps there are f.d.o apis for that...18:19
cachiozyga-x240, nice18:19
cachioI'll check that later18:19
cachioI need to go to the kinesiologist now18:19
zyga-x240kenvandine: ^ do you know if x11/wayland have a way to gracefully ask an application to quit, assume all we know is the PID18:20
zyga-x240cachio: that's allright, it's really early code18:20
zyga-x240kenvandine: something that might trigger an app to ask the user to save a document or confirm18:20
* cachio afk -> kinesiologist18:20
kenvandinezyga-x240: yeah, there is18:21
kenvandinei don't recall what off hand though18:21
zyga-x240kenvandine: \o/18:21
zyga-x240kenvandine: if you recall I'd love to know18:21
zyga-x240I could really use that18:21
kenvandinebut when the user logs out the session triggers all the apps to block until saving18:21
zyga-x240that's perfec18:21
zyga-x240*perfect, I'll try to integrate that18:21
kenvandineit used to be triggered by the gnome-session package18:22
zyga-x240I got lost in various standards and arcane specification18:22
kenvandinebut i am not sure now how it works18:22
kenvandineprobably some glib API18:22
jdstrandkenvandine: fyi, the libreoffice lzo change was merged in the review-tools. emi is working to get that in a store pull. we could do manual reviews for a while if needed18:23
kenvandinehellsworth was waiting on that18:23
kenvandinebut i'll tell her to go for it18:24
jdstrandkenvandine: oh, is she now taking care of libreoffice?18:24
kenvandinezyga-x240: gnome-session-quit does still seem to support it18:26
jdstrandkenvandine: hat's off to her18:26
kenvandineso look at what it does18:26
kenvandinejdstrand: indeed :)18:26
kenvandinejdstrand: she's also maintaining network-manager, tough18:26
zyga-x240apt-get source $(dpkg -S $(command -v gnome-session-quit))18:26
jdstrandambitious. awesome :)18:26
zyga-x240I'm sad to report that using "devel" instead of groovy does not work for source packages18:28
* zyga-x240 spawns a test and takes a break18:31
zyga-x240back in 3018:31
zyga-x240that was not 30 :)19:40
mwhudsonhow do i see the base of a snap?19:57
mwhudsonah info --verbose19:59
zyga-x240one more iteration20:23
zyga-x240mwhudson: note that there are some weird edge cases20:23
zyga-x240like base: core16 being supplied by core20:23
zyga-x240not sure if that's handled there20:24
mwhudsonzyga-x240: ah i don't really mind about taht20:24
zyga-x240yeah, I think core16 never really happened20:24
=== benfrancis5 is now known as benfrancis
rogpeppei don't know if anyone's around, but i've got an odd problem with a raspberry pi running ubuntu core. it's been running for ages without much more issue than the occasional reboot, but just recently (after a power cut), it came back up but its service didn't come up20:46
rogpeppei've just investigated and there's no snap command any more!20:46
rogpeppethe file /usr/bin/snap exists but that's a symlink to /snap/snapd/current and that doesn't exist20:47
zyga-x240rogpeppe: is that a core18 system?20:47
rogpeppeuname -a prints : Linux localhost 4.15.0-1041-raspi2 #44-Ubuntu SMP PREEMPT Wed Jul 3 15:47:01 UTC 2019 armv7l armv7l armv7l GNU/Linux20:48
rogpeppezyga-x240: i've no idea what that means :)20:48
zyga-x240rogpeppe: ok, sorry,20:48
zyga-x240it's kind of late so my questions may be silly20:48
zyga-x240did any snaps fail mounting?20:48
rogpeppezyga-x240: np. your interaction is appreciated!20:48
rogpeppezyga-x240: how can i tell?20:48
zyga-x240systemctl --failed20:49
zyga-x240that may show what's wrong20:49
* zyga-x240 -> shower20:49
zyga-x240rogpeppe: best to file a bug with the details you can collect20:49
zyga-x240I was about to go to sleep20:49
rogpeppeyup, looks like my service failed:20:49
rogpepperogpeppe@localhost:/snap/snapd$ systemctl --failed20:49
rogpeppe  UNIT                           LOAD   ACTIVE SUB    DESCRIPTION20:49
rogpeppe● snap.hydroctl.hydroserver.service loaded failed failed Service for snap application hydroctl.hydroserver20:49
rogpeppeLOAD   = Reflects whether the unit definition was properly loaded.20:49
rogpeppeACTIVE = The high-level unit activation state, i.e. generalization of SUB.20:49
rogpeppeSUB    = The low-level unit activation state, values depend on unit type.20:50
rogpeppe1 loaded units listed. Pass --all to see loaded but inactive units, too.20:50
rogpeppeTo show all installed unit files use 'systemctl list-unit-files'.20:50
zyga-x240is snapd.service running?20:50
zyga-x240ls /snap/snapd/current/20:50
zyga-x240is snapd mounted?20:50
rogpeppehow can i tell?20:50
zyga-x240that ls command should help20:50
rogpeppedf, i guess20:50
zyga-x240current is either a link to a mount point20:50
zyga-x240or a dangling symlink20:50
zyga-x240df works too20:51
rogpeppelooks like it might be: https://paste.ubuntu.com/p/McFB3WDSbP/20:51
rogpeppethe /snap/snapd/current file doesn't exist20:52
rogpeppethat directory just has four directories in with numeric names20:53
zyga-x240next step is to check the status of the mount units20:53
zyga-x240systemctl status snap-snapd-$number.mount20:53
zyga-x240the one that current points to is the most relevant20:53
rogpeppewith the numbers in that dir?20:53
rogpeppethere is no "current" :)20:53
zyga-x240is the current symlink entirely gone?20:54
zyga-x240oh my20:54
zyga-x240pick the largest number20:55
zyga-x240but I suspect something very wrong happened20:55
rogpeppeyes, the symlink is entirely gone20:55
zyga-x240so they are all mounted20:55
zyga-x240that's so weird20:55
zyga-x240can you collect journal logs (if you have persistence)20:55
zyga-x240if not that's futile20:55
zyga-x240I'd re-create the symlink to the largest number20:55
zyga-x240and restart the system to see if things get back to shape20:55
zyga-x240though no guarantees20:56
zyga-x240if that works, snap changes output would be useful20:56
rogpeppethere was a power-cut FYI20:56
zyga-x240I see20:56
zyga-x240well, something to learn each time20:56
rogpeppehow should i get journal logs?20:56
zyga-x240journalctl --list-boots should tell you if you have persistent logging20:56
zyga-x240if there's one entry you do not20:56
zyga-x240following that journalctl can be coerced to render all logs as text20:57
rogpeppelooks like persistence is enabled: https://paste.ubuntu.com/p/2tRSFgP7bB/20:58
rogpeppei can never remember how to use journalctl :) what's the right incantation to get logs out?20:59
zyga-x240rogpeppe: journalctl -b -1 should give you logs from the boot that died due to power cut20:59
zyga-x240please collect logs, I think we can learn a lot from this failure20:59
zyga-x240and to make the system more robust20:59
zyga-x240but to do that, we need to give this problem proper resources (awake developers)20:59
zyga-x240if you re-create current and reboot, that's useful to know as well21:00
rogpeppeunfortunately the machine is a few hundred miles away from me, so i might have to wait before rebooting21:01
zyga-x240Oct 05 12:59:14 localhost kernel: FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.21:01
zyga-x240I see21:01
zyga-x240you don't have to reboot21:01
zyga-x240I'm just tired and I don't remember the right units to start21:02
zyga-x240you may want to fsck that21:02
rogpeppeah, i'll try fsck. hopefully it'll be ok to run that live.21:03
zyga-x240rogpeppe: https://github.com/snapcore/core18/blob/master/static/usr/lib/core18/run-snapd-from-snap21:03
zyga-x240that's what we run on most systems with core18 + snapd snaps21:03
rogpeppeso after i've run fsck, i should try to run that?21:04
zyga-x240what is the status off this service? https://github.com/snapcore/core18/blob/master/static/lib/systemd/system/core18.start-snapd.service21:04
zyga-x240it's invoked from this systemd unit21:04
zyga-x240(that was the part I was not sure of)21:04
rogpeppefsck refuses to run with the filesystem still mounted21:04
zyga-x240rogpeppe: you should be able to umount that briefly, that's boot and it is not used now21:05
rogpeppefsck doesn't seem to like me:21:07
rogpepperoot@localhost:/snap/snapd# fsck /dev/mmcblk0p121:07
rogpeppefsck from util-linux 2.31.121:07
rogpeppewhat does it mean by "21:07
rogpepperoot@localhost:/snap/snapd# fsck /dev/mmcblk0p121:07
rogpeppefsck from util-linux 2.31.121:07
rogpeppewhat does it mean by "fsck from util-linux 2.31.1" ?21:08
zyga-x240not sure21:08
zyga-x240can you check the status of core18.start-snapd.service please21:08
zyga-x240last thing I'd like to know before going to sleep21:08
rogpeppesorry, how do i do that?21:09
zyga-x240systemctl status $unit_name21:09
zyga-x240systemctl status core18.start-snapd.service21:09
rogpeppeah, got it21:09
rogpeppeit's been too long21:10
zyga-x240no worries21:10
zyga-x240ls -ld /var/lib/snapd/state.json21:11
zyga-x240(it should not be empty)21:11
rogpeppe-rw------- 1 root root 43119 Oct  7 15:11 /var/lib/snapd/state.json21:11
zyga-x240can you try one more thing:21:13
zyga-x240please run: /snap/snapd/(pick largest number)/usr/bin/snap debug state /var/lib/snapd/state.json21:13
zyga-x240that will tell us what happened21:14
zyga-x240it looks like snapd disabled itself somehow21:14
rogpeppeq .21:16
rogpepperoot@localhost:/snap/snapd# /snap/snapd/9169/usr/bin/snap debug state /var/lib/snapd/state.json21:16
rogpeppeID   Status  Spawn  Ready  Label  Summary21:16
zyga-x240that's that? no output?21:16
zyga-x240is /var/lib/snapd/state.json empty/corrputed21:17
rogpeppei'd paste you the contents of state.json but it looks like it's got sensitive keys in it21:17
zyga-x240yeah, please dont21:17
rogpeppeit's valid json tho21:17
zyga-x240any nothing in /changes top-level key there?21:17
zyga-x240(inside the json that is)21:18
rogpeppethat's empty21:19
zyga-x240please report this21:19
zyga-x240I don't have a smoking gun at all21:19
zyga-x240we'd have to inspect the state of each snap (in the state)21:19
zyga-x240you could make a copy of state.json in case it changes, even on the device itself21:19
rogpeppewhere should i report it?21:20
rogpeppelol, launchpad :)21:21
rogpeppei've redacted everything that looks auth-related. i could paste the resulting json if you thought that would be useful.21:22
zyga-x240you can file a private bug21:23
zyga-x240and attach that to the bug report (works better for larger files)21:23
rogpeppehave you got a suggestion for what keywords would be good to use in the bug title?21:23
zyga-x240snapd lost current symlink, total system failure21:24
zyga-x240something to catch the eye21:24
zyga-x240we can change that later21:24
rogpeppecool, that's useful, thanks21:24
zyga-x240thank you!21:25
rogpeppezyga-x240: what other info would you suggest is useful to attach to the report?21:27
zyga-x240rogpeppe: hmm, maybe last time when you remember it working?21:28
rogpeppeok, will do21:34
rogpeppezyga-x240: https://bugs.launchpad.net/snapd/+bug/189893421:42
zyga-x240is state.json there sanitized?21:43
zyga-x240I've made the bug private just in case21:43
rogpeppezyga-x240: i will leave the system up and unchanged for a while, in case you want me to do any further investigation, but then i'll need to reinstall or something because this is running an important service (it's running my parents' house heating system!)21:43
rogpeppezyga-x240: yes it is sanitized21:43
zyga-x240rogpeppe: let's chat tomorrow, I think we can recover it21:43
zyga-x240(remotely, without rebooting)21:43
rogpeppezyga-x240: i can reboot if needed - my dad can reboot21:44
rogpeppezyga-x240: one other odd thing: when the system crashes (which is does relatively regularly), it always takes two tries before it comes up properly21:44
zyga-x240right, I remember that21:45
rogpeppezyga-x240: that's almost certainly entirely unrelated though21:45
zyga-x240really not sure why21:45
rogpeppezyga-x240: i'm still intending to try running raspbian for a bit and see if it helps. i don't know if the issues are with ubuntu core or with the hardware itself21:46
* zyga-x240 pushed one more bit to https://github.com/snapcore/snapd/pull/8573 and goes to bed21:46
=== diddledan5 is now known as diddledan

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!