[05:15] <mborzecki> morning
[05:25] <zyga> Good morning
[05:26] <zyga> I’m doing my school run now. See you all later
[06:08] <zyga> back now
[06:08] <zyga> mborzecki: it's cold today
[06:08] <zyga> 13C at most
[06:08] <zyga> brrr
[06:08] <zyga> I hope it won't rain later
[06:08] <mborzecki> and rainy here
[06:09] <zyga> mborzecki: still, all the cars are stuck in traffic
[06:09] <zyga> biking to school is way more robust
[06:09] <zyga> how are we doing today?
[06:09] <zyga> tests were awful yesterday
[06:09] <zyga> failing left and right on random stuff
[06:09] <zyga> store, portals, you name it
[06:14] <zyga> good morning mvo
[06:15] <zyga> a little cold and rainy today :)
[06:15] <zyga> how was your Monday?
[06:19] <zyga> quick breakfast
[06:19] <mvo> hey zyga
[06:21] <mborzecki> mvo:  hey
[06:21] <mvo> hey mborzecki
[06:25] <mup> PR snapd#7425 closed: channel: introduce Resolve and ResolveLocked <Created by pedronis> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/7425>
[06:25] <mborzecki> zyga: can you take another look at #7412 ? looks like we could land it easily
[06:25] <mup> PR #7412: tests: run dbus-launch inside a systemd unit <Test Robustness> <Created by mvo5> <https://github.com/snapcore/snapd/pull/7412>
[06:25] <zyga> sure
[06:26] <mvo> looks like we can merge 7342 too?
[06:26] <mvo> and 7125 needs a second review (should be simple)
[06:27] <zyga> +1 on 7412
[06:27] <zyga> mvo, do you want me to mere or do you want to yourself?
[06:28] <zyga> mvo: +1 on 7342
[06:28] <mvo> I can do the merge, I just noticed it has no second +1
[06:28] <zyga> looking at 7125 now
[06:29] <mup> PR snapd#7412 closed: tests: run dbus-launch inside a systemd unit <Test Robustness> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7412>
[06:31] <mup> PR snapd#7342 closed: fixme: rename ubuntu*architectures to dpkg*architectures <Created by ardaguclu> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7342>
[06:42] <zyga> mvo: reviewed 7125, +1 but please check my comment there
[06:43] <mvo> zyga: thanks, looking now
[06:50] <zyga> thank you
[06:52] <zyga> mvo, mborzecki: I'd like to ask for a review of https://github.com/snapcore/snapd/pull/7435
[06:53] <mup> PR #7435: tests: explicitly restore after using LXD <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/7435>
[06:53] <zyga> it's a blocker for progress on https://github.com/snapcore/snapd/pull/7168
[06:53] <mup> PR #7168: tests: measure testbed for leaking mountinfo entries <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/7168>
[06:54] <zyga> I have one PR slot open so I'll work on finishing and proposing a mount-ns extension that involves a mimic, so that we can properly evaluate https://github.com/snapcore/snapd/pull/7436 later
[06:54] <mup> PR #7436: many: make per-snap mount namespace MS_SHARED <Bug> <Created by zyga> <https://github.com/snapcore/snapd/pull/7436>
[06:57] <zyga> mvo: offtopic, last night I was playing with raspberry pi
[06:57] <zyga> and I think we can slightly improve our watchdog story there
[06:57] <zyga> specifically around try mode boots
[06:59] <mvo> zyga: oh? tell me more
[07:00] <zyga> I read a little about the watchdog on the pi
[07:00] <zyga> it's a bit weird, has fixed 15 second interval
[07:00] <zyga> we could enable it from the boot loader
[07:00] <zyga> so any try mode boot can recover
[07:00] <zyga> I will poke around in free time over evenings
[07:00] <zyga> maybe I will reach something that works
[07:01] <abeato> hey, is anybody experiencing problems with the telegram snap? I've opened https://forum.snapcraft.io/t/telegram-snap-fails-to-start/13132
[07:02] <zyga> abeato: that's new to me
[07:02] <zyga> abeato: ls -ld /mnt ?
[07:03] <abeato> zyga, $ ls -ld /mnt
[07:03] <abeato> drwxr-xr-x 2 root root 4096 jul 19  2016 /mnt
[07:03] <zyga> ok, so regular directory, not a symlink
[07:03] <pstolowski> mornings
[07:05] <zyga> abeato: can you please check how many files matching "*snap-confine*" glob are present in /etc/apparmor.d/
[07:06] <abeato> zyga, https://paste.ubuntu.com/p/xt7NyYgPKr/
[07:06] <zyga> that's that!
[07:06] <zyga> mvo: ^^^^
[07:06] <abeato> sep 11?
[07:06] <zyga> abeato: one of the files is wrong
[07:06] <zyga> it's a bug in our postinst script I believe
[07:06] <zyga> mvo: should that be fixed and released?
[07:06] <abeato> hm, interesting
[07:06] <zyga> abeato: what does "apt-cache policy snapd" say
[07:06] <zyga> that will be most useful to mvo
[07:07] <abeato> zyga, https://paste.ubuntu.com/p/G2JMwHJSJG/
[07:08] <zyga> abeato: the fix for this bug is in 61cc58dbb0a7a1a785e9e3c391b6f593df892839
[07:08] <zyga> Date:   Wed Aug 14 09:43:41 2019 +0200
[07:08] <zyga> it may not be released yet, perhaps
[07:08] <zyga> mvo: ^ can you confirm if 2.40 has this
[07:09] <mvo> abeato: yeah, the /etc/apparmor.d/usr.lib.snapd.snap-confine should not be there :(
[07:09] <mvo> zyga: yeah 2.40 should fix it
[07:09] <zyga> if you want to fix your system please remove the file mvo mentioned and call sudo apparmor_parser -r /etc/apparmor.d/usr.lib.snapd.snap-confine.real
[07:09] <abeato> mvo, zyga snapd journal: https://paste.ubuntu.com/p/wHsRR4R2xD/
[07:09] <zyga> mvo: in that case the fix doesn't work
[07:09] <mvo> zyga: oh well
[07:09] <mvo> zyga: let me look at this again
[07:09] <zyga> thank you!
[07:10] <abeato> do you need more data?
[07:10] <zyga> abeato: can you please update the forum thread with the log from this conversation?
[07:10] <mborzecki> zyga: can you take a quick look at https://github.com/snapcore/snapd/pull/7109 ? pushed some changes there yday
[07:10] <mup> PR #7109: snap-confine: fallback gracefully on a cgroup v2 only system <Created by mvo5> <https://github.com/snapcore/snapd/pull/7109>
[07:10] <mvo> abeato: a bugreport (super small, just the data you already pasted)
[07:10] <abeato> zyga, sure
[07:10] <mvo> abeato: then I will do a sledgehammer fix
[07:10] <zyga> abeato: thank you
[07:10] <zyga> mborzecki: sure, looking now
[07:10] <abeato> mvo, launchpad?
[07:10] <mvo> abeato: plus the content of the /etc/apparmor.d/usr.lib.snapd.snap-confine please
[07:10] <mvo> abeato: yeah
[07:10] <abeato> ok
[07:10] <mvo> abeato: or if its alreaady in the forum thats fine
[07:11] <mvo> abeato: just need a refrence in the PR
[07:11] <abeato> it is already in the forum, yes
[07:11] <abeato> I will update then  the forum post
[07:11] <mvo> abeato: then that should be fine
[07:11] <mvo> abeato: thank you!
[07:11] <abeato> np
[07:12] <mborzecki> anyone else seen this google:debian-9-64:tests/main/snap-service-watchdog to fail recently?
[07:12] <mup> PR snapd#7438 opened: devicestate: add support for base->base remodel <Created by mvo5> <https://github.com/snapcore/snapd/pull/7438>
[07:13] <mborzecki> for soem reason the snap app gets SIGABRT https://paste.ubuntu.com/p/Z6k382RCrD/
[07:17] <abeato> mvo, zyga https://forum.snapcraft.io/t/telegram-snap-fails-to-start/13132 updated
[07:18] <mborzecki> this one is interesting https://paste.ubuntu.com/p/kmGzQzZRHJ/ probably something for Chipaca or pedronis
[07:18] <zyga> store woes?
[07:24] <mborzecki> idk, nonce is logged in the POST request, so it is sent ;)
[07:27] <mvo> abeato: could you please pastebin me /var/lib/dpkg/info/ubuntu-core-launcher.conffiles
[07:27] <mvo> abeato: and snapd.conffile too ?
[07:28] <abeato> mvo, there is nothing starting as ubuntu-core* in /var/lib/dpkg/info/
[07:29] <mvo> abeato: uh, sorry, please see if there is "snap-confine.conffiles"
[07:29] <zyga> mvo: are conffiles retained after a package is removed?
[07:30] <zyga> mvo: that is, they remain until purged?
[07:30] <mvo> zyga: yes
[07:30] <mvo> zyga: correct
[07:30] <abeato> mvo, that file is not there either
[07:30] <abeato> mvo, snapd.conffiles exists
[07:31] <abeato> mvo, https://paste.ubuntu.com/p/pTN8P7Mtt2/ - but note that I already removed the old file and run apparmor_parser to fix the problem
[07:31] <mvo> abeato: any output from grep usr.lib.snapd.snap-confine /var/lib/dpkg/info/*.conffiles
[07:31] <mvo>  ?
[07:31] <abeato> $ grep usr.lib.snapd.snap-confine /var/lib/dpkg/info/*.conffiles
[07:32] <mvo> abeato: yeah, thats fine - I'm mostly trying to figure out if its still leftover in some dpkg files
[07:32] <abeato>  /var/lib/dpkg/info/snapd.conffiles:/etc/apparmor.d/usr.lib.snapd.snap-confine.real
[07:32] <mvo> abeato: thats the only match?
[07:32] <abeato> mvo, yes
[07:33] <mvo> abeato: thanks! I'm slightly puzzled but thats fine, I think I know what to do (even though I'm not sure how this happens, i.e. dpkg should either know about the file or it should be gone :/
[07:33] <abeato> right, it's weird...
[07:39] <zyga> abeato: did you develop snapd on this machine?
[07:39] <zyga> perhaps it came from some earlier hacking
[07:39] <abeato> zyga, no
[07:39] <zyga> mvo: I need to take a break, back-pain after last evening's longer session
[07:39] <zyga> I'll stretch and be back in a few moments
[07:40] <mvo> zyga: sure thing, get well!
[07:43] <mup> PR snapd#7439 opened: packaging: remove obsolete usr.lib.snapd.snap-confine in postinst <Created by mvo5> <https://github.com/snapcore/snapd/pull/7439>
[07:44] <mvo> abeato: -^
[07:45] <abeato> mvo, great!
[08:16] <mborzecki> zyga: https://forum.snapcraft.io/t/significance-of-info-files-in-run-snapd-ns/12938
[08:17] <mborzecki> tests are red again :/
[08:19] <mborzecki> pedronis: hi, i thnk you weren't around when i liked it, interesting failure i stumbled upon today https://paste.ubuntu.com/p/kmGzQzZRHJ/
[08:20] <mborzecki> s/liked/linked/
[08:21] <pedronis> mborzecki: weird error given that we see the nonce in the requests log
[08:21] <mborzecki> mhm
[08:23] <pedronis> for some reason the nonce the store just gave us is considered invalid
[08:23] <pedronis> unless if it's repeats worth poking the store people
[08:23] <pedronis> but we haven't touched anything in that area since a while
[08:28] <Chipaca> we do send the exact thing we get back
[08:28] <Chipaca> fwiw
[08:28] <pedronis> yes
[08:28] <pedronis> so it's not missing
[08:28] <Chipaca> this isn't the first time we've seen this error
[08:29] <Chipaca> i think it's worth chasing down
[08:29]  * Chipaca gets on it
[08:36] <mvo> mborzecki: anything changed that make the tests red?
[08:37] <mborzecki> mvo: no, looks like the usual stuff, desktop portal, occasionally installing snapd deps from package archive or store hiccups
[08:37] <mvo> :(
[08:46] <zyga> mborzecki: thank you, replied
[08:50] <mborzecki> snap/channel/channel_test.go:139:17: undefined: arch.UbuntuArchitecture on master
[08:51] <mborzecki> opening a pr in a bit
[08:52] <pedronis> mvo: I did a pass over the 2.42 PRs (except mine), they all need a little bit more work I fear
[08:52] <pedronis> mborzecki: crossing merges ?
[08:52] <mborzecki> pedronis: yes
[08:53] <mvo> pedronis: ok, I have a look, thank you!
[08:54] <pedronis> mvo: I'll try to tweak the test to check for systemd version in mine when I get a 2nd, worst case it will not make 2.42
[08:54] <mup> PR snapd#7440 opened: snap/channel: fix unit tests, UbuntuArchitecture -> DpkgArchitecture <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7440>
[08:55] <mborzecki> also something weird with tests/unit/go on debian, gofmt is not installed (?)
[08:57] <pedronis> Chipaca: also in #7411 I have two wonderings that maybe you can help with (I @chipaca-ed you on them)
[08:57] <mup> PR #7411: cmd/model: output tweaks, add'l tests <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7411>
[08:57] <Chipaca> pedronis: ack
[08:58] <Chipaca> pedronis: i noticed i didn't notice some of the things you pointed out
[08:58] <Chipaca> will look in a bit
[09:32] <pedronis> mvo: your PRs for 2.42 also need 2nd reviews
[09:57] <rogpeppe> zyga: i wonder if this might have some relevance: Sep 09 19:27:54 localhost snapd[692]: handlers.go:459: Reported install problem for "core18" as f732dba4-d35a-11e9-a660-fa163e6cac46 OOPSID
[09:57] <zyga> rogpeppe: it means that snapd fails to refresh core18
[09:57] <zyga> aborts the transaction
[09:57] <zyga> and rolls back
[09:57] <zyga> that explains your reboot loop
[09:57] <zyga> this is very useful information
[09:58] <rogpeppe> zyga: i'm not seeing a reboot loop currently FWIW
[09:58] <zyga> mvo: ^ can we pull the log from that error tracker entry?
[09:58] <zyga> rogpeppe: it will be re-attempted again
[09:58] <zyga> rogpeppe: until it refreshes successfully
[09:58] <zyga> that report has hints as to what went wrong
[09:58] <rogpeppe> zyga: ah, so that's what explains the fact that it's rebooting periodically, i guess
[09:58] <zyga> yes
[09:58] <zyga> it's the transactional nature
[09:58] <zyga> it's just not immune to problems
[09:59]  * zyga found a bug (in what he was doing since morning)
[09:59] <rogpeppe> zyga: here's another one: Sep 09 07:40:40 localhost snapd[715]: handlers.go:459: Reported install problem for "core18" as fc12d9b2-d2e7-11e9-b568-fa163e102db1 OOPSID
[10:01] <mvo> zyga: INFO Waiting for restart...
[10:01] <mvo> ERROR cannot finish core18 installation, there was a rollback across reboot
[10:01] <zyga> huh
[10:01] <zyga> interesting
[10:01] <zyga> nothing magic
[10:02] <zyga> rogpeppe: what does snap version say?
[10:02] <mvo> looks like the restart is not happening, I see a lot of Waiting for resart
[10:02] <mvo> snapd version is 2.40
[10:03] <mvo> zyga: https://pastebin.canonical.com/p/WHVwWjRTVs/
[10:03] <zyga> this is snapd + core18 arrangement, right?
[10:03] <rogpeppe> zyga:
[10:03] <rogpeppe> rogpeppe@localhost:~$ snap version
[10:03] <rogpeppe> snap    2.40
[10:03] <rogpeppe> snapd   2.40
[10:03] <rogpeppe> series  16
[10:03] <rogpeppe> kernel  4.15.0-1041-raspi2
[10:05] <zyga> rogpeppe: can you run snap changes
[10:05] <zyga> and pastebin that?
[10:06] <rogpeppe> zyga: http://paste.ubuntu.com/p/GbtJH9N6vv/
[10:06] <zyga> how about snap tasks 11
[10:06] <zyga> and snap tasks 13
[10:07] <rogpeppe> zyga: ?
[10:08] <zyga> run the command: snap tasks 11
[10:08] <zyga> and pastebin the output please
[10:08] <rogpeppe> zyga: http://paste.ubuntu.com/p/CnjtBWwrtr/
[10:09] <zyga> oh
[10:09] <zyga> Error   yesterday at 19:25 UTC  yesterday at 19:27 UTC  Automatically connect eligible plugs and slots of snap "core18"
[10:09] <zyga> it failed on auto-connect?!
[10:09] <mborzecki> off to school to pick up the kids, afk for a bit
[10:09] <zyga> pstolowski: ^ can auto-connect prevent a reboot?
[10:09] <rogpeppe> zyga: this is task 13: http://paste.ubuntu.com/p/SvvXP7rP5R/
[10:09] <zyga> same thing happened here
[10:10] <zyga> I think we're getting somewhere now
[10:16]  * Chipaca takes a break
[10:17] <pstolowski> zyga: as any other task handler that errors out and triggers undo. it's implemented to retry on conflicts (which it did a couple of times in that log), but then there is a bunch of things it looks up in the state that can error out. slightly weird we don't see what the error was for this task
[10:17] <zyga> pstolowski: we can presumably ask rogpeppe for the state file
[10:17] <zyga> do you think it would help
[10:18] <pstolowski> yes it might help, maybe we will be able to track down what changed in the state that made task error out
[10:20] <zyga> rogpeppe: can you please report a bug on bugs.launchpad.net/snapd
[10:20] <zyga> rogpeppe: include a rough description of the problem how you see it
[10:21] <pstolowski> rogpeppe: so yes, if you can grab and send us state.json that would be great (don't pastebin as it has your macaroon etc)
[10:21] <zyga> rogpeppe: and then work with pstolowski to attach the logs there
[10:21] <zyga> rogpeppe: and the state file
[10:21] <rogpeppe> pstolowski: where does that file live?
[10:21] <pstolowski> rogpeppe: /var/lib/snapd/
[10:21] <zyga> rogpeppe: please make sure to use a private bug (when reporting it) because as pawel said, the state file contains some shared secrets
[10:21] <zyga> let us know if you need any help with reporting the bug
[10:22] <zyga> we will use it for tracking and eventual regression testing
[10:22] <zyga> mborzecki: found a small-ish bug just now, /etc/ssl is a special case, as you know
[10:22] <zyga> mborzecki: as is /etc/alternatives
[10:22] <zyga> mborzecki: and they don't play nicely with the trespassing detector
[10:23] <rogpeppe> zyga: are the macaroons the only secret things in there?
[10:23] <zyga> yes
[10:24] <rogpeppe> zyga: ok, i've redacted them specifically.
[10:26] <pstolowski> rogpeppe: ty
[10:43] <rogpeppe> zyga, pstolowski: https://bugs.launchpad.net/snapd/+bug/1843417
[10:43] <mup> Bug #1843417: ubuntu core installation goes down regularly <snapd:New> <https://launchpad.net/bugs/1843417>
[10:43] <zyga> thank you very much
[10:43] <zyga> we'll try to get to the bottom of this
[10:43] <zyga> pstolowski: can we use the state file to simulate a refresh somehow?
[10:44] <rogpeppe> zyga: thanks. BTW if it can't be fixed fairly soon, i'll need to move to another distribution, because winter is coming and my parents need heat in the house :)
[10:44] <zyga> rogpeppe: can you try one specific command that may help you
[10:44] <zyga> rogpeppe: snap refresh core18
[10:44] <zyga> that will refresh the base snap
[10:44] <rogpeppe> zyga: ok, trying
[10:44] <zyga> then you can try to refresh snapd
[10:44] <zyga> essentially one-by one
[10:44] <zyga> one-by-one
[10:45] <zyga> rather than all at once
[10:45] <zyga> if there's some kind of conflict happening
[10:45] <zyga> it might be averted this way
[10:45] <rogpeppe> zyga: ok, it's refreshed and is rebooting
[10:45] <rogpeppe> zyga: well, in a minute
[10:46] <rogpeppe> zyga: ok, we'll see if it comes back up
[10:46] <zyga> fingers crossed
[10:46] <zyga> systems at scale is complex
[10:46] <zyga> we wish to make this totally unattended
[10:46] <zyga> but as reality shows, it's not trivial
[10:47] <pstolowski> zyga: in theory possible but lots of mocking
[10:48] <rogpeppe> zyga: i'm somewhat unhappy about the reboot just happening randomly, not under some sort of control, particularly if there's a possiblity that the system might not recover from it
[10:48] <rogpeppe> zyga: in this particular case, if this happened when people were away, it could result in the house not being heated enough and pipes freezing, leading to expensive damage
[10:49] <zyga> rogpeppe: you can schedule updates
[10:49] <zyga> there's a way to update predictably at very precise moments
[10:49] <rogpeppe> zyga: oh? how would i do that?
[10:49] <zyga> https://snapcraft.io/docs/keeping-snaps-up-to-date
[10:49] <zyga> in doubt ask mborzecki, he knows this code very well
[10:50] <zyga> or ask degville about the documentation for suggestions or improvements
[10:50] <pstolowski> i'm looking into the state
[10:50] <rogpeppe> zyga: is there some documentation for the actual refresh timer syntax that isn't just examples?
[10:51] <mborzecki> re
[10:51] <zyga> I don't believe there is, what would you like, a more formal syntax?
[10:51] <rogpeppe> zyga: yup
[10:51] <zyga> I believe it's a set of ranges
[10:51] <zyga> but mborzecki can correct me on this
[10:51] <rogpeppe> zyga: and an explanation of the semantics
[10:51] <zyga> mborzecki: is there a more formal syntax of https://snapcraft.io/docs/keeping-snaps-up-to-date
[10:51] <rogpeppe> zyga: with those docs, i'm left guessing
[10:51] <rogpeppe> zyga: what does the "5" in "fri5" mean, for example?
[10:52] <degville> rogpeppe: I think you're right - we should add a formal explanation of the syntax.
[10:52] <zyga> rogpeppe: the semantics is that snapd will only attempt to refresh in time that fits that schedule
[10:52] <rogpeppe> zyga: it's surely not the 5th friday in the month
[10:52] <mborzecki> rogpeppe: just examples
[10:52] <zyga> rogpeppe: it actually is
[10:52] <zyga> rogpeppe: you can plan a monthly refresh this way
[10:52] <rogpeppe> zyga: most months don't have a 5th friday
[10:52] <zyga> perhaps the example is not the best but that is the intent
[10:53] <zyga> mborzecki: what would happen if you pick the 5th Friday actually?
[10:53] <mborzecki> rogpeppe: exmples list: fri5,23:00-01:00  Last Friday of the month, from 23:00 to 1:00 the next day
[10:53] <mborzecki> rogpeppe: it's day[<weeknum>], 5 means the last week basically
[10:53] <rogpeppe> mborzecki: i don't understand why that's the last friday of the month - that's what i mean when i say that it would be good to actually document the semantics
[10:54] <rogpeppe> mborzecki: if the digit "5" is special, then it should say so
[10:54] <rogpeppe> mborzecki: for example, would fri9 mean the same thing?
[10:55] <mborzecki> rogpeppe: some syntax was initially described here https://forum.snapcraft.io/t/refresh-scheduling-on-specific-days-of-the-month/1239/6 some changes were made along the way though
[10:55] <rogpeppe> mborzecki: more unknowns: would you be allowed to do "22:00~23:00/2" ?
[10:55] <rogpeppe> mborzecki: if so, what's the difference between that and "22:00-23:00/2" ?
[10:56] <zyga> I think those are all good questions, thank you for engaging with us rogpeppe
[10:56] <rogpeppe> mborzecki: basically, i'd like to see some actual description of the semantics, not just a set of examples where i'm left to try to infer the actual rules
[10:57] <mborzecki> rogpeppe: there's this page too: https://forum.snapcraft.io/t/timer-string-format/6562
[10:58] <mborzecki> i gues it could use a little update with some details
[10:58] <mup> PR snapd#7441 opened: asserts,seed/seedwriter: follow snap type sorting in the model assertion snap listings <Created by pedronis> <https://github.com/snapcore/snapd/pull/7441>
[10:58] <rogpeppe> that design doc from niemeyer is a great start. it would be nice if some more of that made it into the actual docs.
[11:00] <rogpeppe> it also mentions some stuff that isn't documented at all, such as `0:00~24:00/6:00`
[11:01] <rogpeppe> but maybe that's not implemented, i guess
[11:02] <rogpeppe> zyga: FYI the pi did not successfully reboot.
[11:02] <rogpeppe> zyga: i'm gonna have to ask for it to be power cycled again
[11:02] <zyga> rogpeppe: did it reboot but fail or failed to reboot?
[11:03] <rogpeppe> zyga: i've got no way of knowing, i'm afraid
[11:03] <zyga> huh, I see :/
[11:03] <rogpeppe> zyga: it said it was going down, my ssh connection was terminated, and it hasn't come back up again
[11:03] <zyga> it has rebooted then
[11:03] <rogpeppe> zyga: well, it shut down :)
[11:03] <zyga> and after someone reboots it again it should roll back
[11:04] <rogpeppe> zyga: and then i'm back in the same position as before?
[11:04] <zyga> I was discussing a way to make that automatic in case of failure on the pi specifically
[11:04] <zyga> yes
[11:04] <rogpeppe> zyga: ok. :-(
[11:04] <zyga> rogpeppe: you can set the schedule to avoid refreshes while we try to understand the cause of the failure
[11:05] <rogpeppe> zyga: can i set the schedule so refreshes are turned off entirely?
[11:05] <zyga> no, that's explicitly not available
[11:05] <rogpeppe> zyga: AFAICS the minimum frequency is once per month
[11:05] <zyga> rogpeppe: you can try one more thing
[11:05] <zyga> you can refresh snapd itself
[11:05] <zyga> that should not reboot
[11:05] <zyga> but give you new software stack
[11:05] <zyga> so that some bugs that were fixed since 2.40 can be applied
[11:06] <zyga> well the fixes that is, not the bugs
[11:06] <zyga> rogpeppe: you can try to refresh snapd to candidate channel to get it
[11:06] <zyga> rogpeppe: with "snap refresh snapd --candidate"
[11:06] <zyga> on core18 systems you no longer need to reboot to get new snapd, fortunately
[11:07] <zyga> using the same strategy you could even refresh to a hotfix branch that contains a fix for your machine
[11:07] <zyga> which would allow you to refresh the rest of the system correctly, once we understand the nature of the failure
[11:09] <rogpeppe> zyga: ok, i'll try that when the system has been restarted, thanks
[11:12] <pstolowski> zyga: so, auto-connect handler errors out because WaitRestart() reports a rollback error, we hit the "// TODO: make sure this revision gets ignored for automatic refreshes" case again, there is a revision mismatch there. this was discussed a few months ago when we hit similiar case. pedronis also did some work around reboots recently but i'm not sure if that's in play here
[11:14] <pedronis> pstolowski: the checking for reboots has been added
[11:14] <pedronis> maybe we didn't remove all the TODOs
[11:14] <mborzecki> zyga: tbh, in situations like this, maybe we should have a mechanism to temporarily disable refreshes, some local assertion or whatnot
[11:14] <pstolowski> zyga: so auto-connect is just a vitim here, problem is elsewhere
[11:14] <zyga> mborzecki: yeah
[11:15] <pedronis> I don't remember when it landed though
[11:16] <rogpeppe> zyga: do you think that the failure to reboot correctly is related to this problem here, or just another problem that happens to be exacerbating the issue?
[11:16] <zyga> I think that it may be a separate problem
[11:17] <zyga> perhaps it'd be good to look at snap boot environment and see what it says
[11:17] <pedronis> mborzecki: you can set refresh.hold no
[11:17] <mborzecki> aahh
[11:17] <mborzecki> right
[11:18] <pedronis> a bit annoying to set, we really need a command for it, but is there
[11:18] <rogpeppe> zyga: ok, i'm back into the system
[11:19] <pedronis> pstolowski: we should drop that todo, the check is now done, is done much earlier, in daemon itself
[11:19] <rogpeppe> zyga: it's maybe interesting that it seems the system only reboots successfully after exactly two power cycles.
[11:19] <zyga> pedronis: refresh.hold? what is that
[11:20] <pedronis> zyga: https://forum.snapcraft.io/t/system-options/87#heading--refresh-hold
[11:20] <pedronis> it's on same page as timer
[11:21] <zyga> ah, it's not on https://snapcraft.io/docs/keeping-snaps-up-to-date
[11:21] <mborzecki> zyga: afaiu it's not intended to be used by the user
[11:22] <pedronis> well, it's annoying but it exists
[11:22] <pedronis> I would not recommend to use it without a reason
[11:22] <pedronis> but it can be used
[11:24] <rogpeppe> am i right that there's no way to specify that the system will refresh on the first day of every month?
[11:25] <zyga> I believe you're correct
[11:25] <pstolowski> pedronis: should the restart check be dropped from auto-connect handler?
[11:25] <rogpeppe> zyga: thanks
[11:25] <pedronis> pstolowski: in which sense?
[11:26] <pedronis> the answer is likely no
[11:26] <pstolowski> pedronis: removing WaitRestart() from auto-connect
[11:26] <pedronis> pstolowski: no
[11:26] <pedronis> it will be hit for a bit until we restart/reboot
[11:26] <pedronis> what I said is checked earlier if when a reboot is triggered
[11:26] <pedronis> whether it happeneed
[11:27] <pedronis> at some point we might be able to not use WaitRestart but that requires changes to taskrunner etc
[11:31] <pstolowski> pedronis: i see. ok, we need to find out what triggers this check to fail sometimes
[11:32] <pedronis> pstolowski: nowadays, if we reboot 3 times and fails it will trigger
[11:32] <pstolowski> i've a power outage, need to power off soon before my ups runs out of battery
[11:32] <pedronis> well, try to reboot 3 times
[11:37] <zyga> 2019 is still the year when I wish lauchpad to support markdown
[11:41]  * pstolowski lunch
[11:42] <zyga> mborzecki: I reported two bugs that I found today https://bugs.launchpad.net/snapd/+bug/1843421 and https://bugs.launchpad.net/snapd/+bug/1843423
[11:42] <mup> Bug #1843421: snap-update-ns doesn't know about the special property of /etc/ssl and /etc/alternatives <snapd:Confirmed> <https://launchpad.net/bugs/1843421>
[11:42] <mup> Bug #1843423: snap-update-ns fails to construct a layout in /etc/test-snapd/foo <snapd:Confirmed> <https://launchpad.net/bugs/1843423>
[11:43] <zyga> ondra: ^ some useful bugs for you
[11:43] <zyga> ogra: in case you run into something like that in the field
[11:44] <zyga> I meant ondra twice but I think ogra may run into things like that as well
[11:45] <mborzecki> zyga: btw. does s-u-n need to know aboutl nsswitch.conf too?
[11:45] <zyga> mborzecki: probably so
[11:46] <ondra> zyga thank you :)
[11:46] <zyga> I'll do my best to fix them obviously
[11:46] <zyga> writing tests is useful
[11:56] <Chipaca> imma go lunch
[11:57] <zyga> enjoy :)
[11:57] <Chipaca> i'll try
[12:11] <zyga> is it just me or are things extra slow today
[12:11] <zyga> setting up main test suite takes about 10 minutes to complete
[12:13] <mborzecki> zyga: and spread test are failing
[12:13] <zyga> how?
[12:14] <zyga> I haven't seen any failures though I'm mostly writing new tests now
[12:14] <zyga> (but no store related failures during that process either)
[12:41]  * zyga quick lunchj
[13:25] <zyga> jdstrand: hello, can you please enqueue https://github.com/snapcore/snapd/pull/7421 for a non-security review but rather a concept review of the idea
[13:25] <mup> PR #7421: cmd/snap-confine: unmount /writable from snap view <Created by zyga> <https://github.com/snapcore/snapd/pull/7421>
[13:38] <jdstrand> zyga: sure
[13:42] <pedronis> mborzecki: I don't understand the spread failures, many of them don't even seem to have clear errors, or I'm not looking right (quite possible)
[13:59] <Saviq> am I doing something dumb?
[13:59] <Saviq> $ snap info multipass | grep installed
[13:59] <Saviq> installed:   0.9.0-dev.171+g7a968814              (x4) 194MB classic
[13:59] <Saviq> $ snap refresh multipass --revision 1125 --amend
[13:59] <Saviq> error: local snap "multipass" is unknown to the store, use --amend to proceed anyway
[14:01] <mup> PR core18#139 opened: hooks: add missing dosfstools to get fsck.fat <Created by mvo5> <https://github.com/snapcore/core18/pull/139>
[14:01] <zyga> rogpeppe: hello
[14:01] <zyga> rogpeppe: we have some more ideas
[14:02] <rogpeppe> zyga: cool!
[14:03] <rogpeppe> zyga: BTW ISTM that the fail-to-reboot issue is the main problem here - i'm not sure that the other issue would be a real problem if the reboot hadn't failed to restart
[14:03] <zyga> rogpeppe: we looked some more and we suspect the boot partition that uses fat is corrupted, we found a bug related to absent fsck on core18 systems
[14:03] <zyga> rogpeppe: we devised a way forward that you should be able to do remotely
[14:03] <zyga> rogpeppe: if you have an app using "core" installed you should have access to fsck.vfat from /snap/core/current/usr/sbin/
[14:03] <zyga> rogpeppe: you can use that to fsck the boot partition
[14:04] <zyga> rogpeppe: you can unmount it for the duration of the check as well
[14:04] <rogpeppe> zyga: given that i re-flashed the card very recently, it seems slightly unlikely that it's corrupted already (within a few hours of first installing) but happy to try
[14:04] <zyga> rogpeppe: mvo looked at some of the error tracker logs and found what I believe was kernel telling us about fs corruption of the FAT partition
[14:05] <zyga> rogpeppe: so that's the first step, I think you know how to run that without hand-holding but please ask for help if you need any
[14:05] <zyga> rogpeppe: try to run it in mode verbose enough for us to see if there were any errors there
[14:05] <rogpeppe> rogpeppe@localhost:~$ ls -l /snap/core/current/usr/sbin/*fsck*
[14:05] <rogpeppe> ls: cannot access '/snap/core/current/usr/sbin/*fsck*': No such file or directory
[14:05] <rogpeppe> rogpeppe@localhost:~$ ls -l /snap/core/current/usr/sbin/*vfat*
[14:05] <rogpeppe> ls: cannot access '/snap/core/current/usr/sbin/*vfat*': No such file or directory
[14:06] <zyga> oh, silly me
[14:06] <zyga> just /snap/core/current/sbin/
[14:06] <zyga> not /usr
[14:08] <rogpeppe> zyga: so i'm planning to run these commands; do they seem right to you?
[14:08] <rogpeppe> umount /boot/uboot
[14:08] <rogpeppe> fsck -V /dev/mmcblk0p1
[14:08] <zyga> yes
[14:08] <zyga> they look good
[14:09] <zyga> (assuming PATH is setup to find the fsck.vfat)
[14:10] <zyga> rogpeppe: as an extra remark, it's sometimes good to stop snapd.service during things like this (hand's on experiments)
[14:10] <zyga> to avoid background activity
[14:10] <rogpeppe> zyga: http://paste.ubuntu.com/p/Fm3RpdC8d2/
[14:11] <zyga> interesting!
[14:11] <rogpeppe> zyga: i'd definitely unmounted the fs
[14:11] <zyga> perhaps uboot and kernel disagree on which boot sector to use and then something gets out of sync later
[14:11] <zyga> for the purpose of experiment, copy original to backup
[14:11] <zyga> I _believe_
[14:12] <zyga> that is what the kernel would use
[14:12] <zyga> but I welcome the advice of mborzecki
[14:12] <zyga> mborzecki: ^
[14:12] <rogpeppe> zyga: and remove dirty bit?
[14:12] <zyga> yes, but please understand my POV of trying to fix the partition and see if that means you can correctly boot out of the problem
[14:12] <zyga> one more idea
[14:12] <zyga> perhaps tarball all of the boot partition
[14:13] <zyga> or even dd the whole partition to ext4 somewhere
[14:13] <zyga> for forensics
[14:13] <zyga> dd is better
[14:13] <rogpeppe> zyga: sure, i could send you a copy
[14:13] <zyga> as you don't have to mount do do
[14:13] <zyga> *to do
[14:13] <zyga> yes, we can then look at it bit by bit in hexedit
[14:13] <zyga> fortunately we don't keep that many files there
[14:15] <rogpeppe> zyga: ok, downloading disk image now; will upload to s3
[14:15] <zyga> thank you
[14:17] <rogpeppe> zyga: i still find it very odd that it only boots up ok once every other time. i can't think of something that might be causing such predictable boot failures.
[14:17] <zyga> I can offer one
[14:17] <mborzecki> zyga: fwiw, i think it's worth checking wether the same incorrectly unmounted warning appears on a cleanly built image
[14:17] <mborzecki> like out of the box
[14:18] <zyga> mborzecki: good idea
[14:18] <zyga> rogpeppe: if uboot reads the FAT differently (we saw that at least once in the past) and sees a different file than linux
[14:18] <zyga> then snapd will configure the boot loader to boot kernel-1, core-1 in "trying" mode
[14:18] <rogpeppe> zyga: BTW the s/w i'm running ran fine without any issues for months on end previously
[14:18] <zyga> the boot loader will go but never see those values
[14:18] <zyga> booting something else
[14:18] <zyga> perhaps something that is removed now
[14:19] <zyga> or perhaps something that is there but disagrees with what snapd expected
[14:19] <zyga> so snapd will change boot configuration again
[14:19] <zyga> and plan another reboot
[14:20] <zyga> rogpeppe: but the point is that the oscillation may be kernel/uboot disagreeing on the contents of a specific file
[14:20] <zyga> and snapd writing to that file in between boots
[14:22] <rogpeppe> zyga: ok, interrresting
[14:23] <mup> PR snapd#7440 closed: snap/channel: fix unit tests, UbuntuArchitecture -> DpkgArchitecture <Simple 😃> <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7440>
[14:25] <rogpeppe> zyga: try this: https://rogpeppe-scratch.s3.amazon.com/bootcopy.gz
[14:25] <zyga> grabbing
[14:25] <zyga> dns issue?
[14:26] <rogpeppe> zyga: i've probably forgotten how s3 urls work
[14:26] <Chipaca> i thought they'd turned off that feature
[14:27] <Chipaca> of being able to just url the stuff
[14:27] <Chipaca> (but i didn't pay too much attention to that email because i don't use it)
[14:29] <rogpeppe> zyga: just usual amazon eventual consistency issues; try again
[14:29] <mborzecki> zyga: fsck does't complain with a pristine core image
[14:29] <zyga> mborzecki: thank you for checking!
[14:29] <zyga> rogpeppe: still nothing
[14:29] <rogpeppe> zyga: ha, it works for me :-\
[14:29] <zyga> my dns may have cached it as gone?
[14:30] <zyga> how big is it?
[14:30] <zyga> can you email me@zygoon.pl
[14:30] <rogpeppe> zyga: 48621287 bytes
[14:30] <zyga> might be faster :)
[14:30] <zyga> I'll go over the bits in the evening to see what's wrong
[14:30] <zyga> meanwhile you can attempt to fix FAT
[14:30] <zyga> or even remake it and copy the files back
[14:31] <rogpeppe> zyga: for the time being, i've just slowed down updates so it'll only update on the first monday of the month
[14:31] <zyga> excellent
[14:31] <zyga> rogpeppe: as pedronis said, you can also use refresh.hold to delay up to 60 days
[14:32] <rogpeppe> zyga: sent
[14:33] <mup> PR snapd#7442 opened: tests: extend mount-ns test to handle mimics <Created by zyga> <https://github.com/snapcore/snapd/pull/7442>
[14:33] <rogpeppe> zyga: i think 60 days isn't enough longer than 30 days to justify the unpredictability
[14:33] <zyga> sure, just saying
[14:33] <zyga> let's try to fix that FAT online
[14:33] <zyga> apply all the fixes that fsck would normally do
[14:34] <rogpeppe> zyga: you think that might fix the issue for good?
[14:34] <zyga> I think it's likely
[14:34] <zyga> but we don't have the data trends from the error tracker AFAIK so perhaps there's more but we're not aware of it
[14:34] <rogpeppe> zyga: this is quite concerning BTW - this was an absolutely pristine image created following the instructions on the web to the letter
[14:35] <rogpeppe> zyga: if i'm seeing this problem, then i'd guess that everyone else using ubuntu core is too
[14:35] <zyga> indeed, we proposed that the boot partition is read only outside of the transactions that need to use it
[14:35] <zyga> so that random power failures don't leave FAT in mounted state
[14:35] <rogpeppe> zyga: that seems like a good idea.
[14:35] <zyga> rogpeppe: there are some issues that can be specific to your device, e.g. the SD card may be just really failing
[14:35] <zyga> I have a number of cards that reliably corrupt a fixed offset
[14:35] <rogpeppe> zyga: it's a near-new SD card too, but i guess that's possible
[14:36] <zyga> you can write zeros or ones, you keep reading that blob that they somehow store
[14:36] <zyga> rogpeppe: the one I have is little-used sandisk pro 32GB card
[14:36] <zyga> rogpeppe: it failed in a few weeks after purchase
[14:36] <rogpeppe> zyga: that would make it both 32GB SD cards that are failing in a similar way then
[14:37] <rogpeppe> zyga: because i saw a similar issue with the Pi 2 and assumed the sd card had gone
[14:37] <rogpeppe> zyga: i actually have that pi with me in fact
[14:37] <zyga> I got the boot image now, thank you
[14:38] <zyga> rogpeppe: try something like 'etcher' to see if you can write a pristine image and read it back correctly if you want to check that
[14:38] <rogpeppe> zyga: yeah, i'll try that
[14:40] <rogpeppe> zyga: i wasn't surprised when the original sd card failed BTW - it had been in constant use for about 3 years.
[14:40] <rogpeppe> zyga: ... if it did fail, of course
[14:40] <zyga> indeed, analysis will reveal the cause
[14:40] <zyga> there are some moving parts
[14:40] <zyga> and some failures on our end
[14:40] <zyga> sil2100: hey, there's one PR for core18
[14:40] <zyga> sil2100: it's related to what we are discussing now
[14:41] <zyga> sil2100: we didn't seed fsck.vfat on core18
[14:41] <zyga> sil2100: do you think you could review it please?
[14:41] <zyga> https://github.com/snapcore/core18/pull/139
[14:41] <mup> PR core18#139: hooks: add missing dosfstools to get fsck.fat <Created by mvo5> <https://github.com/snapcore/core18/pull/139>
[14:48] <mvo> rogpeppe: thank you so much for reporting this - and yes, super concerning to us and we dig into it
[14:48]  * mvo hugs zyga for digging into it
[14:48] <rogpeppe> mvo: thanks for your interaction :)
[14:49] <zyga> rogpeppe: this is what I see on the partition: https://pastebin.ubuntu.com/p/tVXVjCK99Q/
[14:50] <zyga> I will now check the contents of config.txt, cmdline.txt and uboot.env
[14:50] <zyga> for one, I like that mtools exists
[14:50] <zyga> and wish that there was an ext4 variant :)
[14:50] <rogpeppe> zyga: i don't know about mtools...
[14:50] <zyga> rogpeppe: it's a GNU tool for interacting with FAT offline
[14:51] <sergiusens> hello folks, is there a fix upcoming for "- Download snap "crystal" (71) from channel "latest/stable" (stream error: stream ID 1; PROTOCOL_ERROR)" ?
[14:52] <sergiusens> this is really slowing us down
[15:00] <zyga> sergiusens: we don't have a fix, only a workaround mvo worked on
[15:01] <sergiusens> zyga: does the workaround require work (a setting) on our side?
[15:05] <ondra> zyga I found one cloud instance with 4 broken snaps I was not able to remove, with latest snapd, I was able to remove all them :) Great work!
[15:05] <zyga> sergiusens: no, it's automatic
[15:05] <zyga> ondra: thank you so much :)
[15:06] <ondra> zyga thank you for fixing it :)
[15:06] <zyga> ondra: fixing bugs is sometimes very draining, I'm very glad I was able to help you and others in FE
[15:06] <zyga> rogpeppe: this is the hexyl dump of uboot.env, I'll check out what it says next https://paste.ubuntu.com/p/GgScB76RyT/  -- specifically to see if it is in agreement with snapd's state
[15:07] <zyga> though I must say that the colorized output from hexyl is easier to read as it shows NUL bytes and other such stuff in clear, distinct color
[15:08] <zyga> mvo: one other idea, just looking at this, is to have two uboot environment files: one for the fixed program and the other one for just the handful of actual variables we need
[15:08] <zyga> oh, mvo is not online anymore
[15:12] <zyga> rogpeppe: so looking here I see we have the following things: snap_core=core18_1076.snap, snap_kernel=pi-kernel_44.snap, snap_mode= (empty string), snap_try_core=core18_1100.snap, snap_try_kernel=pi-kernel_51.snap,
[15:12] <zyga> I need to reference the boot logic for a second to understand snap_mode=""
[15:12] <zyga> mborzecki: ^ unless you remember
[15:12] <zyga> rogpeppe: do you have the snaps and revisions listed there in /var/lib/snapd/snaps/
[15:13] <zyga> they should all be mounted as well
[15:13] <zyga> and visible in snap list --all
[15:13] <zyga> rogpeppe: (output of snap list --all would help as well)
[15:16] <mborzeck1> zyga: this? https://github.com/snapcore/core-build/blob/master/initramfs/scripts/bootloader-script#L89
[15:17] <mborzeck1> zyga: or this one https://github.com/snapcore/pi3-gadget/blob/16/uboot.env.in#L48
[15:17] <zyga> huh
[15:17] <zyga> https://github.com/snapcore/core-build/blob/master/initramfs/scripts/bootloader-script#L104
[15:18] <rogpeppe> zyga: http://paste.ubuntu.com/p/ktw4XYgrvG/
[15:18] <zyga> I don't understand how that works
[15:18] <zyga> ah, wait
[15:18] <zyga> didn't notice nesting
[15:18]  * zyga re-reads
[15:19] <zyga> rogpeppe: do you have core18 at revision 1100?
[15:19] <zyga> I mean
[15:19] <zyga> I don't see it
[15:19] <zyga> so I assume that's why it fails
[15:19] <zyga> it seems we are trying to get to core18 revision that's simply not here
[15:20] <zyga> that would explain the immediate failure
[15:20] <rogpeppe> zyga: i see core18 at 1076
[15:20] <zyga> though I didn't check what uboot script does if it cannot find that snap
[15:20] <rogpeppe> zyga: what's special about 1100?
[15:20] <zyga> rogpeppe: nothing, it's just referenced from your uboot.env but not present on the system
[15:20] <rogpeppe> zyga: oh, i see
[15:21] <rogpeppe> zyga: i wonder how that happened
[15:21] <zyga> indeed
[15:21] <zyga> though snapd may have undone 1100 transaction
[15:21] <zyga> removing the file from disk
[15:22] <zyga> if you feel lucky, fix the boot partition with fsck
[15:22] <zyga> snap refresh core18
[15:22] <zyga> and check if it manages
[15:22] <zyga> one other lesson from this
[15:22] <zyga> is for snapd to fix any boot variables that are inconsistent with reality
[15:22] <zyga> we have one boot mode
[15:23] <zyga> as in, one variable called snap_mode
[15:23] <zyga> that impacts two variables "trying"
[15:23] <rogpeppe> i'm not feeling very lucky currently :)
[15:23] <zyga> and it's clear that in this case there's a chance one of them will fail
[15:23] <zyga> rogpeppe: I'll collect this in a retrospective
[15:23] <zyga> there's a lot for us to learn from this
[15:23] <zyga> rogpeppe: I would suggest to fix the FAT partition
[15:24] <zyga> that might be enough to fix other issues
[15:24] <rogpeppe> zyga: ok, i'll try that
[15:24] <zyga> I'll collect all of this for a retrospective and share with you
[15:25] <zyga> I was thinking about breaking for dinner now
[15:25] <rogpeppe> zyga: ok, dirty bit  removed. it didn't give me an option to do anything else
[15:25] <rogpeppe> zyga: i suspect i may have done the wrong thing there :-'
[15:25] <rogpeppe> :-\
[15:25] <zyga> oh?
[15:25] <zyga> note, you can always dd it back!
[15:26] <rogpeppe> zyga: good point!
[15:26] <zyga> what did you do?
[15:26] <zyga> try fixing it, mounting it
[15:26] <zyga> and looking at the files
[15:26] <rogpeppe> zyga: http://paste.ubuntu.com/p/MtVGCFGGwk/
[15:26] <zyga> at least the boot.env
[15:27] <rogpeppe> zyga: i suspect i should've said "no" to removing the dirty bit, i think
[15:27] <zyga> ah, no
[15:27] <zyga> I think that's fine
[15:27] <zyga> it's just a bit
[15:27] <zyga> fat has no journal
[15:27] <zyga> so apart from top-down scan there's little to do
[15:27] <rogpeppe> zyga: i thought i'd get the option to address the other issue too
[15:28] <rogpeppe> zyga: so the dirty bit is the only thing that was wrong?
[15:28] <zyga> ah right
[15:28] <zyga> I honestly don't know
[15:28] <zyga> can you fsck again?
[15:28] <zyga> maybe with some --force option
[15:29] <rogpeppe> zyga: a ran it again - it does nothing
[15:29] <rogpeppe> *i* ran it again
[15:30] <zyga> did you use -V?
[15:38] <zyga> rogpeppe: so
[15:38] <zyga> rogpeppe: writing the report I'm not sure we understand what really failed on boot
[15:38] <zyga> rogpeppe: we know that the fat was slightly corrupt
[15:38] <zyga> that it was not unmounted cleanly
[15:38] <zyga> rogpeppe: we do know that core18 revision 1100 was missing from your disk
[15:39] <zyga> rogpeppe: though perhaps it was removed by snapd in its undo path
[15:39] <zyga> rogpeppe: I would like to know if you'd be willing to attempt another reboot, at your convenience, coupled with another reboot done by "snap refresh core18"
[15:40] <rogpeppe> zyga: so one reboot without "snap refresh core18", then run "snap refresh core18", then let that reboot by itself?
[15:40] <zyga> yes
[15:40] <zyga> but only if you have confidence you can recover manually
[15:40] <zyga> and not super inconvenient for you
[15:41] <rogpeppe> zyga: ok, i'll try that. what should i do when it fails to restart after the first reboot?
[15:41] <zyga> power cycle
[15:42] <rogpeppe> zyga: ok
[15:42] <zyga> if that fails we may be SOL but I dont't think it will come to that
[15:42] <zyga> you may want to mount the partition back
[15:42] <zyga> mvo: hey welcome back
[15:42] <rogpeppe> zyga: does that make a difference?
[15:42] <zyga> rogpeppe: in case snapd wishes to write to it
[15:42] <zyga> otherwise no
[15:42] <rogpeppe> zyga: ok, i'll mount it again now
[15:43] <mvo> hey zyga - whats new?
[15:43] <zyga> mvo: we found some things, one sec, I'll share my notes
[15:47] <ijohnson> hey folks, could I get another review on https://github.com/snapcore/snapd/pull/7429 ? mvo maybe if you're not EOD yet and have a couple minutes :-)
[15:47] <mup> PR #7429: wrappers/services: add CurrentSnapServiceStates + unit tests <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7429>
[15:55] <mvo> ijohnson: heh, let me have a look
[15:57] <ijohnson> thanks :-)
[15:58] <zyga> rogpeppe: let us know what you find pleaes
[16:01]  * Chipaca goes for a run
[16:02] <mvo> ijohnson: you have feedback
[16:02] <mvo> ijohnson: should be super simple
[16:02] <ijohnson> yay thanks looking now
[16:04] <rogpeppe> zyga: will do. it might be tomorrow.
[16:04] <zyga> ack, thank you for the note
[16:13] <ijohnson> mvo: thanks I fixed the out of date comment / for loop, but I think it's still nice to have the more verbose/complete systemctl script
[16:14] <mvo> ijohnson: thats fine
[16:14] <mvo> ijohnson: keep it if you prefer it :)
[16:14] <ijohnson> okay, cool so with your and mborzecki's review am I good to merge?
[16:15]  * ijohnson waits on the merge button
[16:15] <mvo> ijohnson: yes
[16:15] <ijohnson> oh well I guess the tests have to pass too
[16:15] <mvo> ijohnson: *cough*
[16:15] <ijohnson> :-O
[16:15] <mvo> ijohnson: :(
[16:15] <ijohnson> they're not failing just haven't started running yet
[16:16] <mvo> ijohnson: ok
[16:17] <ijohnson> thanks mvo, I'll merge sometime in my afternoon then
[16:18] <mvo> ijohnson: good luck!
[16:18] <ijohnson> :-)
[17:25] <rogpeppe> anyone got any idea why `xdg-open` has stopped opening pdf files correctly for me (it opens them in an ebook viewer). I think it might have something to do with the fact that `xdg-mime query filetype anyfile.pdf` doesn't print anything, but I'm not sure how that works.
[17:25] <rogpeppe> oops, wrong channel, sorry!
[17:44] <mup> PR snapcraft#2644 closed: Release changelog for 2.44 <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2644>
[20:14] <mup> PR snapcraft#2709 opened: incorporate content provider snaps in dependency resolution <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/2709>