=== KindTwo is now known as KindOne === KindTwo is now known as KindOne === KindTwo is now known as KindOne === mcphail7 is now known as mcphail [06:06] Good morning [06:06] I will start around 9 [06:49] PR snapd#8842 closed: tests: fix bug waiting for snap command to be ready [06:59] morning [06:59] PR snapd#8816 closed: tests: port 2 uc20 part1 [07:00] mvo_: hi [07:13] Hey everyone [07:13] I’m around, going to address PR feedback now [07:13] I will be off later for the mri [07:19] pedronis: hi, not sure my earlier "hi" made it to you, for some reason I was put into "quiet" mode by freenode [07:19] pstolowski: good morning [07:19] zyga: good luck for later today [07:19] mvo: if your nick is not the registered one (like mvo_) you can't talk here [07:36] Hi there! I've asked that question on #snapcraft but I think it fit best here. I have a snap that break after each reboot. it show me this error https://pastebin.com/X3VCfyua. If I reinstall the snap it work again. I'm running snapd on manjaro-arm on the pinebook pro. Any pointers of what might be going on ? thanks in advance! [07:37] alazred: this means that nothing is loading apparmor profiles [07:38] alazred: what is the status of snapd.apparmor.service? [07:38] alazred: it is responsible for loading snapd apparmor profiles on system startup [07:40] zyga: https://pastebin.com/i1eMy3Md [07:41] zyga: sorry not the good one [07:41] hmm, too bad there are no logs [07:41] do you have persistent journal enabled? [07:41] it would be good to make sure you do (mkdir /var/log/journal) [07:41] and reboot [07:41] and then check the status of that service [07:42] zyga: It's dead https://pastebin.com/eX3ccSgz [07:43] the service is not enabled [07:43] systemctl enable --now snapd.apparmor.service [07:45] Ive enabled it and it's now working ! [07:45] zyga: thanks ! [07:45] alazred: maybe file a bug in manjaro [07:46] alazred: do you remember if it was disabled out of the box? [07:47] zyga: Yeah i will go there also ! The error was not super clear and since the other apparmor.service was enalble i tought I was ok on that side. [07:47] yeah, until apparmor 3 ships we need a custom service [07:47] zyga: I'm pretty sure it was [07:47] apparmor 3 won't need this [07:47] ok [07:48] but since it's arch base I don't know if it enable those things on install. [07:48] I'll try to know more on that. Thanks again for your time! [07:49] alazred: thanks for reaching out :) [07:59] zyga: They dosen't do mention of snapd.apparmor.service in the manjaro wiki. Only to enable snapd.socket. [08:00] alazred: I see, I think it needs fixing [08:00] I reached out to some manjaro friends and let them know [08:00] morning [08:00] zyga: Ok thanks ! [08:05] PR snapd#8841 closed: gadget: drop dead code, hide exports that are not used externally [08:05] PR snapd#8844 opened: asserts/internal: add some iteration benchmarks, potential opt [08:24] pstolowski: I tried to answer your comment in #8823, also made some small changes based on mborzecki feedback [08:24] PR #8823: asserts/internal: limit Grouping size switching to a bitset representation [08:25] thanks [08:48] mvo: I'm seeing this error, not sure if related to my PR or just random [08:49] https://www.irccloud.com/pastebin/Xh7y61FS/ [08:49] zyga: oh, fun. looks like the earlier incorrect loop masked some other problem :( [08:49] zyga: how often/where do you see it? [08:50] just noticed a few systems failed this way on my RFC PR [08:50] https://github.com/snapcore/snapd/pull/8843/checks?check_run_id=756927988 [08:50] PR #8843: [RFC] many: export tools from core/snapd to mount namespaces [08:50] look at the colors on the left [08:50] lots of green, a few reds [08:50] zyga: quire a few reds :/ [08:50] yeah [08:50] I wonder what was going on before [08:50] zyga: ok, I have a look. odd that it did not appear in the original PR [08:50] it doesn't seem like a new bug [08:50] just surfaced [08:51] zyga: yeah, I'm sure the superlong timeout just masked the issue [08:51] it *might* be something my PR does [08:51] but I'm not sure [08:51] one sec [08:51] Wed, 10 Jun 2020 08:30:40 GMT [08:51] retry: next attempt in 1.0 second(s) (attempt 6 of 120) [08:51] 730 [08:51] Wed, 10 Jun 2020 08:30:40 GMT [08:51] + snap wait system seed.loaded [08:51] 731 [08:51] Wed, 10 Jun 2020 08:30:40 GMT [08:52] error: cannot communicate with server: timeout exceeded while waiting for response [08:52] 732 [08:52] with time stamps [08:52] this is not after a long wait [08:52] but this is surprising [08:52] so retry *succeeded* [08:52] after 6 seconds [08:52] zyga: yeah, I suspect snapd is restarting or something [08:53] but the "snap wait" immediately after that fails [08:53] yeah [08:53] maybe snap wait should be more robust [08:53] I think this may be what was randomly failing earlier as well [08:53] btw, those timestamps *rock* [08:54] zyga: yeah, snap should actually retry :/ [08:54] heh [08:54] retry snap wait [08:54] ;) [08:58] we have the wait.go logic but is meant for things that give back a Change [08:58] but we need some bits of it [08:58] for wait [08:58] itself [09:01] this kind of code: https://github.com/snapcore/snapd/blob/master/cmd/snap/wait.go#L96 [09:02] pedronis: thank you [09:45] PR snapd#8845 opened: [RFC] many: add "system.service.snapd-autoimport.disable" setting [09:47] zyga: I can look at the wait issue now [09:47] mvo: thank you [10:00] PR snapd#8808 closed: riscv64: bump timeouts === alazred_ is now known as alazred [10:13] afk [10:24] zyga: are you still using fish? [10:25] zyga: it's all broken [10:27] mvo: what? [10:27] mborzecki: yes [10:27] mvo: what's going on? [10:27] sorry, not having most productive day [10:27] I feel the best in a position where typing is not possible [10:28] zyga: can you try `snap `, does it show the commands and their help? [10:30] zyga: the wait thing you pointed me to is broken in 3 different ways on 16/18/20 :( [10:30] zyga: anyway, I'm looking [10:30] mvo: woah [10:30] mborzecki: trying [10:30] yes [10:30] it super nice way :) [10:30] how do we screenshot on linux? [10:30] hmm [10:30] PR snapd#8846 opened: tests: move update-related tests to snapstate_update_test.go [10:31] mborzecki: check this out https://usercontent.irccloud-cdn.com/file/K78qpiep/fish-tab-completion.png [10:32] yeah, it's in the upstream https://github.com/fish-shell/fish-shell/blob/master/share/completions/snap.fish [10:32] super pretty [10:33] not using bash helpers but I guess for some reasons [10:33] zyga: there was some problems reported in the forum though https://forum.snapcraft.io/t/tab-completion-for-snap-connect-in-fish-spams-errors/18083 [10:33] it wfm though [10:33] yeah [10:33] that is broken [10:34] mhm [10:34] yeah, it's the connect/disconnect thing [10:34] that warning should probably go to stderr [10:34] mborzecki: snap connect SNAFU https://usercontent.irccloud-cdn.com/file/KdWyk66Z/snap-connect-broken-completion.png [10:35] mvo: can you tell some more [10:35] mvo: it's really interesting [10:36] zyga: the first (16) stops with a connection timeout exceeded [10:36] zyga: the second (18) hangs forever [10:36] whaat? [10:36] zyga: and 20 fails with dpkg --purge dir not empty [10:36] and it was just this broken/ [10:36] how did we miss it? [10:36] zyga: I start with 20 now that looks most simple [10:36] zyga: no idea, 20 could be just coincidence [10:36] zyga: heh it calls `snap interfaces $snap | string replace -r '[- ]*([^ ]*)[ ]+([^ ]+)' '$2$1' | string match -v "*Slot*"` [10:38] hm i'll to fix it upstream [10:39] one thing I love in fish [10:39] is that $foo is not unsafe [10:39] as fish does not expand strings this way [10:39] it's weird coming from bash [10:39] but nice [10:41] mvo: sorry for dragging you into this [10:44] zyga: my own fault for doing the original PR [10:44] mvo, mborzecki : is #8784 still a draft? [10:44] PR #8784: snap: add new `snap run --experimental-gdbserver` option [10:45] pstolowski: switched it to 'ready' [10:45] pstolowski: I think we can switch it [10:45] indeed, thanks [10:45] it's probably missing some tests tho, we can add those once we get some reviews [10:54] zyga: this should fix it https://github.com/fish-shell/fish-shell/pull/7104 [10:54] PR fish-shell/fish-shell#7104: completions/snap: workaround snap interfaces deprecation notice [10:54] nice [10:55] hm zomething we could distro patch for instance in ubuntu? [11:00] I'm sure we can [11:00] mvo: cna you dput to ubuntu? [11:01] (once it lands upstream first) [11:01] zyga: dput what? [11:01] mvo: fish [11:02] zyga: I can/could to groovy, focal will need a SRU [11:02] grovy would be good [11:04] i'll file a bug first [11:06] mborzecki: looks like #8830 can be merged? [11:06] PR #8830: bootloader: introduce bootloarder assets, import grub.cfg with an edition marker [11:07] it's green yay! at last [11:07] zyga: and none of the wait issues you saw in the PR is reproducible so far :( [11:07] hmmm [11:08] maybe my PR is at fault [11:08] are you trying on master or on that branch? [11:08] zyga: master [11:08] zyga: let me try your PR [11:08] is the export PR? [11:08] if it's just that PR it's not worth spending time on it [11:08] I thought you found some deeper issues on master [11:10] zyga: ok, I park it for now (if the current run is also not failing) [11:11] PR snapd#8830 closed: bootloader: introduce bootloarder assets, import grub.cfg with an edition marker [11:16] pedronis, zyga a log of the 20.04 "device not ready" we talked about yesterday with the debug log https://github.com/snapcore/snapd/pull/8845/checks?check_run_id=757318867 - I'm now trying to make sense of it [11:16] PR #8845: [RFC] many: add "system.service.snapd-autoimport.disable" setting [11:18] Jun 10 10:09:33 jun100957-926530 snapd[35371]: snapmgr.go:298: cannot read snap info of snap "snapd" at revision 7777: cannot find installed snap "snapd" at revision 7777: missing file /snap/snapd/7777/meta/snap.yaml [11:18] that's super clear IMO [11:18] mvo: this is the bug that we talked earlier about [11:18] mvo: about someone showing a system where booting lost interfaces [11:18] because core was not mounted on time [11:18] but [11:19] how can this be happening!? [11:19] this is what is so myserious about it [11:19] are we racing with mounting that we do ourselves? [11:19] here there is no core snap at all [11:19] to be clear [11:19] there is no snapd snap either [11:19] this is also weird: Jun 10 10:00:06 jun100957-926530 systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing. [11:19] zyga: oh, the mount units could be too slow to start? [11:20] and this killing was after we did this [11:20] Jun 10 10:00:06 jun100957-926530 systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing. [11:20] and this kicks off the snapd.failure unit Jun 10 10:00:06 jun100957-926530 systemd[1]: snapd.service: Triggering OnFailure= dependencies. [11:21] hmmmm [11:21] mvo: is it possible that snapd failing unmounts snap units? [11:21] like undoing some dependency chain or something [11:21] zyga: possibly seems unlikely but I guess we need to explore whatever we can find [11:22] mvo: but remember there should be no snapd at all at that point here [11:22] zyga: hm, two interessting things - the image has snaps seeded already, I think that's the only image we have with that property [11:23] we just reinstalled the snapd deb [11:23] zyga: there is also a error auto-refresh change [11:23] it seems like there's state from something else around [11:23] pedronis: it looks like the image already has snapd in the image (and lxd/core18) [11:23] yea [11:23] and removing snapd doesn't do the right thing [11:24] oh interesting! [11:24] this is prepare.sh [11:24] it was written for systems which just the snapd deb [11:24] but possibly with 20.04 there's a ton of stuff there [11:27] we just install the new deb? [11:27] pedronis: yes [11:27] pedronis: it sounds like we need to purge snapd first [11:27] that probably doesn't work well [11:27] maybe that's enough [11:27] if snapd is in the middle [11:27] of seeding [11:27] there's a real bug here I suppose [11:27] what should happen if you upgrade the deb while snapd is seeding [11:28] right [11:29] not saying that we need to fix that to fix prepare.sh [11:29] but we defintely need to check what happens, what we want to happen [11:29] in that scenario [11:29] yeah [11:30] it seems you can get a fairly weird state [11:34] pedronis: looking at the looks it seems like the seeding is already done, it happend 43 days ago (probably when the image was booted first and preapared for our needs) [11:34] pedronis: I suspect the issue happens when a auto-refresh is in progress while we update the deb [11:35] pedronis: but it's a similar issue I guess(?) [11:35] yes [11:35] I don't think we have thought through this [11:36] maybe it's only relevant if the deb has a newer snapd? [11:36] yeah - and it seems real given that andy had this bug on a ubuntu 20.04 system some days ago [11:36] pedronis: oh, that's interessting [11:39] naively the main issue in prepare.sh is that we have a state.json with seeded [11:39] but it doesn't relate to the installing snapd [11:39] but I don't know if there's other interference possible [11:39] especially if the deb version is newer [11:41] it's a bit confusing because when it works we end up with no snaps except core20 [11:43] pedronis: what is most interessting to me is that we see the mount units vanishing, this looks like the bug we had the other day. I would like to understand this :/ the fix in prepare.sh is easy (like you said) we can purge and things should work. anyway, I have lunch first but this looks interessting and like there is something real lurking here [11:43] * mvo lunch [11:47] mmh, we have also our strange hack that removes the seed [11:47] though I suppose only if it's invalid [12:00] * zyga preps for the MRI [12:00] see you laster [12:00] *later [12:00] good luck [12:01] mvo: purge unmounts [12:01] mvo: so snapd may be doing stuff while we purge and purge and purge [12:01] (just a guess) [12:18] * pstolowski going to the vet, bbl [12:51] hi everyone [12:51] i have a question about building a custom image with ubuntu core 20 [12:51] im specially interested in trying to build with grade:secure option [12:51] i modified my model json file and trying to sign it [12:51] i get this error: error: cannot assemble assertion model: cannot specify a grade for model without the extended snaps header [12:52] now it kinda makes sense because on this page, every snap is put with its identifier: https://docs.ubuntu.com/core/en/releases/uc20 [12:52] but the sign function from snapcraft expects a json [12:52] do i have to somehow insert this information to required-snaps list [12:54] i guess maybe i have to remove "require-snaps" andd replace it in json with "snaps" as a map not just a list of strs [12:55] clmsy: hey, check https://github.com/snapcore/models/blob/master/ubuntu-core-20-amd64.json for an example [12:55] clmsy: there are some more in https://github.com/snapcore/models [12:55] aaah its already in here thanks a lot! === stoopkid_ is now known as stoopkid === alazred_ is now known as alazred [13:34] hmm [13:35] did anyone get pc-kernel snap not found while trying to build a core20 image ? [13:35] clmsy: what branch are you using for your kernel snap you need to be using the 20/ channel [13:36] 20/beta [13:37] i mean surely its there since snap info shows: 20/beta: 5.4.0-34.38.1 2020-06-08 (524) 271MB - [13:37] haha [13:38] clmsy: hmm then I don't know what the cause of that issue is then [13:49] re [13:56] * mvo hands pstolowski the "biggest-PR-of-the-year" award [13:56] /o\ [13:58] pstolowski: heh - not your fault that snapstate_test grew to such monstrous size :) thanks for cleaning it up ! [13:58] * mvo hands pstolowski the "Augean Stables" award instead [13:58] LOL [14:00] haha [14:10] Back home [14:10] Need rest [14:10] Darn pain [14:12] * mvo hugs zyga (gently) [14:17] I won’t be doing those 12 hour flights anytime soon [14:19] Bug #1882957 opened: Snapd `internal error: connection "[slot] [plug]" not found in state` [14:19] pedronis, this is what I updated in spread: https://github.com/snapcore/spread/pull/72 [14:19] PR spread#72: Skip tests [14:19] pedronis, today I'll push another implementation using SKIP command [14:20] I need to make that work better [14:20] before pushing [14:23] im almost able to build a custom image w core20, one last road bump im having [14:23] error: cannot add snap "network-manager" without also adding its base "core" explicitly [14:23] now i have type:base core20 [14:23] do i need to add say core16 without type:base or somehow add core16 base information to snap header of network-manager [14:24] either add "core" to your required-snaps in the model assertion or find a different track for network-manager [14:24] (i'm sure at least a core18 track exists for it) [14:24] ok thank you, ill try figure that out [14:24] clmsy: you need to use type: core for core [14:25] ok i guess as long as type:base is core20 i am allowed to add more type:core [14:25] well, the NM he installs has seemingly "base: core" [14:25] so "core" needs to be in the image as well ... or an NM with core18 or core20 base [14:26] yes, he can do that by having an entry in snaps with type: core name: core id: of core [14:26] thanks for the fast feedback! [14:26] ah, in the model assertion ? (thats new !) [14:26] ogra: in core20 models requires-snaps is called snaps and has much more syntax [14:26] it's a list of maps [14:27] neat [14:27] ! [14:27] (i have admittedly not looked a lot at 20 yet) [14:27] ogra: https://github.com/snapcore/models/blob/master/ubuntu-core-20-amd64.model for an example [14:28] (some docs, howtos and tutorials would be really helpful) [14:28] Bug #1882957 changed: Snapd `internal error: connection "[slot] [plug]" not found in state` [14:28] oh, this is really cool ! [14:28] can you omit the id filed for using local snaps though ? [14:28] yeah i like the syntax and the reason i jumped into trying to utilize is that, one of our main images that we use, i needed to implement full disk encryption and secureboot [14:28] than i saw that with core20 i can utilise grade:secured for that feature [14:29] (local snaps are pretty essential during development) [14:29] ogra: yes, but only in grade: dangerous [14:29] good [14:29] thats good enough [14:29] yes, it's supported exactly for development [14:29] yeah 🙂 [14:31] Bug #1882957 opened: Snapd `internal error: connection "[slot] [plug]" not found in state` [14:33] cachio: silly question - I have one spread run for ubuntu-core-20-64 (https://github.com/snapcore/snapd/pull/8845/checks?check_run_id=757318867) that uses an image that has snaps installed. when I run the same test here with my local spread against google I get a ubuntu 20.04 base image without snaps. any idea how this can happen? why would we get different images for 20.04? [14:33] PR #8845: [RFC] many: add "system.service.snapd-autoimport.disable" setting [14:34] mvo, let me check the pr [14:34] one sec [14:36] mvo: interesting failure in repartition magic [14:36] https://github.com/snapcore/snapd/pull/8829/checks?check_run_id=757631787 [14:36] PR #8829: sandbox/cgroup: add tracking helpers [14:36] PR snapd#8847 opened: tests: fail in setup_reflash_magic() if there are snaps already [14:37] I would add retry test -e /dev/loop1p2 before that settle call [14:37] I suspect we just call settle in a racy way [14:37] since the partition may not exist yet [14:37] (on line 547 in the log) [14:37] mvo, agree with zyga, it has to be a race [14:38] I can send patches later but I really need to rest in a neutral position now [14:38] so away from screen and kbd [14:39] mvo, I'll try to reproduce it here to see how to fix it [14:40] well guys i think you might wanna add multiple core support on the model because the image tool is not allowing this [14:40] error: core snap has unexpected type: base [14:40]