[06:15] Hey [06:15] Back to school day [06:17] morning [06:17] We need to land the critical PR [06:17] zyga: hey, which one? [06:17] Stuff is still broken, eh [06:17] Last one [06:18] 8178 [06:18] uhh [06:19] zyga: that's why pushing snapd to store failed on friday? [06:22] re [06:22] checking what failed [06:22] yes [06:22] sucks, eh? [06:22] I saw some failover failures yesterday [06:23] and eh [06:23] fu** [06:23] https://www.irccloud.com/pastebin/RwhTm8UI/ [06:23] fun? :) [06:23] where do we get foo frm? [06:24] crap from other test? [06:24] yeah, tha'd be my first guess [06:25] pushed a fix [06:25] there's security-private-tmp test [06:26] and it ran before [06:27] mmm [06:27] I'll get coffee and fix this properly [06:27] zyga: maybe as a workaround, we could also take down the setgid flag in postinst? [06:28] hmm? [06:28] postinst of the deb? [06:28] that would ruin the snap, no? [06:28] as in, that would cause the snap to be g-s [06:28] unless I don't understand how it's made [06:29] in either case, it's all temporary [06:29] we'll fix review tools today [06:29] deploy in a day or two [06:29] duh, yeah, that may produce a different result for deb and snap [06:30] hm looks like the snap is built by just extrating snapd, so we'd have to tweak the bit there too [06:30] but that would break review-tools again :/ [06:31] mborzecki: I didn't test-build the deb because i didn't feel comfortable guessing how it's really built [06:31] mborzecki: but the deb, for sure, has no g+s anymore [06:31] er [06:31] *has* [06:31] so with some luck it will pass [06:40] brb [07:07] PR snapd#8179 opened: interfaces: power control interface [07:42] re [07:43] mborzecki: can you review 8178 [07:43] I'll land with one review [07:51] PR snapd#8178 closed: packaging: work around review-tools and snap-confine <⚠ Critical> [07:58] PR snapd#8180 opened: tests: rm -rf /tmp/snap.* in restore [07:58] mborzecki: ^ quick follow-up [07:58] hey mvo [07:58] good morning [07:58] mvo: FYI https://github.com/snapcore/snapd/pull/8178 [07:58] PR #8178: packaging: work around review-tools and snap-confine <⚠ Critical> [07:59] hey zyga [07:59] zyga: looking [08:01] morning [08:01] hey Pawel :) [08:01] mvo: this was pre-approved but I wanted to let you know it landed [08:01] mvo: it allows core/snapd snaps to be published again [08:04] mvo: pstolowski: hey [08:04] hey pstolowski and mborzecki [08:04] zyga: yeah, saw the mails over the weekend [08:05] failover still fails [08:05] zyga: do you know if jamie is aware that the review tools need fixing? [08:05] yes [08:05] PR snapd#8136 closed: boot: write current_kernels in bootstate20, makebootable [08:05] mvo: yes, he said he was working on it [08:05] we talked (Jamie / Pedronis / me) about it together [08:05] cool [08:05] mvo: hey! it occured to me over the weekend that snap-pressed could be made a classic snap, which is what you probably meant when you suggested making it a snap :) [08:06] snapd-failover failure on top of current master https://www.irccloud.com/pastebin/pxEx4fth/ [08:06] pstolowski: yeah, that will work [08:09] PR snapd#8168 closed: snapcraft.yaml: add comments, rename snapd part to snapd-deb [08:10] mvo: good, i'll prep a PR for that (a snapcraft.yaml inside cmd/snap-presseed?) and will make a quick test [08:11] pstolowski: do we know it's needed? I'm probably missing something [08:11] hey pedronis, good morning :) [08:12] pedronis: the only alternative afaict is to use a ppa right now, until a new deb lands in the distro [08:14] pstolowski: it's probably ok to wait for them for feedback first, their build setup is special and may not be suitable for snaps (I honestly don't know). so might be worth to double check before we invest time in this [08:15] btw. trying to find out why spotify core dumps on arch reminded me of the missing feature of having access to debug symbols, that we discussed but never got down to working on it [08:16] mborzecki: there is been work on the snapcraft side on that [08:16] we actually need to play with it [08:17] but not much time atm [08:17] mvo: sure [08:18] pedronis: interesting, is there some info about the snapcraft bits? [08:18] mborzecki: yes [08:18] mborzecki: I can give some pointers in the standup [08:19] pedronis: please do, thanks! [08:39] mvo: I made a comment on #8100, does #8142 need a master merge? [08:39] PR #8100: httputil: add support for extra snapd certs [08:39] PR #8142: client: add support for "ResumeToken", "HeaderPeek" to download [08:42] mborzecki: did you see my remark on #6708 ? [08:43] PR #6708: tests/main/buildmode: verify usability of PIE hardening for deb packages <⛔ Blocked> [08:43] pstolowski: are you waiting on reviews from me atm ? [08:43] pedronis: no. thanks for asking! [08:45] mborzecki: when you have a second could you push master into #8147 ? [08:45] PR #8147: cmd/snap-bootstrap: verify kernel snap is in modeenv before mounting it [08:45] pedronis: sure [08:45] thx [08:58] pedronis: thanks, will followup, 8142 probably needs a master merge [09:28] spread tests should be fixed now in #8156 [09:28] PR #8156: cmd/snap-bootstrap: subcommand to detect UC chooser trigger [09:33] mvo: pedronis: the recovery chooser binary is /usr/lib/snapd/snap-recovery-chooser, does that sound ok? i want to open a PR to subiquity with changes to console-conf-wrapper [09:34] hi, how do I remove a broken (try) snap? it complains because it doesn't find the .service file [09:37] https://paste.ubuntu.com/p/r4H46ycwVq/ is what I get [09:40] oh, it seems umounting the snap before trying to remove it worked [09:54] mborzecki: sorry for the delay, was in a meeting. the name sounds good to me [09:55] ackk: thanks, this feels a bit like a bug on our side, we need to be robust against failures here [09:56] mvo, TBF this was a "snap try" whose prime dir ended up being removed/overwritten, so kinda of an edge case [09:57] ackk: agreed and yet we should handle it better :) but agreed on the edge-case so not that high of a priority right now [09:57] ackk: out of curiosity, did you guys made progress on the run-away supervisor processes? [09:59] mvo, sort of. BjornT experienced something similar installing maas on a rpi4. my gut feeling is that if for some reason there's a high cpu usage and everything slows down, supervisord times out on starting processes, kills and restarts them, making things worse [10:00] mvo, in my case because of squashfuse using all cpu, on the rpi because of limited cpu power [10:00] mvo, there doesn't seem to be any error in logs in these cases, just processes being started and killed [10:00] PR snapd#8181 opened: packaging: revert "work around review-tools and snap-confine" <⛔ Blocked> [10:00] ackk: woah, ok [10:15] mvo: https://github.com/snapcore/snapd/pull/8181 has two patches, only one should be reverted (the other will conflict once https://github.com/snapcore/snapd/pull/8180 lands [10:15] PR #8181: packaging: revert "work around review-tools and snap-confine" <⛔ Blocked> [10:15] PR #8180: tests: rm -rf /tmp/snap.* in restore [10:19] yay, opened https://github.com/CanonicalLtd/subiquity/pull/636 [10:20] PR CanonicalLtd/subiquity#636: bin/console-conf-wrapper: detect whether snapd recovery chooser should run [10:20] mborzecki: looking :) [10:21] i'm thinking that maybe w should have a list of ttys in snapd that the chooser can be invoked on, like /etc/securetty, this can easily be gadget dependent [10:21] mborzecki: looks innocent enough :) [10:21] yeah, it does ;) [10:22] zyga: aha, thanks, feel free to edit and force push, in a meeting righ tnow [10:22] mvo: ack, will do [10:22] /var/lib/snapd/choosertty [10:25] again soemthing with jenkins spread test? [10:26] mborzecki: I saw that today [10:26] their repo times out [10:26] eh [10:26] PR core20#23 opened: hooks: ensure console-conf@ is started after snapd.recovery-chooser-trigger.service [10:27] mborzecki: question [10:27] what happens if that service is disabled? [10:27] namely if you systemctl disable snapd.recovery-chooser-trigger.service [10:27] how does that affect dependency chain? [10:27] zyga: nothing, it's only Before/After ordering, not Requires [10:28] aaah [10:28] that's great [10:28] ok [10:31] mborzecki: wild west idea: a tool that reads from a google spreadsheet, allowing us to put a marker in a specific cell that means jenkins test is skipped [10:31] or that arch or tumbleweed are broken today [10:31] etc [10:31] mborzecki: the sheet could be ACLd and locked [10:36] zyga: we have s3 like storage for out gcp now, we could easily upload a list of disabled tests there, download it to the host, and skip at runtime, then track which run skipped what [10:36] mborzecki: spreadsheets are way easier to work with [10:36] mborzecki: and I bet the ACLs on entire file is not as nice as per-cell ACL [10:38] idk, i prefer plain text files :P [10:38] mborzecki: even changing files in s3-like thing is hard [10:38] and the other thing is self-documenting, you open a URL to a specific cell [10:38] and if you can, you can edit it [10:38] easy [10:46] core 20 setup is sloooooooow [10:48] PR snapd#8133 closed: cmd/snap-confine: deny snap-confine to load nss libs [10:59] PR snapd#8182 opened: build: enable type: snapd <⛔ Blocked> [11:03] zyga: we'll start taking care of some todos that should improve that a bit this week [11:06] that's good to know [11:12] #8176 is simple and green [11:12] PR #8176: tests: adding amazon linux to google backend [11:20] brb, cold and somewhat hungry as a result [11:37] re [11:37] hot tea :) [11:50] did anyone debug why tests/go.{mod,sum} show up [12:00] kids back from school [12:00] also school is back [12:00] whee [12:00] :) [12:13] duh, wonder why nothing is printed out with termbox-go controlling ttyS0 [12:35] mborzecki: maybe it establishes exclusive control over tty? [12:35] there's an ioctl for it [12:35] I used it once [12:35] https://github.com/snapcore/core-build/blob/master/initramfs/testing/init.c#L212 [12:37] mborzecki: some comments and questions on 8156 [13:00] zyga: hm so input works, i'm loogginmg when keys are pressed and so on, but the output does not work [13:00] that's odd [13:01] zyga: are you aware of any mount flag that disables directories bind mounts but won't affect regular file bind mounts? [13:01] cmatsuoka: no, because that's not a property of a mount point [13:02] cmatsuoka: in other words, that's not possible AFAIK [13:02] zyga: I'm having this really weird bug where I can't bind mount directories but files work normally [13:02] I mean, bind mounting files work, but directories fail silently [13:02] cmatsuoka: can you tell me more please [13:02] cmatsuoka: also /proc/self/mountinfo would help [13:02] zyga: ok, so it's a long story. I'll start with the symptoms [13:03] cmatsuoka: sure [13:03] cmatsuoka: do you want to HO or is here find? [13:03] *fine [13:03] in a HO it will be easier to explain [13:04] sure [13:04] I'm in the SU room [13:04] one sec [13:15] cmatsuoka: hasn't xnox made progress (see uc20) [13:16] cmatsuoka: all i can see is that without a valid /etc/crypttab things that require writable mount and bind to it, all get torn down by systemd, because it says dev-mapper-ubuntu-data.device? i don't need that, stop it now. [13:17] cmatsuoka: i'm now redoing to provide a valid /etc/crypttab in the mounted system, to ensure the post-switch-root systemd has left and right half of the brain connected together. [13:21] for reference, we discussed some ideas, including unbindable, and will look at mountinfo data to try to understand what is going on [13:21] xnox: thanks, I think the most interesting piece of information is that switching from a point as late as initrd-cleanup still works [13:22] snap session agent socket activation failed [13:22] https://www.irccloud.com/pastebin/8UEOWyMx/ [13:22] jamesh: ^ [13:27] cmatsuoka: yes, it's the systemd from the rootfs that gets confused. [13:27] cmatsuoka: you can boot with init=/bin/bash => such that switch-root pivots to a bash shell, where everything still looks sane [13:28] xnox: will try that [13:28] mvo: what is happenign here: https://api.travis-ci.org/v3/job/654356068/log.txt there's a SKIP_GOFMT=1 but an error about go fmt ?? [13:30] pedronis: it runs gofmt [13:30] and ignores the result [13:30] that's how I remember it [13:31] (it's kind of weird) [13:31] pedronis: it really failed on FAIL: overlord_test.go:694: overlordSuite.TestEnsureLoopPruneAbortsOld [13:31] pedronis: roughly before the middle of the test log [13:31] ah [13:32] I was confused [13:32] pstolowski: https://api.travis-ci.org/v3/job/654356068/log.txt [13:32] yeah, we should remove that gofmt thing on SKIP_GOFMT [13:32] it's silly [13:32] pedronis: I'll do it later today [13:33] * zyga looks outside at endless rain [13:34] good thing I have tea :) [13:37] xnox: confirmed, it works just after switch root and before init [13:40] zyga: what about SKIP_GOFMT? [13:40] mborzecki: it's really RUN_GOFMT_AND_NOT_FAIL_LOL=1 [13:41] well, that's its purpose [13:41] mborzecki: it's a weird purpose [13:41] mborzecki: it's always noise [13:41] mborzecki: why would you want to look at gofmt from incompatible version of go? [13:41] zyga: not sure i follow [13:41] mborzecki: we set SKIP_GOFMT=1 when we know the result is garbage anyway [13:42] zyga: the problem is that travis builds and cheks with go 1.10, which i don't use locally [13:42] mborzecki: when it is equal to 1 then we _run_ gofmt, print a bunch of diff, and not fail on that [13:42] zyga: right, so that you know what's wrong wrt. 1.10 that travis uses [13:42] mborzecki: are you saying it's always wrong? :) [13:42] zyga: if go didn't break the formatting in 1.11 we wouldn't have the problem, but they did [13:43] mborzecki: anyway, if you look at what the code does now [13:43] mborzecki: I think you will agree that what we do is silly [13:43] mborzecki: we do run the gofmt check with the right go [13:43] mborzecki: (in other places) [13:43] mborzecki: but we always run and print useless diff regardless of anyone asking for it [13:44] as data point I was confused by it and missed the real issue initially [13:44] zyga: but other palces isn't travis unit tests job , maybe it shouldn't check gofmt then [13:45] mborzecki: there's a task that checks gofmt with the right version [13:45] mborzecki: it's not the quick-and-early check [13:45] mborzecki: the quick and early check does run gofmt verification, wasting time, producing confusion and _ignores_ the result because go version mismatch [13:45] mborzecki: that's the problem [13:46] mmh, this was not the early check [13:46] this was unit/go [13:46] afaict [13:46] pedronis: right [13:46] pedronis: because that's the same piece of code [13:46] it runs twice [13:46] zyga: because we run unit tests twice [13:46] I think we should just not run gofmt check at all if SKIP_GOFMT is set [13:46] mborzecki: correct [13:47] zyga: once in travis, then antoher one in spread, the 2nd may run on a new distro where go isn't 1.10 [13:47] mborzecki: that's correct too [13:48] zyga: so shoudl the early check fail, you get a diff of the code wrt. 1.10, and you know what to fix or reformat [13:48] mborzecki: that's not what I'm saying [13:48] mborzecki: unless I'm confused: I think we should just not run gofmt check at all if SKIP_GOFMT is set [13:49] mborzecki: this is sufficient to keep checking exactly what we check now and reduce the confusing diff intermixed with actually relevant unit test failures [13:50] jdstrand: hey [13:50] jdstrand: https://github.com/snapcore/snapd/pull/8123 is ready for a quick re-review [13:50] zyga: ah ok, so yeah, skip the whole thing then [13:50] PR #8123: interfaces/network-control: bring /var/lib/dhcp from host (approach b) [13:50] jdstrand: please check my comments but I think you will find that it implements what you outlined [13:50] zyga: I saw [13:50] well, I saw it was ready for re-review [13:50] jdstrand: I didn't implement the helper that makes the permissions side more understandable [13:50] I haven't gone through it yet [13:50] guten tag friends :-) [13:51] jdstrand: but you indicated that as optional for now [13:51] hey ijohnson :) [13:51] hey jdstrand [13:51] jdstrand: I'll get back to transient scope pr now [13:51] ijohnson: halo :) [13:52] how are you zyga? have a good weekend? [13:52] ijohnson: pretty good, yeah [13:52] ijohnson: winter holidays are over so kids are back to order ;) [13:53] nice, it got really warm here over the weekend (like 5-10 C I think) and we had a bbq at my mom's house, and I found some paczki to share for dessert! [13:53] ijohnson: oh, nice :) [13:53] ijohnson: what's the flavor? [13:54] zyga: it had custard cream in it with some powdered sugar on the outside [13:54] let me see if I can find a picture [13:54] NO :D [13:54] I guess it's a bit "americanized" [13:54] I didn't have lunch yet [13:54] hahaha [13:54] I'll be super hungry all standup :D [13:55] ijohnson: we usually have some jam inside, rose or plum [13:55] ijohnson: and carmelized orange peel on the outside [13:55] https://usercontent.irccloud-cdn.com/file/RhgAaQJe/paczki.png [13:55] (yummy) [13:55] ah that sounds really good [13:55] ^ that's what the ones I found look like [13:55] that's a sizable bag :D [13:55] oh well [13:56] I have some leftovers from dinner to warm up after standup [13:56] (not paczki though) [13:56] :-) [13:57] cmatsuoka: I see the mountinfo data [13:58] pedronis, zyga oh it failed despite the sleep bump last Friday? hmm [13:58] pstolowski: not sure but I think I saw one failure after rebasing [13:59] pstolowski: I don't know the state of the branch that Samuele was looking at [13:59] i see [14:00] cmatsuoka: and when it was in the broken state, what's the mount operation that you tried to perform exactly? [14:00] oh [14:00] standup [14:39] PR snapd#8183 opened: tests: remove tmp dir for snap not-test-snapd-sh on security-private-tmp test [14:41] https://github.com/snapcore/snapd/pull/8180 needs a 2nd review and should be [simple] [14:41] PR #8180: tests: rm -rf /tmp/snap.* in restore [14:43] zyga: ok, responded to PR 8123 [14:43] PR #8123: interfaces/network-control: bring /var/lib/dhcp from host (approach b) [14:43] jdstrand: thank you [14:43] zyga: did you see my comment in PR https://github.com/snapcore/snapd/pull/7731#issuecomment-589829764 [14:43] PR #7731: usersession/userd: add "apt" to the white list of URL schemes handled by xdg-open <⛔ Blocked> [14:44] jdstrand: not yet [14:44] zyga: you might want to see my other comments from friday [14:44] jdstrand: I will, github has some nice new notifications [14:44] I need to start using that again [14:44] zyga: I think we can merge it now, cherry-picking to 2.44 if needed [14:44] (the previous one was just noise and hard to find anything in) [14:45] jdstrand: ack, I'll review both shortly [14:50] meh, my inbox just exploded with snapd edge build failures :( [14:53] yeah [14:53] did you check [14:53] I'm considering checking or having quick lunch ahead of core 20 sync [15:05] does a rabbit qualify as "quick lunch"? [15:05] also - baby oil - which part of the baby does it come from? 🤔 [15:09] roadmr: i literally asked that yesterday [15:12] you were around asking questions about baby oil on #snappy on sunday? [15:14] roadmr: you are on fire today :) [15:14] failure from launchpad builders https://www.irccloud.com/pastebin/i6AZBoZK/ [15:14] and it's only 10:15 am :) [15:14] \o/ [15:17] pstolowski: ^ does that ring a bell [15:18] zyga: have you seen this kind of spread failure ? https://pastebin.ubuntu.com/p/TQ3mkHcQW5/ I've seen this twice now in the past week [15:18] ijohnson: yes, it's fixed [15:18] ijohnson: https://github.com/snapcore/snapd/pull/8180 would help too [15:18] zyga: ack I'll merge master into my PR [15:18]