[03:32] PR snapd#8208 opened: boot_test: add many boot robustness tests for UC20 kernel MarkSuccessul and SetNextBoot [06:40] o/ [08:00] mvo, morning! [08:00] good morning abeato [08:01] mvo, one question, which are the bootloader requirements for UC20? is there any documentation on that? [08:01] morning [08:03] abeato: they are roughly the same as before, ideally the bootloader should be able to chainload but I think we could work without that (it means less robustness though) [08:04] mvo: good morning [08:04] hey zyga [08:04] mvo: I'll be partially off today, need to get haircut and finish that pesky travel thing [08:04] mvo: my goal for today is to publish what I did for sessions and work on prompting demo [08:04] mvo: question [08:04] mvo: last evening pedronis and I noticed a store review block on snapd [08:05] mvo: due to type [08:05] mvo: was that the auto-build? [08:05] mvo: or something new [08:05] mvo, so, basically the same modifications that we need to do today (for u-boot for instance)? [08:05] zyga: an auto-build of 2.43 [08:05] ok [08:05] zyga: but the snap was removed from the queue, no? [08:06] abeato: yes, why do you ask ? [08:06] mvo: I didn't check beyond that [08:06] I'm very tired lately, I try to sleep as much as I can [08:06] abeato: what loader do you have in mind :) [08:06] mvo, customers are asking :) [08:06] abeato: ok [08:06] zyga: good luck, yeah, I feel also not great currently, very strange [08:06] mvo, same bootloaders as usual, but I did not know if UC20 required something new [08:07] abeato: note that core 20 is x86 driven now, arm work will happen later, my suspicion is that for full feature set you may need a better bootloader [08:08] abeato: but I think it's just speculation at this stage [08:08] (or more feature integration) [08:08] abeato: foundations is working on a port of uboot to uc20 currently [08:08] abeato: we willl discuss in FRA what the status is here [08:08] abeato: no plans for lk or anrdoidboot so far though [08:09] sil2100, do you know which is the status of that? ^^ [08:10] abeato: hey! I don't have a full update on that, waveform is working on the raspi port - but I know there was progress last time I checked [08:10] mvo, hm, lk (well, a fork of it, but we use lk support in snapd) is used by nvidia boards which seem to be polular [08:11] abeato: yeah, I'm sure eventually we get lk support it's just not a priority right now [08:12] mvo, ok, thanks for the info [08:13] * zyga feels the urge to start using slack already [08:14] mvo: hi, in #8206 even the amd64 tests, broke ? [08:14] PR #8206: travis.yml: run unit tests on arm64 as well [08:15] pedronis: yeah, it's confusing, I thing I got the syntax for the matrix wrong in some way [08:15] pedronis: I'm looking at the network-retry failure first, then I will get back to this one [08:15] mvo: it's fine, to be understand why that one started failing just now ? [08:16] s/to be/do we/ [08:19] pedronis: getent hosts www.ubuntu.com returns ipv6 now [08:19] pedronis: and the script naively uses "http://$ip" which is invalid for ipv6 of course [08:19] ah [08:19] can't we fix the script ? [08:20] pedronis: it's still a bit puzzling, we could force it to just use ipv4 but I wonder if ipv6 gives a useful error [08:20] yes, same here [08:20] I sort would prefer to test with boths [08:20] pedronis: exactly [08:20] zyga, do you have a pointer to the grub config that will be used for UC20? [08:20] and teach the script to deal with both kind of addresses [08:20] pedronis: that was what I'm playing with right now [08:20] ok, great [08:20] pedronis: the error for connect ipv6 is different but maybe I'm holding it wrong [08:21] pedronis: I litterally got there 20s before we started talking :) [08:21] abeato: I'm not closely involved in uc20 work, sorry [08:22] abeato: I can give you pointers, there are two configs involved (they will move inside snapd soon), but at the moment they are in the gadget [08:22] one sec [08:22] k [08:23] pedronis: meh, it really looks like ipv6 gives a very different net.OpError when it cannot connect, that feels a bit of a rathole "fun" [08:25] abeato: here, https://github.com/snapcore/pc-amd64-gadget grub.cfg-{recovery,boot}, recovery is the first stage, boot is the 2nd used in run mode (vs the various recovery mode of UC20) [08:26] pedronis, great, thanks for the links [08:57] mvo: the fun never ends [09:06] morning [09:07] mborzecki: good morning [09:08] PR snapd#8209 opened: tests: use ipv4 in retry-network to unblock failing master [09:08] mvo: hey, anything interesting going on this morning? [09:12] mborzecki: master red again but beside this not really [09:12] 8209 should unbreak us and I'm looking at an equivalent ipv6 test right now [09:18] mvo: interesting that ipv6 thing [09:18] for me it shows ipv6 on xenial [09:18] specifically [09:18] https://www.irccloud.com/pastebin/etKNfVuq/ [09:18] there are no ipv4 addresses at all [09:18] is that a change in the system config or a change in our domain name? [09:19] notably this xenial machine does not have an ipv6 global address [09:23] zyga: same here, ipv6 only ;) [09:24] mborzecki: it's the future ;) [09:25] yeah, maybe it's a global change, no idea [09:25] zyga: hm maybe the response for AAAA comes back faster than for A? [09:25] dunno, I would have expected the whole response [09:25] it's a DNS query [09:25] you ask for stuff [09:25] or AAAA are queried first [09:25] you get stuff [09:25] (records) [09:26] https://www.irccloud.com/pastebin/4KWXTfnL/ [09:26] check this out [09:27] curious [09:27] dig ANY website-content-cache.canonical.com [09:27] that's giving me AAAA records [09:27] but dig ANY ubuntu.com does not [09:27] but hostent gives both? [09:27] idk [09:27] er, hosts [09:27] maybe the future is now [09:27] the future is broken now [09:31] zyga: actually quite broken, those addresses returned by getent are not really usable, i only got private and link local addresses on my interfaces [09:31] yeah, same here [09:31] I wonder if IS changed something [09:31] or if ubuntu changed something in xenial [09:31] looking at dpkg.log [09:32] nothing, it's pretty much unchanged except installs of local test packages [09:32] imo it's a problem with glibc [09:33] unless i'm misremembering how things work with ipv6 [09:33] I can ask in #is [09:35] mborzecki, zyga asking our is folks [09:35] thank you [09:38] definitely the year of linux desktop ;) [09:39] zyga: btw. try `getent hosts localhost` [09:40] I get ::1 [09:40] that's so weird [09:40] feels broken :) [09:41] zyga: wonder where the preference is encoded, at least i can ping ::1 [09:42] offtopic, sun and snow / hail oscillate every 10-15 minutes [09:42] sunlight with blue sky [09:42] then dark gray and lots of snow [09:42] what a day! [09:58] PR snapd#8210 opened: wrappers: add mount unit dependency for snapd services on core devices [09:59] mvo: ^^ do you think this is 2.44 worthy? [10:03] mborzecki: hmm [10:03] the patch looks buggy [10:03] you have [Unit] and then immediately [Service] [10:03] in the past [Unit] had ExecStart [10:03] is that expected? [10:07] zyga: yeah, ExecStart is part of [Service], so i updated the tests to make them closer to real unit definitions [10:07] aha [10:07] how was it working before? [10:07] compatiblity in systemd? [10:08] zyga: no, it's just the test data, the actual services under data/systemd are correct and have ExecStart in [Service] [10:08] ahhh [10:08] ok, that clears up the mystery ;) [10:08] thanks! [10:09] mborzecki: probably, please tag [10:10] PR snapd#8209 closed: tests: use ipv4 in retry-network to unblock failing master [10:15] PR snapd#8211 opened: tests: run ipv6 network-retry test too [10:16] PR snapd#8207 closed: tests: fix retry network test after image update [10:22] mvo: tagged [10:27] hmm new kernel [10:27] brb [10:34] and running 5.5.6 now [10:44] posh [10:53] mvo: FYI, 14.04 cnf suggests "snap" for "snap" [10:53] could lead to some confusioin [10:54] mvo: what's the state of ESM snapd build? [10:54] I'm testing on 14.04 as well, could get it to see if anything odd pops up [11:01] zyga: it's build in staging [11:55] * zyga -> lunch [12:11] back, [12:18] zyga: can you remind me, when we launch snap-update-ns it's started using fexecve right? [12:18] correct [12:18] we launch all our tools this way [12:18] any issues? [12:18] zyga: trying to update the centos 8 and there's weird selinux denial, let me copy that [12:19] zyga: https://paste.ubuntu.com/p/nDKbryS9tv/ [12:19] zyga: i think it's ld.so tryign to poke ld cache [12:19] mborzecki: yeah [12:19] I suspect so [12:20] * zyga finds selinux gibberish-y [12:20] today I learned one cool feature of dbus is not supported in busctl: https://github.com/systemd/systemd/issues/14954 [12:21] zyga: ha, weird why busctl doesn't support that [12:21] it's just not coded [12:21] there's an error branch and thats's that' [12:21] well, maybe nobody tried it with busctl yet [12:22] zyga: suppose there's limited use case consideing busctl is a command line tool [12:23] mborzecki: shell is still quite versatile, plus you could open python an pass a fd this way :) [12:23] zyga: you can, but should you? [12:23] yes :) [12:24] better than python bindings and in general even bash is sufficient to handle FDs [12:25] zyga: hm why is even snap-update-ns invoked when there's no mount profile? [12:26] mborzecki: for several years it is used to construct one [12:26] zyga: hm need to go through that code again [12:27] zyga: i keep evicing all state info afer a while :P [12:30] zyga: hm still wodnering why this isn't picked up by the other selinux specific tests, jsut this one [12:30] mborzecki: probably because that test is fairly limited [12:30] mborzecki: check out the aggregated denials bug on lauchpad [12:30] mborzecki: or enable robust mount ns [12:30] hm, ubuntu-core-18-64:tests/main/revert:remote seems to be failing currently, I wonder if it's a one-off or a new trend :( [12:30] which exercises the same code [12:30] mborzecki: there's a PR from me about that, it fails on a dozen issues [12:30] mborzecki: I bet they are the same [12:30] mborzecki: they routinely happen in normal use [12:31] mvo: how does it fail? [12:31] it fails due to unexpected denials in the log [12:31] haaa and those happen on remove [12:31] zyga: it says snapd wants to reboot the system [12:32] mvo: huh [12:32] * zyga is overwhelmed by complexity of snapd state handling sometimes [12:33] mvo: how does it fail? [12:33] we haven't changed anything in that area recently [12:33] pedronis: yeah, it looks like there is a pc-kernel refresh that happend just now [12:33] pedronis: so the test runs "snap refresh" and that triggered a pc-kernel refresh :/ [12:34] ayy [12:34] mvo: ah [12:34] fun [12:34] all tests doing snap refresh like that are fragile [12:34] yeah [12:34] until we add holding for single snaps [12:36] * mvo is slightly depressed about this and gets lunch [12:38] mvo: heh [12:38] mvo: anyway we might get some extra test failures when the test snaps get moved as well [12:41] hmmmm [12:42] so when removing a snap and disconnecting the interfaces, we keep recompiling the seccomp profiles on each disconnect [12:42] feels like wasted cpu cycles [12:42] zyga: this also triggers snap-update-ns ^^ [12:43] mborzecki: yeah, I think that's expected in our current design [12:43] mborzecki: we only have one optimization, for mass auto-connect [12:44] mborzecki: we should remember it, but it doesn't sound like something to address right now [12:44] (too many other things) [12:44] pedronis: right, i'll collect a some notes and open a forum topic [12:47] cachio: it seems that tests with the tpm-enabled branch are successfully running on the google-tpm backend, but it's very, very slow [12:49] pedronis: hey, since there were 2 force pushes in as many days for just reviews that I did, I wonder if it makes sense where when you first submit a PR in the section about the CLA if there should be something about force pushes [12:49] that sentence could've used a couple commas === wgrant is now known as wgrant_ === wgrant_ is now known as wgrant [13:11] cmatsuoka, ahh, ok [13:11] do you have a branch to use [13:12] perhaps I can improve it [13:13] cachio: I'm currently testing with the frankfurt-preview-tpm branch in my tree [13:14] cachio: it consolidates all encryption-related branches on top of the current master [13:18] cmatsuoka, nice, thanks [13:18] cmatsuoka, is it everything slow? [13:18] or just a specific test? [13:19] cachio: I've been running it for something like 1h now and it's still at 100/447 [13:19] cmatsuoka, ok [13:21] cachio: also I can't tell if tpm was actually used, if you can shell into the test machine you can check if there's a /writable/system-data/ubuntu-data.keyfile.sealed there. If this file exists the system booted with tpm enabled [13:27] cmatsuoka, nice [13:28] cachio: mvo: the test snaps have been transfered, let's see if something goes wrong with some tests [13:31] PR snapd#8202 closed: cmd/snap-bootstrap,seed: verify only in-play snaps [13:32] ^ this might speed UC20 install/boot a little bit [13:32] *speed up [13:33] cmatsuoka: how many machines are you using? [13:35] the tpm systems have one worker, and our runes take 40 minutes with 4-6 workers [13:35] s/runes/runs/ [13:36] mvo: #8083 should be fine now, i've tagged it with 2.44 since there's a tiny policy update [13:36] PR #8083: spread, data/selinux: add CentOS 8, update policy [13:38] mborzecki: thank you [13:50] pedronis, running tests right now [13:50] cmatsuoka, hey [13:50] how many workers are you using? [13:51] because by default the backend is using 1 [13:51] cachio: oh noes ok [13:52] cmatsuoka: our runes take ~40mins with 4-6 workers [13:52] runs [13:52] cachio: I thought it was using the same number as the rest, but now it's almost finishing [13:53] cachio: ah, it finished. With 279 tasks aborted [14:47] mborzecki: can you check this on your box [14:47] zyga: hm? [14:47] https://www.irccloud.com/pastebin/N9YISpKV/ [14:48] mborzecki: it should behave like "sudo" [14:48] mborzecki: it has -u for picking the user [14:49] mborzecki: it's a bit of a hack [14:51] hmmm [14:51] weird [14:51] zyga: https://paste.ubuntu.com/p/pPfN4F34SK/ [14:51] it still has some warts [14:52] or one annoying one [15:10] PR snapd#8212 opened: tests: updating checks to new test account for snapd-test snaps [15:25] * zyga break [15:33] mmh, github is having issues for me [15:40] * cachio lunch [15:42] yea, status gives degraded hard to do reviews :/ [15:52] pedronis: sorry had my 1:1, but I'm editing the docs now, are you okay if I just make the edits directly or do you want me to propose changes with the suggestions thing on the doc? [15:52] (edits for what's no longer 100% accurate) [15:53] ijohnson: probably just do the edits, unless there is something you are unsure about, in any case they are not that long, I can re-reread them after. Also remove/resolve any stale discussions/comments bits [15:54] pedronis: ack [15:54] thank you [16:05] pedronis: what does the green highlighting mean again? does that mean we already did it? if so perhaps we remove all the green highlighting and just use light red to mean "not done" ? [16:05] or I guess the official name "light green 3" [16:07] what are the usual suspects for seeing "broken" in the notes section for a snap? [16:08] i have lxd (stable) installed, but recently it seemed to break (after dist-upgrade?) and I'm getting errors like apparmor="DENIED" operation="exec" profile="snap.lxd.lxc" name="/usr/sbin/aa-exec" pid=5159 comm="lxc" requested_mask="x" denied_mask="x" fsuid=1000 ouid=0. notes section says "broken" [16:10] pedronis: anyways I made all the edits to update those 2 docs, I'm going to work on a section in the booting" [16:10] I'm going to work on a section "booting" doc to put details about what we discussed this morning about [16:11] cjp256: do you have a "current" symlink in /snap/lxd ? [16:11] yeah, rev 13487, which doesn't actually match the "installed" version according to snap info... [16:12] cjp256: also what about the snap mount unit in systemd `systemctl status snap-lxd-*.mount` ? [16:12] active, mounted 13487 [16:13] snap list shows 13522 as being installed though...? [16:14] do you only have 1 mount unit in that command above ? [16:14] For me, I have 2 mount units, 13487 and 13522 [16:14] yeah snap-lxd-13487.mount - Mount unit for lxd, revision 13487 [16:14] hmm so that's why it's broken, snapd wanted to install 13522, but for some reason you never got a mount unit for it [16:15] it -> that revision [16:15] so in other news, irccloud also started acting up [16:15] apparmor="DENIED" operation="symlink" profile="snap.irccloud.irccloud" name="/home/chris/snap/irccloud/28/.config/ibus/bus" pid=10991 comm="ln" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000 [16:15] what does `snap list lxd --all` show? do you have other revisions listed there (perhaps ones that are disabled) ? [16:16] 13487 "disabled" 13522 "broken" [16:16] cjp256: does irccloud have the desktop-legacy interface connected? === jdstrand_ is now known as jdstrand [16:17] try `snap revert lxd --revision 13487` [16:17] jdstrand: `desktop-legacy irccloud:desktop-legacy :desktop-legacy -` [16:18] cjp256: oh, that is down in SNAP_USER_DATA anyway. that is probably related to a refresh [16:18] ijohnson: that worked like a champ! :D [16:18] cjp256: guessing current is pointing to !28 ? [16:18] now you could try `snap refresh` of lxd again and see how it goes [16:19] well that's interesting [16:19] cjp256: do you have experimental.refresh-app-awareness enabled ? [16:19] irccloud had a similar problem [16:19] --all shows 24 "diabled,broken", and 27 "-" [16:19] ijohnson: yes, green meant done afaiu (sorry was in a meeting) [16:19] but current is a symlink to 28 [16:20] pedronis: thanks [16:20] ijohnson: yes! [16:20] ijohnson: and yes, if it's mostly done, I would remove it and just use read for missing bits [16:20] cool [16:20] s/read/red/ [16:23] ijohnson/jdstrand: though in irccloud's case, a `snap refresh irccloud --stable` did restore order, current is pointed back to 26 [16:23] ijohnson: I'll look at docs now, given that reviewing is a bit not working on github atm [16:24] cjp256: what's `snap changes irccloud`? [16:24] just `143 Done today at 11:21 EST today at 11:21 EST Refresh "irccloud" snap` [16:24] pedronis: ack there is one thing I'm unsure about is whether there's a modeenv in the image when we build it, I had a comment in the doc from a while ago that we were writing one there but perhaps we aren't and it was something else [16:25] ijohnson: i did reboot earlier to see if that resolves the problem, i don't know if changes logs prior boots [16:25] pedronis: the issue that made me write that was that I had a too old `snap prepare-image` and the image I was writing (even though it was writing a newer snapd into the image) wouldn't finish boot [16:25] It got stuck in install mode [16:25] cjp256: no reboots don't clear changes [16:26] cjp256: not sure what caused your problems but it sounded a lot like the ibus input thing jdstrand solved a while ago on irccloud and some bad refresh on the lxd snap [16:27] ijohnson: we don't, modeenv exists only in ubuntu-data and there is no ubuntu-data in UC20 images built (at least for now) [16:28] pedronis: right, then I guess I will delete that, not sure what the problem was with my image a while ago but probably was an artifact of many things changing at once [16:28] ijohnson: now that I have a good idea of what may be happening I can dig in a bit further next time should it happen again... thanks! [16:28] np [16:31] ijohnson: what are the yellow highlights, were they preexisting? [16:31] (me doesn't remember) [16:31] I'm looking at bootenvs right now [16:31] pedronis: yes you added those one day [16:32] pedronis: perhaps we should consolidate the info in that highlighted section and unhighlight [16:32] it [16:32] mmm seems I missed some things in the bootenvs doc [16:34] spotify's UI freezes after sometime [16:34] does that happen to any of you? [16:35] I had also tried reinstalling but issue still exists [16:36] If someone can confirm I can try to downgrade [16:41] ijohnson: I made some suggestions, comments and marked some not yet done bits in light red in the boot envs one [16:41] ijohnson: please review [16:41] pedronis: ok looking [16:43] cachio: the encryption test is failing with google-tpm (/sys/kernel/security/tpm0/binary_bios_measurements: no such file or directory) [16:43] fellows is there a bug reporting place somewhere for a snap? [16:44] cachio: I checked with Chris and the likely cause is secure boot not enabled [16:45] murthy: snap info spotify - the contact field [16:45] cachio: are you using the snakeoil nvram file (OVMF_VARS.snakeoil.fd)? [16:46] pstolowski: checking [16:48] murthy: it seems there is an issue similiar to yours https://community.spotify.com/t5/Desktop-Linux/Spotify-GUI-freezes-in-Ubuntu-19-10/td-p/4900983 [16:51] pstolowski: Ya, I just saw that [16:51] pstolowski: hope they fix it soon [16:51] pstolowski: thanks for the help [16:51] yw [16:51] ijohnson: I did a pass on half of booting, the update bits need some tidying up still [16:52] ijohnson: maybe copy some of the comments from bootstate20.go, not sure [16:52] pedronis: ok doing that now [16:53] ijohnson: I left a comment about were I stopped for now [16:53] ijohnson: would like to eod a bit earlier today [16:53] I will still try to post my reviews later tonight if github behaves [16:53] pedronis: of course, we can look at it all again tomorrow [16:53] ijohnson: thank you [16:58] ijohnson: to be clear, feel free to accept or tweak my suggestions in the two docs [16:58] * pedronis eods [16:58] yes [16:58] have a good evening pedronis [16:59] bye pedronis [17:05] mvo, is it ok https://paste.ubuntu.com/p/HqQP9Pfbhd/ ? [17:06] mvo, because this is making fail the fix for the create user 2 test [17:07] cachio: this smeels like a bug [17:07] cachio: if you run "snap managed" after "snap remove-user", does it report true? [17:07] mvo, exactly [17:08] and initially was false [17:10] cachio: do you have a shell there? [17:10] cachio: the state has all the info we need to grok that [17:10] zyga, sure, 1 second [17:11] cachio: meh, that sounds like a bug, quite annoying [17:11] mvo, zyga https://paste.ubuntu.com/p/Q4D8nM5Hth/ [17:12] cachio: cool, it's even up front [17:12] mvo is logged in :) [17:12] mvo, the good part is that we found it [17:12] mvo: you own plenty of devices :) [17:12] mvo: is that master or release? [17:14] PR snapd#6708 closed: tests/main/buildmode: verify usability of PIE hardening for deb packages <⛔ Blocked> [17:15] zyga: master [17:15]