[01:45] PR snapcraft#2638 closed: remote-build rebased on 3.7 === ogra is now known as Guest89124 [04:44] PR snapd#7149 opened: cmd: add snap model command; daemon: add /v2/model, /v2/model/serial REST APIs === morphis5 is now known as morphis [05:16] morning [05:49] zyga: mount-ns test failed in pedronis' #7147, i've restarted the job, but in case you want to inspect the log it's here: https://paste.ubuntu.com/p/VYHQhc6zhQ/ [05:49] PR #7147: client,cmd/snap: stop depending on status/status-code in the JSON responses in client [06:14] rawhide daily cloud compose doesn't even boot on GCE :/ [06:24] zyga: nvm, i looked at the log and it's /run/netns leaking which should already be fixed in master [06:34] good morning [06:34] mborzecki: I saw a failure last night, something leaks another mounted core [06:35] looking at the log now [06:35] mborzecki: hmm, odd, it is netns [06:35] I'll look [06:35] zyga: btw. that's how rawhide compose boots on GCE: https://paste.ubuntu.com/p/kKMwb6zgFW/ [06:36] mborzecki: quickly :D [06:36] zyga: haha [06:41] I have an idea on how to find broken tests [06:41] But let me get coffee first [06:44] We can use something like HOST.txt to ensure the mount table is not corrupted at each step [06:44] I wish diff had a “first like difference” more [06:44] mode* [06:58] * zyga runs a quick test to look for leaky netns [07:02] hey pedronis, mvo [07:06] zyga: here mount-ns failed: https://api.travis-ci.org/v3/job/562563021/log.txt need to merge master? [07:06] hey zyga [07:06] pedronis: I'm debugging that, I assumed master _was_ merged and that some other test is leaking netns [07:06] pedronis: which PR is this? [07:07] pedronis: (that as in I got pinged by mborzecki earlier today and I'm running a scan of all the tests for netns leaking) [07:07] completely innocent one, the ones that does make(chan error, 1) [07:07] ah, I remember it [07:07] give me 30 minutes please [07:09] we have various red (many for different reasons) [07:12] brb [07:17] re [07:45] hmm, I ran a pass through all tests on Xenial, nothing leaked netns [07:45] * zyga checks the logic [07:45] pedronis: your PR must have the netns unmount patch landed before the mount-ns test landed [07:46] so I must be missing something [07:46] I can try to merge master in it again, if that might help [07:47] zyga: another one failed on mount-ns [07:47] https://api.travis-ci.org/v3/job/562748283/log.txt [07:48] also recent [07:49] oh, that's odd [07:49] something remounted /dev/pts [07:50] different mode, different gid [07:50] oh boy :) [07:50] our tests are doing interesting stuff [07:51] I know snap-confine does things with dev/pts [07:51] I don't see it mentioned in the yaml [07:51] not on the host [07:51] this is the HOST filesystem [07:52] nothing should change that [07:52] zyga: 7148 also fails in mount-s [07:53] zyga: mount-ns with the unified cgroup being "nsdelegate" [07:53] hmmm, perhaps I should switch it to manual [07:53] it seems there are lot of tests that leak here and there [07:53] and running the test dozens of times before was not enough [07:53] ideally I'd run a sequence of (a, mount-ns) for a in all tests [07:53] and fails on ubuntu-core-16 with a bigger diff [07:54] yea, seems premature to have it on [07:57] https://github.com/snapcore/snapd/pull/7150 [07:57] PR #7150: tests: switch mount-ns test to manual [07:57] please merge it [07:58] PR snapd#7150 opened: tests: switch mount-ns test to manual [08:02] at least it's good that we're catching things [08:04] mvo, pedronis do we have a way to use the force-reboot in snapd for non-snap upgrades ... it just struck me that we allow "snap set system someconfigtxt-option foo" on RPi which should theoretically always use a reboot to make the bootloader config change active [08:05] ogra: only upgrades of core or kernels reboot atm [08:05] jdstrand: worked on the idea from your blog post about preparing images for local spread, came up with this cloud init config https://gist.github.com/bboozzoo/f14302d4fc3418200fe28a2d547d3780 [08:05] right, but is it a function that could easily be used here ? [08:05] not without some thinking/design [08:06] it strange to schedule a reboot from just a set [08:06] ah. k [08:06] sure, but in this case it is a bootloader config you are changing [08:07] mvo: you mentioned that Sergio had a branch of spread that can run a sequence without randomised fuss? [08:08] zyga: yes, he spoke about it, you probably can ask him were it lives, don't know if there's PR [08:08] or it's just local to him [08:31] mvo: gentle reminder about #7142 [08:31] PR #7142: boot, o/snapst, o/devicest: limit knowledge of boot vars to boot [08:31] ubuntulog2: ubuntulog3: WAT [08:31] jamesh: ack, I'll look [08:31] zyga: finally running the whole suite locally with fedora rawhide image [08:31] thanks [08:31] jamesh: thank you :) [08:32] zyga: I've updated the icon theme PR now that EnsureTreeState is merged. Once it passes CI, it is probably ready for a real review. [08:32] jamesh: super [08:32] mvo: https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=snapd :-( [08:33] jamesh: thanks [08:33] jamesh: I see what happened there, it's my fault, sorry; I'll re-trigger it after the new test is disabled [08:37] two more buggy tests found [08:45] PR snapd#7151 opened: tests: remove local revision of core [08:45] PR snapd#7152 opened: tests: unmount leftover /run/netns [08:46] Chipaca, pedronis: ^ those are the two leakers [08:47] * zyga looks for more [08:59] ehh, why does spread not include -smp in qemu command line [09:01] mborzecki: I patched my local build but yeah [09:01] mborzecki: also "kvm" is a debian specific thing, I don't have it on suse [09:01] zyga: heh, have a wrapper :P [09:55] PR snapd#7150 closed: tests: switch mount-ns test to manual [10:01] PR snapd#7153 opened: gadget: select the right updater for given structure [10:04] Chipaca: https://github.com/snapcore/snapd/pull/7151/files updated [10:04] PR #7151: tests: remove local revision of core [10:05] found two more leaky tests [10:10] zyga: what happens if you force the mount-ns test to run after everything else? [10:10] Chipaca: how do I force it? [10:10] Chipaca: I didn't try but I'm working around the issue by running a variant of that test in prepare/restore logic [10:10] it triggers instantly on flaky tests [10:11] I'm expanding how much I measure so that I can fix issues one by one rather than see a wall of tests to look at [10:11] zyga: priority: 1 or something? [10:11] Chipaca: uh? [10:11] ah [10:11] no idea [10:11] zyga: priority: -100 [10:11] zyga: default priority is 0 [10:11] zyga: bigger numbers → runs earlier [10:11] I can try [10:11] zyga: negative numbers are supported [10:12] thanks for the idea :) [10:12] this is backwards from what i'd expect of a priority, but it's documented :-) [10:12] zyga: https://github.com/snapcore/spread/#ordering-tasks [10:16] PR snapd#7154 opened: packaging/debian-sid: merge debian upload changes back into master [10:28] PR snapcraft#2637 closed: unit tests: make lifecycle tests more robust [10:30] PR snapd#7152 closed: tests: unmount leftover /run/netns [10:31] PR snapcraft#2539 closed: errors: Add InvalidAppCommand errors for non-existent and not-found [10:32] \0/ [11:07] hm gadget updates is down to 2-3 patches at most now (not counting snap-image) [11:09] PR snapd#7155 opened: tests: remove locally installed core in more tests [11:15] pedronis: https://github.com/snapcore/snapd/pull/7147 is green, merge? [11:15] PR #7147: client,cmd/snap: stop depending on status/status-code in the JSON responses in client [11:16] mborzecki: yes, please [11:16] PR snapd#7147 closed: client,cmd/snap: stop depending on status/status-code in the JSON responses in client [11:25] thx [11:26] 1hr18min travis run [11:26] go, worker, go! [11:26] * Chipaca ⇝ lunch [12:06] anyone seen this? https://paste.ubuntu.com/p/4KyGXgvXpy/ initialize-device failing on f31 in some weird way [12:08] why would f31 (assuming classic) do initialize-device at all ? [12:08] ogra: ? [12:09] isnt that a core thing ? [12:09] no, it's classic too [12:09] oh, i'm mixing it up ... [12:09] sorry [12:09] both seeding and registration exist both on classic and core [12:10] yeah [12:13] mborzecki, pedronis: is that sensitive to gpg on the system? [12:13] PR snapcraft#2639 opened: deprecations: add deprecation notice for version-script (dn10) [12:14] zyga: no, we don't use gpg on prod system, only the development tooling [12:14] uses it [12:14] we use ssh-keygen though [12:14] I think [12:15] aha [12:17] a 2nd review of #7128 would be great (it's small but might need somebody that looked at remodeling code before) [12:17] PR #7128: overlord: DeviceCtx must find the remodel context for a remodel change [12:24] PR # closed: snapd#7142, snapd#7143, snapd#7145, snapd#7148 [12:31] PR snapcraft#2640 opened: cli: improve help for push-metadata [12:34] PR snapcraft#2641 opened: ant plugin: correct default channel and improve help [12:53] pedronis: thanks for the review on 7149 === alan_g_ is now known as alan_g [12:54] pedronis: do you have any recommendations on how to test actually remodelling? is there an easy way to do so on classic, or should I build my own core image with a self signed model assertion and then update the model assertion? [12:55] ijohnson: it's complicated in both cases, we don't let remodel devices that don't have a serial [12:57] well that might be overstating it, but interesting kind of remodeling probably don't work or shouldn't without a serial [13:01] ijohnson: btw regarding output of snap commands, Chipaca is probably the best person to discuss things with [13:01] (related to my yaml-like output comments) [13:01] ack, thanks [13:04] mvo: hey, could you (assign someone to?) respond to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932815 [13:04] not sure why buster wasn't updated for a newer snapd 2.38 and don't know the plans, otherwise I would've [13:04] s/a newer/the newer/ [13:09] mvo: btw, wdyt? https://bugs.launchpad.net/snapd/+bug/1837460 [13:09] mvo: (post-standup) [13:09] Bug #1837460: snap refresh slows down computer dramatically [13:11] ackk: I haven't circled back around to https://launchpad.net/bugs/1831473, but should today [13:12] Bug #1831473: Can't run /usr/bin/systemd-detect-virt from inside snap [13:12] jdstrand, hi, thanks! [13:13] jdstrand, btw is there a way to strace a process run from a "snap run --shell" ? [13:13] jdstrand, I've resorted to creating a shell script that basically prints the pid, calls "read" and then execs the actual command, so I can attach with strace, but it's a bit cluncky :) [13:19] ackk: no [13:19] ackk: because by the time you're in the shell inside you can't strace [13:26] PR snapd#7154 closed: packaging/debian-sid: merge debian upload changes back into master [13:41] jdstrand: I uploaded 2.40 to unstable now [13:41] jdstrand: but that does not help, does it? [13:42] Chipaca: oh, interessting [13:42] PR snapd#7156 opened: packaging/debian: use correct version for apparmor Depends [13:47] PR snapd#7156 closed: packaging/debian: use correct version for apparmor Depends [13:51] mvo: it doesn't help buster, no [13:51] PR snapd#7157 opened: packaging/debian-sid: use correct apparmor Depends for Debian [13:51] mvo: fyi, https://github.com/snapcore/snapd/pull/7157 [13:51] PR #7157: packaging/debian-sid: use correct apparmor Depends for Debian [13:51] brb [13:52] mvo: (unrelated to that Debian bug. just something I noticed was incorrect [13:52] ) [13:52] jdstrand: nice [13:53] zyga to use the spread I sent you just use [13:54] do -> spread -order [13:54] this will execute the tests following the order in the order list [13:54] also you should use -workers 1 [13:55] this will make to use 1 worker so you don't need to update hte spread.yaml [13:55] please, tell me if you have any problem [13:59] Chipaca, I see so I guess the script trick is the only way [14:07] re [14:07] cachio: thanks! is your patch close to being merged usptream? [14:07] zyga, no [14:08] oh, why not? [14:08] zyga, need reviews [14:08] there are 2 PR related to this [14:08] understood [14:08] 1 for the order [14:08] 1 for setting the number of workers [14:09] also you can use -show-output if you want to see the output on real time [14:10] zyga, once we are able to run spread tests on google on any PR it will be really easy to add tests for all these features and make sure they work well [14:11] cachio: ping us for reviews [14:11] I'm sure both me and mborzecki would happily review that [14:12] zyga, ok, once I add the tests for validating those features I'll ping you for a review for sure [14:12] mvo, any news about the spread tests on gce? [14:12] is the key enabled? [14:22] * zyga takes a break while tests run [14:25] PR snapd#7088 closed: tests: manually stop the gvfsd-metadata process [14:28] zyga: running test 72/353 [14:31] 7/350 [14:34] =0.02 [15:17] PR snapd#7146 closed: UC20: cmd/snap-verify: sketch of snap-verify [15:23] PR snapd#7146 opened: UC20: cmd/snap-verify: sketch of snap-verify [15:24] did we ever get back to #6679 [15:24] ? [15:24] PR #6679: many: implement user removal [15:29] PR snapd#7158 opened: tests: part5 making tests work on ubuntu-core-18 [15:30] * cachio lunch [15:48] ARGH ! [15:48] so multipass eats 100% CPU after a reboot of my laptop ... since i dont use it anyway (i did a few times though and there are a few GB of images apparently) i thought i'D remove it ... [15:49] ogra@acheron:~$ snap remove multipass [15:49] Save data of snap "multipass" in automatic snapshot set #1 \error: cannot perform the following tasks: [15:49] - Save data of snap "multipass" in automatic snapshot set #1 (tar failed: context canceled) [15:49] ogra@acheron:~$ snap remove --purge multipass [15:49] error: unknown flag `purge' [15:49] (indeed i ctrl-C'ed the first command) [15:49] how do i proceed to get it gone ? [16:09] Chipaca: not yet, it was preempted by core20 [16:10] ogra: refresh core snap to edge [16:10] then you have --purge [16:10] otherwise you can set the core config option to never save snapshots [16:11] snap set system snapshots.automatic.retention=no [16:13] ijohnson, ah, thanks ... [16:14] * ogra does the latter [16:25] * zyga runs some more tests with more checks [17:11] * zyga is upset a bit [17:11] checking again [17:11] PR snapcraft#2642 opened: remote-build: detect early build errors [17:59] yess [17:59] finally [18:00] PR snapd#7157 closed: packaging/debian-sid: use correct apparmor Depends for Debian [18:40] mborzecki: making good progress here :) [18:41] mborzecki: cannot wait to see this pass everything [18:41] zyga: hm? hm? [18:41] mborzecki: iterating on the dirty test detector, got it in the right spot now, fixing tests as I go :) [18:41] mborzecki: even the diff it produces is super readable [18:41] zyga: aah [18:42] zyga: btw. looking at spread logs, lxd seems to have problems with cgroupsv2 [18:43] mborzecki: would be worth asking stgraber [18:44] zyga: https://paste.ubuntu.com/p/Cz7qszmxPD/ [18:45] IO error [18:45] could be anything [18:45] but yeah, worth looking [18:52] zyga, hey, do you know any way to do iptables on core? [18:52] is there a snap for that? [18:52] cachio: I don't know, perhap with one of the newer tools [18:52] but I haven't used any for a very long time [18:53] nftables? not sure [18:54] zyga, I'll try that [18:54] thanks [18:59] PR snapcraft#2641 closed: ant plugin: correct default channel and improve help [20:00] cyphermox: quick question about "semaphoreci", I pushed pr#93 to netplan but it needs a new build-dependency - can I add this somehow myself (i.e. is there something equivalent to .travis.yml) [20:17] PR snapcraft#2640 closed: cli: improve help for push-metadata [20:50] 419/426 [20:50] all passed [20:54] * zyga tweaks the detector to be more strict and re-runs [20:54] I think now all tests are 100% good in not leaking mounts [20:54] but we'll see [21:21] PR snapd#7159 opened: tests: add functions to make an abstraction for the snaps [22:51] PR snapcraft#2643 opened: project, cli: clean up snap asset messages