[00:31] PR snapd#7151 closed: tests: remove local revision of core [02:21] PR snapcraft#2644 opened: Release changelog for 2.44 [05:09] morning === morphis5 is now known as morphis [06:48] mvo: morning [06:50] hey mborzecki [06:50] mborzecki: lots of red PRs this morning it seems, anything in particular unhappy in the tests? [06:54] mvo: nope, nothing in particular [06:55] mvo: on the interesting-test-failures note, yesterday saw unit tests fail on travis (iirc it was daemon or cmd/snapd package) bc the port we tried to liste on was already busy [06:56] mvo: oh, and the lxd spread test failed in a weird way with cgroupv2 (AFAIK it's supported by lxd 3.x) [06:56] mborzecki: interessting [06:58] mvo: anyways, woke up with some ideas to try out on F31, will discuss those with zyga again probably :) [07:44] hello [07:44] sorry for being late, stopped at 3AM [07:44] hey zyga [07:44] hey :) [07:44] finishing coffee [07:44] I'm eager to run downstairs to see what the test results say [07:45] https://github.com/snapcore/snapd/pull/7155 <- lets merge it [07:45] PR #7155: tests: remove locally installed core in more tests [07:45] it's the same pattern as before, that got reviewed === morphis4 is now known as morphis [07:46] see you at 10 [07:46] I'll get prepped [07:59] zyga: do you think you could have a look at https://github.com/snapcore/core/pull/105 ? [07:59] PR core#105: hooks: empty the configure hook [07:59] zyga: should be pretty easy [07:59] `looking [08:00] mvo: done [08:00] mvo: let's merge https://github.com/snapcore/snapd/pull/7155 [08:00] PR #7155: tests: remove locally installed core in more tests [08:01] PR core#105 closed: hooks: empty the configure hook [08:13] PR snapd#7155 closed: tests: remove locally installed core in more tests [08:25] PR core-build#48 opened: config: remove static files in etc,lib,usr,var [08:46] PR snapd#7160 opened: tests: reformat and fix markdown in snapd-state.md [08:49] a few trivial from my pool [08:49] PR snapd#7161 opened: tests: show stderr only if it exists [08:54] uff 29C in the morning [08:57] pro tip: if you're used to running in the evening, getting up early to run in the morning because the evening is not going to drop below 30 is not necessarily a good plan [08:57] mo'in all, sorry i'm late [08:57] mvo: is https://github.com/CanonicalLtd/netplan/compare/master...mvo5:dbus#diff-a084b794bc0759e7a6b77810e01874f2R2 something I can review in a PR? [08:57] Chipaca: hey :) [08:58] zyga: https://github.com/CanonicalLtd/netplan/pull/93 [08:58] Chipaca: heat wave all around [08:58] PR CanonicalLtd/netplan#93: add dbus based "netplan apply" interface [08:58] zyga: that is the final one [08:58] it reached us now too [08:58] mvo: thank you [08:58] Chipaca: running> yeah heat is bad [08:59] oh, my takeaway is that the morning is bad :-0 [08:59] * zyga wonders what https://semaphoreci.com/cyphermox/netplan is [08:59] Chipaca: yesterday and the day before the other side of my dsl line got a heat stroke at around 16:30. and it takes ages to talk to someone on the telco to reset it :/ [08:59] as in semaphore [08:59] I marked a bunch of my PRs for 2.41 [08:59] pedronis: thank you [09:00] I mean ideally we get all in, but those are the important ones (because they fix something) [09:00] mvo: if you can turn off your modem for ~5 minutes the port you're connected to gets returned to the pool, and you've got a chance of getting a new one [09:00] mvo: there is https://github.com/CanonicalLtd/netplan/blob/master/rpm/netplan.spec but I don't know if that is used for CI [09:00] mvo: also snap one https://github.com/CanonicalLtd/netplan/blob/master/snap/snapcraft.yaml [09:01] Chipaca: oh? that is interessting,I will try that today at ~16:30 when the line goes down again [09:01] mvo: I think james #6954 is ready for merging, do you want to give a final look to the packaging/service bits? [09:01] PR #6954: sessionagent: add a REST interface with socket activation [09:01] zyga: yeah, they use something that I'm not familar with (semaphoreci) - no idea how to add deps there [09:02] pedronis: sure, let me check that [09:02] mvo: OTOH it might be your ISP telling you to work on your work/life balance [09:02] Chipaca: hahaha - do you have your hands in this ;) ? [09:02] * Chipaca whistles innocently [09:09] mvo: I think I got the packaging right in that PR (mostly just followed existing examples), but an extra set of eyes is welcome [09:09] Chipaca: can you look at #7149 when you have a bit of time, especially output formatting in the command and api bits [09:09] jamesh: happy to do this [09:10] PR #7149: [WIP] cmd: add snap model command; daemon: add /v2/model, /v2/model/serial REST APIs [09:10] pedronis: i have time now [09:10] * Chipaca looks [09:11] jamesh: do you happen to know if there is a way to exit dbus services gracefully (and racefree) when they are not needed? context is https://github.com/CanonicalLtd/netplan/pull/93 which is a very simple service that I would like to stop if idle [09:11] PR CanonicalLtd/netplan#93: add dbus based "netplan apply" interface [09:13] mvo: is it problematic if two copies of the daemon run simultaneously for a while? [09:13] jamesh: I don't think so [09:14] mvo: in that case, releasing the name and exiting after current requests complete should be enough [09:14] jamesh: nice! [09:14] mvo: I'm not sure if there is a good example of that using the systemd dbus lib though [09:15] jamesh: no worries, I could change to gio [09:15] mvo: I'm sure there is a good example, I just don't know where it is [09:15] jamesh: do you have one for gio? [09:15] should have been "not sure where there is" [09:16] jamesh: but no worries, I can figure it out :) thanks for the hint! [09:17] mvo: I know I dealt with this a while back for some of the unity-api projects, but can't really remember which [09:17] jamesh: no worries, thanks again [09:17] * mvo goes back to reviewing before diving into this [09:19] mvo: this code is using a bus_event_loop_with_idle() helper: https://github.com/systemd/systemd/blob/master/src/timedate/timedated.c [09:20] that probably represents best practice with sd-dbus [09:20] jamesh: sweet! thanks a lot, this looks like what I need [09:23] hmm [09:23] cannot run daemon: state startup errors: [cannot obtain snap-seccomp version information: invalid format of version-info: "cca7be4711160d1b940e8a87b8d572c8cb18e3dd 2.4.1 8c73f36d3de1f71977107bf6687514f16787f639058b4db4c67b28dfdb2fd3af"] [09:23] what did i break now? :-| [09:25] mvo: https://github.com/systemd/systemd/blob/master/src/shared/bus-util.c#L92 <- this is the utility function. It isn't part of libsystemd, but doesn't appear to be using private API [09:26] Chipaca: your snapd and you snap-seccomp are out of sync? [09:26] pedronis: yeah that was it [09:28] Chipaca: add a symlink from your build-tree to ../snap-seccomp/snap-seccomp [09:29] jamesh: looking [09:29] ijohnson: you there? [09:29] Chipaca: probably not yet [09:29] Chipaca: 4:29 for him [09:29] the 'snap model' PR seems to panic [09:30] mvo: psh, so lazy [09:30] Chipaca: heh :) [09:39] tr [09:39] re [09:39] sorry, interrupts [09:40] (5200/5496) - no failures! [09:40] wow [09:40] with the mount-ns test as last (famous last words, it may still fail) [09:40] PR core#106 opened: core: remove config test, this moved to the snapd code [09:44] PR snapd#6954 closed: sessionagent: add a REST interface with socket activation [09:49] Chipaca: it has no tests atm fwiw (I'm more of a write tests with the code sort of person) [09:50] yeah, this is unlandable as is [09:51] for several reasons [09:51] zyga: if you could review https://github.com/snapcore/core/pull/104 too that would be great. should be easy [09:51] PR core#104: snapcraft.yaml: use remote fc-cache-builder [09:52] PR core#106 closed: core: remove config test, this moved to the snapd code [09:55] brb again [09:56] or maybe someone else can check 104 on core? its really simple :) [10:00] mvo: +1 [10:00] zyga: ta! [10:00] PR core#104 closed: snapcraft.yaml: use remote fc-cache-builder [10:11] PR snapd#7161 closed: tests: show stderr only if it exists [10:12] ppisati, if i use the kdefconfig option in snapcraft.yaml, is that using ./scripts/kconfig/merge_config.sh from the kernel tree in the back or does it simply call the different configs in order ? (i see some weird behaviour between a snap kernel and a manually built one that indicates the configs are different) [10:13] PR snapd#7153 closed: gadget: select the right updater for given structure [10:28] PR snapd#7162 opened: usersession: move userd package to usersession/userd [10:34] PR snapd#7163 opened: tests: measure testbed for leaking mountinfo entries [10:36] ogra: hold on [10:37] zyga: mvo: so lxd attempts some weird things on F31 with unified hierarchy: https://paste.ubuntu.com/p/Z4Qc4qq87v/ [10:38] mborzecki: what is fd=10 [10:38] PR snapd#7164 opened: test: enable and run mount-ns test as last [10:39] mborzecki: https://github.com/snapcore/snapd/pull/7163 [10:39] PR #7163: tests: measure testbed for leaking mountinfo entries [10:39] ideas welcome [10:39] Chipaca: ^ you as well, this is something similar to what you've built before [10:40] ogra: 'make $kdefconfig[0] $kdefconfig[1] $kdefconfig[2] ...' [10:41] ogra: but after the first one is expanded, it should call merge_config.sh by default - that's part of the kernel build rules [10:41] * Chipaca steals cookies [10:42] * Chipaca steals biscuits also, for better i18n coverage [10:43] ppisati, aha ... strange ... when using https://github.com/ogra1/pi4-kernel-hackery/blob/master/build-linux-5.1.sh#L10 vs https://github.com/ogra1/linux-raspberrypi-org/blob/master/snap/snapcraft.yaml#L30 i actually need to specify security=apparmor and apparmo=1 on the cmdline for the snap ... while the native build defaults to using apparmor ... [10:44] the patches and trees are absolutely identical [10:44] PR snapd#7144 closed: tests: measure mount namespaces on Ubuntu 14.04 [10:47] mborzecki: https://www.redbubble.com/people/rodebubbel/works/31716594-i-use-arch?p=sticker [10:47] Chipaca: hahah [10:49] mborzecki: even better, https://www.stickermule.com/uk/products/unixstickers-pro-pack [10:53] Chipaca: aaand placed an order :) [10:54] mborzecki: [11:00] mvo: zyga: https://forum.snapcraft.io/t/lxd-3-15-fails-with-unified-cgroup-hierarchy/12462 the lxd thing [11:02] mborzecki: thanks [11:26] how do you spell "ALL THE LUNCH" [11:26] * Chipaca all the lunches [11:37] * zyga does reviews for a while [11:37] PR snapd#7160 closed: tests: reformat and fix markdown in snapd-state.md [11:50] some unfortunate project name, can anyone try `go get github.com/ojii/gettext.go`? [11:51] the repo name causes a panic in fedora packaging tools :/ [11:54] mborzecki: perhaps we should fork it [11:54] I think the .go suffix on a project name is a bit unfortunate [11:54] zyga: hm there's a fork under snapcraft/go-gettext [11:54] is it up to date? [11:54] duh, snapcore/go-gettext [11:56] zyga: yeah, perhaps we should just package it, dropping all i18n support feels to heavy [11:58] yeah [11:58] I think so [11:58] we should craft a plan to support (maintain) our deps in Debian and Fedora [11:58] we really need that [11:59] mborzecki: can you look at https://github.com/snapcore/snapd/pull/7163 [11:59] PR #7163: tests: measure testbed for leaking mountinfo entries [11:59] it's super short and will hopefully open the path towards leakproof tests [12:16] and it found stuff I wasn't testing... 18.04 is full of fun [12:16] on it :) [12:19] PR snapd#7163 closed: tests: measure testbed for leaking mountinfo entries [12:22] PR snapd#7164 closed: test: enable and run mount-ns test as last [12:43] hey Chipaca [12:44] (and everyone else) [12:48] Chipaca: thanks for the detailed review, I have in fact been running that PR, but I have been cherrypicking the commits on top of the 2.39 branch because that was easier to test without dealing with the system-key format and I naively assumed that master would work the same as that branch [12:49] * Chipaca hands one of the cookies back [12:50] remodeling work has changed quite a few details about DeviceManager/devicestate [12:51] mvo: won't be at the standup today, there's an interesting session at the same time slot [12:52] mborzecki: https://github.com/snapcore/snapd/pull/7165 [12:52] ijohnson: https://github.com/snapcore/snapd/pull/7165 :) [12:53] PR #7165: tests: restore cpuset clone_children clobbered by lxd [12:53] ijohnson: I'm fighting tests that break the testbed by doing changes to the mount table [12:53] PR snapd#7165 opened: tests: restore cpuset clone_children clobbered by lxd [12:53] Chipaca: Model was taking the lock for you in 2.39, it doesn't anymore [12:53] the older behavior had its reasons but was kind of weird (we don't take the lock for such small methods usually) [12:54] zyga: nice, I can take a look in a bit, still reading through all of my cmd/model comments that I will need to address [12:55] ijohnson: 2.39 isv very different from master, if you look at changelog of 2.40 already lots have happened [12:55] ijohnson: no worries :) [12:58] * Chipaca grudgingly returns the rest of the cookies [12:58] * Chipaca now needs to find his own [13:00] Chipaca: costco near me sells a big box of 16 cookies for $4 [13:00] * ijohnson hands Chipaca some cookies [13:01] * Chipaca discovered the meaning of win-win [13:27] PR snapd#7162 closed: usersession: move userd package to usersession/userd [13:36] I'll grab some food and return to reviews [13:50] a 2nd review for 7066 would be great (it's not urgent but it's old and there is one more on top) [13:55] fedora tests look good but core is full of nasties [14:01] zyga: mvo_: left a note under https://bugzilla.redhat.com/show_bug.cgi?id=1438079#c6 [14:01] I saw that :) [14:02] (mail notification ahead of IRC) [14:02] thank you [14:02] and good luck with flock [14:02] zyga: mvo_: also https://gist.github.com/bboozzoo/76b1535c93686a27bb7fdbaad0f560f7 got updated [14:02] mborzecki: note, freezer is available since 5.2 or 5.3 [14:02] as in the freeze feature [14:03] but as we discussed, we should switch to safe mount api [14:03] and not freeze [14:03] zyga: mhm, i doubt anyone on older kernels will switch to unified anyway :) [14:03] yeah, just saying [14:12] travis is starving us now [14:22] PR snapcraft#2643 closed: project, cli: clean up snap asset messages [14:37] pedronis: I did a pass over https://github.com/snapcore/snapd/pull/7066#pullrequestreview-266656756 [14:37] PR #7066: overlord/assertstate: add Batch.Precheck to check for the full validity of the batch before Commit [14:41] * zyga can smell lovely curry from the kitchen and goes for a break [15:01] hi guys, i'm trying to install Ubuntu Core on an embedded computer with an i5 CPU. I copied the ubuntu-core-18-amd64.img.xz image on the mSata disk and the boot the PC. When I press enter to make the configuration I got an error: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 31.. etc.. and the PC promt again "press enter to c [15:01] onfigure"what can I do? [15:02] PR snapd#7166 opened: client: add doTimeout to http.Client{Timeout} [15:03] fgiulianiint: ouch [15:06] fgiulianiint: hmmm, I think that image is DOA, where did you get it from? [15:07] is `make hack` still supposed to work/be used? [15:08] ijohnson: yes, though YMMV [15:08] ijohnson: send patches to improve it [15:08] I use it regularly [15:08] but I have a local patch that's hard to upstream, to make it work on suse [15:08] where the .real suffix is not used [15:08] ok, yeah I'll send a small patch to make it work when using the go snap, because it's using options that don't work with read-only go install [15:09] zyga: i downloaded from the official repo (at least I think) http://cdimage.ubuntu.com/ubuntu-core/18/stable/current/?_ga=2.266211528.651887194.1563966098-1627888589.1561621113 [15:09] ijohnson: read-only go install? [15:09] the go snap? [15:09] ijohnson: I'm looking forward to the patch [15:09] go build runtime/cgo: open /snap/go/4100/pkg/linux_amd64/runtime/cgo.a: read-only file system [15:09] zyga: what DOA stands for? [15:09] fgiulianiint: ouch, not sure what to do about it, can you raise the topic on a forum and we can see what to do from there; p [15:10] dead on arrival [15:10] I think that image is busted [15:10] is the text localised in any way? [15:10] zyga: thx, updated, I did something slightly different [15:10] +1 [15:11] zyga: yes, the only problem I can't copy the error becouse the monitor is cleared every time it prompts again "Press enter to configure" [15:11] fgiulianiint: no worries, that's okay [15:12] ok than [15:12] fgiulianiint: thank you for reporting that, I think we should check how that happened, there's supposed to be QA around that [15:13] mvo_: ^ who manages the images that are produced? [15:16] zyga: thx, I'll land it if/when it gets green. from uc20 spike work I learned that we should probably make Batch more versatile and move it to asserts. something maybe to try during travel [15:17] +1 [15:17] pedronis: thank you [15:19] PR snapd#7167 opened: cmd/Makefile.am: support building with the go snap [15:20] zyga: see 7167 ^ [15:21] ijohnson: looking [15:21] thanks [15:22] +1 [15:23] btw is there a convenient "make unhack" to undo the changes from the make hack command? I just copied all of /usr/lib/snapd and was gonna re-copy it back when done [15:23] not sure if there's a more convenient way to do that [15:28] PR snapd#7166 closed: client: add doTimeout to http.Client{Timeout} [15:29] PR snapcraft#2639 closed: deprecations: add deprecation notice for version-script (dn10) [15:29] * cachio lunch [15:31] ijohnson: no, just reinstall the package [15:31] ijohnson: it's a quick and easy fix [15:32] I see, I hadn't thought of that, but that sounds easy [15:32] :) [15:32] +1 on making that in make unhack [15:32] if you feel like wanting that [15:32] it would make it easy [15:32] I'm also gonna add some more info to HACKING.md since it is a little dusty I think [15:33] +1 [15:33] yeah, go for it :) [15:33] I wish we'll reach a point where the makefile won't have to hide in the cmd directory [15:33] one day [15:34] * zyga restarted some local tests and returns to the kitchen where it is cooler [15:36] zyga, hey, I'm doing some testing by bind mounting snapd, but I found this error: [15:36] Jul 25 15:34:45 localhost snapd[5256]: cannot run daemon: state startup errors: [cannot obtain snap-seccomp version information: invalid format of version-info: "762c8972ff78d5e5d00275810e4dba7ae0ae0ea4 2.4.1 8c73f36d3de1f71977107bf6687514f16787f639058b4db4c67b28dfdb2fd3af"] [15:36] abeato: hmmmm [15:36] ah [15:36] hey abeato, I ran into the exact same thing [15:36] zyga, any idea on how can I workaround this? [15:36] yeah, new snapd needs matching snap-seccomp [15:37] I fixed it by doing make hack in cmd/ subdir [15:37] bind mount snap-seccomp in the _same_ directory as snapd [15:37] it uses /proc/self/exe [15:37] if you develop from source ijohnson's advice is good [15:37] make hack is your friend [15:37] ijohnson, nice, so then I need to bind mount snap-confine or other binaries? [15:37] abeato: make hack installs them [15:38] abeato: if you just care about snapd then just providing snap-seccomp is sufficient to get past that [15:38] abeato, go with what zyga says :-) [15:38] zyga, ok, will do that, I am compiling in a different machine to where I am testing, as it is an UC system [15:38] aha [15:38] yeah, a symlink would work too [15:39] whatever is convenient [15:39] just need a "recent enough" snap-seccomp [15:39] it's not that they have to be built together [15:39] ok, will try that thanks ijohnson and zyga [15:39] :) [15:44] PR snapd#7165 closed: tests: restore cpuset clone_children clobbered by lxd [15:46] PR snapd#7168 opened: tests: measure testbed for leaking mountinfo entries [15:49] * ijohnson goes to go look at a used car [15:56] you have weird hobbies [15:56] PR snapd#7166 opened: client: add doTimeout to http.Client{Timeout} [16:34] ok, I'm out [16:34] AC is no longer keeping up [16:34] send help [16:34] * Chipaca heads for the showers [16:39] * zyga is running away from the office [16:39] it's the hottest part of the house now [16:55] running another pass [16:55] I found another general class of leaky tests [16:55] core18 is often leaked on core systems [16:55] because cleanup is different there [16:55] this is now fixed, let's see what else is uncovered [16:55] meanwhile, I *really* need to run away from the office [16:55] ttyl [17:10] is it expected / by design that snaps cannot run when snapd is unable to start? [17:17] ijohnson: it's not exactly like that [17:17] ijohnson: in general snaps *can* almost always run without snapd [17:17] ijohnson: your case is different because you use a local build [17:17] ijohnson: in that case you always need snapd to compute matching security profiles [17:17] ijohnson: (matching to snap-run) [17:17] I should specify, this is for a classic snap [17:18] zyga: yes I know why my local snapd can't start [17:18] the logic doesn't check for snap confinement type [17:18] zyga: I'm just wondering if it's the case that something in snap-run/snap-confine needs to talk to snapd the daemon in order to continue [17:18] yes [17:18] but only in a specific case [17:18] snap run computes the "system key" [17:18] and if mismatches what's on disk (written by snapd) [17:18] it waits [17:19] or pokes snapd, I forgot [17:19] but it waits anywa [17:19] *anyway [17:20] I see, so if the system key matched, but snapd still couldn't start for some other reason the snap would be able to start? [17:21] no [17:22] okay, so it does look like it's at least expected that the snap wouldn't be able to run, but perhaps that's not an explicit design choice [17:22] I'm aware of the explicit design choice of snapd not blocking bootup and not getting in the way of other things running on the system [17:22] I wasn't sure if there was a similar thing for running snaps specifically [17:23] ijohnson: you are right with the design choice of not blocking startup [17:23] ijohnson: but except for kernel or snapd changes [17:24] ijohnson: and that's what's going on, effectively here [17:24] ijohnson: running snaps is in the same area, since the boot up problem is really "snapd needs to work for system to boot" [17:31] PR snapd#7169 opened: tests: remove core18 if installed by tests [17:32] PR snapd#7170 opened: tests: fix typo "current" [17:35] zyga: okay that makes sense [17:44] PR snapd#7169 closed: tests: remove core18 if installed by tests [17:51] PR snapd#7066 closed: overlord/assertstate: add Batch.Precheck to check for the full validity of the batch before Commit [18:11] PR snapd#7171 opened: tests: improve how the system is restored when the upgrade-from-2.15 test fails [18:20] * cachio afk [18:43] found another bug, installing the "classic" snap clobbers the host [18:43] fun fun fun [18:51] uUUUUh [18:51] omg [18:51] man [18:51] this is bad [18:53] ah, wait [18:53] I misread [18:53] it's not bad [18:53] geez :| [18:53] zyga: chill! [18:54] hard to do with your heat wave, come down south to our cold weather :-) [18:54] sergiusens: polandball colors are somewhat appropriate now [18:54] (being upside down) [19:21] Issue classic-snap#32 opened: Using classic clobbers /dev/pts on ubuntu-core 16 systems [19:44] PR snapd#7172 opened: tests: work around classic snap affecting the host [19:45] ogra: do you know who maintains the classic snap? [19:45] jdstrand: ^ interesting PR [19:45] jdstrand: tl;dr; security implication is that mounts that don't propagate to the other namespaces can still remount a mount point in the other namespaces if they have the permission (or attack vector) to do so [19:45] * zyga takes a break [19:50] zyga: note devpts is special. I haven't read the code but it would surprise me if it honored propagation events. you just mount what you want at the place iirc and the kernel gives you that [19:51] zyga: and the classic snap is performing mount -o mode=666,ptmxmode=666 -t devpts devpts /dev/pts [19:51] jdstrand: that's odd, I tried "mounting over" but it would return EBUSY unless I passed -o remount [19:52] jdstrand: did you strace the classic snap or inferred this from the source code [19:52] jdstrand: as it does use -o remount in some if-then-else tree [19:52] but anyway, it's interesting and was non-obvious to me what is the cause of this [19:52] jdstrand: the mountinfo-tool work and the recent testbed-tool work is really paying off [19:52] classic is doing a weird: mount -o mode=666,ptmxmode=666 -t devpts devpts /dev/pts ; $SCRIPT && umount /dev/pts [19:52] note && is a bug most likely [19:53] it's horrible and should be rewritten away from shell IMO [19:53] zyga: as mentioned, I didn't look at anything. I just know that devpts is special in many ways and it wouldn't surprise me if this was one of them [19:53] jdstrand: ack, I'll read the kernel source to see what it does [19:54] I just wanted to share it because I was definitely not expecting this and I learned something today :) [19:54] jdstrand: in case you are interested where this is coming from: https://github.com/snapcore/snapd/pull/7168/files [19:54] PR #7168: tests: measure testbed for leaking mountinfo entries [19:55] zyga: it is interesting, but I don't consider it a security issue within snapd case the classic-support interface isn't meant to provide a security barrier and strict snaps can't do the mounting (and classic snaps are limited by DAC (which is only useful for non-root commands of course)) [19:56] ie, if you can do arbitrary devpts mounts, then well, I'm not surprised that can affect things [19:57] jdstrand: I wonder if our apparmor policy that allows a particular mount also allows you to remount, in other words, if you don't say options=(remount), do you have that? [19:57] jdstrand: or in other words, should way always say options=() to avoid having the effect of options=(*) [19:58] zyga: remount is a different rule [19:58] mount, umount, remount [19:58] really? I didn't know that [19:58] I recall we have some remount code [19:58] * zyga looks [20:00] https://github.com/snapcore/snapd/blob/7721bb9df44a136f31920515fc5d1a3c2e1e9850/cmd/snap-confine/snap-confine.apparmor.in#L296 [20:00] we say options=(remount, ...) here [20:00] or did you mean that? [20:00] based on what you said I was expecting "remount ..." as a top-level rule [20:00] I do see options=(remount ro bind) etc in there [20:00] there is a remount rule [20:01] are they equivalent? [20:04] zyga: it probably depends on how it was mediated. I can say that we use options=(...) which means that the mount options must exactly match. you can't omit or inject [20:05] jjohansen: zyga and I were curious when 'remount ...' is used, vs 'mount options=(remount, ...)' [20:09] zyga: also note that systemd-run is in play with the classic snap, so it might've done something too [20:10] zyga: the classic snap is not using 'newinstance' either... [20:10] anyway, I need to get back to what I was working on [20:11] jdstrand: remount is the same as options=(remount,...) [20:12] its just a convenience keyword [20:13] zyga, classic should go away in favour of lxd [20:13] jjohansen: ack, thanks (cc zyga ^) [20:14] it comes from a time where we needed a quick way to use debs on core for development, it has never been more than a hack [20:16] zyga, also note that there has never been a stable release for a reason ;) [20:19] snapd ate all of my AWS CPU credits. It was running for several minutes. I restarted the VM. What was that thing doing? It made the VM unresponsive. [20:19] I'm not even using snaps on this VM. :-\ [20:19] It looks like it was doing a lot of IO 'cause some kernel disk process was topping the charts. [20:26] JordiGH: can you run "snap changes" to let us know? [20:26] systemd logs could also be useful (of snapd.service) [20:30] $ snap changes [20:30] error: no changes found [20:30] $ sudo journalctl -u snap.service [20:30] -- No entries -- [20:31] Oh, sorry, snapd. [20:31] Does this mean anything to you? http://paste.debian.net/1093102/ [20:33] looking [20:33] is snapd still using CPU? [20:33] so far all it did seems nurmal [20:33] *normal [20:33] No, it's not doing anything now. [20:33] But what could it possibly have been doing? [20:34] Is there some scheduled task? [20:34] initially it seeds snaps [20:34] snaps are on disk but are not installed [20:34] Even on a system that, afaict, has no snaps? [20:34] I see you have two snaps [20:34] snap list? [20:34] Oh, I do? [20:34] I think so [20:34] Ugh, Amazon has started to use snaps. [20:34] amazon-ssm-agent [20:34] so it installed the two snaps, the core is quick and easy but the amazon-ssm-agent might required some CPU startup time to configure the sandbox [20:34] That, eh? [20:34] yeah [20:35] *might have required [20:35] when you said it ate all your credits, what was the actual amount [20:35] I believe those are baked into the images [20:35] did it run for minutes, hours? [20:35] I'm really worried that Ubuntu is really trying to get rid of or hide apt. [20:35] roadmr: yes [20:35] zyga: It ran for about 30 minutes before I noticed and terminated the instance. [20:35] JordiGH: we are doing neither, I can assure you; snaps do make a lot of sense for many payloads though [20:36] JordiGH: ouch! that's certainly bad [20:36] was it snapd itself, or the agent snap? if the latter, this is probably best raised with amazon themselves, right? [20:36] JordiGH: can you share the details please: what kind of instance, which image, etc [20:36] JordiGH: so that we can reproduce and fix this [20:36] roadmr: I don't know, I saw snapd in top(1) as well as a kernel IO process. I'm not sure if it was kswapd. [20:36] JordiGH: right, that doesn't look like the agent itself [20:37] Not the agent itself? Something other than snapd? [20:37] snapd was doing something that was doing a lot of disk writing. [20:37] JordiGH: ideally we can reproduce this and determine the cause [20:37] zyga: t2.large [20:38] zyga: I was told that yeah, the goal was to make people stop using apt. [20:38] JordiGH: can you collect this in a bug report please, I'll make sure the cloud people investigate this [20:38] Was I mislead? Canonical employees told me this. [20:38] JordiGH: who told you this? [20:38] it wasn't me :) [20:38] I'm a canonical employee and I think both snaps and classic packages have their uses [20:39] yeah, apt is for building snaps, right? [20:39] Users should only use snaps. [20:39] it depends [20:39] I don't remember who said this. [20:39] it's not as simple as that [20:39] for some use cases one is more appropriate than the other [20:39] for end user applications, despite the relative immaturity, snaps do make a lot of sense [20:40] I thought the point is that Canonical was going to be spending all of its energy on making snap packages and make apt packaging secondary. [20:40] Anyway, what am I going to report and to whom? [20:40] for certain types of services and in headless consumer electronic stuff snaps make a lot of sense [20:40] JordiGH: I think whoever told you that was sharing his personal view and not a company policy [20:40] I don't have a lot of data. I don't even remember exactly what I was seeing in top(1). [20:40] JordiGH: please collect the information and report it either on forum.snapcraft.io or on bugs.launchpad.net/snapd [20:41] in either case please share the link, I'll get some eyes on it [20:41] Link to what? [20:41] JordiGH: region, instance type, image used, that should be enough to see if it happens all the time [20:41] to the bug report or forum post you make [20:42] I'm going to make a bug report? [20:42] please, if you can [20:42] eu-west-1a, t2.large, xenial-based image, [20:42] Where should I make a bug report? [20:42] xenial-based? [20:42] was it an official image or something custom? [20:43] I'm only asking to know if we can reproduce it on the stock one [20:43] We grab the xenial image, add stuff to it, and deploy VMs based on that. [20:43] I see [20:43] JordiGH: please report the bug on https://bugs.launchpad.net/snapd [20:44] ogra (cc zyga): is lxd viable for the workflow you described (ie, building things in an armhf container). are there official lxd images for armhf and arm64? [20:44] jdstrand: I'm sure there are [20:45] jdstrand: I used lxd on both [20:45] hmm.. I recall wanting them at one point and not having them [20:45] but that was a while ago [20:45] zyga: Can't remember my password, my email reminder isn't showing up, I'm already cross and don't want to bother with this anymore. [20:45] I think at this time you just install lxd and launch a container from it [20:46] If it happens again I'll come back. [20:46] JordiGH: thank you [20:46] JordiGH: if you want to, a forum post on forum.snapcraft.io is also useful [20:46] just anything that we can share with you as a way to continue the conversation [20:47] That probably also requires creating accounts, and I'm already short-tempered, ttyl! [20:48] zyga (cc ogra): sudo lxc launch ubuntu:16.04 xenial is working on arm65 [20:48] arm64* [20:48] jdstrand: cool :) [20:48] so, nm [20:48] jdstrand: I switched off my arm machines to (ironically) save some power today so I cannot ssh into anything to check quickly [20:48] heh [20:48] and I'm too lazy to go down and see the office today [20:48] it's 22:48 and here I am, working [20:49] zyga: you should stop that :) [20:49] I know but then we would not have this conversation about remount :) [20:52] heh [20:54] and sudo lxc launch ubuntu:16.04 xenial works on armhf too [21:07] jdstrand: btw, if you could comment on https://github.com/snapcore/snapd/pull/7172 I would love to fix this and carry on [21:07] PR #7172: tests: work around classic snap affecting the host [21:07] the context is that there's a new tool that shouts loudly if a test clobbers the machine [21:07] so we get to see things like that easily [22:30] PR snapd#7170 closed: tests: fix typo "current" [22:44] jdstrand, yeah, absolutely, i use it every day [22:45] jdstrand, and if you add your user into /var/lib/extrausers/group to the lxd line you can even get away without sudo ;)