[03:46] PR snapd#3857 opened: tests: fix lxd test for external backend [05:13] * Son_Goku grumbles [05:13] somehow, I just woke up at 1am :( [05:13] so... morning, I guess === JoshStrobl|Work is now known as JoshStrobl|zzz [05:41] mwhudson: hey, thanks for helping with the ppc64el link error. do you know if there is any workaround? [05:41] mwhudson: this is a blocker for 2.28 currently and I wonder what options there are [06:32] mwhudson: nevermind, I think I found a workaround [06:39] PR snapd#3858 opened: snap-confine,snap-update-ns: add -no-pie to fix FTBFS on ppc64el [06:49] o/ [06:52] meep [06:54] you know, I *really* don't like how go does compiler flags [06:54] it's a really dumb way to do things [06:55] good morning Son_Goku (not actually morning for you I guess ;) [06:55] PR snapd#3841 closed: Do not emit cloud-init.disabled; cloud-init only runs if datasource is present [06:55] and good morning to you zyga-ubuntu [06:55] [01:12:33 AM] Son_Goku grumbles [06:55] [01:12:40 AM] somehow, I just woke up at 1am :( [06:55] [01:12:53 AM] so... morning, I guess [06:55] :/ [06:56] this is what I get for going to sleep at 8:30pm [06:57] Son_Goku: autsch! [06:57] Son_Goku: that is going to be a coffee fueled day then :) [06:57] I was feeling pretty bad yesterday, so I took some Advil (headache medicine) and went to sleep [06:57] much earlier than I usually do [06:57] yeah, I might wind up drinking coffee today [06:57] I usually don't... === chihchun_afk is now known as chihchun [07:00] Son_Goku: aha, yeah. hope you make it well through the day, I get cranky when I did not sleep enough [07:00] well, my work day begins in 6 hours [07:00] at which point, I'm going to be tired as hell [07:00] yeah :) [07:00] zyga-ubuntu: some easy reviews tagged with 2.28 *hint* ;) [07:01] mvo, not having pie across the board for snapd 2.28 kind of sucks, though [07:02] Son_Goku: its just for two small helpers [07:02] snapd is made up of "small helpers" :P [07:02] Son_Goku: and if you have !go1.7 you can simply reomove it [07:02] I wonder if this is even broken on Fedora 25 for ppc64le [07:02] we have go 1.7 there too [07:03] Son_Goku: 2.28 probably is, 2.27 had a slightly different cflags/ldflags iirc for snap-seccomp [07:04] good morning :) [07:04] hmm [07:04] mvo: I'll have a look [07:04] 1.7.4 ~ 1.7.6 [07:04] I guess there's only one way to find out... [07:06] * Son_Goku logs into Fedora ppc64le test machines [07:07] mvo: please check out https://github.com/snapcore/snapd/pull/3854#pullrequestreview-60818594 [07:07] PR #3854: corecfg: mock "systemctl" in all corecfg tests [07:07] mvo: I need a re-review from jamie on https://github.com/snapcore/snapd/pull/3621 and I need to fix one opensuse packaing quirk (still fighting natively) and with merged master it should go green [07:07] PR #3621: cmd/snap-{confine,update-ns}: apply mount profiles using snap-update-ns [07:08] mvo: but since it is tagged for 2.28, could you please have a look? [07:08] mvo: note that we must wait for jdstrand's review for the child profile [07:08] mvo: so if you want to release earlier, please drop it from the release [07:10] zyga-ubuntu: I dropped it, I am happy to review but unfortunately it missed the 2.28 cycle [07:10] zyga-ubuntu: I replied to 3621 [07:12] mvo, I'm running through a test build of current master on Fedora 25 ppc64le [07:12] which has golang 1.7.6 [07:12] Son_Goku: nice [07:20] man... ppc64le is slow [07:21] ok [07:21] Son_Goku: it has to be, there is only one machine on the whole planet, everything else is virtualzed and over-committed ;-) [07:21] haha [07:22] you're probably not far off from the truth [07:22] at least I'm not debugging arm crap [07:22] I hate doing that [07:22] arm SoCs are even slower [07:23] * zyga-suse just had a brilliant business idea [07:24] * Son_Goku shoots the light bulb over zyga-suse's head [07:24] in some future where everything is running on one big machine, to sustain old business models we will still pay for traffic from A to B even though A is B, [07:24] oh god [07:24] I didn't shoot it fast enough [07:25] maybe we need a payasyougo.ko to track this [07:25] oh god damnit [07:25] someone switched the import path for cheggaaa/pb [07:26] yeah, it got switched to the canonical path [07:26] * Son_Goku groans [07:27] zyga-suse, file a bug against the fedora cheggaaa/pb package [07:27] it needs the gopkg.in import path supported [07:28] ok, what should the bug report say? [07:28] just "please support the canonical import path via gopkg.in"? [07:29] yeah [07:29] hmm, I seem to have plenty of stale kernels [07:30] Son_Goku: ok, let me try to report that [07:30] describe what the new import path is, and why we need it added (snapd 2.28 requires it) [07:30] ok [07:35] Son_Goku: for rawhide? [07:36] mark the fedora release as rawhide, but ask for it to be added for all releases the package is provided for in the bug report [07:37] ok [07:37] poor Jan [07:37] he just updated cheggaaa/pb, too [07:37] mvo: hi, thanks for the review of the overlord.Mock PR [07:39] mvo: snapd git master compiles just fine today with Fedora 25 golang 1.7.6 [07:40] Son_Goku: https://bugzilla.redhat.com/show_bug.cgi?id=1488747 [07:41] I'm surprised Fedora tests haven't started failing [07:41] or maybe they have and no one said anything [07:42] if that's the case, why bother having the tests... :( [07:43] mvo: #3727 seems to have a real unit tests issue now, probably related to some recent merge to master [07:43] PR #3727: daemon, snapstate: move ensureCore from daemon/api.go into snapstate.go [07:45] oh geez [07:45] it worked somehow [07:46] oh, no [07:46] let me guess [07:46] zyga-suse: are we giving a vendorized tarball to the snapd build in Fedora? [07:46] I don't think we do [07:46] (especially because that progress bar package was older in fedora and had UI corrputions) [07:46] *corruptions [07:47] and this was fixed in subsequent commits upstream [07:47] I mean in spread [07:47] in spread yes, we do [07:47] FUCK [07:47] damn it [07:48] the damned build tests are useless then [07:48] that is bound to be always helpful [07:48] *to not be [07:48] okay, then I need to break this [07:49] crap, this means I need to make it so patches apply cleanly with the vendor tarball [07:49] no one notices when builddeps are missing because of the stupid usage of vendor stuff [07:50] Son_Goku: indeed, that's a real problem [07:50] Son_Goku: our debian build suffers from the same thing [07:51] actually, I can just rm -rf vendor/* in %prep, I suppose [07:51] minus the vendor.json [07:51] Son_Goku: I think even with the vendor json, nothing needs that (just don't run get-deps) [07:51] pedronis: thanks, checking [07:51] zyga-suse: remember that I need to be able to apply patches [07:52] and I'd like to not have to edit patches more than I already have to when backported [07:52] *backporting [07:52] it was somewhat annoying to backport userd to 2.27.5, but I did [07:52] what's the experience, does it work? [07:52] you tell me [07:53] the updates are sitting in bodhi right now [07:53] well, koji [07:53] for some reason they haven't made it to updates-testing yet [07:53] Son_Goku: hm, zesty has go 1.7.4 wonder if thats related but aiui only 1.8 fixes the issue, I will dig [07:54] mvo: https://src.fedoraproject.org/rpms/golang/tree/f25 [07:54] that should help [07:55] Son_Goku: how should I remove stale kernels from my fedora box? [07:55] did you change /etc/dnf/dnf.conf for installonlypkgs limit? [07:55] it should be set to 3 by default [07:56] rebooting [07:56] something broke so I booted into the recovery kernel and updated [07:56] and ... [07:56] the logo ... [07:57] and... [07:57] the same [07:57] I'll leave it for 5 minutes in case it's some timeut [07:57] but it looks broken :/ [07:58] ~_~ [07:58] if it's Fedora 26 or newer, then `dnf install dnf-utils; package-cleanup --oldkernels --count=3` [07:59] Fedora 25 and older will require you to actually do the work yourself :P [08:00] fedora 26 [08:00] but broken at boot, let me try recovery again [08:12] PR snapd#3859 opened: packaging/fedora: Ensure vendor/ is empty for builds [08:12] zyga-suse: there ^ [08:13] if spread isn't failing after we do this, then there's a problem [08:14] pstolowski: did you remotely access my machine last night? Because it's working now o_o [08:14] davidcalle: maybe he did (automatic snap refresh maybe?) [08:17] mvo: doesn't look like it, I had install hooks failing 100% of the time. On stable core, artful snapd, so no recent changes on this front afaik... Unless another snap was messing with everything [08:17] mvo: anyway, bug seems to be fixed, so... [08:21] pstolowski: just confirmed that the install hook was being called as expected, thanks for the help [08:22] * zyga-suse walks daughter to school [08:28] is Chipaca off today? [08:33] davidcalle, waaat [08:33] :) [08:36] davidcalle, fyi, I investigated it further yestaerday's evening / this morning.. your state.json didn't show any apparent problem, only confirmed the issue (the error you saw was there). the only *theoretical* explanation I had was that for whatever reason an old version of snap-exec was used (as judging from the error message, snap-exec didn't know about install hooks). but since you didn't disable reexec, I'm clueless... [08:38] pedronis, hey, are you familiar with CommandManager / cmdstate? [08:39] pstolowski: I think I might have reviewed it, what's the question? [08:41] pedronis, for some reason in a spread test I'm hitting timeout (which is very generous), as if change wasn't set to ready after finished - https://github.com/snapcore/snapd/pull/3852/files#diff-8c0b309b451b46cd7f7dac1f80654270R113 ; this is working fine in unit test where I'm not actually executing command, but mocking it and set change to ready explicitely [08:41] PR #3852: hooks: commands for controlling own services from snapctl [08:47] pedronis, perhaps there is something about taskrunner / changes that I do not know or misunderstand [08:47] re [08:47] but afaiu it should be set ready automatically when tasks are finished [08:48] zyga-suse, hmm, what changed in snap-confine ? seems a lot of users get "cannot locate the base snap: ubuntu-core: No such file or directory" ... grepping the source i only see this message in snap-confine/mount-support.c [08:49] (there are at least three threads on the forum wherre this seems to be an issue) [08:50] ogra_: hmmmm [08:50] ogra_: it's a small patch that landed that lets people use an alternate base snap as an opt-in [08:50] i assume it isnt snap-confine itself but smething hands it "ubuntu-core" [08:50] mvo: ^ [08:50] pstolowski: you are keeping the lock in runServiceCommand, how can anything else run? [08:50] ogra_: ooh [08:50] ogra_: ubuntu-core you say [08:50] well, read the error above :) [08:51] right, I see now [08:51] that is ... curious [08:51] whats even more curious is that some users in these threads show snap list with core 16-2.26.14 [08:52] pstolowski: I don't what's happenign in your tests but for sure that code doesn't look right [08:52] (though that might just be a rre-exec thing ... running a newer deb version) [08:52] pstolowski: I mean all lock mgmt in runServiceCommand seems off [08:53] pedronis, ahh, thanks [08:53] pstolowski: seems you are mocking too much [08:53] btw [08:54] that's probably related to why the tests fowkrs [08:55] ogra_: core 2.26.14 with snapd 2.27.4 [08:55] yeah [08:55] ogra_: but that's not possible on debian stable, it must be sid [08:55] because the stable core wasnt updated yet [08:55] * zyga-suse booted debian 9 [08:55] ogra_: it was, last night AFAIR [08:56] yup [08:56] ogra@anubis:~/datengrab/devel/branches/snapd$ snap info core|grep stable [08:56] stable: 16-2.27.5 (2774) 85MB - [08:56] i stand corrected [08:56] yes [08:56] but that user didnt have it :) [08:56] (refresh can take up to 24h iirc) [08:56] (depending on the timers) [08:57] I'm reading https://forum.snapcraft.io/t/castersoundboard-snap-requires-deprecated-ubuntu-core/1999 [08:57] here the user did have 2.27.4 [08:57] theer is also https://forum.snapcraft.io/t/call-for-testing-warzone-2100/2001/4 [08:57] so that's definitely a debian unstable [08:57] and cjwatson's LXD build announcement [08:57] (and i think i saw it once more but i cant find it) [08:59] pstolowski: maybe you want to give a look at #3856 , it might be useful for your tests there [08:59] PR #3856: overlord: introduce Mock which enables to use Overlord.Settle for settle in many more places === chihchun is now known as chihchun_afk [09:00] zyga-suse, oh ... same thing but different https://forum.snapcraft.io/t/cannot-locate-core-snap-error/884 [09:03] Chipaca: hi [09:03] pedronis: o/ [09:03] pedronis, ack, thanks. [09:04] Chipaca: #3727 and #3856 could use 2nd reviews [09:04] PR #3727: daemon, snapstate: move ensureCore from daemon/api.go into snapstate.go [09:04] PR #3856: overlord: introduce Mock which enables to use Overlord.Settle for settle in many more places [09:05] mvo: I forsee 2.27.6 [09:05] yeah :/ [09:05] pedronis: ack [09:05] zyga-suse: :'( [09:07] does the travis log loading bog down your whole browser, or is my laptopt showing its age? [09:08] if I wait for it to load it's multiple minutes; even getting to click the static log button takes a few seconds (the static logs load almost instantly) [09:08] it takes kind of forever === chihchun_afk is now known as chihchun [09:12] Chipaca: it's travis + javascript suxxines [09:12] or maybe our logs are too verbose :-D [09:12] zyga-suse: spread on opensuse seems broken because of an actual issue in the scripts [09:12] + go install -s -v -p 4 -x macro doesnt allow us to pass any additional parameters '#' so we we have to invoke '`go' 'install`' here manually. export GOPATH=/usr/src/packages/BUILD/go:/usr/lib64/go/contrib export GOBIN=/usr/src/packages/BUILD/go/bin '#' Options used are the same as the /usr/lib/rpm/golang.sh build macro does but as it '#' doesnt allow us to amend new flags we have to repeat them github.com/snapcore/snapd/ [09:12] here: [09:12] Chipaca, use lynx :P [09:12] can't load package: package macro: cannot find package "macro" in any of: [09:13] pedronis: where? can you give me a link to the failure please? [09:13] zyga-suse: here for example: https://travis-ci.org/snapcore/snapd/builds/272361316?utm_source=github_status&utm_medium=notification [09:13] * zyga-suse looks [09:13] seems like a comment ends up as data [09:14] pedronis: I'm debugging something similar right now, it's the %gobuild macro that is really both go build and go install [09:14] pedronis: I'm looking for a solution [09:14] pedronis: (note that this only affects opensuse) [09:14] how did it make to master? [09:15] pedronis: how was this merged [09:15] pedronis: good question, my branch that is affected has not landed yet [09:15] pedronis: I'll check this out soon, looking at debian sid and possible regression [09:15] zyga-suse: mvo: master is broken like this :( [09:16] how was it merged? [09:16] or something has changed in suse itself? [09:17] pedronis: nothing like that would change inside leap [09:18] well master is red and broken [09:18] sure [09:18] I think it may be a commented-out macro [09:18] macros cannot be commented out [09:18] because of how rpm works [09:18] * zyga-suse looks quickly [09:18] something doesn't compute here [09:19] I how did it land either way? [09:19] ogra_: interestingly, snapd in sid is not reexecing [09:19] pedronis: no idea [09:19] oh [09:21] zyga-suse: there's a comment like this in the spec: # The %gobuild macro [09:21] I suppose it gets interpreted anyway? [09:21] I see [09:21] though this didn't fail before [09:21] I just removed them, let's make a quick PR [09:22] zyga-suse, I feel like I should leave this broken: https://github.com/snapcore/snapd/pull/3859 [09:22] PR #3859: packaging/fedora: Ensure vendor/ is empty for builds and fix spec to build current master [09:24] pedronis: I just made a PR, let's see what happens [09:24] PR snapd#3860 opened: packaging: don't include any marcos in comments [09:26] zyga-suse: seems definitely something that changed under our feet because those comments are still from Simon [09:27] pedronis: I'll check changelogs [09:32] huh ? [09:32] ogra_: I cannot reproduce the issue, I'll report back but I suspect it's something funky on the reporter's machine [09:32] how did 2.28 into beta ? [09:32] *get into [09:32] ogra_: mvo uploaded it [09:32] ogra_: after 2.27.5 was stable [09:33] ogra_: its 2.28~rc1 fwiw [09:33] why ? [09:33] ah, [09:33] it doesnt say -rc1 [09:33] the thing that builds the versions cuts a but too aggressively it seems [09:33] ah [09:35] zyga-suse: meh, do you have a handle on this issue already? [09:36] mvo: which of the recent ones? [09:36] ogra_: probably worth fixing, its slightly annoyying [09:36] yeah, will confuse people [09:36] zyga-ubuntu: the name where snap-confine stops working for some people, that is rather criticial [09:36] mvo: not yet, I cannot reproduce it on debian 9 and sid [09:36] mvo: I asked for more information [09:37] zyga-ubuntu, pedronis: I would say we disable suse for the moment and deal with the other crisis first, ok? [09:37] mvo: and I'll check the code, what would trigger this [09:37] zyga-ubuntu: thanks a bunch - so all reports from debian so far? [09:37] mvo: technically it looks like snapd passes ubuntu-core as the name of the base snap [09:37] mvo: I sent a PR that might cure suse, if it fails I'll disable it [09:38] mvo: there are three reports, one is from debian "9" (but really sid apparently), the rest is unknown for now [09:38] zyga-ubuntu: ta [09:39] PR core#55 closed: Create mount points for use in exposing host system fontconfig [09:40] zyga-ubuntu: thanks for the suse fix, interessting bug [09:40] mvo: not sure if that's really it [09:40] mvo: it's rather mysterious [09:40] mvo: I was building suse package very often and I dind't hit this [09:41] zyga-ubuntu: it sounds likely [09:41] mvo: but my packaging is different and I'm on tumbleweed, not leap [09:41] zyga-ubuntu: yeah, like pedronis said, mysterious why it is happeing just now [09:41] mvo: I'll check in detail later, the core regression is more serious [09:41] zyga-ubuntu: +100 [09:41] from the logs is seems like it's running macros from inside the comments [09:41] mvo: golang packaging is being changed but I don't think those changes would be backported to leap [09:41] mvo: maybe I'm just wron [09:41] *wrong [09:42] I'll sync suse packaging later to see what's not in trunk [09:42] pedronis: yeah, this is what zyga-ubuntu fixed in his branch [09:53] zyga-ubuntu: are all reports from the same user? [09:59] mvo: so far yes [09:59] mvo: my suspicion is that something odd is going on on his machine [10:00] zyga-suse, well, seems to be sid ... [10:01] yes, it must be sid [10:01] (surprise: unstable is unstable :) ) [10:01] because it has 2.27.4-1 [10:01] mvo: it looks like we append "info.Base" [10:02] mvo: as base snap name [10:04] zyga-suse: yes, iirc if info.Base != "" [10:04] zyga-suse: which seems reasonable, no? [10:06] yes [10:07] unless you're re-execing? [10:07] then it should break due to mismatch [10:12] zyga-suse: fwiw, I tried with my debian9 install and had no luck reproducing the issue [10:12] same [10:23] sep 06 12:11:36 magpie kernel: audit: type=1400 audit(1504692696.878:207): apparmor="DENIED" operation="open" profile="snap.hexchat.hexchat" name="/run/systemd/resolve/stub-resolv.conf" pid=6685 comm="hexchat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=102 [10:23] uh oh... [10:59] heh, prune failed on ppc again [10:59] I bet PPC has even weirder tick/scheduling [11:00] PR snapd#3861 opened: interfaces: add NETLINK_KOBJECT_UEVENT to kernel-module-control [11:01] mvo: hmm, that feels ill-placed for that interface [11:02] mvo: I wonder if we are just missing an interface for something more specific [11:02] mvo: or if we should add a quirk system that lets us inject snippets for a specific snap [11:02] mvo: (ideally via store assertions) [11:02] mvo: so we could fix people in the wild [11:02] zyga-ubuntu: yeah, I agree. i asked jdstrand for suggestions [11:02] (though I don't think we refresh assertions, do we?) [11:03] complexity, schmoplixity [11:03] we can do a lot of stuff [11:03] dont' know if more moving parts will save us in the end though [11:03] zyga-ubuntu: we could udisks2 or modem-manager or any of the kobject_uevent interfaes to livepatch [11:03] mvo: "add" I presume [11:04] pedronis: I think that for things like this [11:04] oh, why did my laptop quit? [11:04] for things like this, post-releaes issues wrt security [11:04] zyga-ubuntu: correct [11:04] it would be good to 1) fix existing specific snaps [11:04] and 2) create new interfaces and let people upgrade [11:04] (people=snap devs) [11:04] I think we broke many things when netlink mediation landed [11:05] because we started to reject things previously allowed [11:05] and then proceeded to samp the extra permissions in various places [11:05] as I said we can do many things, but complexity is a proiblem in itself [11:05] pedronis: I don't disagree in any way :) [11:05] I think suse download servers are broken now [11:06] I see it both at home and on linode [11:06] we don't even apply any of those kind of changes immediately, atm [11:06] you need a new revno either way [11:06] I see [11:10] I think we need interface hooks done before we add anything more in that area [11:21] mvo: hey, noob question here: what can I monitor after dputting something to sid? [11:23] zyga-suse: yes, suse servers seems down [11:23] :/ [11:23] time to put it to manual until other fires are out I think [11:23] given master is broken atm [11:25] yes [11:25] * zyga-suse pushes branch [11:25] FYI: https://status.opensuse.org/ [11:28] pedronis: ah, it seems we both did it [11:28] btw, mup seems off/slow [11:28] that kind of day [11:28] PR snapd#3862 opened: spread.yaml: turn suse to manual given that it's breaking master [11:28] PR snapd#3863 opened: spread: disable openSUSE on linode backend [11:28] mvo: can you please merge one of the two instantly [11:29] apparently fedoraproject.org DNS in Europe is having problems, too [11:29] :) [11:29] well [11:29] it must be that kind of day then [11:30] mvo: so are we doing 2.27.6? [11:30] likely [11:30] mvo: with fixes for canonical-livepatch [11:30] I think we are waiting to discuss what's best with jdstran-d [11:30] mvo: and whatever may be causing that ubuntu-core back-from-hell [11:30] mvo: we could do a hack where we never pass ubuntu-core to snap-confine, and just change that to "core" [11:31] mvo: I'm looking at theories why this could ever happen [11:34] mvo: hmm, I think I know what may be going on [11:36] mvo: my bet is that the reporter lost the current symlink [11:36] pedronis: whatever the outcome it feels like we will have a new release === ShalokShalom_ is now known as ShalokShalom [11:51] mvo: I won though whatever took out /snap/core/current also blew everything else out of the water [11:52] mvo: I'm worried that it might be some crazy upgrade script that does rm -rf /snap [11:54] you mean in a classic snap? [11:54] or somebody using such a script on their system? [11:54] I meant the upgrade script in debian (.postinst and such) [11:54] or part of packaging? [11:55] maybe there's some bug that causes it to run the prune code [11:55] * zyga-suse looks [11:58] on ubuntu we just have postrm that rm things only on prune [11:58] sorry, on purge [11:59] right, but I think there's a delta with debian, looking at that now [12:07] zyga-suse: I merged mine turn off suse given it got green [12:07] ok [12:07] please close the other [12:08] PR snapd#3862 closed: spread.yaml: turn suse to manual given that it's breaking master [12:09] PR snapd#3863 closed: spread: disable openSUSE on linode backend [12:13] thank you [12:14] the postrm script look sane [12:16] pedronis, mvo: I wonder if the snap manager should ensure that all snap mount points exist and the units are mounted in its ensure loop [12:16] having something like that would give us a healing system that would fix whatever has affected that particular user [12:17] similarly to how snapd ensures security profiles are correct [12:21] maybe, doesn't sound 2.27+ material [12:22] * zyga-suse break [12:29] mvo: #3727 needs a master merge [12:29] PR #3727: daemon, snapstate: move ensureCore from daemon/api.go into snapstate.go [12:30] pedronis: sure, doing that now [12:31] zyga-suse: you should get an accept mail after dputing (sorry for the delay) [12:34] mvo: no worries [12:34] mvo: I got the mail (REJECTED, but that's another topic) [12:34] mvo, pedronis: I think that after we fix what affects canonical-livepatch we *must* teach snapd to start any services that are stopped/broken [12:34] as I bet the background service is broken on various running machines now [12:35] and while the .6 update will fix it it would only auto-start on the next reboot, defeating the purpose [12:37] Chipaca: ^ [12:38] zyga-suse: we will update livepatch too (the snap) [12:39] woah [12:39] http://paste.ubuntu.com/25478060/ [12:39] do we never clean up when core gets updated ? [12:41] mvo: that's another solution [12:41] * ogra_ wonders when users will start running out of inodes [12:42] ogra_: known bug, I think [12:42] ah, k, then i wont file it ... looks shocking nontheless [12:43] there's a bug open for hit [12:43] for it* [12:44] PR snapd#3861 closed: interfaces: add NETLINK_KOBJECT_UEVENT to kernel-module-control [12:44] mvo: is there any interface that allows just kobject uevent? [12:45] mvo: if no let's add it to some netlink-kobject-uevent maybe and then patch the livepatch snap to use it [12:46] I don't think that's the spirit, we don't have interfaces with so low granularity afaict [12:47] anyway seems jdstrand is thinking/looking into this [12:47] let's not do that [12:47] mvo and I have a plan and a workaround for livepatch [12:49] PR snapd#3864 opened: interfaces: add udev netlink support to hardware-observe [12:55] jdstrand: hey [12:55] jdstrand: what is the plan? [12:55] that pr ^ [12:55] zyga-suse: that achieves what you suggested in an existing interface. and that interface is the correct place for the policy [12:55] jdstrand: thanks [12:56] mvo: I commented on the PR, would love it if you could revise the commit message [12:58] zyga-suse: sure [12:58] * jdstrand thought the comment was clear (and commented on it) [12:58] but whatever you come up with is fine [12:59] ogra_: are those "just" empty mount points? [12:59] mvo, i think so, let me check [13:00] mvo, confirmed [13:02] Chipaca, pedronis: standup [13:02] omw [13:12] PR snapcraft#1519 closed: lxd: use a unique temporary folder [13:20] Son_Goku: fwiw, 2.27 has working base snaps, I would love to try this with your fedora work [13:20] Son_Goku: aiui you have a fedora base already :)? [13:20] what do you mean, working base snaps? [13:20] I have a script for making core/base snaps, yes: https://gitlab.com/Conan_Kudo/snapcore-mkrpmdistcoresnap [13:20] I stopped working it a while back because I can't test the output [13:21] now you can [13:21] :) [13:38] re [13:39] Son_Goku: I can explain [13:39] Son_Goku: but yes, we can now try [13:39] I'm going to guess that my YAML needs to be different for the base snap [13:41] needs to be base'ic ... (muhahaha) [13:44] davidcalle, hey, one more question re your issue and how it disappeared this morning - have you rebooted your box? [13:46] pstolowski: yes, I tend to reboot frequently, yesterday, the day before... [13:46] zyga-ubuntu, ^ [13:47] pstolowski: so I know what the problem is IMO [13:48] pstolowski: I can revive a branch that fixes it and we can sit on the bug in apparmor that this exposes [13:48] PR snapd#3642 closed: cmd/snap: get keys or root document [13:53] pstolowski: let me know if you want to have a step back chat about Plug/SlotData today or tomorrow ? [13:53] mvo, I see this error https://paste.ubuntu.com/25478387/ [13:53] in most of the ubuntu cores [13:53] pedronis, sure, will let you know, probably tomorrow. thanks [13:54] zyga-suse, oh, so it goes deepeer [13:54] zyga-ubuntu, ^ [13:54] pstolowski: it's pretty simple but the fix ran into a bug in the kernel [13:54] I reproduced it [13:54] zyga-ubuntu, is this about namespaces? [13:54] pstolowski: the bug is from 2016 [13:54] pstolowski: its open on LP [13:54] mvo, but not sure it is a bug or not [13:54] zyga-ubuntu: 2016? psh, amateur :-p [13:54] mvo, it is in the border line === jkridner|pd is now known as jkridner [13:55] zyga-ubuntu, why couldn't I reproduce it? and it was happening to davidcalle every day (till this morning)? [13:55] * zyga-ubuntu just had a very interesting question from tech-illiterate friend [13:55] pstolowski: I'll explain in a sec [13:56] question was: should I download amd64 or i386 build of ubuntu if I have a celeron procesor, he was going for i386 because his box is not amd [13:56] i'm glad though it's not a fundamental issue with install hook though [13:56] I wonder how many people use 32bit systems because of that [13:56] pstolowski: so the hook is in the core snap [13:56] pstolowski: which is the root fs [13:57] zyga-ubuntu, no, the hook is in a reguar snap [13:57] pstolowski: to reproduce have two revisions of a snap and two revisions of the core [13:57] pstolowski: (sorry, I meant snap-exec) [13:57] right [13:57] pstolowski: start on old-core and old-snap and run the snap (or any hook) so that we have a mount namespace [13:57] pstolowski: now refresh core to new-core (with new hook support in snap-exec) and refresh to new-snap [13:58] pstolowski: when core changes we don't have a mechanism for refreshing existing mount namespaces [13:58] pstolowski: the fix, even once my branch lands, won't always refresh it either [13:58] pstolowski: because we don't do it for valid reasons [13:58] pstolowski: a simple fix is to run snap-exec via ... and bare with me... /var/lib/snapd/hostfs/snap/core/current/usr/lib/snap-exec [13:58] pstolowski: because *that* will be always fresh [13:59] pstolowski: it should be a one liner in snap run and one liner in snap-confine.apparmor.in [13:59] pstolowski: try writing a case that reproduces this first [13:59] pstolowski: it's a very interesting bug [13:59] pstolowski: NOTE: it's sufficient to test a subset [13:59] pstolowski: stat snap-exec before and after core refresh [13:59] pstolowski: and if the inode is not changed, we are running the wrong one [14:00] pstolowski: if you can come up with a full test that would be nice but I think it's not really needed given how hard it is to set this up [14:00] pstolowski: as long as we run snap-exec via hostfs we're fine [14:00] pstolowski: one more thing is that new base snap support makes this more complex [14:00] pstolowski: but I think bases need to have hostfs so the fix is universal [14:00] pstolowski: that's all [14:00] mvo: ^ [14:01] * zyga-ubuntu returns to 14.04 debugging [14:01] zyga-ubuntu: uh, interessting [14:03] zyga-ubuntu, wow, that's interesting bug indeed. thanks for explanation! I need to copy it somewhere.. [14:09] a second review for 3854 would be great (should be trivial) [14:11] * zyga-ubuntu looks [14:12] mvo: question: why use mockCommand, instead of mocking SystemctlCmd? [14:12] ah, I already did [14:12] mvo: another question: why aren't i asking this in the PR? [14:12] mvo: thank you for the explanation [14:12] PR snapcraft#1531 opened: Release changelog for 2.34 [14:12] ah, corecfg [14:12] ah [14:12] Chipaca: I guess corecfg actually would need some work to do it this way, might not be a bad idea to convert it [14:13] * zyga-ubuntu had an idea to have a way for a given module to offer to mock itself out [14:13] so that we don't have to know how to mock internal details in tests of something else that just happen to use the public API of that first thing [14:14] mvo: corecfg is already using systemd.SystemctlCmd [14:14] zyga-ubuntu: ^ [14:14] it's not doing its own exec [14:14] Chipaca: sure but there's no way to say .MockEverythingExternal() [14:14] Chipaca: so if uses it needs to know too much in its tests [14:15] ideally this would be exposed for tests in the public api but not clutter the real public api [14:15] I think that's hard in go [14:15] zyga-ubuntu: so you're saying that knowing the details of the implementation of something, and mocking one of the details, is better than knowing the API that thing exposes and using that in your tests? [14:15] Chipaca: mvo: seems to be doing both [14:16] pedronis: where's it calling systemctl itself? [14:16] it's also using s.coreCfgSuite.SetUpSuite(c) [14:16] sorry [14:16] Chipaca: I no, I'm saying that if I work on A that uses public api of B I should not need to know internal details of B that so that I can mock it out (again: external interactions of B) [14:16] Chipaca: maybe I'm wrong but I feel it would be nice [14:16] Chipaca: I mean the tests are using mocking SystemctlCmd as well [14:16] Chipaca: aha, interessting, so I mocked the wrong thing [14:16] Chipaca: thanks, I will tweak that [14:17] zyga-ubuntu: it sounds like we agree on the premise [14:17] Chipaca: I don't if the code is using both, I asked that in the review though [14:17] Chipaca: +1 :) [14:17] * zyga-ubuntu is starving [14:17] I'll skip 14.04 debugging and get a meal [14:17] zyga-ubuntu: so based on that, setting SystemctlCmd is the right thing to do, and using MockCommand to replace systemctl, the wrong thing [14:17] zyga-ubuntu: gah! go get food :-) [14:17] Chipaca: yes [14:17] Chipaca: though it'd be nice if we could standardize that more [14:18] Chipaca: .MockIt() or whatever [14:18] zyga-ubuntu: just as soon as we standardize all modules to be .DoIt() [14:19] Chipaca: I think zyga-ubuntu as a point here [14:19] Chipaca: modern style is MockFoo(foo) (restore func()) [14:19] or at least I hope that was his point [14:20] pedronis: in export_test yes; would we want to make that part of the public api also? [14:20] we do sometimes [14:20] ah [14:20] having a global that you can clobber isn't much better [14:21] oh, agreed, it's not pretty [14:21] Chipaca: release.MockOnClassic for example [14:21] etc [14:21] man, this battery lasts ~10 minutes :-( [14:21] if we need this across packages we do it [14:21] I mean put it in the pkg public api [14:21] pedronis: sounds like an easy PR [14:22] they all are, they say (at the beginning) :) [14:22] not looking to add another cleanup to my current pile [14:23] i'll do it at some point maybe [14:24] is there an interface that allows to take a screenshot of the desktop ? (I am not aware of technicalities on what needs read permission under X11 for that) [14:25] om26er: x11 is probably good enough for that [14:25] om26er: but it won't work on wayland; i don't think we've got anything that'll work there yet [14:26] Chipaca: yeah, I am only concerned about X11 for now. Will check if just connecting x11 works. === chihchun is now known as chihchun_afk [14:27] pedronis: that was the point [14:28] Chipaca: ~10min on your laptop? [14:28] zyga-ubuntu: it's an 8yr old battery [14:28] yes [14:28] Chipaca: this is about adding "systemd.MockSystemctlCmd()"? [14:28] ah [14:28] it's good it hasn't exploded or deformed [14:28] mvo: not in that PR [14:28] Chipaca: I missed most of the earlier discussion [14:29] Chipaca: it might make sense, should be easy(ish) [14:29] mvo: bottom line for your PR: replace SystemctlCmd everywhere, don't use MockCommand [14:29] mvo: how about systemd.MockExternalInteractions [14:29] MEI [14:29] zyga-ubuntu: not in that PR! [14:29] MEIBI [14:29] Chipaca: fair enough [14:29] ExternalInteractions [14:29] what are those? [14:29] Chipaca: yes, that part I got out of it [14:29] mvo: that's all, for your pr [14:29] the rest is a bigger change that needs more work [14:29] pedronis: vague stuff including things like running processes [14:29] MockBoundaryCrossingPokings [14:29] +1 [14:30] zyga-ubuntu: pedronis: that would mean providing two functions at least [14:30] ? [14:30] SystemctlCMd and JournalctlCmd [14:30] Chipaca: we mock both with SystemctlCmd now ? [14:31] or are you saying there are two globals? [14:31] I don't care honestly, as long as what one has to do is understanble [14:31] ExternalInteraction seems a bit too vague for my taste [14:31] there are two globals [14:31] pedronis: yes, that was a bit my point [14:32] we're trying to pretend everything is homogeneousible (?) and it clearly isn't [14:33] jdstrand: hey, can you please review the changes made to https://github.com/snapcore/snapd/pull/3621 -- I'd like to land it when you give it a +1 [14:33] PR #3621: cmd/snap-{confine,update-ns}: apply mount profiles using snap-update-ns [14:35] jdstrand: I wrote the child profile, should be uncontroversial apart from broad/open umount [14:36] jdstrand: I can tighten that further with more PRs or inside this one, as you prefer [14:36] jdstrand: but landing this would allow me to propose useful enchancements [14:36] Chipaca: an alternative would be to simpliy use "testutil.MockCommand("systemctl")" maybe? less elegant I guess [14:37] mvo: what do you think about my idea to let a module offer to mock its internals out as a public API? [14:38] mvo: i see MockCommand as a last resource; the path fiddling it does is rather racy [14:38] zyga-ubuntu: yeah, I think that is consistent with other uses we have [14:39] Chipaca: ok [14:39] offtopic, I read how getenv/setenv work and OMG :/ [14:39] that's not a nice API [14:39] zyga-ubuntu: hehe [14:39] not nice because designed with stupid assumptions eons ago [14:39] zyga-ubuntu: wait, did you look at it in go, or in glibc? [14:39] but less setenv we do, the better [14:39] in glibc [14:39] zyga-ubuntu: look at it in go sometimes [14:39] sometime* [14:39] ah, will do [14:40] zyga-ubuntu: that's the source of the raciness i mentioned [14:40] Chipaca: the glibc one has other issues and I suspect gets called eventually [14:40] OTOH even setenv is MT-Unsafe [14:40] yes [14:40] * zyga-ubuntu just checked [14:41] which probably explains why tests fail when started with -count=1000 [14:41] getenv just borks [14:41] and mocked commands don't get used [14:41] yep [14:41] zyga-ubuntu: so, we could work around that [14:41] especially coreconfig tests [14:42] yes, I think we ought to [14:42] by tweaking the path super early [14:42] design it not to fail so easily [14:42] and having a single test path env [14:42] I wish golang had a warning for setenv [14:43] like gcc linker scripts have for gets [14:47] zyga-ubuntu: just don't wish too hard [14:47] zyga-ubuntu: look what happened to setuid [14:48] /o\ [14:59] cachio_: if you have a moment could you please have a look at spread-cron and in particular the snapd-vendor-sync branch? it seems like its not doing things automatically anymore. it should sync the lp:snapd-vendor branch on each merge to master but we had several merges today and nothing got merged since 21h. I think the last merge happend when I clicked a manual rebuild. maybe the actual cron thing is no longer running? [15:00] mvo, I saw the same and restarted the snap like 30 mins ago [15:00] it should be running again [15:01] snapd-vendor-sync is on the queue to be executed [15:01] mvo, not sure what happened, I'll take a look [15:03] mvo, I am gonna disconnect now [15:03] backing from the vpn [15:04] cachio_: thank you [15:07] Chipaca: is http://paste.ubuntu.com/25478666/ what you had in mind for better mocking of the systemctl thing? [15:09] mvo: that looks an awful lot like what pedronis and zyga were suggesting, which I recommended happen in a different PR [15:09] mvo: or is this a different PR [15:09] pedronis: 3858 should be ready for another look [15:10] Chipaca: it is a different pr, its nothing at all right now, I was waiting for tests and that seemed to be about the level of mental capacity I had left :/ so if it looks sane(ish) I can turn it into a PR [15:10] mvo: ah ok :) [15:11] mvo: now the same but for journalctl ;-D [15:12] Chipaca: yeah, much less exposure that one === cachio is now known as cachio_lunch [15:31] PR snapd#3865 opened: many: provide systemd.MockSystemctl() helper === chihchun_afk is now known as chihchun [15:41] mvo: was having a walk, seems it gets dark earlier and if I wait after dinner for my walk I end up not walking at all [15:42] pedronis: no worries, thank you [15:42] pedronis: enjoy the walk! [15:42] mvo: I'm back [15:42] mvo: #3727 seems mergeable to me [15:42] PR #3727: daemon, snapstate: move ensureCore from daemon/api.go into snapstate.go [15:42] but I let you do the honors if you want/need to tweak the commit message [15:45] pedronis: I think I take a short break now myself, feel free to merge otherwise I will when I'm back [15:46] pedronis: your 3856 also looks ready for merging (not sure if you want to tweak further but definitely enough +1 :) [15:47] zyga-ubuntu: this continues to be on my todo list. it is not forgotten. yesterday had a lot of things I had to respond to over the weekend. note I need to review more than the child profile-- I need to do a careful review of the arg parsing (and env, but I expect that part to go fast) [15:47] mvo: I wanted to merge #3727 because I expect conflicts/needing some tweaks [15:47] PR #3727: daemon, snapstate: move ensureCore from daemon/api.go into snapstate.go [15:47] zyga-ubuntu: it is also extremely high on said list [15:47] with #3856 [15:47] PR #3856: overlord: introduce Mock which enables to use Overlord.Settle for settle in many more places [15:50] mvo: fyi, https://github.com/snapcore/snapd/pull/3864/files#r137309433 [15:50] PR #3864: interfaces: add udev netlink support to hardware-observe [15:51] mvo: I'd like you to add 'bind' to the seccomp policy. this isn't for livepatch since it gets it elsewhere, but it is so hardware-observe can standalone [15:51] PR snapd#3727 closed: daemon, snapstate: move ensureCore from daemon/api.go into snapstate.go [15:52] mvo: I would've done it in another PR but it looks like you didn't update that one yet, so thought I'd mention it. if you prefer to just merge it as is, that is fine, I'll do a followup PR [15:59] PR snapd#3847 closed: tests: run the tests/unit/go everywhere [16:02] Chipaca: Hey, which release did we get tab-completion fully working again? Isn't that 2.27? I probably missed it in the notes [16:02] niemeyer: what do you mean by "fully working" [16:02] Chipaca: Usable by snaps without any local changes [16:03] niemeyer: 2.27.2 had snippets, meaning it works without local changes on classic [16:03] and falls over flat on core [16:03] niemeyer: 2.27.4 skips creating snippets on core [16:03] or maybe that's 2.27.5 [16:03] yeah, .5 [16:03] Chipaca: Aha, yeah, so it's 2.27 indeed.. I will fix the notes [16:04] but 2.27 doesn't have it [16:04] there are features in the .s of 2.27 [16:05] Chipaca: Yeah, sorry.. there's a bit of confusion around that.. I've made a mroe general 2.27 announcement since this was really when we've sorted all small blockers and updated the core snap [16:05] ok === cachio_lunch is now known as cachio [16:10] mvo, about hte base snaps test, do you know why it did not fail on linode but it failed on the external:ubuntu-core? [16:36] PR snapd#3866 opened: many: implement fetching sections and package names periodically [16:36] it lives! [16:40] hmm [16:41] so in my update-manager snapd is showing up as "Tool to interact with Ubuntu Core Snappy" in the UI [16:41] i wonder if we should change that to something less specific :) [16:53] Feel free to steal my summary and description from my package [16:58] cachio: yes, it failed because on external it uses a snap that was build ~6month ago and on linode it uses our self-build snapbuild [16:59] jdstrand: if you mention it in the PR I will address it next (adding bind) [16:59] mvo: I did [16:59] mvo, ok [17:00] mvo, ok, thanks [17:01] jdstrand: excellent, thank you [17:04] hi, please could somebody address https://forum.snapcraft.io/t/snapcraft-autopkgtest-failure/2007 [17:07] * zyga-ubuntu breaks for dinner [17:08] doko: hey, good to see you [17:09] finally doko found his way into the proper channels :) [17:25] mvo: ok, I've looked at this quite a bit. I looked to see what had network netlink rules and udev and all interfacecs that had both are fine (these are all slot implementations) [17:26] mvo: I then looked at avahi and found that what it is missing is NETLINK_ROUTE, but it gets that with 'network' so no regression in that snap (I'll still fix that interface for comleteness) [17:27] mvo: I then checked all the interfaces with just a 'network netlink' rule. these were all slots that were heavily checked before and have been verified by CE to work without regressions, so that should be fine [17:27] PR snapcraft#1532 opened: tests: update the meson test to latest meson requirements [17:28] mvo: there were three plugging interfaces with 'network netlink'-- account-control, network-control and network-observe [17:28] mvo: account-control is correct and doesn't need to be changed [17:30] mvo: I then looked in the store to see what plugged account-control, network-control or network-observe and found that about half plug neither hardware-observe, x11 or unity7 (ie, things that give NETLINK_KOBJECT_UEVENT) [17:30] mvo: so I investigated if we should just add NETLINK_KOBJECT_UEVENT to network-control and network-observe, and looking at kernel sources in net/core, I think we should [17:32] mvo: therefore, I will proactively send up a PR for network-control and network-observe for NETLINK_KOBJECT_UEVENT, even though we have no regression reports on these. this will make it so nothing in the stable channel in the public store will regress on NETLINK_KOBJECT_UEVENT, because everything that could've gotten it for free with bare 'socket' before will get the new rule via an interface that they already plugs === JoshStrobl|zzz is now known as JoshStrobl [17:38] jdstrand: sounds good, I guess we want to also pull this into 2.27.6 then, right? [17:39] mvo: if you are rolling a 2.27.6, yes [17:39] jdstrand: I think it makes sense [17:39] jdstrand: if only for canonical-livepatch - even though we workaround it it feels better to do a proper fix [17:39] mvo: I'll be sending up the PR for network-control and network-observe in just a bit [17:40] jdstrand: thank you, I will look at it in my morning [17:40] mvo: it'll be separate from any other policy work so it is easy to pull in [17:40] mvo: should I target master, 2.28 and 2.27? [17:42] jdstrand: we will need it in 2.27 and 2.28 and master :/ but if I can cherry pick it then one PR is enough and I just cherry-pick it [17:46] mvo, is it gonna come a new 2.27 today? [17:46] a beta one? [17:48] mvo, I have physiotherapy now, I can start as soon you have something === cachio is now known as cachio_afk [18:23] PR snapd#3867 opened: interfaces/network-{control,observe}: allow receiving kobject_uevent() messages [18:39] PR snapd#3869 opened: interfaces/network-{control,observe}: allow receiving kobject_uevent() messages [18:42] PR snapd#3870 opened: interfaces/network-{control,observe}: allow receiving kobject_uevent() messages for 2.27 [18:43] mvo: ok, fyi ^ (3867, 3869 and 3870) === chihchun is now known as chihchun_afk === cachio_afk is now known as cachio [19:28] PR snapcraft#1532 closed: tests: update the meson test to latest meson requirements [21:58] PR snapd#3871 opened: tests: fix regex to check core version on snap list [23:26] PR snapcraft#1533 opened: demos: update the name of the remote mqtt part [23:38] PR snapcraft#1534 opened: Fix UnboundLocalError for chmod_path in jhbuild plugin