=== plars_ is now known as plars === Raboo__ is now known as Raboo === chesty_ is now known as chesty [01:48] hello [01:48] I just threw up this question on discourse: https://forum.snapcraft.io/t/lxd-profile-for-content-interface-bind-mounts/15344 [01:48] trying to get some hits [01:49] possibly ill dig a little deeper and ask on the lxd side of things too as it seems to be more of a lxd/lxc thing [01:50] thought I would start here [01:50] thanks! [06:30] morning [07:51] meh, wish we could access the console of spread nodes [07:59] hey, is there a way to make snapd skip confinement when: [07:59] system does not fully support snapd: apparmor detected but insufficient permissions to use it [08:02] Saviq: i don't think so, where did you get that message? [08:02] morning [08:02] pstolowski: hey [08:03] mborzecki: I've a VPS (a container actually) at a friend's [08:04] it's his custom kernel, though [08:08] Saviq: looking at the code, we're trying to poke /sys/kernel/security/apparmor/profiles as root with some comments that problems can happen in unprivileged lxd [08:10] Saviq: anyways, it's a sanity check so further snapd operation will be blocked sadly, we dont' really have a mechanism to degrade apparmor support in this scenario [08:11] ack, thanks :/ [08:11] mborzecki: so the solution would be to get permissions to actually set up apparmor inside the container? [08:12] Saviq: yes, i believe so [08:13] Saviq: simplest check would be to run aa-status as root inside the container [08:13] "You do not have enough privilege to read the profile set." [08:31] Saviq: yeah, that's what snapd is hitting too, maybe zyga remembers more details about that [08:32] mvo: pedronis: hi [08:32] hey mborzecki [08:32] and good morning pedronis [08:32] mborzecki: thanks, will negotiate with the hoster ;) [08:32] mvo: i was looking through 'reflash magic' setup, did an update to grub script, but there's more problems [08:32] mborzecki: uh, like what? [08:36] mvo: I fixed Stop, bit ugly (indeed it needs to use syscall.Select) [08:37] mvo: ah, w8, maybe not :P sorry to cause panic, thought we added --image-size to the u-i [08:37] pedronis: thank you, did you push it already? [08:37] mvo: no, I need to clean up what I have [08:37] pedronis: cool [08:37] pedronis: yeah, sleeping over it I think without a proper fix this is too nasty so thank you so much for fixing it [08:38] mvo: I don't know about nasty, but is definitely confusing [08:38] because a Stop that doesn't stop is confusing :) [08:38] pedronis: exactly :) [08:39] mvo: have you tried this maybe https://gist.github.com/mvo5/fa14638782fe30949998096d7b3c6314#gistcomment-3167661? [08:39] mborzecki: did you try it for the netlink socket? [08:39] mborzecki: let me quickly give it a go [08:40] mborzecki: I think the issue is that go is "smart" and determines if it can poll the socket or not and for netlink there is just no support [08:40] mvo: right, the change i did is make it nonblocking before wrappint ig as os.NewFile(), the implementation chceks whether the fd i non-blocking and sets up a poller for it [08:40] mvo: I tried that [08:41] sorry [08:41] mborzecki: I tried that [08:41] but go doesn't consider it non-blocking [08:41] and doesn't set up netpoll for it [08:41] for a netlink socket [08:42] because it's actually the reverse [08:42] pedronis: i stepped through it with debugger, and was hitting the non-block kind [08:42] pedronis: which go version did you use? [08:42] mborzecki: yea, but that's not relevant [08:42] it's whether netpoll is happy with [08:42] which it isn't indeed [08:42] go wil set up non blocking itself [08:42] if netpoll is happy [08:44] it's interessting, I tried mborzecki version with netlink and it seems to stop the socket https://paste.ubuntu.com/p/Y2z9dh8rDT/ [08:44] * mvo scatches head [08:44] different go versions? [08:45] pedronis: I have 1.12 on one and 1.13 on the other, let me try again [08:45] on 1.10 [08:45] it didn't work for me [08:45] * mvo tries that too [08:47] pedronis: mvo: https://github.com/golang/go/commit/ea5825b0b64e1a017a76eac0ad734e11ff557c8e 1.11+ [08:48] that would explain what pedronis saw [08:48] anyway we are still 1.9 [08:48] :/ [08:48] yeah [08:48] that makes sense now [08:48] slightly sad [08:49] mvo: do we care if it leaks though? [08:49] mborzecki: it's messy interface wise [08:49] especially in tests [08:50] we would need to change the interface to not have a Stop [08:50] etc [08:50] it's not pleasant to reason about [08:51] pedronis: maybe we could do some extra mocking for tests [08:51] mborzecki: as I said I have a fix [08:51] is not beatiful, but I will make localized and leave a TODO for when we are 1.11+ or something [08:53] pedronis: mvo: do we need to check udev monitor as well? [08:53] mborzecki: there's an indirection there that doesn't make the problem apparent [08:53] but yes osutil/udev has the same kind of bug [08:58] mvo: do you know if there's a way to access the console of spread system when cachio is not around yet? [08:59] mborzecki: afaik there is none :( [09:07] PR snapd#8097 opened: tests/lib/prepare: fix hardcoded loopback device names for UC images [09:08] mvo: trivial change ^^ [09:08] mborzecki: ta, looking [09:26] PR snapd#8092 closed: timeutil: add a unit test case for trivial schedule [09:53] PR snapd#8098 opened: httputil: remove workaround for redirect handling in go1.7 [09:58] PR snapd#8099 opened: httputil: remove go1.6 transport workaround [10:20] Hey, just waving. Still sick and weak. Sorry [10:21] zyga: hey, nice to see you! get well [10:21] Uh, I hate this state I’m in. I’ll check back later. [10:22] zyga: yeah, being sick sucks [10:24] pstolowski: could you please re review the uio branch. We removed some code. [10:27] zyga: will do [10:37] mvo: I pushed this branch: https://github.com/pedronis/snappy/tree/netlink-channel-n-stop [10:40] pedronis: thanks, looking [10:41] pedronis: if you guys want I have a epoll.go package [10:48] pedronis: this is super nice, thanks for this [10:50] cp -a "$SNAPD_UNPACK_DIR/usr/lib/snapd/snap-bootstrap" unpacked-initrd/main/usr/lib/snapd/snap-bootstrap [10:50] #8097 trivial pr, needs 2nd review [10:50] PR #8097: tests/lib/prepare: fix hardcoded loopback device names for UC images [10:50] and a bit of my clipboard :) [10:50] mborzecki: haha [10:50] pedronis: do you want to propose it as a separate pr or should I cherry pick it into my netlink pr? [10:51] mvo: what's funny is that the same repackign code running in qemu builds a working kernel snap, but one on gcp panics with no init, probably borked initrd [10:51] mvo: mmh, ideally we should split in everyhting but the new netlink bits [10:52] and then you could pick those [10:53] glad at least we can download the build uc20 image from spread system [10:54] pedronis: sounds good,I can take your code and chunk it into smaller PRs, I guess this is what you asked? [10:55] mvo: yes, one PR with the non netlink-bits [10:55] and the rest picked up by your PR [10:55] mvo: thanks if you can do that, I'm doing spec work atm [10:56] mvo: sorry, I mean the not new netutil netlinks bits [10:56] basically osutil/udev and o/ifacestate [10:57] pedronis: thanks, yeah, happy to do that [11:14] PR snapd#8097 closed: tests/lib/prepare: fix hardcoded loopback device names for UC images [11:23] sergiusens: did you see my query on the forum? (re. github actions) [11:26] jamesh: hi, did you see my last bit of comments on #7456 ? [11:26] PR #7456: usersession/client: add a client library for the user session agent [11:28] pedronis: yeah. I'm just tidying up some of the tests for it. In these cases we have an error message from the agent, so I'm not sure whether we want to lose that when the error value is badly formed [11:28] PR snapd#8100 opened: httputil: add support for extra snapd certs [11:29] PR snapd#8054 closed: snap: add `snap pack --compression=` options [11:31] mborzecki: I'll miss SU today and be out for 2ish hours, so if you want me to try anything more on uc20 spread testing, let me know, I might not be back til after your EOD [11:31] mvo: are you going to tweak those comments you commented on? when you make the PRs [11:36] * pedronis lunch [11:37] ijohnson: i'll leave notes in the standup doc for you [11:40] Sounds good [11:54] pedronis: yes, happy to [12:04] PR snapd#8101 opened: netlink: fix/support stopping goroutines reading netlink raw sockets === mpt_ is now known as mpt [12:15] meh, more uc20 image woes https://paste.ubuntu.com/p/S3TP4BBPQT/ [12:16] mborzecki: did the kernel/initrd change or is this in the dev branch? [12:18] mvo: nothing changed, supposedly the same kernel image i'm using, somehow when built on gcp this does not mount [12:19] mborzecki: so this is the repackaging and it's not quite working? [12:19] mborzecki: is this with the new snap-bootstrap? maybe we regressed here :( [12:20] PR snapd#8102 opened: o/ifacestate: move ResolveDisconnect to ifacestate [12:20] mvo: i called snap-bootstrap manually, and the line is correct, tells to mount /dev/disk/by-label/ubuntu-seed under /run/mnt/ubuntu-seed [12:20] pedronis: ^ i hope this captures the idea we discussed [12:20] pstolowski: thanks, will look [12:21] pedronis: re 8100> picking up the certs immediately, we could do what we do for the proxy and have a "Certs func(*http.Request) (*x509.CertPool)" (just like we do for proxy. is that what you have in mind here? [12:22] bbiab [12:26] mvo: I don't know , just fearing it will be a bit messy [12:26] pedronis: I can explore the callback option, then at least it's equally messy to how we support the proxy :) [12:38] mvo: thanks for pushing #8101, of course I cannot review it :) [12:38] PR #8101: netlink: fix/support stopping goroutines reading netlink raw sockets [12:48] hmm i think i know what's going on [13:08] pstolowski: see #8101 [13:09] PR #8101: netlink: fix/support stopping goroutines reading netlink raw sockets [13:26] pstolowski: is this correct https://github.com/snapcore/snapd/pull/7863#discussion_r375827782 ? [13:26] PR #7863: interfaces: add uio interface [13:27] yay, got a working image in gcp [13:27] \o/ [13:27] re === ricab is now known as ricab|lunch [13:31] well, at least i did manage to download the image and make sure it boots and seeds inside qemu locally ;) [13:32] haha too soon, seeding not complete [13:33] PR snapd#8103 opened: snap-bootstrap: store encrypted partition recovery key [13:34] at least a familiar problem https://i.imgur.com/BOn7yOX.png [13:35] but it did work on gcp [13:35] yay! [13:42] PR snapcraft#2912 opened: meta: do not prime commands with adapter == "none" [13:45] PR snapcraft#2913 opened: spread: disable journal debug dump unless configured [14:45] mborzecki: any progress ? [14:45] * ijohnson is around for a little while [14:46] ijohnson: got it to work [14:46] ijohnson: pushing changes [14:47] ijohnson: still something off about ubuntu-core-initramfs, somehow i need to use the /usr/lib/ubunto-core-initramfs/main as skeleton of the rootfs, not the unpacked one [14:47] mborzecki: that's great that you got it to work [14:47] are you booting from 20.04 now or did you revert to 18.04 ? [14:47] ijohnson: 20.04 [14:47] ijohnson: it needed a little tweak in grub script [14:47] ah okay, you fixed the grub.cfg and script then [14:48] great [14:49] ijohnson: please take a look at the PR, i've tried to comment all the quirks [14:49] looking now [14:49] need to leave now, back in 2h or so [14:49] ack [15:01] PR snapd#8098 closed: httputil: remove workaround for redirect handling in go1.7 [15:01] PR snapd#8099 closed: httputil: remove go1.6 transport workaround === ricab|lunch is now known as ricab [15:06] mvo: hey, I don't know the status of 2.43 point releases, but if you are doing another one, cherry-picking https://github.com/snapcore/snapd/pull/8091#issuecomment-582935623 might make sense. we have (at least) two different forum topics and a bug on it... [15:07] PR #8091: interfaces/greengrass-support: add /dev/null -> /proc/latency_stats mount [15:07] mvo: and the url above is to a comment discussing that possibility in case you want to respond [15:08] jdstrand: thanks, will cherry pick [15:08] jdstrand: there will be another one to include uio [15:09] jdstrand: I added the 2.43 milestone, than kyou [15:09] mvo: having a break and then will work on the uio backporting prep [15:09] mvo: thank *you* :) [15:10] pedronis: thank you, I focus on reviews and download now [15:12] pedronis, mvo: fyi, I have one more 2.44 review (it's a big one though). I hope to do that tomorrow. I'll then make a concerted effort to squeeze in the non-2.44 reviews while I work on other things [15:21] * cachio lunch [15:22] jdstrand: thanks [15:36] PR snapd#8104 opened: interfaces/builtin: backport verifySlotPathAttribute (2.43) [15:39] ijohnson: #8091 can be merged, right? and then backported [15:39] PR #8091: interfaces/greengrass-support: add /dev/null -> /proc/latency_stats mount [15:43] sorry was a bit afk [15:44] pedronis: let me look at the PR, but did it change? [15:44] ijohnson: ? [15:44] pedronis: that PR is fine, it can be merged I wasn't sure why you were asking me if it could be merged [15:45] ijohnson: because I haven't looked into it at all, but you reviewed and is all green [15:45] pedronis: yes it has +1 from jdstrand so it's fine [15:45] I tend to not merge things that I didn't even skim [15:45] even if they are all green [15:45] ack, np [15:45] I can merge it [15:45] thanks jdstrand [15:45] I've been told I can merge simple things like that :) [15:46] ijohnson: but note, I discussed this with mv o eralier and he will cherrypick [15:46] jdstrand: cool yeah I was just going to say I defer to mvo about 2.43 vs 2.44 [15:47] ijohnson: done. it is milestoned for 2.43 (by mvo) [15:47] yeah [15:48] mvo: sorry, seems I wasn't clear in #8063, anyway I can work on it at some point soon [15:48] PR #8063: cmd/snap: implement 'snap remove-user' [15:48] PR snapd#8091 closed: interfaces/greengrass-support: add /dev/null -> /proc/latency_stats mount [15:48] 8091 is now cherry picked [15:49] pedronis: oh, sorry, I misunderstood [15:49] pedronis: I can look at this too (or download) either way is fine [15:50] mvo: download probably needs more attention giving it seems we need to understand if we have a bug or not [15:50] *given [15:50] pedronis: ok, I'm looking at this right now anyway so I will just keep doing that [16:34] pedronis: fwiw, I also see what john sees, I don't get 206 from the cdn, only 200 with the old and the new code, investigating now [16:36] mvo: fun with FdSet being platform dependent in the netlink stuff, trying to see if I can avoid need +build stuff [16:37] pedronis: uh, fun :( [16:38] re [16:40] hey mborzecki I ran a spread test of your branch and it passed on gce for me [16:40] \o/ [16:40] yay :P [16:40] I don't think the tests on travis have been run yet however [16:40] that's ok, they'll get their chance [16:40] yeah still in "received" state [16:41] ijohnson: but i understand the base/modeenv branch worked for you right? [16:41] it would be good to get someone else to +1 your branch and I can merge it in my PM when it goes green (hopefully 🤞) [16:41] mborzecki: which PR ? [16:42] 8073 [16:42] err [16:42] #8076? [16:42] PR #8076: boot: add TryBase and BaseStatus to modeenv; use in snap-bootstrap [16:42] ijohnson: 8075++ [16:43] haha, I have not tried it in spread, but it worked locally with qemu before, I was going to merge master into thos PR's after the spread tests are working so hopefully we can get a real spread test of the changes [16:47] ijohnson: are 8075 and 8076 even landable alone, or we can only land 8077 ? [16:48] pedronis: 8075 is fine because kernel= in the modeenv is not used anywhere anymore [16:49] pedronis: 8076 is fine because it's just snap-bootstrap reading base_status and try_base, but nothing will write those into the modeenv until 8077 [16:49] so yes both of those PR's are standalone [16:49] I still need a 2nd review on 8076 however [16:49] 8075 could be merged today if mbrozecki's branch is green [16:50] (it has 2 +1s) [16:50] ijohnson: my plan was to review 8076 after 8075 is in [16:50] pedronis: sounds good, I am hopefully I can get 8075 in today so you could review 8076 tomorrow [16:51] mvo: mabye I fixed it, cross/go-build passes now [16:51] hm, so "curl -v -L -o /dev/null -r 1000000 https://api.snapcraft.io/api/v1/snaps/download/TJEfggNhgEJ4XKJ8o7ahsvRklz5kRK5w_29.snap" does not seems to work, it seems our cdn just ignores the range header [16:52] * mvo is puzzled if he misses anyhing [16:52] pedronis: oh, nice! [16:53] mvo: maybe ask the store people [16:59] pedronis: yeah, asking there now [17:00] PR snapd#8069 closed: tests: build the initramfs + kernel snap for UC20 spread tests [17:09] PR snapcraft#2914 opened: meta: ensure Application passthrough is scrubbed for snap.yaml [17:29] ijohnson: i'm logging out for today, feel free to push fixes to 8094 if needed and land it, once it's merge i'd like to investigate why there's a problem using unpacked initrd 'main' root, perhaps we're still missing something there (or just a bug in u-c-i) [17:30] mvo: of course it needs to do something sane even if range doesn't work [17:33] pedronis, mvo i'm having failure of nested/classic/hotplug tests on 8101; i'm going to check if it regressed with master too. fwtw i see this: Feb 06 17:30:26 feb061718-477082 snapd[11297]: udevmon.go:110: udev monitor stopping timed out [17:33] Feb 06 17:30:27 feb061718-477082 snapd[11478]: udevmon.go:148: udev event error: Internal error: bad file descriptor [17:34] but perhaps it's shutdown code and unrelated to the test failure [17:34] pstolowski: that migh be a prexisting behavior [17:34] pedronis: yeah, part of the problem is that we don't check for 206 on range so we already do the wrong ting [17:34] cachio: can you check if we're running nested tests for hotplug nightly? [17:35] pstolowski: I think the disconnect in udevmon might be a bit of a hammer [17:35] pstolowski, passed 14h ago [17:35] https://travis-ci.org/snapcore/spread-cron/builds/646683311 [17:36] pstolowski: it might need more tweaks [17:37] cachio: thanks, we're running it on 16.04 only though? [17:39] pstolowski, 16 and 18 [17:39] cachio: i don't see 18 in that log, is it a separate log? [17:39] pstolowski, 16 -> core and classic [17:39] 18 -> core [17:39] pstolowski, https://travis-ci.org/snapcore/spread-cron/builds/646683311#L5829 [17:41] cachio: was there a specific reason we don't run it on 18 classic [17:41] ? [17:41] pedronis: cachio: what are your thoughts on organizing the spread tests to have a dir "tests/core/..." with all the UC specific tests in there and then having tests in there specific to say uc20 just be named so like "tests/core/uc20-basic" and filter with systems: in the task.yaml ? [17:42] ijohnson: yes, something like that [17:42] I just found a couple of tests that still are only run on uc16, but really should be run on uc18 as well as now uc20, and I think organizing the tests that way would make it less likely we run into this kind of thing for uc20 and uc22 etc. [17:42] cachio: hmm maybe it was something with qemu, i don't remember anymore... [17:42] I may prototype that a bit this PM [17:42] pstolowski, checking [17:44] pstolowski, we are running ../bin/spread -v google:tests/nightly/ google-nested:ubuntu-16.04-64:tests/nested/classic/ google-nested:ubuntu-16.04-64:tests/nested/core/ google-nested:ubuntu-18.04-64:tests/nested/core/ [17:45] I don't rememer why we are not running classic suite on 18 [17:45] pstolowski: from those logs it sounds like it was stuck in ReadEvent, which means the select was confused, or ReadEvent isn't quite doing what we expect [17:45] cachio: yeah that's weird, it's even listed in spread.yaml under nested/classic [17:45] pstolowski, I can add it [17:45] pstolowski, in 1 minutes [17:46] cachio: it may fail right now, i'm checking it with master [17:46] pstolowski: you might have to add more logging to understand what is happening [17:48] pstolowski, added [17:51] pedronis: it didn't fail on master [17:52] pedronis: but i'll re-run it a couple of time with & without the changes [17:52] spread -debug google-nested:ubuntu-18.04-64:tests/nested/classic/hotplug [17:53] ^ for the record [17:53] cachio: thanks [17:53] pstolowski, np [18:01] pstolowski: as I said it's quite possibly that is not working, consider that udevmon tests fake the underlying bits [18:02] pstolowski: you'll need more debuggin logs in conn.go to know what's happening [18:03] pedronis: yep, i'll invesitgate, will pick it up tomorrow morning [18:09] pstolowski: thank you === pstolowski is now known as pstolowski|afk === ijohnson is now known as ijohnson|lunch === ijohnson|lunch is now known as ijohnson [18:51] PR snapd#8104 closed: interfaces/builtin: backport verifySlotPathAttribute (2.43) [18:52] PR snapd#8105 opened: store: detect if server does not support http range headers [18:56] pedronis: thanks so much for 8104 [18:56] pedronis: sorry that I had to open another PR but I think we need to deal with 200 vs 206 better [19:08] was #8080 backported ? [19:08] PR #8080: dirs: manjaro-arm is like manjaro [19:13] pedronis: mvo: are there any high priority PR's for me to review? I'm still reviewing pawel's preseed PR, but seems there are other things in motion right now that might be important to review too ? [19:13] oh I guess mvo is offline now [19:18] ijohnson: mostly Pawel's PRs atm, and anything old that can be reviewed or things about test themselves [19:19] pedronis: ack [19:19] ijohnson: there's quite a bit of in-progress stuff that is not yet ready for review [19:19] right [19:19] * ijohnson is waiting on many spread runs to test fixes for the uc20 spread PR [19:49] PR snapcraft#2912 closed: meta: do not prime commands with adapter == "none" [19:52] PR snapcraft#2913 closed: spread: disable journal debug dump unless configured [20:07] PR snapcraft#2915 opened: rust plugin: respect Cargo.lock if present in project [20:17] ijohnson: I think #8105 can also be reviewed [20:17]