[05:20] morning [07:06] mornings! [07:18] pstolowski: hey, welcome back, well rested? [07:19] mborzecki: yep! i had great time, thanks [07:19] pstolowski: nice ;) [07:20] mborzecki: how was it here? [07:20] pstolowski: few of us left this week, mvo on vac, pedronis & zyga attending a sprint [07:20] pstolowski: rather quiet [07:52] PR snapd#7183 opened: [RFC] many: drop snap.ReadGadgetInfo wrapper [08:07] unit tests hanging on travis again, hmm [09:04] hah! finding bugs during a refactor is fun [09:08] Chipaca: bugs == fun [09:08] Chipaca: unless they bite, then run [09:08] PR snapd#7184 opened: daemon: do not modify test data in user suite [09:09] Chipaca: can you take a look ^^ ? [09:14] mborzecki: in a mo' [09:14] Chipaca: sure, thanks! [09:14] PR snapd#7185 opened: boot, etc.: refactor boot to have a lookup with different imps [09:32] brb [09:48] Good morning [09:49] zyga: hey [09:49] zyga: how was the trip? [09:52] hey [09:53] my flight had 3 hour delay because they had to fly in a replacement [09:53] but otherwise it was totally uneventful and pleasant [09:54] hey zyga! [09:54] hey pawel [09:55] guys, a small heads up [09:55] despite the sprint I will try to land bug fixes [09:55] I think core and ubuntu are green now [09:55] fedora still has some interactions with lxd that I need to address [09:55] my goal is to make https://github.com/snapcore/snapd/pull/7168 landable [09:55] PR #7168: tests: measure testbed for leaking mountinfo entries [09:56] sorry about the noisy PRs, I'll try to garden them throughout the day [09:56] if you need anything from me urgently please use telegram [09:56] ack [09:56] if you don't have me on telegram ping me on IRC, i'll try to check from time to time [10:10] pstolowski: hey! wb :-) [10:10] Chipaca: o/ [10:10] Chipaca: thanks for helping land some of my PRs while i was off [10:11] pstolowski: sorry we couldn't manage to land them all :) [10:12] Chipaca: nah, it was better than expected! [10:13] Chipaca: afaiu the PR about snapd type & snapcraft needs passthrough for now, was this something you discussced & agreed on? [10:14] pstolowski: i think using passthrough was deemed acceptable for now [10:14] pstolowski: but i'm not 100% positive on that [10:15] k. i'll give it one more try on spread and see if i can find anything before trying passthrough [10:35] PR snapd#7186 opened: gadget: ensure filesystem labels are unique [10:35] I like that Fedora Workstation automatically creates a mugshot for your user, based on your initials [10:36] I particularly like it because, as per my usual approach, the user's name is Fedora User [10:36] simple one ^^ pstolowski maybe? [10:36] so password prompts are like 'FU gimme your password' [10:36] i like distros with attitude [10:37] hahaha [10:39] anyone noticed unit tests getting stuck recently? [10:43] not me [10:43] mborzecki: where? [10:44] Chipaca: go test ./... as run on travis seems to get stuck randomly, i can't reproduce it locally though [10:54] mborzecki: yeah i think i've just experienced it on my PR [10:58] pstolowski: yay [10:59] pstolowski: did you get a bracktrace by any chance? [10:59] mborzecki: nope, it was on travis [10:59] pstolowski: there were some changes recently around socket listen in deamon, i'm suspecting that might be the cause [11:23] mborzecki: can you try go test with race detector? [11:34] zyga: nothing comes up [11:39] zyga: more problems with lxd & unified cgroups https://forum.snapcraft.io/t/lxd-3-15-fails-with-unified-cgroup-hierarchy/12462/10 [11:42] mborzecki: hey [11:42] looking [11:47] * Chipaca lunches [11:55] pstolowski: pinged you in https://github.com/snapcore/snapd/pull/7187 [11:55] PR #7187: data/selinux: allow snap-confine to read entries on nsfs [11:55] zyga: is this something you saw in your tests too? ^^ [11:55] PR snapd#7187 opened: data/selinux: allow snap-confine to read entries on nsfs === morphis0 is now known as morphis [12:03] mborzecki: re [12:03] mborzecki: I don't believe so but I didn't focus on fedora yet, it's the next pass [12:03] mborzecki: I didn't get a clean run on fedora yet [12:04] zyga: ok [12:18] I'm working on fixing the fedora leaks now [12:26] hi, it seems my 'core' is in the state 'broken', how do I fix that? [12:26] zyga: I don't do it this way: https://www.telegraph.co.uk/politics/2019/07/28/dominic-cummings-one-strike-ban-cabinet-leaks-leaked-daily-telegraph/ [12:27] micah: what kernel are you running? [12:27] seems I needed to remove core and install core [12:28] Chipaca: 4.14.119 [12:28] micah: hm, that should be working fine (5.2 has some issues) [12:29] micah: how did this happen? [12:38] Chipaca: i dont know how... but it seems when I restart, its broken again [12:39] micah: can you do: snap info cpre | pastebinit [12:39] er [12:39] micah: can you do: snap info core | pastebinit [12:40] micah: or even better, [12:40] micah: can you do: snap info --verbose core | pastebinit [12:42] Chipaca: https://pastebin.com/b4dWfzT9 [12:42] "snap version2 too [12:43] err [12:43] "snap version" [12:43] (little weakness of the shift key) [12:43] 4.14.119 sounds like some random mainline kernel ... [12:44] ogra: hmm, sounds like an LTS kernel tbh [12:45] https://pastebin.com/2mRCq9K7 [12:46] micah: systemctl --all --failed | grep snap ? [12:46] mborzecki, well ... 4.14.119-2.pvops.qubes.x86_64 [12:46] ogra: 4.14.119 with qubes patches on top likely [12:47] mborzecki, definitely not an ubuntu kernel (though this seems to be debian ... but also not a debian archive kernel for sure) [12:47] mborzecki, sure, but you cant know what config options are set, if the apparmor patches are complete etc [12:48] it is simply not "some distro standard kernel" [12:48] ogra: or whether it defaults to selinux like the mainline builds did/do :) [12:49] yep :) [12:49] ogra: i'm not going to abort any support as soon as the kernel isn't on a known-good list [12:49] Chipaca, i didnt ask that anyne stops supporting ... it is just an important fact to know about during support [12:49] *anyone [12:49] micah: please pastebin the output of that? [12:52] Chipaca: seems systemctl --all --failed | grep snap doesn't give me anything [12:52] micah: so what's at /snap/core/current ? (try with ls -l) [12:52] lrwxrwxrwx 1 root root 4 Jul 29 08:27 /snap/core/current -> 7270 [12:53] micah: and in 7270? [12:53] * Chipaca goes for tea [12:53] and 7270 is empty [12:56] micah: does 'systemctl --all | grep snap' give you anything? [12:58] Chipaca: it does [12:59] micah: can i see? [13:01] pstolowski: cachio: standup? [13:01] Chipaca: https://pastebin.com/pNsjmY9u [13:02] degville: standup? [13:10] micah: systemctl status snap-core-7270.mount ? [13:10] chipaca: https://pastebin.com/hySYpyXv [13:12] micah: systemctl start snap-core-7270.mount ? [13:13] Chipaca: that seems to start it [13:14] and 7270 now has things in it [13:17] micah: ok, systemctl restart snapd.service, to get things back in shape [13:18] micah: something went wrong somewhere with systemd for that to be broken, but if you don't know what ¯\_(ツ)_/¯ [13:18] grr go 1.13 runtime now pokes /sys/kernel/mm/transparent_hugepage/hpage_pmd_size during startup, making selinux complain each time you run snap* command [13:19] stgraber: do you have a link to the 5.3 mount api breakage? [13:23] Chipaca: yeah, definately... too bad I dont know what [13:29] micah: would be good to know if it's still ok on reboot [13:33] Chipaca: i'll try in a second [13:34] micah: no rush, we'll be here tomorrow [13:34] :) [13:43] https://github.com/snapcore/snapd/pull/7187 unit tests hanging on travis again :/ [13:43] PR #7187: data/selinux: allow snap-confine to read entries on nsfs [13:45] cachio: hey, would you consider having a `ubuntu-devel` image for spread? [13:46] Our at least build a 1910 one? [13:46] *Or [13:46] Saviq, sure [13:47] Saviq, but also you can used the one published by canonical [13:48] Chipaca: https://lkml.org/lkml/2019/7/26/388 [13:49] cachio: could you review our list please https://github.com/MirServer/mir/blob/master/spread.yaml#L14 ? [13:50] * Saviq tries to remember why we don't use the official images in the first place… [13:51] Saviq, just run: gcloud compute images list --project ubuntu-os-cloud-devel [13:51] this will give you all the devel imags [13:52] stgraber: thanks [13:52] you just can use any of these by doing image: ubuntu-os-cloud-devel/ [13:53] cachio: thanks, I'll try, I never got gcloud working properly here… [13:53] Saviq, https://paste.ubuntu.com/p/z9ckvYwScZ/ [13:53] tihs is the list of images [13:53] cachio: thanks :) [13:54] take a look and if there is not any image that works for you I can create 1 with the lates changes from devel [13:54] Saviq, your spread.yaml seems to be ok [13:54] the ubutnu images that you use are the official ones [13:58] PR snapd#7188 opened: data/selinux: allow read on sysfs [13:58] pstolowski: heh ^^ another one [14:21] Chipaca: on reboot, same problem [14:22] micah: ok, look at 'journalctl --system', and look for anything red [14:23] Chipaca: i see nothing red there that is related (some firmware/acpi things) [14:26] hey, any news? [14:27] micah: what's systemctl status snap-core-7270.mount ? [14:28] zyga: about what? [14:28] in general [14:28] guys, the go unit tests are flaky [14:28] can we see if a branch from last week would be green [14:28] I have a feeling it is something we merged mid week [14:28] is anyone on this? [14:29] zyga: attempted to reproduce this, but only got this https://github.com/snapcore/snapd/pull/7184 obviously the problem doesn't happen locally (at least for me) [14:29] PR #7184: daemon: do not modify test data in user suite [14:30] mborzecki: can you try with GOMAXPROCS=2 [14:30] or something "low" [14:34] Chipaca: i put the output of the journalctl --system and the systemctl status into this paste: https://pastebin.com/MknBeW9T [14:38] mborzecki: the tests that use lxd should reboot to clean up [14:38] that's the bottom line from the lxd team [14:38] zyga: sgtm [14:38] not sure how to do it yet [14:39] zyga: only a handfull of those [14:39] I know we have REBOOT and REBOOT_COUNT, need to play with that [14:39] zyga: i can look into that, there's a REBOOT in a couple of tests I added already [14:39] mborzecki: thank you [14:39] I'll work on other tests [14:40] though honestly, right now unit tests being red is more problematic [14:41] zyga: mborzecki: do you have an example run of the tests hanging? [14:41] no, only in travis [14:41] most recent PRs are red [14:41] zyga: that's what i mean [14:41] (because of this) [14:41] Chipaca: https://travis-ci.org/snapcore/snapd/builds/564940409?utm_source=github_status&utm_medium=notification [14:42] zyga: not necessarily because of this [14:42] mborzecki: thanks [14:42] Chipaca, micah, wow, so qubes seems to essentially be a livecd kind of setup (everything readonly, parts of the system are rw bind mounts) ... perhaps some dirs needed by snapd are simply missing (or not there when snapd starts due to races etc) [14:43] (it is also interesting that your kernel was built on fedora while your rootfs is debian based apparently ... but thats just a fun fact) [14:45] PR core#38 closed: Add another pi-config option [14:45] PR core#83 closed: move most of the ubuntu-core config deb into the snap snap build [14:45] ogra: Chipaca it appears qubes does this to make it work: https://github.com/QubesOS/qubes-app-linux-snapd-helper [14:46] PR core#38 opened: Add another pi-config option [14:46] PR core#83 opened: move most of the ubuntu-core config deb into the snap snap build [14:46] wow https://github.com/QubesOS/qubes-app-linux-snapd-helper/blob/master/qubes-snapd-mount [14:47] so it re-creates all snap mount units [14:47] ... on the fly ... [14:50] should it have a pre-dependency on something? [14:50] because it seems like in some boots, it works, and some it does not [14:50] micah: looks like snapd.service should have an after: whatever it is that creates the mounts [14:51] yeah, my guess would be that snapd is already running while this mount unit recreation still runs [14:52] it is also rather dangerous what they are doing there, typically snapd creates these mount units ... if anything chenags on the snapd side, the qube script wont pick it up [14:52] *changes [14:53] if they need to chnage the pre-snap mount units, they should better take the original ones as input [14:53] *per-snap [14:54] Jul 29 10:32:03 browser systemd[1]: snap-core-7270.mount: Succeeded. [14:54] Jul 29 10:32:03 browser systemd[1]: Unmounted Mount unit for core, revision 7270. [14:54] Jul 29 10:32:03 browser systemd[1]: Started Rebuild the non-persistent snappy systemd mounts. [14:54] Jul 29 10:32:03 browser systemd[1]: Starting Snappy daemon... [14:54] Jul 29 10:32:03 browser systemd[1]: Unmounting Mount unit for firefox, revision 243... [14:54] Jul 29 10:32:03 browser systemd[1]: rw-bind\x2ddirs-snap-firefox-243.mount: Succeeded. [14:55] they are definitely stepping on each others toes [14:55] hey Chipaca, degville: I'm finalizing a refactoring of the instructions for running spread in HACKING.md, should I make all the text under 80 chars there? [14:55] The file isn't consistent, not sure if we have a style we try to maintain for markdown [14:57] ijohnson: as long as it looks ok in github i don't mind [14:57] * Chipaca is not a purist [14:57] cool! sounds good [14:57] make it 78.5 chars .. the you can still have a terminal scrollbar [14:57] :P [14:57] *then [14:58] zyga: cpufreq'ed my laptop down to 800MHz and turned off all my cores except 1 [14:58] zyga: trying to repro the travis thing [14:58] thank you [14:59] ogra: how many times have you read markdown from a github repo in a terminal in the past month? [14:59] i haven't had a work machine this slow since the first one i built myself [14:59] :D [14:59] ijohnson, every time i used w3m ... [14:59] 200MHz pentium mmx :') [14:59] * ijohnson searches what w3m is [14:59] ogra: yeah, that does look like its stepping on itself there... i'm not familiar enough with what is going on here to know what I should do to make this not happen [14:59] ijohnson, (like ... never :P ... and i wa indeed joking) [14:59] ah got it [15:00] ijohnson / Chipaca: It's not my jurisdiction, I'm sure, but editing md in vim is definitely easier for me when it's <80 wide. [15:01] hmm, let me see how it looks in GitHub when it's at 80 lines [15:01] we do want to make docs easy for degville to edit :-) [15:01] it should be fine. [15:01] the width shouldn't affect the output. [15:02] those sound like famous last words [15:02] ahahaha! [15:02] degville: in preformatted text it does [15:02] that is, in ```s [15:02] ^ except that of course :) [15:05] micah, well, it would be interesting to know what they try to achieve by duplicating the snap mount units in the first place [15:06] it also seems their rw-bind script tries to mount the snap dirs rw alongside ... which is pointless given everything is a squashfs anyway [15:08] ok, what looks better in GitHub: https://github.com/anonymouse64/snapd/blob/feature/HACKING-md-updates/HACKING.md [15:08] or https://github.com/anonymouse64/snapd/blob/feature/HACKING-md-updates-unlimited-chars/HACKING.md [15:08] perhaps systemd is simply clever enough to recognize that this is producing a mess and simply unmounts them all [15:08] ijohnson: ooh, that needs to say 1.9 now [15:09] I actually don't see a difference [15:09] Chipaca: ah you're right [15:09] honestly, the formatting looks the same whether it's 80 chars or not, so I think degville will get to keep easily editing in vim [15:10] Chipaca: do you know what release of snapd started requiring go 1.9? [15:10] ijohnson: not offhand [15:10] let me check the logs [15:10] huh ? why would vim have anything to do with 80chars ? [15:11] Chipaca: i got it once with GOMAXPROCS=2 and -count=20, but didn't have GOTRACEBACK set :/ [15:12] ijohnson: 2.38 [15:13] thanks Chipaca [15:13] ogra: see degville's comment [15:13] ijohnson: that was 'git log' to find #6294, and then 'git tag --contains 618450bb0fb3064a9f310456047993017828556d' [15:13] PR #6294: packaging/ubuntu: build with golang 1.10 [15:14] how did you know which PR to look for though? [15:14] ijohnson, we should do some compny wide crowdfunding to pay a proper post-VGA monitor for degville then :) [15:14] ijohnson: I looked for 1\.9 [15:14] ah [15:14] :) [15:15] ogra: it's only because I have been used to editing tw=79 for a long time. Probably the only cache size my brain can handle. [15:15] haha [15:20] ogra: i'm trying to come up with a clear, concise summary of the problem so I can report it to qubes [15:21] ogra: think this is accurate: "Qubes automounts conflict with Snap mounts, causing snaps not to work and appear broken." [15:22] micah, yeah, kind of ... i think whet they should do is to simply "ln -sf /var/lib/snapd/snap /" and just leave the mount units alone [15:23] *what [15:23] ănd they shoudl do that linking before any of the mount units or snapd even attempt to start) [15:24] i see [15:24] btw I think we should update the spread snap to look for the user's real $HOME dir instead of $HOME/snap/spread/current [15:25] it seems that they do all this re-creation dance just because they do not know if /snap exists or not [15:26] ijohnson: ? [15:26] there are other options ... they could pretend they are fedora, then /var/lib/snapd/snap would be the default and snapd would be taking care that the mount units do the right thing ... [15:26] well the HACKING.md says to get spread, you should install the snap, but it also says that you should put images inside $HOME/.spread/qemu [15:27] so the instructions don't work if you follow them exactly [15:27] or they could provide a patch to snapd upstream to put their OS into a list that makes snapd DTRT [15:27] you either need to move qemu images into $SNAP_HOME (or whatever env points there I forget), or you need to export HOME before running spread [15:28] though it seems their OS actually pretends to be debian even though it is a derivative of some kind [15:28] so it might be hard to detect for snapd that you run QubesOS [15:31] Chipaca: in case you're playing with the unit tests getting stuck, https://github.com/golang/go/issues/27189 [15:35] mborzecki: yay [15:36] Chipaca: heh https://paste.ubuntu.com/p/QGhSdgQWSC/ that's a bit unexpected too [15:37] i'm going to leave this running and go for a run myself [15:38] niemeyer: https://forum.snapcraft.io/t/rfc-plug-slot-rules-plug-names-slot-names-constraints/12439 [15:38] THanks [15:42] Chipaca: another interesting find https://paste.ubuntu.com/p/yFNJcPBjfg/ [15:43] mborzecki: /o\ [15:43] * Chipaca really goes afk now [15:43] PR snapd#7189 opened: HACKING.md: update spread section, other updates [15:43] ogra: well basically its a debian VM, running under xen [15:46] ogra: could I construct a systemd unit file that just does this symlink and is run before the the mount units or snap units and try and see if that works? [16:09] * cachio lunch [16:12] Chipaca: was about to leave when noticed the tests got stuck: https://paste.ubuntu.com/p/yhGfmdRvj7/ [16:12] leaving for real now :) [16:12] zyga: ^^ === pstolowski is now known as pstolowski|afk [16:20] ogra: The rw-bind script just makes sure that the snaps are saved in the appvm storage rather than being lost on shutdown on the appvm [16:23] PR snapcraft#2646 opened: cli: only say sorry for in-snapcraft issues [16:29] micah, well, try it (the symlink creation) .. i'd also disable the rewrite unit then [17:15] Re [17:54] I'm just installing Windows Insider build 18945 to see what the state of affairs is for snaps now that squashfs is in the shipped kernel [17:57] diddledan: I'm running that [17:57] \o/ [17:58] diddledan: it's not enough but I read that some people managed to run systemd inside a hand-made container [17:58] diddledan: and then run snapd inside that [17:59] yeah you need to trick systemd to believe it's pid1 [18:29] PR snapcraft#2646 closed: cli: only say sorry for in-snapcraft issues [18:52] PR snapd#7190 opened: tests: set GOTRACEBACK=1 when running tests [19:03] snap version: https://www.irccloud.com/pastebin/VHgAuLOx/ [19:03] diddledan: neat [19:03] diddledan: can you tell me more about how you got that? [19:04] start systemd with: $ sudo daemonize /usr/bin/unshare -fp --mount-proc /lib/systemd/systemd --system-unit=basic.target [19:05] find the process id of the unshare process and run $ sudo nsenter -t $SYSTEMD_PID -m -p /bin/login -f dllewellyn [19:05] then just `snap version` [19:05] mmm [19:05] can you snap install xkcdwebserver and see it from windows? [19:08] seems to timeout accessing via either localhost or the wsl instance's ip [19:10] diddledan: is the service up? [19:10] yup [19:10] diddledan: it's a composite test that checks if many things together actually worked [19:11] curious, ok [19:11] thanks! [19:11] is it on 80 or some other port? [19:11] systemctl status snap.xkcd-webserver.xkcd-webserver.service: https://www.irccloud.com/pastebin/jhhaJwvU/ [19:11] port 80 [19:12] there also seems to be a mono service running but that's from the base image rather than via a snap [19:12] tcp 0 0 0.0.0.0:8084 0.0.0.0:* LISTEN 1289/mono [19:12] that _does_ respond [19:15] googler snap (cli for google searches) works [19:16] theloungeirc responds from windows on port 9000 [19:16] diddledan: neat [19:16] something specific to xkcd is failing tho [19:16] maybe port 80 is just "busy" with windows stuff? [19:16] possibly [19:18] PR snapd#7191 opened: tests: remove installed snap on restore [19:19] aah, stopping and restarting the service with `snap stop` and `snap start` appears to have disloged the blockage (I was looking to see what the port did on windows side when it was stopped) [19:19] https://usercontent.irccloud-cdn.com/file/aB6fZBpA/image.png [19:19] that's coming from the snap in wsl2! [19:23] diddledan: awesome work! [19:23] * ijohnson relocates to coffee shop [19:23] what's the debugging command to get the confinement status of a system again? [19:23] i.e. to output what confinements the snapd is seeing as available for use [19:24] the one that shows the apparmor flags and stuff [19:26] aha. found it [19:26] https://www.irccloud.com/pastebin/w0UFjDgb/ [19:26]