[00:00] and if so, how does it run performance wise? [00:04] bashingboi: I haven't tried on debian yet [00:04] I can try tomorrow [00:04] today I should really hit the bed :) [00:05] ha, np. I will try for myself and check how it runs [02:17] Laney: yeah i guess unpacking repacking the initramfs is even more steps [02:18] Laney: it's at https://github.com/CanonicalLtd/subiquity/blob/master/scripts/inject-subiquity-snap.sh although bits of that are hardcoded for subiquity, the general bits could be extracted i think === chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk [06:10] morning [06:14] o/ [06:16] mborzecki: can you please review https://github.com/snapcore/snapd/pull/4912 [06:16] PR #4912: overlord/configstate: change how ssh is stopped/started (2.32) [06:16] mborzecki: if you have a kvm or a pi at home it's best to try it for real too [06:17] zyga: looking [06:18] zyga: wasn't sshd.service -> ssh.service already addressed by a patch from mvo? [06:18] zyga: or maybe it was the other way around [06:19] it was [06:19] but it broke badly [06:19] once sshd is disabled [06:19] it cannot be enabled [06:19] because it is an alias [06:20] zyga: snapd alias or some sort of system alias? [06:21] systemd alias [06:22] look at systemctl cat ssh.service [06:22] sshd is an alias there [06:22] if you disable ssh.service [06:22] the alias goes away [06:22] and cannot be enabled, started or stopped [06:22] we used sshd instead of ssh because of the issue with writable paths [06:22] if you just switch to ssh everything is fine [06:22] but then you reboot [06:22] and ssh is re-enabled [06:22] because writable paths is terrible [06:23] and ssh is enabled by default so it bring the file back [06:23] this is why I used two things: sshd -> ssh so that enable/disable works [06:23] and the conditional file to actually control being off or on [06:24] i'm reading up the systemd.unit manpage [06:24] ok [06:24] nice, > aliases specified through Alias= are only effective when the unit is enabled. [06:24] I may have missed something, hence the request for review [06:24] it's an urgen issue affecting customer deployments [06:32] zyga: wow that service file is interesting [06:32] zyga: sshd_not_to_be_run is a debian thing right? [06:34] Probably [06:52] zyga: a similar attemp was there in configure hook in the core snap https://github.com/snapcore/core/commit/dc782a37a3aef3fb11f256260f5205f2ff256acd [06:53] oh, that's interesting [06:53] we we had this [06:53] and lost this? [06:54] yeah, i was looking for the actual contents of writable paths and found this instead :P [07:09] PR snapd#4903 opened: tests: [WIP] split prepare-restore.sh into prepare-restore.d modules [07:36] PR snapd#4913 opened: interfaces,release: probe seccomp features lazily [07:38] hmmm, my PR just got a massive TLS handshake failure with everything (google, linode) [07:38] all tasks aborted [07:38] hope this is not a norm today [07:39] zyga: ok, played a bit with core image, if i understand it right since we disable the service, the symlink is removed from etc, but then it's still restored from core snap because /etc/systemd/system is synced in writable-paths? [07:40] yes [07:40] that's the issue [07:40] that's the reason we switched to sshd service [07:40] but this had unintended consequences [07:40] from what i see disabling/masking sshd does not work either [07:46] yep [07:46] I'm testing another mode now [07:46] where sshd was disabled [07:46] thus breaking people [07:47] and chceking if the new core with my fix heals that [07:48] hah, now i'm locked out of the ssh, and serial console only shows the prompt with ssh login :P, how can i log in? [07:50] mborzecki: ha [07:50] yank the card [07:50] set a password [07:50] and put it back in [07:50] mvo: so to recap [07:51] mvo: I tested the scenario where sshd.service was disabled [07:51] and that the new patched snapd can fix that state simply by rebooting (which happens on update) and starting ssh.service [07:51] by setting the core config value [07:51] so I think this is a fix for the issue in general and for the machines affected in particular [07:52] zyga: we do more than just disable, we also mask, I think this will result in a different outcome [07:53] my feeling is that we need to at least unmask (opportunisticly and ignore the error) [07:55] I will test that now [07:55] yes, perhaps [07:56] mvo: I think masking doesn't work [07:56] zyga@localhost:~$ sudo systemctl disable sshd.service [07:56] zyga@localhost:~$ sudo systemctl mask sshd.service [07:56] Failed to execute operation: File exists [07:57] the initial state was that neither sshd nor ssh were masked [07:57] zyga: let me check [07:57] hmm [07:57] maybe I'm wrong [07:57] let me reset everything [07:58] zyga: it used to be the case, we masked ssh.service and then sshd.service would reappear via the writable-magic. but let me double check :) [07:58] ok [07:58] tested that now [07:59] my vm will take a moment to be ready, it has an older core [07:59] initial state https://www.irccloud.com/pastebin/BMckHA77/ [07:59] so sshd is disabled and masked [07:59] now I'm setting the core config [07:59] and ssh starts and I can log in [07:59] ah [07:59] because I *start* it [08:00] not enable it [08:00] let's reboot now [08:00] I mean, the hack just starts ssh directly [08:00] so even if sshd is masked [08:00] that's not an issue [08:00] I think we should unmaks it though [08:00] but it's not for corectness [08:00] but more for purity [08:01] mvo: it's also good after reboot [08:01] even though sshd is masked [08:01] zyga: even with the mask? [08:01] lrwxrwxrwx 1 root root 9 Mar 23 2018 /etc/systemd/system/sshd.service -> /dev/null [08:01] zyga: wuuut? crazy [08:01] (this is after reboot) [08:02] woah [08:02] https://www.irccloud.com/pastebin/LHAlPevl/ [08:02] look at this [08:02] sshd.service changed on disk [08:02] this is after reboot, no touching at all [08:02] how come? [08:02] I'll reboot again just to be 100% sure [08:03] zyga: I think its confusion from ssh.service vs sshd.service but let me double check [08:03] so +1 to unmask it [08:03] zyga: I'm so happy we have the /etc/ssh_no_start thing, this unit aliases seem to be crazy [08:04] yes, I have the same feeling [08:04] testing the new build now [08:07] I need to buy some more serial ports === pstolowski|afk is now known as pstolowski [08:07] mornings [08:07] this makes testing so much easier [08:08] mvo: pstolowski: morning guys [08:09] hey mborzecki [08:09] zyga: mvo: could we straighten out the situation with ssh instead of using that ssh_no_run.. monstrosity? [08:09] mborzecki: maybe [08:09] mborzecki: so there are multiple issues [08:09] mborzecki: do you have a proposal? [08:10] I mean, yes, but not trivially [08:10] good morning [08:10] we'd have to remove writable-path sync on systemd services [08:10] mborzecki: our ssh.service is using Alias=sshd.service [08:11] mborzecki: our core snap has "/etc/systemd/system/sshd.service" which is in writable-paths marked as "sync". this means that when the file is missing it will be copied from the core snap [08:11] as i understand we could fix it in the core snap, have just one ssh.service and no aliases [08:11] mborzecki: so we need to unmask both ssh and sshd service [08:11] mborzecki: yes, I think that is a good idea, fix the core snap to a) remove the alias b) not use /etc/ but /lib for the unit [08:12] mborzecki: but its more risky than what zyga is doing [08:12] mborzecki: what I really like about the zyga PR is that its dead-simple [08:12] mvo: force pushed with the unmask [08:12] tested that it works all the way [08:12] ta [08:12] mhm, it is :) [08:13] mborzecki: but gustavo also wants to talk about this so maybe it will be solved differently :) [08:13] mvo, mborzecki: one note, we need to tie this to core16 [08:13] with core18 the approach should be direct and without hacks [08:13] zyga, mborzecki I think its fine to revisit this very soon and fix core16 directly, i.e. fix our ssh install and go back to straight systemctl stop/disable ssh [08:14] as far as i'm considered, it's +1, but ideally we should fix this and use ssh.service (or sshd, just make sure that it's a single one, no aliases or whatnot) [08:14] agreed [08:14] (sshd cannot be used) [08:14] we must use ssh.service [08:14] and get rid of that sync stanza on that director [08:14] *directory [08:14] mborzecki: whats also annoying is that even if we do stop/disable/mask ssh and sshd it fails on enable because the operations are not symetric [08:15] mborzecki: i.e. unmask/enable/start ssh sshd will fail during enable sshd because it says there is a symlink alsready [08:15] mvo: yeah, i did this in the console :P [08:15] mborzecki: which boggles my mind, but maybe I'm missing something [08:15] and then locked out myself :P [08:15] haha [08:15] :D [08:15] well [08:15] mborzecki: like how can you sanely stop a service with aliases [08:15] mborzecki: haha [08:15] I'm happy with this patch [08:15] * mvo hugs zyga [08:15] I won't touch it until we talk to gustavo [08:15] * mvo hugs zyga harder [08:17] mvo: 'Failed to execute operation: Too many levels of symbolic links' maybe it's something in system(d,ctl) too [08:19] mborzecki: yeah [08:19] mborzecki: like "wuuuuut"? [08:19] mborzecki: what kind of error is that to begin with [08:19] mvo: hi, there's quite bit of PRs marked for 2.32, anything in particular that I could help with a review? [08:20] mborzecki: anyway, sorry, I'm still perplexed how difficult it is to disable ssh via systemctl [08:20] zyga: pushed an update to #4902 [08:20] PR #4902: cmd/snap-confine: nvidia: preserve globbed file prefix [08:20] pedronis: let me unmark some [08:20] mvo: disabling is simple, making sure that it never gets enabled is a bit harder :P [08:20] pedronis: 4882 might be a good one, that is the most critical one right now [08:21] mborzecki: heh, exactly [08:22] I'll push a unmask of ssh.service now [08:22] just testing that after reboot [08:24] mborzecki: quick question about 4902 - that fixes nvidia issues on arch? [08:25] mvo: does 4882 need a follow up to use 4911 ? [08:25] mvo: yes, there's another pr for ubuntu #4908 (though still RFC) that includes the patch from 4902 [08:25] pedronis: yes, I will talk to gustavo about it [08:25] PR #4908: [RFC] cmd/snap-confine: attempt to detect if multiarch host uses arch triplets [08:26] pedronis: I think technically 4882 will solve the problems we are seeing but gustavo was keen to add the second layer of check (snap run talking to snapd on system-key mismatch) [08:26] mborzecki: ta [08:27] mvo: unmasking both now [08:27] zyga: ta [08:28] PR snapd#4909 closed: interfaces: harden snap-update-ns profile (2.32) [08:30] PR snapd#4907 closed: advisor: deal with missing commands.db file [08:30] PR snapd#4913 closed: interfaces,release: probe seccomp features lazily [08:34] PR snapd#4914 opened: advisor: add comment why osutil.FileExists(dirs.SnapCommandsDB) is needed [08:35] PR snapd#4915 opened: interfaces,release: probe seccomp features lazily [08:38] mvo: left some questions/comments [08:38] pedronis: yay,thank you! [08:54] mborzecki: https://github.com/snapcore/snapd/pull/4902/files#r176670685 [08:54] PR #4902: cmd/snap-confine: nvidia: preserve globbed file prefix [08:56] zyga: responded :) [08:57] replied too [08:59] zyga: ok, makes sense, btw. i was surprised by it too [09:01] mvo: not urgent but I'm confused/see some duplication of things in the roadmap page (for 2.32 and 2.33 and after) [09:03] mvo: did you notice https://github.com/snapcore/core/commit/dc782a37a3aef3fb11f256260f5205f2ff256acd [09:03] mborzecki pointed me to it [09:05] zyga: force pushed [09:05] thank you [09:08] damn, i feel sick, finally caught something from the junior [09:12] zyga: no, huh [09:12] mborzecki take it easy [09:12] mvo: fun, right? [09:12] where is that, it's gone now? [09:13] it was lost when we went to in-core config [09:13] zyga: yeah :( [09:13] zyga: sadly [09:13] full circle [09:14] 'sup? [09:14] Chipaca: fire, smoke, screams and tears [09:14] zyga: I didn't ask about your bedroom [09:17] Chipaca: it's slowly getting better onw [09:18] zyga: good to hear :-) [09:18] what did we lose by moving to in-core config? [09:18] * Chipaca curious [09:19] Chipaca: ssh control [09:19] https://github.com/snapcore/snapd/pull/4912 [09:19] PR #4912: overlord/configstate: change how ssh is stopped/started (2.32) [09:23] so, I have a question about the ssh-keys interface. I added it to my snap, did `snap connect` to enable it but when the snap runs I'm getting `Permissions 0644 for '/home/rumo/.ssh/id_local' are too open.` [09:24] The permissions that ls reports within --shell as the same as on the real system, though. And of course the permissions are correct on the real system. [09:26] zyga: quick question about reverts from 2.32 -> 2.31. the new/updated security profiles will break reverts, no? I mean, suppose 2.32 has the stricter profiles and it loads the per-snap update-ns profiles. now on revert the old snap run will start network-manager and that will use the new 2.32 profiles which are strict and won't let n-m run, is that correct? [09:27] on revert snap-confine will load the old profile which is permissive for content and doesn't support layouts [09:27] kalikiana: that's very weird [09:27] kalikiana: is the snap in the store? i'd like to take a look [09:27] mvo: the snap-confine profile is from core directly [09:27] zyga: awsome [09:27] mvo: and the issue didn't affect the regular profiles, just s-u-n profile [09:28] zyga: so no problem here, just wanted to double check [09:28] mvo: I think it should work [09:28] but I did not check [09:28] zyga: its part of our standard tests so we will know [09:28] ok [09:28] Chipaca: I haven't pushed that bit yet since it's not working, but I can push it [09:29] kalikiana: is it an easy to build snap? [09:29] i mean, just run snapcraft and wait sort of thing? [09:29] in that case i could use just the snapcraft.yaml :-) [09:30] to be clear, this is just me being lazy: i'd rather tinker with it to figure out what's going on, than think about it in the abstract [09:43] zyga: hey, does https://paste.ubuntu.com/p/jxtYz7z9CD/ ring any bells? [09:43] mmm [09:43] zyga: happend on master [09:43] is that on release [09:43] aww [09:43] I didn't push the debug thing to master [09:45] mvo: can you please force-land that https://github.com/snapcore/snapd/pull/4916 [09:45] PR #4916: tests: change debug for layout test [09:46] PR snapd#4916 opened: tests: change debug for layout test [09:56] kalikiana: so, I can't reproduce your issue [09:56] kalikiana: wondering what's different between here and there [09:58] kalikiana: one thing though, i don't think your snap needs to include openssh-client, as that's part of core and ssh-keys gives you access to it (at least to /usr/bin/ssh) [09:58] but here it works with both the ssh in the snap, and the one in core [10:00] Ah, I didn't realize that. So I can drop the package. [10:00] kalikiana: but that doesn't answer where the 0644 error is coming from :-) [10:01] Chipaca: Could it be a bug in snap connect then? Vaguely thinking of content interface issues depending on the order of install/connect/snap executation [10:01] kalikiana: wait, can you ssh from within --shell? [10:01] that's what i am doing and it works, if it works for you as well the issue is even weirder [10:02] Chipaca: No, same error, even on a simple "ssh helga" (helga being an arbitrary host that works outside confinement) [10:03] phew [10:04] Chipaca: Also getting a `Control socket connect(/home/rumo/.ssh/socket-root@***.***.***.***:22): Permission denied`with just ssh, before the other error [10:04] kalikiana: just to make sure, can you ‘find /home/rumo/.ssh -ls’? [10:06] Chipaca: That works and lists each key with -rw-r--r-- [10:06] kalikiana: inside, or outside? [10:07] Chipaca: both are the same [10:07] kalikiana: why are your private keys world-readable? [10:08] kalikiana: (see also: how is your ssh 'outside' not complaining about it all the time?) [10:08] Hmmm now that you mention it, they probably shouldn't be... [10:08] Chipaca: Outside is fine [10:09] kalikiana: fine how? [10:09] As in, I get no errors [10:09] right [10:09] But lemme try and change them [10:09] kalikiana: but you should :-) [10:10] kalikiana: only thing I can think of is that you have seahorse auto-adding your ssh keys on session start, and it's less picky about permissions, and ssh never looks at your actual keys because by the time you run it they're in the agent [10:11] kalikiana: if that's the case, it sounds like a bug in seahorse to me ;) [10:12] kalikiana: but also, sounds like an easy thing to detect and warn about from your snap [10:12] ie, can i access the keys? no: tell user about snap connect, yes: are they the right perms? no: tell user about it [10:13] Dropping the group/world reads seems to fix it indeed. [10:14] Now ssh just tells me `bind: Permission denied` and `unix_listener: cannot bind to path` [10:14] zyga: i think i need to check for a specific library in autodetection, one thats' always there, libnvidia-glcore.so.. looks like a good candidate [10:14] But it unlocked the key/passphrase fine [10:14] * kalikiana tests git push from the snap [10:16] kalikiana: that socket might be a control master thing, which isn't supported from inside [10:16] kalikiana: (the thing that lets you multiplex ssh sessions over a single connection) [10:21] Chipaca: You were right, disabling it explictly works [10:25] Chipaca: Thanks a lot for helping me track this down! [10:26] Saviq, a new fedora image with the correct size is uploaded [10:26] Now if it'd actually use the ssh agent, that'd be sweet, but it works perfectly [10:26] cachio: ack, me tries [10:27] Saviq, great, tell me how if goes [10:27] cachio: -64 suffix, too? [10:28] kalikiana: talk with jdstrand, but I suspect much laughter to be had :-) [10:29] :-D [10:40] Saviq, fedora-27-64 [10:40] this is the name you have to use [10:41] Saviq, you don't need to specify the image [10:44] zyga: can you take another look at #4902? [10:44] PR #4902: cmd/snap-confine: nvidia: preserve globbed file prefix [10:45] zyga: also i'm finishing with fixes for 4908 and will be switching back to ubuntu to check if things didn't break :) [10:45] zyga: might #1757284 be an instance of the 'interface connections go away' bug? [10:45] Bug #1757284: Several snap apps fail to launch

[10:46] (now that i think about it, it probably is -- but don't know that bug #) [10:48] mborzecki: looking [10:49] PR snapd#4916 closed: tests: change debug for layout test [10:49] * Chipaca attempts to dad up and get out of bed [10:50] Chipaca: updated the bug with a question [10:57] cachio: yeah, it's lookin' good https://travis-ci.org/MirServer/mir/builds/357325003 [11:01] * Chipaca AFK for a while (physio, etc) [11:04] pedronis: hrm, hrm, my system-key pr has a bit of a problem on core. on firstboot on core snapd starts, generates the system-key and the core revision is empty (because things are not seeded yet). then stuff gets seeded but that means the systme-key changes because the core revision is part of the inputs [11:05] zyga: (cc -^) [11:05] * mvo will need to think about this while taking a short lunch break [11:05] hmm hmm hmm [11:05] mvo: if we drop core revision? [11:05] we keep it because apparmor profiles are there (includes) [11:06] hmm :/ [11:06] zyga: yeah, feels strange to just drop it, but maybe we can drop it on core? [11:07] well, if apparmor gets updated [11:07] zyga: let me try that [11:07] and snapd doesn't [11:07] it's not correct then [11:07] zyga: hm, good point [11:08] will apparmor come from the snapd snap? [11:08] fun questions [11:09] Saviq, nice [11:09] good to hear that [11:09] don't we check the feature of apparmor becuase of that? [11:10] pedronis: partly, the abstractions are also part of the mix and those come from the core snap [11:11] pedronis: apparmor has two inputs: the kernel capabilities and the abstractions with pre-writen rules [11:11] we could force a re-gen of the profiles after seeding but thats a bit ugly [11:11] full of dragons this pr [11:12] https://paste.ubuntu.com/p/77k9YJbVQK/ [11:12] humanSuite.TestHuman failed here, but was passing yesterday [11:12] pedronis: yes, apparmor will come from core snap (on core) [11:13] pedronis: I mean the include statements, we check for kernel features but each profile we make has include statements that reach to apparmor profiles from either native package or from the core snap on all-snap boxes [11:14] Chipaca: DST change https://paste.ubuntu.com/p/77k9YJbVQK/ [11:14] LOL [11:14] stuff will never stop to amaze me [11:15] otoh, i had no clue that we're changing clocks on saturday ;) [11:22] mvo: also the issue with seeding exists also on classic [11:22] mvo: we support seeding there with snaps [11:23] mvo: I have an idea [11:23] mvo: it's a bit unclear to me why are we writing a key before being seeded ? [11:24] mvo: we should wait/skip that, if there are not snaps, we are not seeded [11:24] mvo: instead of revision we can inspect a single file in /meta/manifest [11:24] (we don't have that file yet) [11:24] but as long as it encodes the version of every package [11:25] it caputres the essence of core [11:25] pedronis: we need that on boot to setup profiles for core itself [11:25] pedronis: on snapd daemon startup actually [11:25] mborzecki: on the one hand, huzzah it works [11:25] mborzecki: on the other, booo [11:25] zyga: ok, there's a recursion problem here [11:25] mborzecki: on the last one, a test that works 350 days a year is fine, right? [11:25]