| bashingboi | and if so, how does it run performance wise? | 00:00 |
|---|---|---|
| zyga | bashingboi: I haven't tried on debian yet | 00:04 |
| zyga | I can try tomorrow | 00:04 |
| zyga | today I should really hit the bed :) | 00:04 |
| bashingboi | ha, np. I will try for myself and check how it runs | 00:05 |
| mwhudson | Laney: yeah i guess unpacking repacking the initramfs is even more steps | 02:17 |
| mwhudson | Laney: it's at https://github.com/CanonicalLtd/subiquity/blob/master/scripts/inject-subiquity-snap.sh although bits of that are hardcoded for subiquity, the general bits could be extracted i think | 02:18 |
| === chihchun_afk is now known as chihchun | ||
| === chihchun is now known as chihchun_afk | ||
| mborzecki | morning | 06:10 |
| zyga | o/ | 06:14 |
| zyga | mborzecki: can you please review https://github.com/snapcore/snapd/pull/4912 | 06:16 |
| mup | PR #4912: overlord/configstate: change how ssh is stopped/started (2.32) <Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/4912> | 06:16 |
| zyga | mborzecki: if you have a kvm or a pi at home it's best to try it for real too | 06:16 |
| mborzecki | zyga: looking | 06:17 |
| mborzecki | zyga: wasn't sshd.service -> ssh.service already addressed by a patch from mvo? | 06:18 |
| mborzecki | zyga: or maybe it was the other way around | 06:18 |
| zyga | it was | 06:19 |
| zyga | but it broke badly | 06:19 |
| zyga | once sshd is disabled | 06:19 |
| zyga | it cannot be enabled | 06:19 |
| zyga | because it is an alias | 06:19 |
| mborzecki | zyga: snapd alias or some sort of system alias? | 06:20 |
| zyga | systemd alias | 06:21 |
| zyga | look at systemctl cat ssh.service | 06:22 |
| zyga | sshd is an alias there | 06:22 |
| zyga | if you disable ssh.service | 06:22 |
| zyga | the alias goes away | 06:22 |
| zyga | and cannot be enabled, started or stopped | 06:22 |
| zyga | we used sshd instead of ssh because of the issue with writable paths | 06:22 |
| zyga | if you just switch to ssh everything is fine | 06:22 |
| zyga | but then you reboot | 06:22 |
| zyga | and ssh is re-enabled | 06:22 |
| zyga | because writable paths is terrible | 06:22 |
| zyga | and ssh is enabled by default so it bring the file back | 06:23 |
| zyga | this is why I used two things: sshd -> ssh so that enable/disable works | 06:23 |
| zyga | and the conditional file to actually control being off or on | 06:23 |
| mborzecki | i'm reading up the systemd.unit manpage | 06:24 |
| zyga | ok | 06:24 |
| mborzecki | nice, > aliases specified through Alias= are only effective when the unit is enabled. | 06:24 |
| zyga | I may have missed something, hence the request for review | 06:24 |
| zyga | it's an urgen issue affecting customer deployments | 06:24 |
| mborzecki | zyga: wow that service file is interesting | 06:32 |
| mborzecki | zyga: sshd_not_to_be_run is a debian thing right? | 06:32 |
| zyga | Probably | 06:34 |
| mborzecki | zyga: a similar attemp was there in configure hook in the core snap https://github.com/snapcore/core/commit/dc782a37a3aef3fb11f256260f5205f2ff256acd | 06:52 |
| zyga | oh, that's interesting | 06:53 |
| zyga | we we had this | 06:53 |
| zyga | and lost this? | 06:53 |
| mborzecki | yeah, i was looking for the actual contents of writable paths and found this instead :P | 06:54 |
| mup | PR snapd#4903 opened: tests: [WIP] split prepare-restore.sh into prepare-restore.d modules <Created by zyga> <https://github.com/snapcore/snapd/pull/4903> | 07:09 |
| mup | PR snapd#4913 opened: interfaces,release: probe seccomp features lazily <Created by zyga> <https://github.com/snapcore/snapd/pull/4913> | 07:36 |
| zyga | hmmm, my PR just got a massive TLS handshake failure with everything (google, linode) | 07:38 |
| zyga | all tasks aborted | 07:38 |
| zyga | hope this is not a norm today | 07:38 |
| mborzecki | zyga: ok, played a bit with core image, if i understand it right since we disable the service, the symlink is removed from etc, but then it's still restored from core snap because /etc/systemd/system is synced in writable-paths? | 07:39 |
| zyga | yes | 07:40 |
| zyga | that's the issue | 07:40 |
| zyga | that's the reason we switched to sshd service | 07:40 |
| zyga | but this had unintended consequences | 07:40 |
| mborzecki | from what i see disabling/masking sshd does not work either | 07:40 |
| zyga | yep | 07:46 |
| zyga | I'm testing another mode now | 07:46 |
| zyga | where sshd was disabled | 07:46 |
| zyga | thus breaking people | 07:46 |
| zyga | and chceking if the new core with my fix heals that | 07:47 |
| mborzecki | hah, now i'm locked out of the ssh, and serial console only shows the prompt with ssh login :P, how can i log in? | 07:48 |
| zyga | mborzecki: ha | 07:50 |
| zyga | yank the card | 07:50 |
| zyga | set a password | 07:50 |
| zyga | and put it back in | 07:50 |
| zyga | mvo: so to recap | 07:50 |
| zyga | mvo: I tested the scenario where sshd.service was disabled | 07:51 |
| zyga | and that the new patched snapd can fix that state simply by rebooting (which happens on update) and starting ssh.service | 07:51 |
| zyga | by setting the core config value | 07:51 |
| zyga | so I think this is a fix for the issue in general and for the machines affected in particular | 07:51 |
| mvo | zyga: we do more than just disable, we also mask, I think this will result in a different outcome | 07:52 |
| mvo | my feeling is that we need to at least unmask (opportunisticly and ignore the error) | 07:53 |
| zyga | I will test that now | 07:55 |
| zyga | yes, perhaps | 07:55 |
| zyga | mvo: I think masking doesn't work | 07:56 |
| zyga | zyga@localhost:~$ sudo systemctl disable sshd.service | 07:56 |
| zyga | zyga@localhost:~$ sudo systemctl mask sshd.service | 07:56 |
| zyga | Failed to execute operation: File exists | 07:56 |
| zyga | the initial state was that neither sshd nor ssh were masked | 07:57 |
| mvo | zyga: let me check | 07:57 |
| zyga | hmm | 07:57 |
| zyga | maybe I'm wrong | 07:57 |
| zyga | let me reset everything | 07:57 |
| mvo | zyga: it used to be the case, we masked ssh.service and then sshd.service would reappear via the writable-magic. but let me double check :) | 07:58 |
| zyga | ok | 07:58 |
| zyga | tested that now | 07:58 |
| mvo | my vm will take a moment to be ready, it has an older core | 07:59 |
| zyga | initial state https://www.irccloud.com/pastebin/BMckHA77/ | 07:59 |
| zyga | so sshd is disabled and masked | 07:59 |
| zyga | now I'm setting the core config | 07:59 |
| zyga | and ssh starts and I can log in | 07:59 |
| zyga | ah | 07:59 |
| zyga | because I *start* it | 07:59 |
| zyga | not enable it | 08:00 |
| zyga | let's reboot now | 08:00 |
| zyga | I mean, the hack just starts ssh directly | 08:00 |
| zyga | so even if sshd is masked | 08:00 |
| zyga | that's not an issue | 08:00 |
| zyga | I think we should unmaks it though | 08:00 |
| zyga | but it's not for corectness | 08:00 |
| zyga | but more for purity | 08:00 |
| zyga | mvo: it's also good after reboot | 08:01 |
| zyga | even though sshd is masked | 08:01 |
| mvo | zyga: even with the mask? | 08:01 |
| zyga | lrwxrwxrwx 1 root root 9 Mar 23 2018 /etc/systemd/system/sshd.service -> /dev/null | 08:01 |
| mvo | zyga: wuuut? crazy | 08:01 |
| zyga | (this is after reboot) | 08:01 |
| zyga | woah | 08:02 |
| zyga | https://www.irccloud.com/pastebin/LHAlPevl/ | 08:02 |
| zyga | look at this | 08:02 |
| zyga | sshd.service changed on disk | 08:02 |
| zyga | this is after reboot, no touching at all | 08:02 |
| zyga | how come? | 08:02 |
| zyga | I'll reboot again just to be 100% sure | 08:02 |
| mvo | zyga: I think its confusion from ssh.service vs sshd.service but let me double check | 08:03 |
| zyga | so +1 to unmask it | 08:03 |
| mvo | zyga: I'm so happy we have the /etc/ssh_no_start thing, this unit aliases seem to be crazy | 08:03 |
| zyga | yes, I have the same feeling | 08:04 |
| zyga | testing the new build now | 08:04 |
| zyga | I need to buy some more serial ports | 08:07 |
| === pstolowski|afk is now known as pstolowski | ||
| pstolowski | mornings | 08:07 |
| zyga | this makes testing so much easier | 08:07 |
| mborzecki | mvo: pstolowski: morning guys | 08:08 |
| mvo | hey mborzecki | 08:09 |
| mborzecki | zyga: mvo: could we straighten out the situation with ssh instead of using that ssh_no_run.. monstrosity? | 08:09 |
| mvo | mborzecki: maybe | 08:09 |
| mvo | mborzecki: so there are multiple issues | 08:09 |
| zyga | mborzecki: do you have a proposal? | 08:09 |
| zyga | I mean, yes, but not trivially | 08:10 |
| kalikiana | good morning | 08:10 |
| zyga | we'd have to remove writable-path sync on systemd services | 08:10 |
| mvo | mborzecki: our ssh.service is using Alias=sshd.service | 08:10 |
| mvo | mborzecki: our core snap has "/etc/systemd/system/sshd.service" which is in writable-paths marked as "sync". this means that when the file is missing it will be copied from the core snap | 08:11 |
| mborzecki | as i understand we could fix it in the core snap, have just one ssh.service and no aliases | 08:11 |
| mvo | mborzecki: so we need to unmask both ssh and sshd service | 08:11 |
| mvo | mborzecki: yes, I think that is a good idea, fix the core snap to a) remove the alias b) not use /etc/ but /lib for the unit | 08:11 |
| mvo | mborzecki: but its more risky than what zyga is doing | 08:12 |
| mvo | mborzecki: what I really like about the zyga PR is that its dead-simple | 08:12 |
| zyga | mvo: force pushed with the unmask | 08:12 |
| zyga | tested that it works all the way | 08:12 |
| mvo | ta | 08:12 |
| mborzecki | mhm, it is :) | 08:12 |
| mvo | mborzecki: but gustavo also wants to talk about this so maybe it will be solved differently :) | 08:13 |
| zyga | mvo, mborzecki: one note, we need to tie this to core16 | 08:13 |
| zyga | with core18 the approach should be direct and without hacks | 08:13 |
| mvo | zyga, mborzecki I think its fine to revisit this very soon and fix core16 directly, i.e. fix our ssh install and go back to straight systemctl stop/disable ssh | 08:13 |
| mborzecki | as far as i'm considered, it's +1, but ideally we should fix this and use ssh.service (or sshd, just make sure that it's a single one, no aliases or whatnot) | 08:14 |
| mvo | agreed | 08:14 |
| zyga | (sshd cannot be used) | 08:14 |
| zyga | we must use ssh.service | 08:14 |
| zyga | and get rid of that sync stanza on that director | 08:14 |
| zyga | *directory | 08:14 |
| mvo | mborzecki: whats also annoying is that even if we do stop/disable/mask ssh and sshd it fails on enable because the operations are not symetric | 08:14 |
| mvo | mborzecki: i.e. unmask/enable/start ssh sshd will fail during enable sshd because it says there is a symlink alsready | 08:15 |
| mborzecki | mvo: yeah, i did this in the console :P | 08:15 |
| mvo | mborzecki: which boggles my mind, but maybe I'm missing something | 08:15 |
| mborzecki | and then locked out myself :P | 08:15 |
| zyga | haha | 08:15 |
| zyga | :D | 08:15 |
| zyga | well | 08:15 |
| mvo | mborzecki: like how can you sanely stop a service with aliases | 08:15 |
| mvo | mborzecki: haha | 08:15 |
| zyga | I'm happy with this patch | 08:15 |
| * mvo hugs zyga | 08:15 | |
| zyga | I won't touch it until we talk to gustavo | 08:15 |
| * mvo hugs zyga harder | 08:15 | |
| mborzecki | mvo: 'Failed to execute operation: Too many levels of symbolic links' maybe it's something in system(d,ctl) too | 08:17 |
| mvo | mborzecki: yeah | 08:19 |
| mvo | mborzecki: like "wuuuuut"? | 08:19 |
| mvo | mborzecki: what kind of error is that to begin with | 08:19 |
| pedronis | mvo: hi, there's quite bit of PRs marked for 2.32, anything in particular that I could help with a review? | 08:19 |
| mvo | mborzecki: anyway, sorry, I'm still perplexed how difficult it is to disable ssh via systemctl | 08:20 |
| mborzecki | zyga: pushed an update to #4902 | 08:20 |
| mup | PR #4902: cmd/snap-confine: nvidia: preserve globbed file prefix <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4902> | 08:20 |
| mvo | pedronis: let me unmark some | 08:20 |
| mborzecki | mvo: disabling is simple, making sure that it never gets enabled is a bit harder :P | 08:20 |
| mvo | pedronis: 4882 might be a good one, that is the most critical one right now | 08:20 |
| mvo | mborzecki: heh, exactly | 08:21 |
| zyga | I'll push a unmask of ssh.service now | 08:22 |
| zyga | just testing that after reboot | 08:22 |
| mvo | mborzecki: quick question about 4902 - that fixes nvidia issues on arch? | 08:24 |
| pedronis | mvo: does 4882 need a follow up to use 4911 ? | 08:25 |
| mborzecki | mvo: yes, there's another pr for ubuntu #4908 (though still RFC) that includes the patch from 4902 | 08:25 |
| mvo | pedronis: yes, I will talk to gustavo about it | 08:25 |
| mup | PR #4908: [RFC] cmd/snap-confine: attempt to detect if multiarch host uses arch triplets <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4908> | 08:25 |
| mvo | pedronis: I think technically 4882 will solve the problems we are seeing but gustavo was keen to add the second layer of check (snap run talking to snapd on system-key mismatch) | 08:26 |
| mvo | mborzecki: ta | 08:26 |
| zyga | mvo: unmasking both now | 08:27 |
| mvo | zyga: ta | 08:27 |
| mup | PR snapd#4909 closed: interfaces: harden snap-update-ns profile (2.32) <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4909> | 08:28 |
| mup | PR snapd#4907 closed: advisor: deal with missing commands.db file <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4907> | 08:30 |
| mup | PR snapd#4913 closed: interfaces,release: probe seccomp features lazily <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4913> | 08:30 |
| mup | PR snapd#4914 opened: advisor: add comment why osutil.FileExists(dirs.SnapCommandsDB) is needed <Created by mvo5> <https://github.com/snapcore/snapd/pull/4914> | 08:34 |
| mup | PR snapd#4915 opened: interfaces,release: probe seccomp features lazily <Created by mvo5> <https://github.com/snapcore/snapd/pull/4915> | 08:35 |
| pedronis | mvo: left some questions/comments | 08:38 |
| mvo | pedronis: yay,thank you! | 08:38 |
| zyga | mborzecki: https://github.com/snapcore/snapd/pull/4902/files#r176670685 | 08:54 |
| mup | PR #4902: cmd/snap-confine: nvidia: preserve globbed file prefix <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4902> | 08:54 |
| mborzecki | zyga: responded :) | 08:56 |
| zyga | replied too | 08:57 |
| mborzecki | zyga: ok, makes sense, btw. i was surprised by it too | 08:59 |
| pedronis | mvo: not urgent but I'm confused/see some duplication of things in the roadmap page (for 2.32 and 2.33 and after) | 09:01 |
| zyga | mvo: did you notice https://github.com/snapcore/core/commit/dc782a37a3aef3fb11f256260f5205f2ff256acd | 09:03 |
| zyga | mborzecki pointed me to it | 09:03 |
| mborzecki | zyga: force pushed | 09:05 |
| zyga | thank you | 09:05 |
| mborzecki | damn, i feel sick, finally caught something from the junior | 09:08 |
| mvo | zyga: no, huh | 09:12 |
| zyga | mborzecki take it easy | 09:12 |
| zyga | mvo: fun, right? | 09:12 |
| zyga | where is that, it's gone now? | 09:12 |
| zyga | it was lost when we went to in-core config | 09:13 |
| mvo | zyga: yeah :( | 09:13 |
| mvo | zyga: sadly | 09:13 |
| zyga | full circle | 09:13 |
| Chipaca | 'sup? | 09:14 |
| zyga | Chipaca: fire, smoke, screams and tears | 09:14 |
| Chipaca | zyga: I didn't ask about your bedroom | 09:14 |
| zyga | Chipaca: it's slowly getting better onw | 09:17 |
| Chipaca | zyga: good to hear :-) | 09:18 |
| Chipaca | what did we lose by moving to in-core config? | 09:18 |
| * Chipaca curious | 09:18 | |
| zyga | Chipaca: ssh control | 09:19 |
| zyga | https://github.com/snapcore/snapd/pull/4912 | 09:19 |
| mup | PR #4912: overlord/configstate: change how ssh is stopped/started (2.32) <Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/4912> | 09:19 |
| kalikiana | so, I have a question about the ssh-keys interface. I added it to my snap, did `snap connect` to enable it but when the snap runs I'm getting `Permissions 0644 for '/home/rumo/.ssh/id_local' are too open.` | 09:23 |
| kalikiana | The permissions that ls reports within --shell as the same as on the real system, though. And of course the permissions are correct on the real system. | 09:24 |
| mvo | zyga: quick question about reverts from 2.32 -> 2.31. the new/updated security profiles will break reverts, no? I mean, suppose 2.32 has the stricter profiles and it loads the per-snap update-ns profiles. now on revert the old snap run will start network-manager and that will use the new 2.32 profiles which are strict and won't let n-m run, is that correct? | 09:26 |
| zyga | on revert snap-confine will load the old profile which is permissive for content and doesn't support layouts | 09:27 |
| Chipaca | kalikiana: that's very weird | 09:27 |
| Chipaca | kalikiana: is the snap in the store? i'd like to take a look | 09:27 |
| zyga | mvo: the snap-confine profile is from core directly | 09:27 |
| mvo | zyga: awsome | 09:27 |
| zyga | mvo: and the issue didn't affect the regular profiles, just s-u-n profile | 09:27 |
| mvo | zyga: so no problem here, just wanted to double check | 09:28 |
| zyga | mvo: I think it should work | 09:28 |
| zyga | but I did not check | 09:28 |
| mvo | zyga: its part of our standard tests so we will know | 09:28 |
| zyga | ok | 09:28 |
| kalikiana | Chipaca: I haven't pushed that bit yet since it's not working, but I can push it | 09:28 |
| Chipaca | kalikiana: is it an easy to build snap? | 09:29 |
| Chipaca | i mean, just run snapcraft and wait sort of thing? | 09:29 |
| Chipaca | in that case i could use just the snapcraft.yaml :-) | 09:29 |
| Chipaca | to be clear, this is just me being lazy: i'd rather tinker with it to figure out what's going on, than think about it in the abstract | 09:30 |
| mvo | zyga: hey, does https://paste.ubuntu.com/p/jxtYz7z9CD/ ring any bells? | 09:43 |
| zyga | mmm | 09:43 |
| mvo | zyga: happend on master | 09:43 |
| zyga | is that on release | 09:43 |
| zyga | aww | 09:43 |
| zyga | I didn't push the debug thing to master | 09:43 |
| zyga | mvo: can you please force-land that https://github.com/snapcore/snapd/pull/4916 | 09:45 |
| mup | PR #4916: tests: change debug for layout test <Created by zyga> <https://github.com/snapcore/snapd/pull/4916> | 09:45 |
| mup | PR snapd#4916 opened: tests: change debug for layout test <Created by zyga> <https://github.com/snapcore/snapd/pull/4916> | 09:46 |
| Chipaca | kalikiana: so, I can't reproduce your issue | 09:56 |
| Chipaca | kalikiana: wondering what's different between here and there | 09:56 |
| Chipaca | kalikiana: one thing though, i don't think your snap needs to include openssh-client, as that's part of core and ssh-keys gives you access to it (at least to /usr/bin/ssh) | 09:58 |
| Chipaca | but here it works with both the ssh in the snap, and the one in core | 09:58 |
| kalikiana | Ah, I didn't realize that. So I can drop the package. | 10:00 |
| Chipaca | kalikiana: but that doesn't answer where the 0644 error is coming from :-) | 10:00 |
| kalikiana | Chipaca: Could it be a bug in snap connect then? Vaguely thinking of content interface issues depending on the order of install/connect/snap executation | 10:01 |
| Chipaca | kalikiana: wait, can you ssh from within --shell? | 10:01 |
| Chipaca | that's what i am doing and it works, if it works for you as well the issue is even weirder | 10:01 |
| kalikiana | Chipaca: No, same error, even on a simple "ssh helga" (helga being an arbitrary host that works outside confinement) | 10:02 |
| Chipaca | phew | 10:03 |
| kalikiana | Chipaca: Also getting a `Control socket connect(/home/rumo/.ssh/socket-root@***.***.***.***:22): Permission denied`with just ssh, before the other error | 10:04 |
| Chipaca | kalikiana: just to make sure, can you ‘find /home/rumo/.ssh -ls’? | 10:04 |
| kalikiana | Chipaca: That works and lists each key with -rw-r--r-- | 10:06 |
| Chipaca | kalikiana: inside, or outside? | 10:06 |
| kalikiana | Chipaca: both are the same | 10:07 |
| Chipaca | kalikiana: why are your private keys world-readable? | 10:07 |
| Chipaca | kalikiana: (see also: how is your ssh 'outside' not complaining about it all the time?) | 10:08 |
| kalikiana | Hmmm now that you mention it, they probably shouldn't be... | 10:08 |
| kalikiana | Chipaca: Outside is fine | 10:08 |
| Chipaca | kalikiana: fine how? | 10:09 |
| kalikiana | As in, I get no errors | 10:09 |
| Chipaca | right | 10:09 |
| kalikiana | But lemme try and change them | 10:09 |
| Chipaca | kalikiana: but you should :-) | 10:09 |
| Chipaca | kalikiana: only thing I can think of is that you have seahorse auto-adding your ssh keys on session start, and it's less picky about permissions, and ssh never looks at your actual keys because by the time you run it they're in the agent | 10:10 |
| Chipaca | kalikiana: if that's the case, it sounds like a bug in seahorse to me ;) | 10:11 |
| Chipaca | kalikiana: but also, sounds like an easy thing to detect and warn about from your snap | 10:12 |
| Chipaca | ie, can i access the keys? no: tell user about snap connect, yes: are they the right perms? no: tell user about it | 10:12 |
| kalikiana | Dropping the group/world reads seems to fix it indeed. | 10:13 |
| kalikiana | Now ssh just tells me `bind: Permission denied` and `unix_listener: cannot bind to path` | 10:14 |
| mborzecki | zyga: i think i need to check for a specific library in autodetection, one thats' always there, libnvidia-glcore.so.<maj>.<min> looks like a good candidate | 10:14 |
| kalikiana | But it unlocked the key/passphrase fine | 10:14 |
| * kalikiana tests git push from the snap | 10:14 | |
| Chipaca | kalikiana: that socket might be a control master thing, which isn't supported from inside | 10:16 |
| Chipaca | kalikiana: (the thing that lets you multiplex ssh sessions over a single connection) | 10:16 |
| kalikiana | Chipaca: You were right, disabling it explictly works | 10:21 |
| kalikiana | Chipaca: Thanks a lot for helping me track this down! | 10:25 |
| cachio | Saviq, a new fedora image with the correct size is uploaded | 10:26 |
| kalikiana | Now if it'd actually use the ssh agent, that'd be sweet, but it works perfectly | 10:26 |
| Saviq | cachio: ack, me tries | 10:26 |
| cachio | Saviq, great, tell me how if goes | 10:27 |
| Saviq | cachio: -64 suffix, too? | 10:27 |
| Chipaca | kalikiana: talk with jdstrand, but I suspect much laughter to be had :-) | 10:28 |
| kalikiana | :-D | 10:29 |
| cachio | Saviq, fedora-27-64 | 10:40 |
| cachio | this is the name you have to use | 10:40 |
| cachio | Saviq, you don't need to specify the image | 10:41 |
| mborzecki | zyga: can you take another look at #4902? | 10:44 |
| mup | PR #4902: cmd/snap-confine: nvidia: preserve globbed file prefix <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4902> | 10:44 |
| mborzecki | zyga: also i'm finishing with fixes for 4908 and will be switching back to ubuntu to check if things didn't break :) | 10:45 |
| Chipaca | zyga: might #1757284 be an instance of the 'interface connections go away' bug? | 10:45 |
| mup | Bug #1757284: Several snap apps fail to launch <amd64> <apport-bug> <bionic> <snapd (Ubuntu):Confirmed> <https://launchpad.net/bugs/1757284> | 10:45 |
| Chipaca | (now that i think about it, it probably is -- but don't know that bug #) | 10:46 |
| zyga | mborzecki: looking | 10:48 |
| mup | PR snapd#4916 closed: tests: change debug for layout test <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4916> | 10:49 |
| * Chipaca attempts to dad up and get out of bed | 10:49 | |
| zyga | Chipaca: updated the bug with a question | 10:50 |
| Saviq | cachio: yeah, it's lookin' good https://travis-ci.org/MirServer/mir/builds/357325003 | 10:57 |
| * Chipaca AFK for a while (physio, etc) | 11:01 | |
| mvo | pedronis: hrm, hrm, my system-key pr has a bit of a problem on core. on firstboot on core snapd starts, generates the system-key and the core revision is empty (because things are not seeded yet). then stuff gets seeded but that means the systme-key changes because the core revision is part of the inputs | 11:04 |
| mvo | zyga: (cc -^) | 11:05 |
| * mvo will need to think about this while taking a short lunch break | 11:05 | |
| zyga | hmm hmm hmm | 11:05 |
| zyga | mvo: if we drop core revision? | 11:05 |
| zyga | we keep it because apparmor profiles are there (includes) | 11:05 |
| zyga | hmm :/ | 11:06 |
| mvo | zyga: yeah, feels strange to just drop it, but maybe we can drop it on core? | 11:06 |
| zyga | well, if apparmor gets updated | 11:07 |
| mvo | zyga: let me try that | 11:07 |
| zyga | and snapd doesn't | 11:07 |
| zyga | it's not correct then | 11:07 |
| mvo | zyga: hm, good point | 11:07 |
| pedronis | will apparmor come from the snapd snap? | 11:08 |
| pedronis | fun questions | 11:08 |
| cachio | Saviq, nice | 11:09 |
| cachio | good to hear that | 11:09 |
| pedronis | don't we check the feature of apparmor becuase of that? | 11:09 |
| mvo | pedronis: partly, the abstractions are also part of the mix and those come from the core snap | 11:10 |
| mvo | pedronis: apparmor has two inputs: the kernel capabilities and the abstractions with pre-writen rules | 11:11 |
| mvo | we could force a re-gen of the profiles after seeding but thats a bit ugly | 11:11 |
| mvo | full of dragons this pr | 11:11 |
| mborzecki-ubuntu | https://paste.ubuntu.com/p/77k9YJbVQK/ | 11:12 |
| mborzecki-ubuntu | humanSuite.TestHuman failed here, but was passing yesterday | 11:12 |
| zyga | pedronis: yes, apparmor will come from core snap (on core) | 11:12 |
| zyga | pedronis: I mean the include statements, we check for kernel features but each profile we make has include statements that reach to apparmor profiles from either native package or from the core snap on all-snap boxes | 11:13 |
| mborzecki-ubuntu | Chipaca: DST change https://paste.ubuntu.com/p/77k9YJbVQK/ | 11:14 |
| zyga | LOL | 11:14 |
| zyga | stuff will never stop to amaze me | 11:14 |
| mborzecki-ubuntu | otoh, i had no clue that we're changing clocks on saturday ;) | 11:15 |
| pedronis | mvo: also the issue with seeding exists also on classic | 11:22 |
| pedronis | mvo: we support seeding there with snaps | 11:22 |
| zyga | mvo: I have an idea | 11:23 |
| pedronis | mvo: it's a bit unclear to me why are we writing a key before being seeded ? | 11:23 |
| pedronis | mvo: we should wait/skip that, if there are not snaps, we are not seeded | 11:24 |
| zyga | mvo: instead of revision we can inspect a single file in /meta/manifest | 11:24 |
| zyga | (we don't have that file yet) | 11:24 |
| zyga | but as long as it encodes the version of every package | 11:24 |
| zyga | it caputres the essence of core | 11:25 |
| zyga | pedronis: we need that on boot to setup profiles for core itself | 11:25 |
| zyga | pedronis: on snapd daemon startup actually | 11:25 |
| Chipaca | mborzecki: on the one hand, huzzah it works | 11:25 |
| Chipaca | mborzecki: on the other, booo | 11:25 |
| pedronis | zyga: ok, there's a recursion problem here | 11:25 |
| Chipaca | mborzecki: on the last one, a test that works 350 days a year is fine, right? | 11:25 |
| pedronis | the double build-id doesn't make sense then | 11:25 |
| * Chipaca now really goes | 11:26 | |
| zyga | pedronis: can you explain? | 11:26 |
| pedronis | zyga: we generate profile with a key id without build-id for a thing that by definition has one | 11:26 |
| Chipaca | mborzecki: actually that might be a real bug, need to look when i get back | 11:26 |
| pedronis | then we regenerate just because | 11:26 |
| zyga | pedronis: why without a build-id? | 11:26 |
| pedronis | zyga: because there's no /snap/core/current | 11:27 |
| pedronis | at that point | 11:27 |
| pedronis | at least that's what I understood was the problem | 11:27 |
| pedronis | mvo mentioned | 11:27 |
| zyga | pedronis: on core we could just use /usr/lib/snapd/snapd | 11:27 |
| zyga | no need for core's build-id | 11:27 |
| pedronis | zyga: as I said the same issue exists with classic seeding | 11:27 |
| pedronis | first boot is not a core only thing | 11:27 |
| zyga | on classic we can use /proc/self/exe | 11:28 |
| zyga | but we cannot capture apparmor changes in the distro | 11:28 |
| zyga | but I think that is done by apparmor itself, it has a trick to detect that in the init script | 11:28 |
| pedronis | zyga: did you look at the PR, the system-key has two build-ids in it | 11:28 |
| pedronis | atm | 11:28 |
| zyga | yes, we discussed that last night | 11:28 |
| mup | PR snapd#4917 opened: repo: added repo ConnectionsInfo method (for the new snap connections API) <Created by stolowski> <https://github.com/snapcore/snapd/pull/4917> | 11:28 |
| zyga | I think we can evolve that pattern | 11:28 |
| pedronis | on core we need only one | 11:28 |
| pedronis | but on classic | 11:28 |
| pedronis | we have a problem | 11:28 |
| zyga | on classic we also need only one but we did two because that's easier | 11:29 |
| pedronis | but then seeding | 11:29 |
| zyga | (because we don't know which) | 11:29 |
| pedronis | it's not easy anymore | 11:29 |
| zyga | mvo: what do you think? | 11:29 |
| zyga | pedronis: I agree | 11:29 |
| zyga | pedronis: no easy way out yetr | 11:29 |
| zyga | *yet | 11:29 |
| pedronis | one issue is also that we think reexec | 11:29 |
| pedronis | is something you can turn on and off | 11:30 |
| pedronis | easily | 11:30 |
| pedronis | but is quite true | 11:30 |
| pedronis | especial if you are outside snapd and the inside snapd | 11:30 |
| pedronis | are far apart | 11:30 |
| pedronis | *is not quite true | 11:30 |
| zyga | mborzecki: https://github.com/snapcore/snapd/pull/4908#issuecomment-375392384 | 11:45 |
| mup | PR #4908: [RFC] cmd/snap-confine: attempt to detect if multiarch host uses arch triplets <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4908> | 11:45 |
| * cachio afk | 11:45 | |
| mup | PR snapd#4903 closed: tests: [WIP] split prepare-restore.sh into prepare-restore.d modules <Created by zyga> <Closed by zyga> <https://github.com/snapcore/snapd/pull/4903> | 11:46 |
| mborzecki | zyga: it's alredy fixed, i also improved detection and look for a specific nvidia lib (libnvidia-glcore.so), the packaging is also updated, now i'm cleaning up s-c apparmor profile | 11:47 |
| mborzecki | zyga: https://github.com/bboozzoo/snapd/commits/bboozzoo/nvidia-glob-prefix-ubuntu-triplet-wip if you want to take a look | 11:48 |
| pedronis | mvo: zyga: I made a comment, one idea could be to simply delay writint the key when we are not yet seeded | 11:50 |
| zyga | thanks, | 11:50 |
| zyga | but this is going to change one important property | 11:51 |
| zyga | (perhaps) | 11:51 |
| zyga | that daemon doesn't respond to api requests before having stable security profiles | 11:51 |
| pedronis | ? | 11:51 |
| zyga | maybe if we are not seeded (rare) we should just always rewrite profiles on startup (mvo: opinion?) | 11:51 |
| zyga | actually, I'm confused: writing the system key is not the same as computing it in memory | 11:51 |
| zyga | so perhaps that's okay | 11:51 |
| pedronis | if we are seeded there no snap profiles | 11:51 |
| pedronis | sorry | 11:52 |
| zyga | that's true | 11:52 |
| pedronis | if we are not seeeded | 11:52 |
| zyga | yes, that's a good point | 11:52 |
| pedronis | I'm saying, if not seeded delay until we really need it | 11:52 |
| pedronis | which is about when we generate profiles for core | 11:52 |
| pedronis | (which is the first thing we seed) | 11:52 |
| zyga | right, I understand that now, I forgot that not seeded == nothins is installed | 11:52 |
| zyga | so I think this is okay | 11:52 |
| pedronis | I might be missing something | 11:53 |
| zyga | mborzecki: interesting https://build.opensuse.org/request/show/590390 | 11:56 |
| zyga | pstolowski: https://github.com/snapcore/snapd/pull/4917#pullrequestreview-106469715 | 12:16 |
| mup | PR #4917: repo: added repo ConnectionsInfo method (for the new snap connections API) <Created by stolowski> <https://github.com/snapcore/snapd/pull/4917> | 12:16 |
| mborzecki-ubuntu | zyga: that opensuse review looks good, maybe advise him to update to 2.31.2 while at it | 12:23 |
| zyga | I'll merge it and encourage him to iterate | 12:24 |
| zyga | mborzecki-ubuntu: the package is not buildable on leap though | 12:25 |
| zyga | actually, I didn't merge it but I added some comments | 12:27 |
| * zyga lunch | 12:32 | |
| mborzecki-ubuntu | zyga: i'll force push to #4908 | 12:33 |
| mup | PR #4908: [RFC] cmd/snap-confine: attempt to detect if multiarch host uses arch triplets <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4908> | 12:33 |
| zyga | k | 12:33 |
| mborzecki-ubuntu | and pushed | 12:35 |
| mborzecki-ubuntu | ohmygiraffe works with nvidia and confinement now ;) | 12:36 |
| mborzecki-ubuntu | let me try the the flare snap | 12:37 |
| niemeyer | Glad for your giraffe :P | 12:41 |
| niemeyer | Morning all | 12:41 |
| mvo | pedronis: delaying sounds reasonable, let me have a look at this | 12:46 |
| pedronis | Chipaca: is you that moved the waitMixin stuff to its own wait.go file? | 12:50 |
| Chipaca | pedronis: that sounds like me | 12:58 |
| Chipaca | pedronis: why | 12:58 |
| pedronis | Chipaca: there's a function in cmd_snap_op.go that probably should have moved too, given that is used only in the wait code, lastLogStr | 12:58 |
| Chipaca | pedronis: ah, good call | 12:59 |
| pedronis | Chipaca: I noticed because I'm solving conflicts in one of my old PRs | 12:59 |
| mvo | pedronis, zyga yet another `snap run` system-key issue on core. when we install a new core we update "current". but we only reboot ~10min later. during that time the system-key on disk and the one that snap run calcualtes are different because /snap/core/current is part of the inputs | 13:25 |
| pedronis | mvo: I'm fixed the 10 minutes things but that's part of the problem | 13:25 |
| pedronis | *that's only | 13:26 |
| mvo | pedronis: | 13:26 |
| mvo | pedronis: yes | 13:26 |
| pedronis | mvo: I suppose there the endpoint might help | 13:26 |
| sergiusens | mvo: is there a way to `snap run --shell core18....`? | 13:30 |
| Chipaca | sergiusens: no, snap run only runs apps* | 13:36 |
| Chipaca | * and hooks but you don't want to do that | 13:37 |
| mvo | sergiusens: a trivial snap that just has a /bin/sh and uses base: core18 | 13:38 |
| mvo | sergiusens: might be the solution | 13:38 |
| sergiusens | mvo: yeah, I am doing just that; to verify core18 is in good shape (using the test job I created, but using yours should be fine as well, I couldn't find it from the store just yet though) | 13:39 |
| sergiusens | python/asciinema base: core18 and install to test the story end2end | 13:39 |
| sergiusens | mvo ok, everything is working now | 13:42 |
| mvo | sergiusens: sys | 13:43 |
| mvo | sergiusens: eh, I mean yay | 13:43 |
| sergiusens | lol, I was trying to decipher that :-P | 13:44 |
| pedronis | Chipaca: this bit of code seems nonsensical or am I confused?: https://github.com/snapcore/snapd/blob/master/cmd/snap/cmd_snap_op.go#L543 | 13:44 |
| zyga | E: Type 'curity' is not known on line 50 in source list /etc/apt/sources.list | 13:45 |
| zyga | E: The list of sources could not be read. | 13:45 |
| zyga | how did we get 'curity' out of 'security' | 13:45 |
| Chipaca | pedronis: that looks broken indeed | 13:45 |
| zyga | https://api.travis-ci.org/v3/job/357284871/log.txt | 13:45 |
| pedronis | Chipaca: I find it by chance, I suppose not all the error paths are tested :/ | 13:47 |
| zyga | mborzecki-ubuntu: https://github.com/snapcore/snapd/pull/4908 broken | 13:51 |
| mup | PR #4908: [RFC] cmd/snap-confine: attempt to detect if multiarch host uses arch triplets <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4908> | 13:51 |
| mborzecki-ubuntu | zyga: force pushed just now | 13:51 |
| mborzecki-ubuntu | zyga: `dpkg-architecture -a i386` doesn't work on 14.04, you need to do `dpkg-architecture -ai386` :/ | 13:52 |
| Chipaca | pedronis: the wait one is hard to test i guess | 13:52 |
| Chipaca | seems to be the only one with that particular bug though :-) | 13:52 |
| pedronis | yes | 13:53 |
| pedronis | funnily enough my test used that one command | 13:53 |
| niemeyer | Hangouts going crazy | 13:53 |
| Chipaca | pedronis: do you want me to fix it, or will you? | 13:56 |
| pedronis | Chipaca: I can make a PR in a bit, also moving that other function | 13:58 |
| pedronis | Chipaca: trying to finish the merge right now | 13:58 |
| Chipaca | pedronis: k | 13:58 |
| mvo | cachio: for the sru validation, could you pastebin the failures again for me please (or link) | 14:02 |
| mborzecki | Chipaca: if you're willing to try the nvidia stuff, please grab https://github.com/snapcore/snapd/pull/4908 | 14:12 |
| mup | PR #4908: [RFC] cmd/snap-confine: attempt to detect if multiarch host uses arch triplets <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4908> | 14:12 |
| Chipaca | mborzecki: actual nvidia, right? | 14:13 |
| Chipaca | I need to restart my session for that, give me a mo' | 14:13 |
| mborzecki | Chipaca: yes | 14:13 |
| ackk | mvo, hi, WRT the discussion at the sprint about having a "nobody" user snaps, has there been further discussion on whether that's acceptable? | 14:14 |
| Chipaca | mborzecki: what do i need to build against this to test? | 14:17 |
| mborzecki | Chipaca: just build and reinstall the package, add SNAP_REEXEC=0 to /etc/environment to make sure you're runing the right thing | 14:18 |
| mborzecki | Chipaca: once snapd deb is installed, please snap install graphics-debug-tools-bboozzoo too | 14:20 |
| Chipaca | mborzecki: … i'm getting the DST change error also | 14:28 |
| mborzecki | Chipaca: i've commented it out locally :) sorry | 14:29 |
| Chipaca | mborzecki: is this your revenge for that code :-) | 14:29 |
| * Chipaca runs it agian with DEB_BUILD_OPTIONS=nocheck | 14:29 | |
| mborzecki | Chipaca: can you also do `cat /sys/module/nvidia/version`? | 14:34 |
| Chipaca | mborzecki: 384.111 | 14:34 |
| Chipaca | mborzecki: snapd build and running | 14:35 |
| Chipaca | mborzecki: an' now? | 14:35 |
| mborzecki | Chipaca: have you installed graphics-debug-tools-bboozzoo --edge? | 14:35 |
| * mborzecki forgot to mention --edge before | 14:36 | |
| Chipaca | mborzecki: graphics-debug-tools-bboozzoo 1.0 3 edge maciek-borzecki - | 14:36 |
| cachio | mvo, https://paste.ubuntu.com/p/KB7SyvgRn4/ | 14:36 |
| mborzecki | Chipaca: SNAP_CONFINE_DEBUG=1 SNAPD_DEBUG=1 snap run graphics-debug-tools-bboozzoo.glxinfo |& tee log and paste the log somewhere | 14:36 |
| cachio | these are the spread tests | 14:36 |
| mborzecki | oh, and SNAP_REEXEC=0 | 14:36 |
| cachio | mvo, give me 5 minutes | 14:37 |
| mborzecki | Chipaca: SNAP_CONFINE_DEBUG=1 SNAPD_DEBUG=1 SNAP_REEXEC=0 snap run .. | 14:37 |
| cachio | mvo, also many of the autopkgtets failed because of "package context: unrecognized import path "context" (import path does not begin with hostname)" | 14:38 |
| cachio | seem to be a different go version installed there | 14:39 |
| Chipaca | mborzecki: http://paste.ubuntu.com/p/94hmf6KcHN/ | 14:40 |
| Chipaca | mborzecki: do you also want me to test it with the intel board (ie with prime turned off) | 14:40 |
| Chipaca | or turned to nvidia | 14:40 |
| mborzecki | Chipaca: looking good, can you try some snap that does graphics? eg. ohmygiraffe | 14:40 |
| * Chipaca doesn't know what these things are called | 14:41 | |
| Chipaca | mborzecki: ohmygiraffe, supertuxkart and minecraft worked | 14:41 |
| pedronis | Chipaca: I managed to add tests, not super interesting tests but better than nothing | 14:41 |
| * Chipaca hugs pedronis | 14:42 | |
| Chipaca | pedronis: thank you | 14:42 |
| Chipaca | mborzecki: so as far as i'm concerned there is no regression with this :-) | 14:42 |
| mborzecki | Chipaca: one more thing, can you sudo nsenter -m/run/snap/ns/ohmygiraffe.mnt | 14:43 |
| mborzecki | Chipaca: and then `cat /proc/self/mountinfo` and paste it | 14:43 |
| Chipaca | mborzecki: snapd, but yes | 14:43 |
| mborzecki | right /run/snapd/ns/.. | 14:44 |
| Chipaca | mborzecki: whoa, leaving the mountspace makes x unhappy for a bit | 14:44 |
| mborzecki | hm? how so? | 14:45 |
| mborzecki | Chipaca: if you could also check that intel works :) | 14:48 |
| mborzecki | zyga: mvo: do we want the nvidia fixes for 2.32? | 14:48 |
| Chipaca | mborzecki: http://paste.ubuntu.com/p/4GhMJmt22t/ | 14:52 |
| Chipaca | mborzecki: or http://paste.ubuntu.com/p/Rp7TQVjsDY/ | 14:52 |
| Chipaca | mborzecki: I can try nvidia in a bit | 14:53 |
| mborzecki | Chipaca: great, so it doesn't regress and mounts the right thing :) | 14:53 |
| Chipaca | mborzecki: wrt leaving the mount namespace, it's as if i'd gone to console and back | 14:53 |
| Chipaca | even redshift gets reset | 14:54 |
| mborzecki | Chipaca: that's weird, maybe it's because the env is still there | 14:54 |
| Chipaca | mborzecki: you don't see that? | 14:54 |
| mborzecki | nope | 14:54 |
| Chipaca | ok, switching to nvidia | 14:54 |
| * Chipaca in a rush | 14:54 | |
| Chipaca | mborzecki: http://paste.ubuntu.com/p/yKb7RhVffw/ | 14:56 |
| Chipaca | mborzecki: with intel | 14:56 |
| Chipaca | mborzecki: but … the ns are still there, i guess, maybe that's the problem? | 14:56 |
| Chipaca | mborzecki: http://paste.ubuntu.com/p/8xhdRfjnZw/ | 14:56 |
| Chipaca | now 3d apps don't work | 14:57 |
| mborzecki | Chipaca: is the driver loaded now? lsmod|grep nvidia? | 14:58 |
| Chipaca | mborzecki: i've got to go, but i'll be back in ~45 | 14:58 |
| Chipaca | mborzecki: it is not | 14:58 |
| mup | PR snapd#4918 opened: cmd/snap: fix one issue with noWait error handling logic, add tests <Created by pedronis> <https://github.com/snapcore/snapd/pull/4918> | 14:58 |
| * Chipaca runs | 14:59 | |
| mborzecki | Chipaca: sudo /usr/lib/snapd/snap-discard-ns graphics-debug-tools-bboozzoo | 14:59 |
| mborzecki | ah ok :) i'll leave you some notes here | 14:59 |
| mborzecki | Chipaca: thank you for testing this | 14:59 |
| mup | PR snapcraft#2022 opened: Polish our landing page ✨ <Created by evandandrea> <https://github.com/snapcore/snapcraft/pull/2022> | 15:12 |
| pedronis | mvo: zyga: on classic we use the apparmor abstractions from the host? or from core ? | 15:24 |
| mvo | pedronis: host | 15:24 |
| pedronis | mvo: that is not really part of the system key atm, no? | 15:25 |
| pedronis | then | 15:25 |
| mvo | pedronis: yes, we just discussed htis that we can drop | 15:25 |
| mvo | pedronis: which makes the whole problem much easier | 15:28 |
| mvo | pedronis: also on core we apparently don't need this input because one of the init scripts (apparmor init) will rebuild the profiles if the abstraction change | 15:29 |
| mvo | pedronis: this makes the problem on core go away when the current symlink is missing | 15:29 |
| mvo | pedronis: eh, missing or updated too early | 15:29 |
| mvo | pedronis: which is really nice, I will update my PR now with this, I hope we finally have figured it out | 15:29 |
| Chipaca | zyga: ping | 15:30 |
| Chipaca | ah, maybe nothing | 15:31 |
| pedronis | Chipaca: I pushed #4918 | 15:31 |
| mup | PR #4918: cmd/snap: fix one issue with noWait error handling logic, add tests plus other cleanups <Created by pedronis> <https://github.com/snapcore/snapd/pull/4918> | 15:31 |
| Chipaca | pedronis: ok | 15:31 |
| zyga | I’m on a walk | 15:32 |
| cachio | mvo, there? | 15:34 |
| pedronis | mvo: let me know when/if I should look again | 15:34 |
| mvo | cachio: yes, sorry | 15:35 |
| mvo | pedronis: will do | 15:35 |
| cachio | np | 15:35 |
| cachio | so, about the sru | 15:36 |
| mvo | yes | 15:36 |
| cachio | mvo, did you see the expect error? | 15:36 |
| mvo | cachio: what was the pastebin again, sorry, missed it was in a very long meeting | 15:36 |
| cachio | np | 15:37 |
| cachio | mvo, many execs failed because of this https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/i386/s/snapd/20180319_122711_e1003@/log.gz | 15:37 |
| cachio | in autopkgtests | 15:37 |
| mvo | cachio: uhhh, I don't know where this one comes from :( | 15:38 |
| cachio | mvo, in xenial all the execs failed because of that | 15:39 |
| cachio | seem to be a problem with the go version | 15:39 |
| pedronis | mvo: cachio: seems like spread doesn't build with 1.6 anymore | 15:40 |
| pedronis | that's the issue I think | 15:40 |
| cachio | pedronis, yes | 15:40 |
| pedronis | it's using context directly | 15:40 |
| cachio | I am builing with 1.9 | 15:40 |
| cachio | to avoid issues on spread | 15:40 |
| pedronis | well but that test | 15:40 |
| pedronis | is trying to build spread | 15:41 |
| pedronis | on xenail | 15:41 |
| pedronis | go get -u github.com/snapcore/spread/cmd/spread | 15:41 |
| pedronis | boom | 15:41 |
| cachio | but, why it is doing go get instead of downloading spread? | 15:41 |
| pedronis | downloading from where? | 15:41 |
| pedronis | this is autopkgtests | 15:42 |
| cachio | from aws | 15:42 |
| pedronis | not even sure they will let us download a random binary | 15:42 |
| cachio | or the snap | 15:42 |
| pedronis | the snap might be an option I suppose | 15:42 |
| pedronis | or spread need to be made to build 1.6 again | 15:43 |
| pedronis | with 1.6 | 15:43 |
| cachio | pedronis, well that's something for niemeyer | 15:43 |
| pedronis | though I'm not quire sure if the autopackage test for snapd can use snap | 15:44 |
| pedronis | mvo: we cannot build spread anymore on xenial | 15:44 |
| mup | PR snapcraft#2008 closed: Release changelog for 2.40 <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2008> | 15:45 |
| mvo | Chipaca: what was the status of "human_tsest.go:67" ? it fails here with http://paste.ubuntu.com/p/nyxKqcQFqQ/ | 15:56 |
| Chipaca | :-( | 15:56 |
| Chipaca | mvo: yeah, mborzecki alerted me of that, and indeed it's failing here as well | 15:56 |
| Chipaca | so there's two bugs in it | 15:57 |
| mborzecki | mvo: it will be fixed tomorrow, or a day after | 15:57 |
| Chipaca | one in the test, which happily pretends it'll never get run on dst, and one in the code because there's a day missing | 15:57 |
| mborzecki | until the next dst change | 15:57 |
| Chipaca | need to look into that | 15:57 |
| Chipaca | mborzecki: glxinfo is still not finding the intel driver, but the games work | 15:58 |
| Chipaca | mborzecki: http://paste.ubuntu.com/p/XcmQZmvN5z/ | 15:58 |
| Chipaca | mborzecki: and http://paste.ubuntu.com/p/cZBYzXvX73/ | 15:59 |
| mborzecki | Chipaca: hmm interesting, can you check if it behaves the same with the regular snapd? | 15:59 |
| Chipaca | mborzecki: it did | 15:59 |
| mborzecki | Chipaca: ah ok, then there's no regression :) | 16:00 |
| Chipaca | bah, i can double check just in case | 16:00 |
| mvo | mborzecki: I can't wait that long … | 16:01 |
| Chipaca | mborzecki: hmm, hold on, some of these tests were run without SNAP_REEXEC=0 :-( | 16:02 |
| Chipaca | augh | 16:02 |
| cachio | mvo, also there errors I saw in linode exec for sru https://paste.ubuntu.com/p/KB7SyvgRn4/ | 16:09 |
| cachio | mvo, many "Permission denied" | 16:09 |
| mvo | cachio: I think these are the ones i'm concerned about | 16:10 |
| Chipaca | mborzecki: yeah, same | 16:10 |
| cachio | mvo, I am gonna debug those | 16:10 |
| zyga | mvo: did we land https://github.com/snapcore/snapd/pull/4877 for 2.32? | 16:10 |
| mup | PR #4877: snap-confine: fallback to /lib/udev/snappy-app-dev if the core is older <Created by mvo5> <https://github.com/snapcore/snapd/pull/4877> | 16:10 |
| zyga | mvo: I think we need too | 16:10 |
| zyga | to too | 16:10 |
| mborzecki | Chipaca: that's sort of great :) thank you for checking | 16:10 |
| Chipaca | mvo: I can fix that human time bug | 16:10 |
| Chipaca | mvo: probably :-p | 16:11 |
| mvo | Chipaca: please | 16:11 |
| mvo | zyga: I think so | 16:12 |
| mvo | zyga: but not for 2.31 .( | 16:12 |
| mvo | zyga: that might be it | 16:12 |
| zyga | all the failures there look crazy | 16:13 |
| zyga | I restarted it now | 16:13 |
| mvo | zyga: I suspect its the symlink for snappy-app-dev to snap-device-helper | 16:15 |
| zyga | mvo: symlink => copy | 16:15 |
| zyga | actually, symlink is fine | 16:15 |
| zyga | I think | 16:15 |
| mvo | zyga: I think we did not do that | 16:15 |
| mup | PR snapd#4891 closed: tests: add support for phased prepare-restore logic <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4891> | 16:16 |
| mup | PR snapd#4914 closed: advisor: add comment why osutil.FileExists(dirs.SnapCommandsDB) is needed <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4914> | 16:16 |
| mup | PR snapd#4915 closed: interfaces,release: probe seccomp features lazily <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4915> | 16:16 |
| pedronis | zyga: did you merge #4891 with one review? | 16:19 |
| mup | PR #4891: tests: add support for phased prepare-restore logic <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4891> | 16:19 |
| zyga | mvo: I will tweak the ssh fix now | 16:19 |
| zyga | pedronis: yes, because it's tiny and I have useful fixes on top | 16:19 |
| zyga | pedronis: (and doesn't do anything yet) | 16:19 |
| pedronis | did you discuss this with niemeyer? | 16:20 |
| pedronis | I remember we had long code org discussion around this | 16:20 |
| zyga | I wasn't aware of that, what was the discussion about specifically? | 16:21 |
| pedronis | how stuff should be organized | 16:21 |
| pedronis | the usual contentious topic | 16:21 |
| zyga | was the outcome captured somewhere? | 16:22 |
| pedronis | no | 16:22 |
| zyga | well, I can still back that out, I just want to do something because we have real issues and no solution in sight | 16:23 |
| pedronis | I'm just saying that merging somethign that changes deeply how we structure spread bits | 16:23 |
| zyga | well, it's not doing that yet | 16:23 |
| pedronis | with one review and without niemeyer input | 16:23 |
| pedronis | is perilious | 16:23 |
| pedronis | zyga: also is the usual issue with do nothing code, it's hard to tell how it will be used in practice | 16:24 |
| niemeyer | zyga: Yeah, please revert this.. | 16:24 |
| zyga | sure, in a moment, just working on ssh now | 16:25 |
| niemeyer | zyga: This sounds like a recipe for shell sausage | 16:25 |
| zyga | I explained how it will be used in the example file but I'm fine with reverting it | 16:25 |
| niemeyer | zyga: Yes, it's an example of sausage | 16:26 |
| zyga | I don't know what a sausage is | 16:26 |
| niemeyer | zyga: It's also not the right way of doing these changes.. please hear pedronis | 16:26 |
| niemeyer | zyga: I'll send you a picture later | 16:26 |
| zyga | our code is a mess there, I want someone to own it and fix it; I may do it in a way some people don't agree with but my motivation was to fix this | 16:27 |
| niemeyer | zyga: We have a forum here: https://forum.snapcraft.io | 16:27 |
| zyga | I don't mind if someone fixes our test suite to run better, really | 16:28 |
| niemeyer | zyga: Merging these changes, which sound controversial and unclear to begin with, with no actual discussion despite the fact we spent hours talking today, on a Friday, before a release, when people are leaving on holiday... man... | 16:28 |
| niemeyer | With one review! | 16:28 |
| zyga | niemeyer: none of that is in the release, none of that is contoversial or unclear, did you see what that code did? IMO that is an over reaction now | 16:29 |
| niemeyer | zyga: Please step back and breath.. it's not the way we do things.. | 16:30 |
| niemeyer | zyga: If you want to reorganize that logic, please open a forum topic and let's talk | 16:30 |
| niemeyer | zyga: As perdronis insightfully pointed out, we are very careful to not create a mess of shell on our tests.. we simplified things out of seemingly innocent practices several times.. | 16:31 |
| pablo_ | Hi: I'm trying to use wifi-ap snap on ubuntu core in Raspberry Pi 3, but status keeps changing to active=false after a couple of seconds i activate it. Any hints? | 16:33 |
| cachio | zyga, any news on https://travis-ci.org/snapcore/snapd/builds/357331248#L8138 ? | 16:34 |
| zyga | cachio: ha, interesting | 16:34 |
| mvo | cachio: can you look at the denial for the permission denied failures please? | 16:35 |
| zyga | cachio: only that I still don't understand something there | 16:35 |
| mvo | cachio: can you pastebin those? | 16:35 |
| cachio | 1 sec | 16:36 |
| cachio | [Fri Mar 23 11:08:29 2018] audit: type=1400 audit(1521803309.374:626): apparmor="DENIED" operation="open" profile="snap-update-ns.test-snapd-layout" name="/proc/sys/kernel/seccomp/actions_avail" pid=16648 comm="3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 | 16:36 |
| cachio | [Fri Mar 23 11:08:29 2018] audit: type=1400 audit(1521803309.402:627): apparmor="DENIED" operation="mount" info="Failed name lookup - deleted entry" error=-2 profile="snap-update-ns.test-snapd-layout" name="/etc/group" pid=16648 comm="3" flags="rw, bind" | 16:36 |
| zyga | cachio: the 1st one is fixed in another pr | 16:36 |
| cachio | today at least 3 builds failed on master because of this :( | 16:37 |
| zyga | cachio: the 2nd one says something removed /etc/group while we were looking and I don't understand it | 16:37 |
| zyga | jdstrand: ^ do you know what could cause the denial above? | 16:37 |
| zyga | (the second one) | 16:38 |
| mup | PR snapd#4919 opened: Revert "tests: add support for phased prepare-restore logic" <Created by niemeyer> <https://github.com/snapcore/snapd/pull/4919> | 16:39 |
| zyga | niemeyer: I pushed the ssh changes | 16:39 |
| niemeyer | cachio: After you're done with the current topic, what spread images are ready to merge? | 16:39 |
| mup | PR snapd#4919 closed: Revert "tests: add support for phased prepare-restore logic" <Created by niemeyer> <Merged by niemeyer> <https://github.com/snapcore/snapd/pull/4919> | 16:40 |
| zyga | is that what you had on your mind earlier? | 16:40 |
| zyga | https://github.com/snapcore/snapd/pull/4912/files | 16:40 |
| mup | PR #4912: overlord/configstate: change how ssh is stopped/started (2.32) <Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/4912> | 16:40 |
| niemeyer | cachio: Sorry.. I mean, what snapd PRs moving images to Google | 16:40 |
| niemeyer | zyga: Looking | 16:40 |
| cachio | niemeyer, opensuse is almost ready, I think today will be ready to be merged | 16:40 |
| cachio | then, just fedora is missing, we need to see how to fix those deniails from selinux | 16:41 |
| pedronis | zyga: I'm aware that we have issues with tests, it's a bit unclear that they are organisation problem though, the feeling in that prepare/restore do a lot, unclear if doing a lot differently is what we need, or just try to be more careful/streamline | 16:42 |
| zyga | yes, I know | 16:42 |
| niemeyer | zyga: One comment and LGTM | 16:42 |
| zyga | anyway, we can discuss this later when there's more time to see what I wanted to do | 16:42 |
| cachio | niemeyer, well then just missing debian sid which is currently set as manual | 16:42 |
| zyga | I have a refactor of the original code, that does what it did before but is more readable, that I wanted to use a base for refactoring | 16:43 |
| zyga | to restore the property that setup sets stuff up from pristine and restore undoes that to pristine | 16:43 |
| zyga | with hard checks that ensure this is happening for real | 16:43 |
| niemeyer | pedronis: Right, exactly.. the moment we allow ourselves to just throw files at a directory, which in fact *need* to be ordered because they depend on each other, is the day we lose complete control over the sanity of something that is already not as clean as it should be | 16:43 |
| cachio | niemeyer, are we gonna try arch or centos ? | 16:43 |
| zyga | niemeyer: they don't depend on each other much; I'd love to discuss this but I think you are jumping to conclusions now | 16:44 |
| niemeyer | cachio: Yeah, mborzecki had arch working already.. I closed the PR because it was still on Linode, but you two should be able to quickly get it up again on Google once you have everything else sorted | 16:44 |
| zyga | updating ssh again, just a sec | 16:44 |
| cachio | niemeyer, ok | 16:44 |
| niemeyer | zyga: I'm not jumping into conclusions.. I'm concluding from experience from many years looking at similar systems evolve | 16:44 |
| niemeyer | zyga: What's the rationale for the change in the first place? | 16:45 |
| cachio | niemeyer, then we need to discuss the changes needed to make fedora work on google with selinux | 16:45 |
| cachio | niemeyer, that's missing to move fedora | 16:45 |
| mborzecki | cachio: ping me when you'd want to give arch a try, IIRC there was just a couple of tests failing there (one of those was merged usr which got fixed in master not long ago) | 16:45 |
| niemeyer | cachio: Ok.. but gback to the original question: what can I merge? | 16:46 |
| zyga | niemeyer: pushed ssh, | 16:46 |
| zyga | the rationale... | 16:46 |
| zyga | " This patch will help to modularize the prepare/restore logic and hopefully make it more testable and easy to reason about." | 16:46 |
| cachio | niemeyer, opensuse will be ready but later doday | 16:47 |
| zyga | also that related prepare/restore code is in one file | 16:47 |
| niemeyer | cachio: So nothing yet? | 16:47 |
| cachio | niemeyer, #4886 | 16:47 |
| mup | PR #4886: tests: adding opensuse-42.3 to google <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/4886> | 16:47 |
| zyga | and not far away from each other | 16:47 |
| niemeyer | zyga: Yep.. so exactly what I said.. | 16:47 |
| cachio | yesterday we merged debian | 16:47 |
| zyga | niemeyer: I disagree but I dont' want to argue abut it this way now | 16:47 |
| niemeyer | zyga: Me neither.. that was part of the point | 16:48 |
| cachio | niemeyer, I pushed a fix for opensuse, tests are waiting to be executed | 16:48 |
| zyga | niemeyer: is https://github.com/snapcore/snapd/pull/4912/commits/1213460ccc6b96baf28f478bde752d25b8212dbb ok? I think that's it for this fx | 16:48 |
| mup | PR #4912: overlord/configstate: change how ssh is stopped/started (2.32) <Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/4912> | 16:48 |
| niemeyer | zyga: Yeah, that was all, thank you | 16:48 |
| zyga | excellent, now what's left for the system key work | 16:48 |
| zyga | mvo: how can I help? | 16:48 |
| niemeyer | mvo, zyga: I had a quick question/comment.. can we hang out for a few (hopefully short) minutes? | 16:49 |
| zyga | yeah | 16:49 |
| mvo | zyga, niemeyer sure | 16:50 |
| niemeyer | I'm there | 16:50 |
| pablo_ | ogra_: hi, i followed your wifi-ap tutorial, but i cannot make it work... could you help me pls? | 16:58 |
| ogra_ | pablo_, on what board is that ? | 16:58 |
| ogra_ | (did you check jrounalctl for errors ?) | 16:58 |
| ogra_ | *journalctl | 16:59 |
| pablo_ | ogra_: raspberrypi 3. I'll check right now | 16:59 |
| ogra_ | any other network related snaps installed ? | 16:59 |
| pablo_ | ogra_: nmcli | 17:01 |
| ogra_ | ah, i think that will take over the wlan device ... remove NM | 17:01 |
| ogra_ | (if youo need to configure ethernet, use the config in /etc/netplan/ | 17:02 |
| ogra_ | = | 17:02 |
| ogra_ | ) | 17:02 |
| pablo_ | ogra_:journalctl shows lots of messages i don't understand.. i'll pastebin them. | 17:02 |
| pablo_ | ogra_: i'm using raspberry through ssh | 17:02 |
| ogra_ | well, likely that wifi-ap cn not switch the interface to ap mode | 17:02 |
| pablo_ | if i remove nmcli would i be disconnecteD? | 17:02 |
| ogra_ | well, if you did the initial setup with console-conf (to get the user set up etc) you should have a working config for eth0 already ... but you can also double check in /etc/netplan/ | 17:03 |
| pablo_ | ogra_: i'll check... if eth0 is configured there i can safely remove NM? then i install wifi-ap? | 17:04 |
| ogra_ | yes, that should work | 17:04 |
| cachio | jdstrand, I see this denial https://paste.ubuntu.com/p/8ZQVRQjZvZ/ running the test security-device-cgroups | 17:06 |
| cachio | it is happening when execute test-snapd-tools.env | 17:06 |
| cachio | jdstrand, any idea about what could be causing that? | 17:07 |
| pablo_ | ogra_, i removed it, then executer wifi-ap.setup_wizard. Connection keeps geting ap.active=false | 17:07 |
| ogra_ | did you reboot ? | 17:08 |
| pablo_ | no :S sorry | 17:08 |
| ogra_ | the wlan driver might be in a weird state | 17:08 |
| pablo_ | ogra_: same thing... after a couple of seconds, ap.active changes to false | 17:10 |
| ogra_ | well, then lets check the log ... | 17:10 |
| pablo_ | ogra_: how do i check the logg? sorry i'm very noob with this | 17:12 |
| pablo_ | journalctl? https://pastebin.com/hcRmcvbq | 17:14 |
| ogra_ | Mar 23 17:10:07 ema systemd-networkd[577]: wlan0: DHCPv4 address 192.168.2.111/24 via 192.168.2.1 | 17:15 |
| ogra_ | thats clearly in client mode | 17:15 |
| ogra_ | do you have an entry for wlan0 in your netplan config ? | 17:16 |
| ogra_ | (in /etc/netplan/00-snapd-config.yaml ) | 17:16 |
| === pstolowski is now known as pstolowski|afk | ||
| pablo_ | ogra_, yes i have... it was created when i flashed ubuntu core.. | 17:18 |
| pablo_ | ogra_, how i change it? | 17:18 |
| ogra_ | well, if you use the wlan device in clinet mode it can not run as AP | 17:18 |
| ogra_ | *client | 17:18 |
| pablo_ | I cannot even use the wifi as client right now.. how can i change client mode? | 17:19 |
| ogra_ | you can re-run console-conf with "sudo console-conf" and make sure to only configure wired ... or you can edit /etc/netplan/00-snapd-config.yaml and remove the "wifis" block (but make sure you keep ethernet configured to still get into the system) | 17:20 |
| pablo_ | thanks! i'll try with console-conf | 17:20 |
| Chipaca | mvo: 4920 fixes the human thing | 17:21 |
| mup | PR snapd#4920 opened: timeutil: in Human, count days with fingers <Created by chipaca> <https://github.com/snapcore/snapd/pull/4920> | 17:21 |
| pablo_ | ogra_, in console-conf, should i erase everything in wlan0? is that enough? or how should i config it to work as ap | 17:22 |
| ogra_ | just pick "do not use" | 17:22 |
| pablo_ | ogra_, i can select "do not use" in ipv4 and ipv5 settings, but it says "associated to mywifi" | 17:24 |
| ogra_ | well, then change the wifi settings too | 17:24 |
| ogra_ | not sure how the "dont use" option in there is named | 17:24 |
| pablo_ | there isn't... i try to erase configuration. | 17:25 |
| mvo | pedronis: 4882 is ready for a second look I think | 17:29 |
| pablo_ | ogra_, excellent! now it works! thanks a lot.. | 17:29 |
| ogra_ | :) | 17:29 |
| ogra_ | enjoy | 17:29 |
| pedronis | mvo: ok, almost dinner here | 17:30 |
| pedronis | though | 17:30 |
| mvo | pedronis: yeah, same here | 17:30 |
| pablo_ | ogra_, is it possible to connecto trhough ssh if i connecto to the ap also? i mainly want it to do a simple webhost, but connecting trhough ssh could be helpful | 17:30 |
| ogra_ | sure, as long as you use the same key on the client side it doesnt matter through which network device you connect | 17:31 |
| pablo_ | thanks a lot! i'll disconnect a little time to connect to the boards ap | 17:32 |
| === sparkieg` is now known as sparkiegeek | ||
| mup | PR snapcraft#2023 opened: repo: catch error due to broken build packages <Created by kalikiana> <https://github.com/snapcore/snapcraft/pull/2023> | 17:43 |
| Chipaca | the BBC is wondering about Ubuntu's future: https://www.bbc.co.uk/cbbc/quizzes/beyond-bionic-mega-quiz-1 | 17:43 |
| * kalikiana wrapping up for the day | 17:43 | |
| niemeyer | kalikiana: Heya.. didn't forget our conversation.. still need to write the notes into the forum.. let me do that now | 17:50 |
| niemeyer | kalikiana: Enjoy the weekend | 17:50 |
| niemeyer | cachio: Haha :) | 17:51 |
| niemeyer | Oops.. that was Chipaca | 17:51 |
| kalikiana | niemeyer: Grand! Looking forward to reading it. | 17:53 |
| Chipaca | I think this is a mostly-EOD from me | 17:54 |
| Chipaca | I'm off to do housework, might pop back in when the neighbours object to the hoovering | 17:55 |
| mup | PR snapcraft#2022 closed: Polish our landing page ✨ <Created by evandandrea> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2022> | 17:58 |
| niemeyer | Chipaca: Sweet.. I'm might have a review done for you, but you shouldn't care anyway because it's your evening isn't it :) | 18:00 |
| Chipaca | niemeyer: sorry i can't hear you over all the hoovering | 18:00 |
| niemeyer | :) | 18:00 |
| === alan_g is now known as alan_g|EOW | ||
| mup | PR snapcraft#1999 closed: tests: document the SRU testing process <Created by elopio> <Closed by sergiusens> <https://github.com/snapcore/snapcraft/pull/1999> | 18:04 |
| cachio | mborzecki, hey I'll start preparing the image | 18:08 |
| mborzecki | cachio: ok | 18:08 |
| cachio | mborzecki, which version should I use? | 18:09 |
| cachio | mborzecki, I think we can use this https://github.com/GoogleCloudPlatform/compute-archlinux-image-builder | 18:10 |
| mborzecki | cachio: take the last one, 2018.03.01, it's a rolling release anyway, so we'll have to update the image (bi-)weekly | 18:10 |
| cachio | the ones tht are prebuilt are pretty old | 18:10 |
| mborzecki | cachio: you can start with those scripts, if they work then we'll have an image, if not we'll try another way :) | 18:11 |
| cachio | mborzecki, ok, I'll try those scripts and see | 18:14 |
| cachio | mborzecki, shnould we try first with some stable image? | 18:15 |
| cachio | or it is ok with the last one? | 18:15 |
| mborzecki | cachio: there's no stable image per se, it's just a snapshot of current repos, it's always updating | 18:16 |
| cachio | mborzecki, ah, ok | 18:16 |
| mborzecki | cachio: https://www.archlinux.org/download/ 2018.03.01 is a good starting point | 18:16 |
| cachio | let's try with this last one | 18:16 |
| cachio | mborzecki, nice thanks | 18:16 |
| mborzecki | cachio: let me check one more thing though, there was some discussion in the mailing liast about a cloud/vagrant image of arch | 18:16 |
| mborzecki | cachio: does google require cloud init? | 18:19 |
| cachio | mborzecki, I don't think so | 18:19 |
| cachio | it requires its own cloud dependencies | 18:19 |
| cachio | mborzecki, why? | 18:20 |
| mborzecki | cachio: do you know what these deps are? | 18:20 |
| cachio | mborzecki, some of these ones https://github.com/GoogleCloudPlatform/compute-image-packages | 18:22 |
| mborzecki | cachio: ok, let me know if you run into problems, mayble i'll need to package something | 18:23 |
| mup | PR snapd#4902 closed: cmd/snap-confine: nvidia: preserve globbed file prefix <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4902> | 18:26 |
| pedronis | mvo: back from dinner, I will look at the PR now | 18:28 |
| mvo | pedronis: great, thank you | 18:28 |
| zyga | Thank you pedronis | 18:32 |
| niemeyer | Here is a short teaser | 18:41 |
| niemeyer | root@spread-cluster:/proc# cat cpuinfo | grep processor | wc -l | 18:42 |
| niemeyer | 96 | 18:42 |
| niemeyer | root@spread-cluster:/proc# ls -ls kcore | 18:42 |
| niemeyer | 0 -r-------- 1 root root 141905719468032 Mar 23 18:42 kcore | 18:42 |
| mup | PR snapcraft#2024 opened: errors: improve the UX for sending error data <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2024> | 18:43 |
| pedronis | mvo: left a few comments, I'm wondering what the move from yaml to json means in terms of the behavior on 2.31->2.32 transition | 18:45 |
| cachio | niemeyer, awesome | 18:45 |
| cachio | 141905719468032 is bigggg | 18:45 |
| niemeyer | cachio: Yeah, and wrong.. sorry.. :) It's close to the real one, but kcore shows virtual memory.. the actual size is 128GB, which is close, but in GB not TB | 18:46 |
| cachio | niemeyer, so, spread should be in charge of creating vms inside this server? | 18:47 |
| cachio | niemeyer, is it possible to create vms with arm32 architecture too? | 18:48 |
| mvo | pedronis: yeah, thats a critical point indeed | 18:48 |
| niemeyer | cachio: Yeah.. I'll do a tiny daemon that basically allows a conversation like "give me a server" => "okay, server 142 has ssh at ...:8782" => "thanks, destroy 142" | 18:48 |
| mvo | pedronis: I think we need to handle this as a mismatch without | 18:48 |
| pedronis | mvo: bug we ingore mismatches | 18:48 |
| pedronis | s/bug/but/ | 18:49 |
| mvo | pedronis: I mean, I think we need to return that we have a mismatch and no errror here which sucks a bit | 18:49 |
| mvo | pedronis: or we could first try json and if that fails try nyaml it | 18:49 |
| pedronis | but the content is different, no? | 18:50 |
| mvo | pedronis: yes, the content is different so it would be a case of "there is a mismatch, try to talk to snapd before contiue" | 18:50 |
| cachio | niemeyer, opensuse tests passed 100% but I had to retrigger because the layout test failed for ubuntu-sore | 18:50 |
| niemeyer | cachio: Due a known issue or hiccup? | 18:51 |
| mvo | pedronis: maybe that is actually the answer, on error we try to reach snapd instead of silently ignoring the issue? | 18:51 |
| cachio | niemeyer, there are 2 issues related, for one there is a PR | 18:51 |
| cachio | niemeyer, the second one is under research | 18:51 |
| niemeyer | cachio: Ack | 18:51 |
| cachio | niemeyer, zyga has more info for sure | 18:52 |
| mup | PR snapd#4921 opened: skip test if no user "daemon" in build jail <Created by rkitover> <https://github.com/snapcore/snapd/pull/4921> | 18:52 |
| pedronis | mvo: the issue is that there are various scenarios | 18:52 |
| pedronis | the race described in the comment is only one of them | 18:52 |
| pedronis | there's also switchin on or off REEXEC | 18:52 |
| pedronis | potentially | 18:52 |
| pedronis | and this yaml to json transition | 18:53 |
| zyga | mvo: I sent my review | 18:54 |
| mvo | pedronis: indeed, at least for the race described in the comment and for the yaml -> json handling this by waiting for snapd should be ok. i.e. on error try to reach snapd and then continue | 18:54 |
| mvo | zyga: cool, thanks | 18:54 |
| zyga | sorry for taking so long, I took a break from this | 18:54 |
| mvo | zyga: no problem, thanks for the review. will address things in a little bit | 18:55 |
| mvo | pedronis: I'm not sure about the switching on/off of re-exec though | 18:56 |
| pedronis | mvo: there is nothing we can do in general there | 18:56 |
| * zyga EOWs | 18:56 | |
| pedronis | except for testing/development it shouldn't really be done | 18:57 |
| pedronis | mvo: anyway hopfully people don't have things trying to run while they switch | 18:58 |
| zyga | we just have to keep the packages up to date ;-) | 19:00 |
| niemeyer | mvo: Just reviewed 4882 as well.. very nice, and looks super close | 19:07 |
| mup | PR snapd#4918 closed: cmd/snap: fix one issue with noWait error handling logic, add tests plus other cleanups <Created by pedronis> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/4918> | 19:20 |
| mvo | niemeyer: yay, thank you all. fixing the comments now | 19:31 |
| cachio | mborzecki, the images that that project is creating are about 100GB | 19:31 |
| cachio | 107 GB | 19:32 |
| cachio | I am aborting this, creating arch linux image from scratch | 19:32 |
| pedronis | mvo: niemeyer: the comment about connection to old snapd means we really need #4911 too ? | 19:42 |
| mup | PR #4911: daemon,client: add build-id to /v2/system-info <Created by mvo5> <https://github.com/snapcore/snapd/pull/4911> | 19:42 |
| cachio | niemeyer, #4886 in green | 19:48 |
| mup | PR #4886: tests: adding opensuse-42.3 to google <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/4886> | 19:48 |
| niemeyer | cachio: Is it stable, or are there issues to sort out still? | 19:48 |
| cachio | niemeyer, opensuse is stable | 19:48 |
| cachio | I ran more than 10 times and I didn't see any error | 19:49 |
| cachio | last 3 executions in travis 0 errors for opensuse | 19:49 |
| niemeyer | cachio: LGTM then! | 19:51 |
| * niemeyer needs to step out.. back later | 19:53 | |
| Syntist | As Snap is container based solution, does it effect the performance? | 19:59 |
| mvo | niemeyer, pedronis updated the PR | 20:04 |
| mup | PR snapd#4912 closed: overlord/configstate: change how ssh is stopped/started (2.32) <Critical> <Squash-merge> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4912> | 20:14 |
| mvo | zyga: quick question - 4912 needs a port to master, right? | 20:14 |
| mup | PR snapd#4922 opened: overlord/configstate: change how ssh is stopped/started (2.32) (#4912) <Created by mvo5> <https://github.com/snapcore/snapd/pull/4922> | 20:16 |
| zyga | Re | 20:27 |
| mup | PR snapd#4886 closed: tests: adding opensuse-42.3 to google <Created by sergiocazzolato> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/4886> | 20:31 |
| * zyga checks 4912 | 20:43 | |
| zyga | mvo: yes b | 20:43 |
| mvo | zyga: I added it | 20:43 |
| zyga | mvo: I assumed it would not be an issue if this lands with the beta release you plan on making today | 20:43 |
| mvo | zyga: all except 4882 are now in | 20:43 |
| mvo | zyga: its fine | 20:44 |
| mvo | zyga: I was just asking to not duplicate work :) | 20:44 |
| zyga | sorry I wasn't around, I didn't hear the pings | 20:44 |
| zyga | I was on headphones watching a movie | 20:44 |
| mvo | zyga: toally fine | 20:45 |
| mvo | zyga: no worries | 20:45 |
| * mvo hugs pedronis for his review | 20:46 | |
| * zyga makes tea and reads the reviews | 20:46 | |
| mvo | zyga: just 4882 | 20:49 |
| mvo | zyga: :) | 20:49 |
| zyga | mvo: I added one nitpick and +1 | 20:49 |
| zyga | it's not important though | 20:49 |
| zyga | not worth waiting another round of tests if they go green | 20:49 |
| * mvo hugs zyga | 20:52 | |
| mvo | zyga: the fearful kernel | 20:52 |
| zyga | haha, did I write fearful? | 20:52 |
| mvo | zyga: no but that is what I read at first | 20:52 |
| zyga | Yeah they sound kind of similar | 20:54 |
| zyga | So after this branch lands then what? | 21:01 |
| zyga | Do you need a PPA build | 21:01 |
| mvo | zyga: yeah, when it lands-> ppa build | 21:05 |
| zyga | oooooh | 21:24 |
| zyga | I reproduced the /etc issue!!!!!! | 21:24 |
| zyga | for the first time, I have debug shell | 21:24 |
| zyga | this is like candy | 21:24 |
| zyga | but diet | 21:24 |
| mvo | zyga: yay | 21:27 |
| * mvo hugs zyga | 21:27 | |
| pedronis | mvo: do we need #4877 ? | 21:28 |
| mup | PR #4877: snap-confine: fallback to /lib/udev/snappy-app-dev if the core is older <Created by mvo5> <https://github.com/snapcore/snapd/pull/4877> | 21:28 |
| zyga | and I have a strace | 21:31 |
| mvo | pedronis: thats the master version of a pr that landed in 2.32 already | 21:31 |
| zyga | [pid 2951] mount("/tmp/.snap/etc/group", "/etc/group", 0xc82010fb6b, MS_BIND, NULL) = -1 ENOENT (No such file or directory) | 21:31 |
| zyga | this is what fails | 21:31 |
| pedronis | mvo: ah, ok | 21:31 |
| mvo | pedronis: its "funny" becasue the tests for this one in master fail all the time with snap-confine.dsajlkjfslda (tmpname) leftovers | 21:31 |
| mvo | pedronis: and its not clear what is writing these leftover files | 21:31 |
| pedronis | mvo: I have a PR that gets layout problems most of the time, interestingly follow up PRs are all green | 21:32 |
| * zyga has a feeling this is a kernel bug | 21:35 | |
| zyga | I'll paste the strace | 21:35 |
| zyga | https://pastebin.ubuntu.com/p/qKyPqTnT3N/ | 21:37 |
| mvo | pedronis: hm,hm,I saw the layout ones too but not on recent test runs | 21:37 |
| * cachio afk | 21:38 | |
| zyga | super interesting facts: | 21:39 |
| zyga | 1) /dev/sda3 on /etc/group type ext4 (rw,relatime,data=ordered) | 21:39 |
| zyga | : /etc/group is a bind-mount | 21:39 |
| zyga | hmmm | 21:39 |
| pedronis | is that the result of threspassing ? | 21:40 |
| zyga | no | 21:40 |
| zyga | this is just /etc/ on core | 21:41 |
| zyga | I don't get this thing: [pid 2951] lstat("/etc/group", 0xc8201097c8) = -1 ENOENT (No such file or directory) | 21:41 |
| zyga | [pid 2951] mount("/tmp/.snap/etc/group", "/etc/group", 0xc82010fb6b, MS_BIND, NULL) = -1 ENOENT (No such file or directory) | 21:41 |
| zyga | this last error is the error we are seeing | 21:42 |
| zyga | we carefully create /tmp/.snap/etc/group | 21:42 |
| zyga | and it exits, we can see that on line 1478 in the pastebin | 21:42 |
| zyga | but when we try to bind mount /tmp/.snap/etc/group over to /etc/group, something fails and we get ENOENT | 21:43 |
| zyga | that pastebin is really weird, | 21:43 |
| pedronis | we are bind mounting over a bind-mount? | 21:43 |
| zyga | I will need to study it | 21:43 |
| zyga | no, technically we are mount --rbind /etc /tmp/.snap/etc | 21:44 |
| zyga | mounting tmpfs over /etc | 21:44 |
| zyga | and bind-mounting things back. one by one | 21:44 |
| zyga | at this stage /etc/group should be an empty file | 21:44 |
| zyga | but I think the code gets confused | 21:44 |
| pedronis | but you said it's a bind-mount? | 21:44 |
| zyga | because we check if it's a file or directory | 21:44 |
| zyga | on the host | 21:44 |
| zyga | on core it seems to be | 21:45 |
| zyga | let me look on my dragon | 21:45 |
| zyga | hmm | 21:45 |
| zyga | it's not | 21:45 |
| zyga | it's squashfs there | 21:46 |
| * zyga wonders what's the google core-16 image then | 21:46 | |
| pedronis | we build the image | 21:46 |
| pedronis | in the tests | 21:46 |
| zyga | mountinfo on google ubuntu-core-16 https://www.irccloud.com/pastebin/h9nNyYdf/ | 21:46 |
| pedronis | mmh | 21:47 |
| zyga | ahhh | 21:47 |
| zyga | I see what's wrong now | 21:47 |
| zyga | 117 25 8:3 /system-data/var/lib/extrausers/group//deleted /etc/group rw,relatime shared:7 - ext4 /dev/sda3 rw,data=ordered | 21:47 |
| zyga | ! | 21:47 |
| zyga | after booting | 21:47 |
| zyga | something wiped /writable | 21:47 |
| zyga | and all hell broke loose | 21:47 |
| zyga | it's a very peculiar thing in mounts | 21:48 |
| zyga | I need to add code to detect that | 21:48 |
| zyga | and bail early | 21:48 |
| zyga | maybe then we'll know more | 21:48 |
| zyga | so....... | 21:48 |
| zyga | I would say that some of our tests killed /writable | 21:48 |
| zyga | this broke /etc in the real system | 21:48 |
| zyga | and then my code doesn't cope with a file being there but not really being there because it's a //deleted mount | 21:48 |
| zyga | I need to check the kernel what the semantics is then | 21:49 |
| zyga | but I bet a beer that this is broken prepare/restore | 21:49 |
| pedronis | zyga: notice that we do special things to the image to get the spread user in it | 21:49 |
| pedronis | let me find a pointer | 21:49 |
| pedronis | zyga: we add special units that bind mount /var/lib/extrausers/x to /etc/x | 21:50 |
| zyga | I don't understand why /etc/group is the extrausers/group in this image but not in my dragon | 21:50 |
| zyga | ooooh | 21:50 |
| zyga | where is this? | 21:50 |
| zyga | man, | 21:50 |
| zyga | I didn't know that | 21:50 |
| pedronis | one sec | 21:50 |
| zyga | sure, thank you! | 21:50 |
| zyga | I bet it's suite-level restore | 21:51 |
| zyga | as I only ran this test (layout) with -repeat | 21:51 |
| pedronis | zyga: no, it's done with the image | 21:51 |
| pedronis | is not a restore thing | 21:51 |
| pedronis | zyga: https://github.com/snapcore/snapd/blob/master/tests/lib/prepare.sh#L388 | 21:51 |
| zyga | but it doesn't always fail | 21:52 |
| pedronis | that is a different question | 21:52 |
| zyga | it must be the ordering or tests so that layout runs after something restored suite in the same system | 21:52 |
| pedronis | but this operation is done with the image once | 21:52 |
| zyga | when I ran layout (and just layout) with repeat | 21:52 |
| pedronis | but the mounts will happen at boot each time | 21:52 |
| zyga | yes, but it doesn't hurt normally | 21:52 |
| zyga | what is wrong is that something removed them | 21:52 |
| zyga | so reset_all_snap has this at the end | 21:54 |
| zyga | https://www.irccloud.com/pastebin/SPpOyIr1/ | 21:54 |
| zyga | -rw-r--r-- 1 root root 453M Mar 23 21:22 /home/gopath/src/github.com/snapcore/snapd/snapd-state.tar.gz | 21:56 |
| zyga | this doesn't feel right | 21:56 |
| zyga | 450MB? | 21:56 |
| zyga | ha | 21:56 |
| zyga | it's not tar.gz | 21:56 |
| zyga | it's tar | 21:56 |
| zyga | what? | 21:56 |
| zyga | some this is clearly wrong here | 21:56 |
| pedronis | ? | 21:57 |
| pedronis | it's probably mostly /var/lib/snapd/snaps that make up that size | 22:03 |
| pedronis | zyga: I see a few tests that manipulate things in /var/lib/extrausers but nothing removes things totally | 22:14 |
| pedronis | ah but | 22:15 |
| zyga | re | 22:22 |
| zyga | but? | 22:23 |
| zyga | https://github.com/snapcore/snapd/pull/4923 may help us find the culprit | 22:23 |
| mup | PR #4923: spread.yaml: look for signs of //deleted mounts <Created by zyga> <https://github.com/snapcore/snapd/pull/4923> | 22:23 |
| mup | PR snapd#4923 opened: spread.yaml: look for signs of //deleted mounts <Created by zyga> <https://github.com/snapcore/snapd/pull/4923> | 22:23 |
| zyga | I need to check what the kernel semantics is | 22:23 |
| zyga | but I think we are much closer to understanding the issue | 22:24 |
| zyga | where //deleted comes from https://www.irccloud.com/pastebin/jtTaEdO3/ | 22:26 |
| * pedronis -> rest | 22:28 | |
| zyga | pedronis: let's catch up with this next week, have a great weekend | 22:28 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!