| === braderhart is now known as Guest13679 | ||
| === chihchun_afk is now known as chihchun | ||
| === chihchun is now known as chihchun_afk | ||
| === chihchun_afk is now known as chihchun | ||
| mborzecki | morning | 05:07 |
|---|---|---|
| === chihchun is now known as chihchun_afk | ||
| mborzecki | interesting failure in unit tests: https://paste.ubuntu.com/p/PB3FhvyfT7/ | 06:07 |
| === chihchun_afk is now known as chihchun | ||
| === chihchun is now known as chihchun_afk | ||
| === chihchun_afk is now known as chihchun | ||
| mborzecki | mvo: hey | 07:00 |
| mvo | mborzecki: hey, good morning | 07:02 |
| mborzecki | mvo: have you seen something like this? https://paste.ubuntu.com/p/PB3FhvyfT7/ | 07:03 |
| mvo | mborzecki: yes, that rings a bell - not sure though what is causing this, maybe some incorrect mocking, i.e. your device is already registered | 07:05 |
| mborzecki | mvo: happened once on travis, couldn't reproduce it locally, feels like some race between mock server being started and the actual code accessing it | 07:07 |
| mvo | mborzecki: yeah, that sounds likely | 07:07 |
| === chihchun is now known as chihchun_afk | ||
| === chihchun_afk is now known as chihchun | ||
| === chihchun is now known as chihchun_afk | ||
| mborzecki | tests are super unhappy again | 07:52 |
| mborzecki | new failure in unit gccgo unit tests https://paste.ubuntu.com/p/NfYDvfwGdg/ | 07:53 |
| mvo | mborzecki: woah, strange. what distro/system is that? | 07:54 |
| mborzecki | Chipaca may be right that we're messing something up with fds in the tests | 07:54 |
| mborzecki | mvo: 2018-07-25 06:25:13 Error executing google:ubuntu-16.04-64:tests/unit/gccgo : | 07:54 |
| mborzecki | mvo: https://api.travis-ci.org/v3/job/407668444/log.txt full log | 07:55 |
| mborzecki | mvo: the auto import unit test failed there too | 07:55 |
| pedronis | we didn't see this errors a lot before | 07:57 |
| pedronis | wondering what changed | 07:57 |
| zyga | good good morning | 07:57 |
| === chihchun_afk is now known as chihchun | ||
| zyga | mvo: hey, welcome back | 08:01 |
| zyga | I'm sorry for being late today, we were stargazing last night and ... well :) | 08:01 |
| zyga | we saw Mars, red and bright like a cherry on the sky :) | 08:01 |
| zyga | we also saw the ISS, I think, I never saw it before | 08:02 |
| cjwatson | it's very distinctive, quite a sight | 08:02 |
| zyga | it was also the last call as it's a rare sight over here and it's expected to be cloudy for the next week | 08:04 |
| mvo | zyga: hey hey | 08:08 |
| mborzecki | mvo: will you be publishing busybox-static for more arches? | 08:12 |
| mvo | mborzecki: yes, working on it as we speak | 08:12 |
| mborzecki | mvo: got it, great :) | 08:12 |
| mvo | mborzecki: its a bit of fiddling/clicking around in LP, but should be ready soon | 08:13 |
| cjwatson | embrace the API :) | 08:13 |
| mvo | cjwatson: *cough* I really should use it more | 08:14 |
| * mvo is sometimes a bit set in his ways | 08:14 | |
| mborzecki | mvo: btw. i've seen apt-hooks test fail a couple of times on travis | 08:26 |
| mvo | mborzecki: yeah, this is what 5548 should fix | 08:27 |
| mvo | mborzecki: or rather, what do you mean with "fails" | 08:27 |
| mvo | mborzecki: do you have an example? | 08:27 |
| mvo | hrm, except that 5548 failed with the "MATCH" failure | 08:28 |
| mvo | :( | 08:28 |
| mborzecki | mvo: the output did not contain information about aws-cli snap | 08:29 |
| mbeneto | Hey guys, sorry to interrupt, just a stupid question. Is the $SNAPCRAFT_STAGE variable empty when building a "classic" snap? | 08:29 |
| mborzecki | mvo: unfortunately i didn't notice anyting specific in the debug log | 08:29 |
| mvo | mborzecki: ohhh | 08:30 |
| mvo | mborzecki: thats interessting, it might be a race with the update-catalog code | 08:30 |
| mvo | mborzecki: also grrrr @snapcraft, it insistend on having a linker in the base, I'm looking for a workaround now | 08:32 |
| mbeneto | I have a CMakeLists containing a $SNAPCRAFT_STAGE variable that works well when in a devmode (it takes the value the proper path) but remains empty when building it in classic | 08:36 |
| mborzecki | wondering if those connection refused errors may be caused by http server taking a longer while to start accepting on the socket, looking in stdlib there is no synchronization between the http.Server.Start() and actual http.Server.Serve(), so just calling http.NewServer() there's a chance that the if you do a request the server won't be waiting in Accept() yet | 08:49 |
| mborzecki | heh https://paste.ubuntu.com/p/Pw77jthXgQ/ | 09:06 |
| mvo | mborzecki: test-snapd-busybox-static should be ready on all arches now, might be easiest to just force push the base removal PR without the last commit | 09:18 |
| mborzecki | mvo: pushed and restarted the build | 09:20 |
| mvo | mborzecki: ta | 09:21 |
| zyga | mvo: hey | 09:30 |
| zyga | did you have a chance to look at the kernel->fixrtc->apparmor issue? | 09:30 |
| zyga | specifically I think we must do something about it before the release | 09:31 |
| zyga | we can either undo the snapctl change so that existing profiles remain compatible | 09:31 |
| zyga | we could change apparmor caching but that's a bigger (beyond snapd) work | 09:31 |
| zyga | I think we could also fix fixrtc (heh) | 09:32 |
| zyga | since it involves several subsytems it's going to be a coordination issue more than bugfixing issue | 09:32 |
| === chihchun is now known as chihchun_afk | ||
| mvo | zyga: sure, let me check | 09:34 |
| zyga | mvo: there's a trivial tweak to fixrtc to use the more appropriate timestamp | 09:34 |
| zyga | mvo: and if we could somehow fix the shutdown process to write the last-write-time timestamp on shutdown/reboot we would not need any changes to apparmor | 09:35 |
| zyga | mvo: I can send you debdiffs | 09:36 |
| zyga | mvo: then those need review from ubuntu developers and need to be SRUd | 09:36 |
| zyga | mvo: the shutdown fix is more subtle, no idea how to fix that yet | 09:36 |
| zyga | mvo: we could (terrible terrible idea) hack something so that we write the last-write-time ourselves on core | 09:37 |
| mvo | zyga: we have our own shutdown C program | 09:37 |
| zyga | mvo: or perhaps instead of using last-write-time of the rootfs, instead look for mtime of a canary file in /writable | 09:37 |
| mvo | zyga: maybe that helps? | 09:37 |
| zyga | mvo: that would be an non-invasive way to fix it | 09:37 |
| zyga | mvo: yes, we could use it to touch /writable/.timestamp | 09:37 |
| zyga | and look at that from fixrtc | 09:38 |
| zyga | mvo: longer term we should ask the kernel team for opinion/fix (maybe just foundations) | 09:38 |
| mvo | zyga: lets fix fixrtc first, that seems a pretty obvoious one | 09:38 |
| zyga | mvo: do you have a pi at home? I didn't bring any here | 09:38 |
| zyga | mvo: note that fixing fixrtc alone won't solve the issue | 09:38 |
| mvo | zyga: I have a pi | 09:38 |
| zyga | but sure, let me start with that | 09:38 |
| zyga | excellent | 09:39 |
| mvo | zyga: let me look at our custom shutdown, maybe we can hook into that | 09:39 |
| zyga | mvo: can you boot your pi and just do a sanity check please | 09:39 |
| zyga | mvo: use "dumpe2fs -h" on the root filesystem | 09:40 |
| zyga | mvo: and pastebin the result | 09:40 |
| zyga | mvo: I'm also curious how we are special, my fedora system (using lvm and ext4) is not affected by that | 09:40 |
| zyga | mvo: so something works correct in this case (timestamps updated) but not on core | 09:40 |
| mborzecki | Chipaca: hi, did you get to anything debugging the connection refused issue? | 09:40 |
| Chipaca | mborzecki: I was able to reduce its incidence a lot, but not eliminate it, meaning that the changes I did were not finding the root cause, just making it less probable | 09:41 |
| mborzecki | Chipaca: tried this locally https://paste.ubuntu.com/p/Pw77jthXgQ/ | 09:41 |
| mvo | zyga: the remount -o ro - is that done by systemd? | 09:41 |
| mborzecki | Chipaca: but at times i got no panic and the http requests still failed with connection refused | 09:42 |
| mborzecki | Chipaca: stracing it now | 09:42 |
| zyga | mvo: I _think_ so, not sure if intrinsic part of systemd or just some service we have | 09:42 |
| Chipaca | mborzecki: Hmm. I'll add this to the changes I've done, let's see what gives | 09:42 |
| zyga | mvo: I just recall seeing that message | 09:42 |
| * zyga reboots | 09:42 | |
| mvo | Chipaca: good morning! do you think we can merge 5531 (it got plenty of +1) or shall we ask for a gustavo review first? | 09:42 |
| zyga | so my fedora system is _not_ affected | 09:46 |
| zyga | trying with ubuntu now | 09:46 |
| Chipaca | mborzecki: cmd/snap/main_test.go:132: url.Hostname undefined (type *url.URL has no field or method Hostname) | 09:46 |
| Chipaca | mborzecki: i think i might be losing it | 09:46 |
| mborzecki | Chipaca: i'm using go 1.10.1 | 09:46 |
| Chipaca | mvo: I don't have a strong opinion either way; I think I described the pr in the standup on monday and he didn't flag it, fwiw | 09:47 |
| Chipaca | mborzecki: hah | 09:48 |
| mborzecki | Chipaca: url.Host instead? | 09:48 |
| Chipaca | mborzecki: so url.Host is host+port | 09:48 |
| Chipaca | yeah | 09:48 |
| mvo | Chipaca: ok, I leave it to you, we can either double check in the standup or just go ahead (easy enough to revert if needed) | 09:48 |
| Chipaca | mborzecki: ... value *os.PathError = &os.PathError{Op:"close", Path:"/tmp/check-4087160113880545930/34/var/lib/snapd/system-key.WWtDQKfmnsvk~", Err:0x9} ("close /tmp/check-4087160113880545930/34/var/lib/snapd/system-key.WWtDQKfmnsvk~: bad file descriptor") | 09:50 |
| mborzecki | Chipaca: right, seen this one too | 09:51 |
| Chipaca | mborzecki: WAT!!wat | 09:51 |
| Chipaca | that's wat^(wat^....^wat) wat times | 09:51 |
| zyga | Chipaca: close EBADFD - it got closed elsewhere, you have descriptor usage bugs? | 09:52 |
| mborzecki | zyga: it's in master :P | 09:52 |
| zyga | is golang protecting against close(-1) ? | 09:53 |
| Chipaca | zyga: this is runnig unit tests for cmd/snap on master | 09:53 |
| zyga | Chipaca: who would ever do that | 09:53 |
| Chipaca | zyga: I've got a PR that fails with one particular race every time, but the race is on master already | 09:54 |
| Chipaca | I think I described the issue yesterdya already | 09:54 |
| Chipaca | eh | 09:54 |
| Chipaca | new one: ... value *os.PathError = &os.PathError{Op:"read", Path:"/tmp/check-5611194952709039976/48/mountinfo", Err:0x9} ("read /tmp/check-5611194952709039976/48/mountinfo: bad file descriptor") | 09:54 |
| Chipaca | but, importantly, different | 09:54 |
| Chipaca | but, strangely, stll in auto_import | 09:54 |
| mborzecki | Chipaca: i see int quite often here | 09:54 |
| Chipaca | (where other bugs have been seen) | 09:54 |
| Chipaca | mborzecki: you see int? | 09:55 |
| mborzecki | Chipaca: yeah, int is all i see | 09:55 |
| mborzecki | so much int :) | 09:55 |
| Chipaca | isn't that an '80s ballad | 09:55 |
| zyga | jdstrand: hey, 4.18 kernel seems to be misbehaving wrt apparmor again: https://bugs.launchpad.net/snappy/+bug/1709155 | 09:56 |
| zyga | jdstrand: I recall you said something about that a few weeks ago | 09:56 |
| zyga | any bells? | 09:56 |
| mup | Bug #1709155: Better error message for unsupported kernel <Snappy:Triaged> <Ubuntu:Confirmed> <https://launchpad.net/bugs/1709155> | 09:56 |
| Chipaca | mborzecki: have you seen panics, with this change? | 09:58 |
| pedronis | Chipaca: mmh, I think we EBADF errors somewhere else at some point, trying to remember where | 10:00 |
| mborzecki | Chipaca: twice | 10:00 |
| mborzecki | Chipaca: strace when aliases failes with connection refused | 10:01 |
| mborzecki | Chipaca: https://paste.ubuntu.com/p/NcVssQP7sY/ | 10:01 |
| * zyga cries a little wrt shell in fixrtc | 10:02 | |
| zyga | pedronis: I recall (a long time ago) when we didn't realize some part of golang did dup(2) on a FD and we were leaking fds in the server | 10:03 |
| Chipaca | mborzecki: what're the openat … /json things? | 10:05 |
| mvo | zyga: my pi is unhappy right now, looking into it, but can't ssh into it right now | 10:06 |
| Chipaca | seems to be the AuthFile | 10:06 |
| zyga | that's okay | 10:06 |
| zyga | mvo: do you have a serial port adapter? | 10:06 |
| mborzecki | Chipaca: maybe because of successive connect attempts | 10:08 |
| Wimpress | Morning. | 10:13 |
| mvo | zyga: yes, it tells me the ip but no ssh, i may have used this to test disabling the ssh login :/ | 10:13 |
| Chipaca | mbeneto: what do you see if you run the tests under a 'ulimit -H -n 200'? | 10:13 |
| Chipaca | mborzecki: ^ | 10:13 |
| Chipaca | mbeneto: sorry :-) | 10:13 |
| zyga | mvo: ah, that bug :) | 10:13 |
| zyga | mvo: heh, well | 10:13 |
| Wimpress | Just want to confirm that only content snaps published by Canonical can be connected to by 3rd parties. | 10:14 |
| zyga | Wimpress: or snaps with an appropriate store side declaration | 10:14 |
| Wimpress | Yep. | 10:14 |
| Wimpress | In this case we have a new content snap we want 3rd parties to be able to use freely. | 10:14 |
| Wimpress | So publishing as Canonical is the way to go, right? | 10:15 |
| Chipaca | mborzecki: ok now i got a panic in httptest/server itself | 10:16 |
| pedronis | Wimpress: to be clear Canonical is not special here in code, we setup a declaration either way | 10:16 |
| pedronis | it's a policy question | 10:16 |
| mborzecki | Chipaca: in listener? | 10:17 |
| zyga | Wimpress: what content is that? will canonical maintain it? | 10:17 |
| mborzecki | Chipaca: err i mean when setting up listener? | 10:17 |
| Chipaca | mborzecki: https://pastebin.ubuntu.com/p/ryWkGQp3FD/ | 10:17 |
| mvo | zyga: dumpe2fs output https://paste.ubuntu.com/p/jzVHMqRJjP/ | 10:24 |
| zyga | Filesystem created: Wed Jan 10 21:18:38 2018 | 10:25 |
| zyga | Last mount time: Wed Jan 10 21:19:40 2018 | 10:25 |
| zyga | Last write time: Wed Jan 10 21:19:40 2018 | 10:25 |
| zyga | now reboot | 10:25 |
| zyga | and look again | 10:25 |
| zyga | if those don't change in a sensible way (they are already off IMO) it's confirmed | 10:25 |
| zyga | mvo: so in the early boot phase your device think it's still winter | 10:25 |
| zyga | mvo: I'm looking at vanilla ubuntu, fedora is certainly not affected somehow | 10:26 |
| zyga | mvo: classic ubuntu is also not affected | 10:26 |
| zyga | though let me triple check to be certain | 10:27 |
| zyga | I have a pi running raspbian as well, let me check that | 10:27 |
| Wimpress | zyga pedronis I've not actually been involved in assertions for content snaps. | 10:28 |
| Chipaca | ... value client.ConnectionError = client.ConnectionError{error:(*url.Error)(0xc82000a630)} ("cannot communicate with server: Post http://127.0.0.1:42000/v2/aliases: dial tcp 127.0.0.1:42000: getsockopt: connection reset by peer") | 10:28 |
| Chipaca | EVERYTHING IS TERRIBLE | 10:29 |
| mborzecki | Chipaca: stopped with dlv at a breakpoint in TestAliasesNoneFilterSnap | 10:29 |
| mborzecki | Chipaca: there don't seem to be many fds open | 10:29 |
| Wimpress | The content is the chromium-ffmpeg codecs. Will be published by Canonical since Canonical are the MPEG-LA license holder. | 10:29 |
| Wimpress | We have an initial browser vendor who will connect to the that snap. | 10:29 |
| zyga | mvo: interestingly raspbian _is_ affected | 10:29 |
| zyga | Filesystem created: Thu Jan 1 00:00:35 1970 | 10:30 |
| zyga | Last mount time: Sun Jul 22 19:02:57 2018 | 10:30 |
| zyga | Last write time: Thu Nov 3 17:16:45 2016 | 10:30 |
| zyga | but my dates are even more crazy than yours | 10:30 |
| zyga | last write time is equally bogus there | 10:30 |
| zyga | this machine is using a few snaps | 10:30 |
| zyga | I will remove them now to see if there is any relation to shutdown without snaps | 10:30 |
| pedronis | Wimpress: you will need to ask jdstrand to setup the declaration | 10:31 |
| Wimpress | pedronis: Thanks, will do. | 10:31 |
| * zyga rebooted his raspbian pi | 10:32 | |
| zyga | mvo: so last-write-time is unreliable in raspbian (somehow) but not on ubuntu (laptop) | 10:34 |
| zyga | I wonder if lack-or-presence of rtc is a factor here | 10:34 |
| zyga | I will look if I can capture the shutdown sequence somewhere | 10:35 |
| mvo | zyga: ta | 10:36 |
| mvo | zyga: fwiw, the downloaded base image I was using is from 10 Jan, so that might explain the date | 10:37 |
| mvo | zyga: I rebooted now but the last-write-time is still the same | 10:37 |
| zyga | yeah | 10:37 |
| zyga | you can replicate the issue with just loopback ext4 | 10:37 |
| zyga | just mount -o remount,ro | 10:38 |
| zyga | and then unmount | 10:38 |
| zyga | I think the proper fix for now is a canary file | 10:38 |
| zyga | we will need to patch shutdown scripts but that is ok | 10:38 |
| zyga | well, the shutdown _helper_ | 10:38 |
| zyga | and fixtrc in our ppa | 10:38 |
| zyga | mvo: the fixed fixrtc will restore clock based on the canary file | 10:39 |
| zyga | mvo: and that should be it really | 10:39 |
| zyga | then we pass this to foundations to untangle and fix properly with SRU candence | 10:39 |
| zyga | mvo: I think that's the best case scenario for now | 10:39 |
| zyga | mvo: I wonder if any other magic bugs can be attributed to devices waking up and thinking time is standing still | 10:41 |
| zyga | mvo: also it would be interesting to have a core device metric with (has-hardware-rtc, has-working-ntp) | 10:42 |
| mvo | zyga: it looks like this is somewhat specfic to core, on ubuntu my last-write-time looks vageuly ok | 10:44 |
| mvo | zyga: so maybe the best case is we fix this? | 10:44 |
| zyga | mvo: I agree but I also think there is something missing | 10:46 |
| zyga | is the remount ro specific to core? | 10:46 |
| zyga | is that a workaround for broken shutdown from 15.04 | 10:46 |
| zyga | and we just carry it? | 10:46 |
| mvo | zyga: I think that is what we need to find out | 10:46 |
| zyga | yeah, | 10:46 |
| zyga | zyga@pi3-1:~ $ journalctl --list-boots | 10:51 |
| zyga | -1 5a859aca24244b1aa150eadb4461c434 Wed 2018-07-25 10:49:49 UTC—Wed 2018-07-25 10:49:55 UTC | 10:51 |
| zyga | 0 a78a995e1b794fc3829e3ff69c90cc92 Thu 2016-11-03 17:16:45 UTC—Wed 2018-07-25 10:50:47 UTC | 10:51 |
| zyga | enabling persistent journal shows interesting dates | 10:52 |
| ogra_ | zyga, abeato has a patch for canary file support in fixrtc we havent merged | 10:54 |
| zyga | ogra_: oh, nice, do you have a link for that? | 10:54 |
| ogra_ | ... should consider using that one ;( | 10:54 |
| ogra_ | err | 10:54 |
| ogra_ | ;) | 10:54 |
| * abeato searching... | 10:54 | |
| ogra_ | not off the top of my head, thats why i pinged abeato ;) | 10:54 |
| zyga | thanks abeato | 10:55 |
| ogra_ | hmm, interesting ... i'm doing a snap build, but snapcraft clean fails with permission denied (files of the build in prime/ are root only apparently) i wonder how thats possible given i didnt use sudo or anything | 10:56 |
| zyga | mvo: so interesting, there's a systemd unit | 10:58 |
| zyga | Jul 25 10:55:19 pi3-1 systemd[1]: Stopped Remount Root and Kernel File Systems. | 10:59 |
| zyga | it runs just before actual shutdown, inspecting it now | 10:59 |
| zyga | systemctl cat systemd-remount-fs.service | 11:00 |
| ogra_ | thats an upstream one AFAIK | 11:00 |
| zyga | that's just done on startup | 11:01 |
| * zyga keeps looking | 11:01 | |
| zyga | (this is a non-issue) | 11:01 |
| zyga | Jul 25 10:55:19 pi3-1 systemd[1]: Failed to set timeout to 600s: Invalid argument | 11:02 |
| zyga | Jul 25 10:55:19 pi3-1 kernel: systemd-shutdow: 26 output lines suppressed due to ratelimiting | 11:02 |
| zyga | I need to find the rate limit sysctl toggle | 11:02 |
| ogra_ | printk_ratelimit | 11:03 |
| zyga | that's in sysctl.conf? | 11:04 |
| ogra_ | in sysctl.conf it is kernel.printk_ratelimit=X | 11:05 |
| ogra_ | (X is seconds, default is 5) | 11:05 |
| abeato | zyga, ogra_ this was the additional initrd script I used in some projects: https://paste.ubuntu.com/p/mNw7qKpbyx/ | 11:06 |
| zyga | ok rebooting | 11:06 |
| zyga | abeato: thanks | 11:07 |
| abeato | it looks for dates on some files, when the mount date was not really reliable | 11:07 |
| abeato | np | 11:07 |
| ogra_ | it sadly makes everything snap specific ... | 11:08 |
| ogra_ | so nothing for upstream | 11:08 |
| abeato | right, although it can be modfified easily | 11:09 |
| ogra_ | yeah | 11:09 |
| zyga | ogra_: fixrtc is not upstream | 11:10 |
| mborzecki | Chipaca: heh, it stopped reproducing now | 11:11 |
| Chipaca | mborzecki: on updated master? | 11:12 |
| zyga | ha | 11:12 |
| zyga | Chipaca: are you on 16.04? | 11:12 |
| Chipaca | zyga: yes | 11:12 |
| mborzecki | Chipaca: no, the same code, just stopped appearing | 11:12 |
| zyga | Chipaca: can you run: lsblk -a and find your rootfs ext4 | 11:12 |
| zyga | Chipaca: then please run sudo dumpe2fs -h /dev/sdaX | 11:12 |
| zyga | and look at the three timestamps and paste them here | 11:12 |
| zyga | mvo: maybe it depends on systemd version | 11:12 |
| zyga | and is fixed in bionic / fedora 28 | 11:13 |
| zyga | but not in xeial | 11:13 |
| zyga | *xenial | 11:13 |
| zyga | and not in my old raspbian | 11:13 |
| Chipaca | zyga: i can do that on encrypted and non-encrypted / | 11:13 |
| zyga | Chipaca: non-encrypted please | 11:13 |
| zyga | though should not matter as ext4 doesn't see the encryption layer | 11:13 |
| Chipaca | zyga: i mean, i have both, on separate machines (obviously) | 11:13 |
| zyga | and this is an ext4 superblock feature | 11:13 |
| Chipaca | zyga: http://paste.ubuntu.com/p/vqFY25f3zs/ | 11:14 |
| zyga | ha | 11:14 |
| zyga | Filesystem created: Sat Feb 24 17:42:06 2018 | 11:14 |
| zyga | Last mount time: Fri Jul 13 21:11:46 2018 | 11:14 |
| zyga | Last write time: Fri Jul 13 21:11:46 2018 | 11:14 |
| zyga | is this machine up since the 13th of July? | 11:14 |
| Chipaca | zyga: probably yes | 11:16 |
| zyga | hmmm, would you be ok rebooting it? | 11:16 |
| ogra_ | zyga, ubuntu upstream | 11:16 |
| Chipaca | zyga: up 11 days | 11:16 |
| zyga | ok | 11:16 |
| Chipaca | zyga: http://paste.ubuntu.com/p/3KghN9nj7v/ | 11:16 |
| Chipaca | zyga: that one's up 8 days | 11:16 |
| jdstrand | zyga: I don't recall knowing anything about https://bugs.launchpad.net/snappy/+bug/1709155/comments/4 | 11:16 |
| mup | Bug #1709155: Better error message for unsupported kernel <Snappy:Triaged> <Ubuntu:Confirmed> <https://launchpad.net/bugs/1709155> | 11:16 |
| zyga | so somehow classic xenial is not broken | 11:16 |
| zyga | jdstrand: thank you | 11:16 |
| Chipaca | zyga: I would be ok rebooting it if needed | 11:17 |
| zyga | unless you never rebooted before it should be consistent | 11:17 |
| Chipaca | zyga: i've rebooted lots of times | 11:18 |
| Chipaca | just not in the past few weeks | 11:18 |
| zyga | ok | 11:18 |
| jdstrand | jjohansen: hey, the bug description is wrong, but starting at comment #4, people are complaining about "cannot perform readlinkat() on the mount namespace file descriptor of the init process" in 4.18. this reminds me of https://bugs.launchpad.net/apparmor/+bug/1656121, but people aren't talking about complain mode | 11:19 |
| mup | Bug #1656121: unexpected errno=13 and disconnected path when trying to open /proc/1/ns/mnt from a unshared mount namespace <verification-done-xenial> <verification-done-yakkety> | 11:19 |
| mup | <AppArmor:Confirmed> <linux (Ubuntu):Incomplete> <linux (Ubuntu Xenial):Fix Released> <linux (Ubuntu Yakkety):Fix Released> <https://launchpad.net/bugs/1656121> | 11:19 |
| jdstrand | jjohansen: I've asked for more info (in particular, denials). does the 4.18 have the sauce patches? | 11:19 |
| jdstrand | jjohansen: actually, seems most how are complaining are talking about mainline/upstream | 11:20 |
| jdstrand | s/how/who/ | 11:20 |
| * Chipaca ~> lunch | 11:21 | |
| zyga | mvo: raspbian uses "fake-hwclock" | 11:25 |
| zyga | but it runs in mid-early userspace (not in intrd) | 11:27 |
| Chipaca | mborzecki: merged master insto the green cehck mark and didn't get the connection error this time (got others instead) | 11:29 |
| Chipaca | now running tests in a loop with the race detector on | 11:29 |
| Chipaca | we'll see | 11:29 |
| * Chipaca ~> really lunch | 11:29 | |
| pedronis | Chipaca: btw the advise tests do direct FD manipulations in goroutines | 11:30 |
| pedronis | there might be something there | 11:30 |
| Chipaca | pedronis: ack | 11:30 |
| ogra_ | zyga, yeah, that wont help with what initially triggered the fixrtc creation ... ("mount time is in the future yadda yadda, boom") | 11:33 |
| zyga | mhm | 11:33 |
| ogra_ | zyga, though perhaps combining both might work | 11:34 |
| zyga | I'm trying to understand why last-write-time is bad on raspbian | 11:34 |
| zyga | and not on ubuntu | 11:34 |
| ogra_ | but that will indeed result in jumpy clock | 11:34 |
| zyga | is it caused by some systemd shutdown logic | 11:34 |
| ogra_ | s/jumpy/more jumpy/ | 11:34 |
| zyga | are we using systemd in the initrd in classic ubuntu? | 11:35 |
| zyga | AFAIK fedora does now | 11:36 |
| ogra_ | no | 11:36 |
| ogra_ | there were attempts from ... hmm, how was he called ... james something | 11:36 |
| ogra_ | he spent 6 months on it and gave up | 11:36 |
| ogra_ | i'd actually love if we could perhaps go the android approach one day ... | 11:36 |
| ogra_ | initrd as rootfs ... and just mount all subdirs from disk ... i.e. no rootfs pivoting at all | 11:37 |
| ogra_ | that would surely work with systemd ... | 11:37 |
| ogra_ | migrating todays initramfs logic will be a lot harder | 11:37 |
| ogra_ | (especially since there are tons of mudular bits that get dynamically added or removed ... every package can enhance the initrd) | 11:38 |
| pedronis | Chipaca: I see some code similar to the one we had problem with here: ac0523dadf923f4dc6bb | 11:39 |
| jdstrand | ogra_: james hunt? | 11:40 |
| ogra_ | jdstrand, ah, yeah, him ! | 11:41 |
| zyga | ok, so somewhat closer now | 11:41 |
| zyga | I'm looking at source of part of systemd | 11:41 |
| jdstrand | mborzecki: hey, did you implement timers? if so, fyi, https://forum.snapcraft.io/t/system-options/87/12 | 11:42 |
| pedronis | Chipaca: mborzecki: I think there's a chance the issues come from https://github.com/snapcore/snapd/pull/5122 | 11:42 |
| mup | PR #5122: snap: add support for `snap advise-snap --from-apt` <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/5122> | 11:42 |
| pedronis | s/the issues/some of the issues/ | 11:42 |
| jdstrand | mborzecki: but the reason I mention it is to ask if they are expected to work on 14.04? it seems it doesn't support 'systemctl list-timers' and though the timer file is generated, it doesn't ever seem to fire | 11:43 |
| jdstrand | mborzecki: it is fine (with me anyway) if it isn't supported, but seems like something that should be documented. maybe it is and I just don't know where | 11:44 |
| mborzecki | jdstrand: i'm aware of list-timers not working, but timers as such should likely work on 14.04, unless systemd is that old there | 11:45 |
| mborzecki | jdstrand: well, it's possible to enable a timer unit and start it | 11:46 |
| jdstrand | mborzecki: it has 229-4ubuntu21.2. I used 'timer: 0:00-24:00/24', but it never fired after reboot afaics. it worked as expected on 16.04 | 11:47 |
| jdstrand | mborzecki: I could start it. I thought I even enabled it... but it wouldn't fire hourly | 11:48 |
| mborzecki | jdstrand: journalctl -u *.timer ? | 11:49 |
| jdstrand | mborzecki: it's gone now. I guess I can create bug/topic. which do you prefer? | 11:50 |
| mborzecki | jdstrand: let's do a topic first | 11:50 |
| * zyga -> walk, see you at the standup | 12:05 | |
| jdstrand | mborzecki: ok, it seems the problem was actually with the logging to rsyslog on 14.04 | 12:06 |
| mborzecki | phew :) | 12:07 |
| jdstrand | mborzecki: I created a timer that fires every minute and updates a file in SNAP_COMMON and echos to stdout. the file is updated, rsyslog is not. on 16.04, I see both | 12:07 |
| jdstrand | mborzecki: journalctl shows it. so this is configuration issue with logging | 12:08 |
| jdstrand | as it happens, the snap that I was using was specifically trying to generate a syslog entry | 12:09 |
| jdstrand | I could use logger, etc. but between that and list-timers, I thought there was an issue. but there isn't | 12:09 |
| mborzecki | well, at least we won't have to debug systemd | 12:10 |
| mborzecki | jdstrand: btw. there are some timer related properties if you do systemctl show foo.timer | 12:10 |
| jdstrand | mborzecki: I was surprised to see how long the timer file was for the pathological 00:00-24:00/1440 | 12:10 |
| mborzecki | jdstrand: but i don't recall if those are present in older releases | 12:10 |
| mborzecki | jdstrand: yes, our syntax is more expressive, we could do some optimization and be smarted about generated schedules | 12:11 |
| * jdstrand nods | 12:11 | |
| jdstrand | ok, thanks! | 12:12 |
| mborzecki | hm something weird about how snaps run in amazon linux | 12:13 |
| mborzecki | zyga: /etc/os-release inside a snap should come from the core snap right? | 12:16 |
| zyga | Yes | 12:29 |
| zyga | But by accident | 12:29 |
| zyga | Because it is a slimlink | 12:29 |
| zyga | On typical etc | 12:30 |
| zyga | That goes to /lib | 12:30 |
| abeato | jdstrand, hi, I think this MP is waiting for your review: https://github.com/snapcore/snapd/pull/5469 | 12:34 |
| mup | PR #5469: interfaces/apparmor: (un)load profiles in one apparmor_parser call <Created by alfonsosanchezbeato> <https://github.com/snapcore/snapd/pull/5469> | 12:34 |
| mborzecki | Chipaca: anything came up with the race detector? | 12:37 |
| Chipaca | mborzecki: just one minor thing in cmd_wait_test, so far | 12:38 |
| cachio | mvo, did you see this one? | 12:38 |
| cachio | https://travis-ci.org/snapcore/snapd/builds/407953474#L5654 | 12:38 |
| Chipaca | (cmd_wait_test uses a rather funny way of doing the check it does, fwiw...) | 12:38 |
| Chipaca | i might just rewrite it out of spite | 12:38 |
| Chipaca | mborzecki: infuriatingly, twice i've hit the 'connection refused' error with no race detection | 12:40 |
| mborzecki | heh | 12:40 |
| mborzecki | Chipaca: also funny how even a tiny change in code makes it disappear to pop up in another test | 12:41 |
| jjohansen | jdstrand: there are no sauce patches in ubuntu or upstream around mount | 12:42 |
| Chipaca | mborzecki: another one that I hit frequently is the 'queueing for later' (which is just another connection error) | 12:42 |
| mborzecki | Chipaca: yup, same here | 12:42 |
| jjohansen | the only patch missing upstream is af_unix | 12:43 |
| mojtaba | Hello, after running df, I am seeing this: /dev/loop1 640 640 0 100% /snap/gnuchess/9 Could you please explain what it is and how can I unmount it to free some space? | 12:43 |
| mborzecki | ok, interestingly the snaps with commands that use relative paths, eg. ../../../bin/sh fail on amazon linux, it's clearly doing something wrong with the paths and calling eg. /var/lib/snapd/bin/sh instead of /bin/sh | 12:44 |
| mborzecki | zyga: any clues ^^ ? | 12:44 |
| Chipaca | mojtaba: why would just unmounting it free some space? | 12:45 |
| Chipaca | mojtaba: but, to answer your question with an explanation, look at the output of "snap list --all gnuchess" | 12:45 |
| Chipaca | mojtaba: (pastebinit so I can see it as well) | 12:45 |
| abeato | has anybody seen error when snapcraft downloads packages lately? like https://paste.ubuntu.com/p/M9mfqB9WcR/ | 12:50 |
| mborzecki | zyga: hm i guess it has something to do with the hosts /etc/os-release appearing inside the mount namespace, but i don't know why it happens yet | 12:51 |
| mojtaba | Chipaca: http://paste.ubuntu.com/p/fChWFPp7Ct/ | 12:53 |
| zyga | mborzecki: is it a file? | 12:53 |
| zyga | mborzecki: if so, that's why | 12:53 |
| zyga | ogra_: ping | 12:53 |
| zyga | ogra_: I need one more bit of help from your pi | 12:53 |
| zyga | ogra_: can you spare 3 minutes | 12:53 |
| zyga | ogra_: journalctl -u apparmor.service | 12:53 |
| zyga | that's all I need | 12:53 |
| zyga | thanks! | 12:53 |
| Chipaca | mojtaba: so, snapd will keep a number of previous releases of the snaps around, to fall back to in case something breaks | 12:53 |
| mojtaba | Chipaca: http://paste.ubuntu.com/p/JpyVcmYcmc/ | 12:54 |
| mojtaba | There is another one for core | 12:54 |
| Chipaca | mojtaba: yes, by default it keeps a maximum of 3 around | 12:54 |
| Chipaca | mojtaba: you can lower that to two, that is, it'll keep the current one and up to one older one (so when it refreshes it removes the previous old one) | 12:54 |
| Chipaca | mojtaba: you can also remove them by hand if you need even more space | 12:55 |
| mojtaba | Chipaca: I am running out of space in my root directory. Can I unmount the first two? | 12:55 |
| Chipaca | mojtaba: unmounting won't free up space | 12:55 |
| mojtaba | Chipaca: How can I change that to 2 and remove the first one by hand? | 12:55 |
| Chipaca | mojtaba: and, note that snaps are heavily compressed, so if you're using "du" or similar, it's not giving you the whole picture | 12:55 |
| mojtaba | Chipaca: What should I use then? | 12:55 |
| Chipaca | mojtaba: (you can see the .snap files in /var/lib/snapd/snaps/) | 12:55 |
| Chipaca | mojtaba: i'm getting there | 12:55 |
| Chipaca | mojtaba: let me look up the command to change the default | 12:56 |
| Chipaca | 1 se | 12:56 |
| Chipaca | c | 12:56 |
| mojtaba | Chipaca: I am there, should I remove the files? like gimp_38.snap? | 12:56 |
| Chipaca | no, do not remove those files by hand | 12:57 |
| Chipaca | mojtaba: snap set system refresh.retain=2 | 12:57 |
| Chipaca | ^ that sets it to keep max of 2 revisions | 12:57 |
| Chipaca | mojtaba: (but it only kicks in during refresh) | 12:57 |
| Chipaca | mojtaba: to remove a revision by hand, "snap remove --revision=<the revision> thesnap" | 12:57 |
| Chipaca | mojtaba: e.g., given your 'snap list gnuchess' output, 'snap remove --revision=9 gnuchess' | 12:58 |
| mojtaba | Chipaca: I am getting this error: error: cannot perform the following tasks: | 12:58 |
| mojtaba | - Run configure hook of "core" snap (run hook "configure": cannot set "core.refresh.retain": unsupported system option) | 12:58 |
| Chipaca | mojtaba: ah, maybe it's not in stable yet, sorry :-) | 12:58 |
| Chipaca | should be there soon though | 13:00 |
| ogra_ | zyga, my pi is currently building a kodi snap ... that will keep it busy for another 1-2h (and it is already 2h in and i dont want to stop it) | 13:00 |
| mojtaba | Chipaca: Do you know how can I change it? Or it is not possible to change it to 2? | 13:00 |
| jdstrand | abeato: yes, on my list for today actually. was sprinting last week | 13:00 |
| Chipaca | mojtaba: meanwhile there's a script that uses 'snap list' output to remove all "disabled" revisions | 13:00 |
| zyga | Chipaca: can you ssh in and just run that or would that be a risk? | 13:00 |
| abeato | jdstrand, cool, thanks | 13:00 |
| Chipaca | zyga: ? | 13:01 |
| ogra_ | (keypresses on it take minutes to return atm) | 13:01 |
| zyga | er | 13:01 |
| zyga | that was for ogra_ | 13:01 |
| zyga | ah o | 13:01 |
| Chipaca | mojtaba: if the 'set' command failed, your snapd does not have that option yet (so it's hardcoded to 3) | 13:01 |
| mojtaba | Chipaca: What should I do know? | 13:01 |
| mojtaba | Can I edit the config file? | 13:01 |
| Chipaca | mojtaba: give me a moment | 13:01 |
| mojtaba | Chipaca: thanks | 13:02 |
| Chipaca | mojtaba: https://superuser.com/a/1330590/63234 | 13:03 |
| thresh | uhm | 13:12 |
| thresh | hello! | 13:12 |
| thresh | so I want to use my pkcs11-enabled smart card with a snapped firefox | 13:12 |
| thresh | is there a way to grant it access to a real / ? | 13:12 |
| ogra_ | crazy ideas ! | 13:12 |
| ogra_ | :) | 13:12 |
| thresh | well | 13:12 |
| zyga | thresh: it is in /var/lib/snapd/hostfs but it's not really available due to confinement | 13:13 |
| ogra_ | (i guess that would need some kind of smartcard interface ... i have seen people ask about it before) | 13:13 |
| zyga | thresh: yes, we'd need a new interface | 13:13 |
| zyga | we could craft one based on the denials you are seeing | 13:13 |
| thresh | I don't have any denials, firefox just says it fails to load the library | 13:17 |
| mojtaba | Chipaca: I could remove the first one of the three, but I just wonder if I could change the default value to 2. So in future it would limit to 2 versions. | 13:17 |
| mojtaba | Chipaca: I could not find it in that link. | 13:17 |
| thresh | (I need it to import /usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so) | 13:17 |
| thresh | https://forum.snapcraft.io/t/snapped-firefox-unable-to-use-smart-card/5719 seems related | 13:18 |
| pedronis | Chipaca: https://github.com/snapcore/snapd/blob/master/cmd/snap/cmd_advise_test.go#L129 vs https://github.com/snapcore/snapd/commit/ac0523dadf923f4dc6bb4fcecff8f2632d54020c I think | 13:20 |
| mborzecki | zyga: os-reelase on amazon -> https://paste.ubuntu.com/p/ZsyX7stm74/ | 13:20 |
| mborzecki | zyga: it's an actual file :( damn | 13:20 |
| mborzecki | that's why snap-exec may be getting confused with paths | 13:21 |
| Chipaca | mojtaba: that configuration option is there from snapd 2.34, which isn't yet in "stable" | 13:21 |
| mojtaba | Chipaca: Ok, thanks | 13:21 |
| Chipaca | mojtaba: so you could look at switching to snapd candidate (snap refresh core --candidate) to get the feature | 13:21 |
| thresh | no denials afaict in journalctl | grep audit | 13:22 |
| Chipaca | mojtaba: or, wait a bit :-) | 13:22 |
| mojtaba | Chipaca: thanks a lot for the info. | 13:22 |
| Chipaca | mojtaba: note 2.34.1 might have issues (there's already a 2.34.2 in the works), so take care | 13:22 |
| Chipaca | i mean, there are reasons it's not stable yet :-) | 13:23 |
| Chipaca | pedronis: tests are at loop #30 with no failures just by skipping TestAdviseFromAptIntegration | 13:25 |
| mborzecki | zyga: should we fix /etc/os-release as part of quirks in snap-confine? | 13:25 |
| Chipaca | pedronis: so, yeah, i think that's the one | 13:25 |
| zyga | mborzecki: ish... it's longer than it seems | 13:26 |
| pedronis | Chipaca: good, I think the 2nd link should have an idea what we can do | 13:26 |
| pedronis | if I understand things | 13:26 |
| mborzecki | zyga: something like if /etc/os-release is not a symlink to /usr/lib/os-release mount core /usr/lib/os-relase on top of it? | 13:27 |
| zyga | that's racy :) | 13:27 |
| mborzecki | zyga: in snap-confine? | 13:27 |
| zyga | yes | 13:27 |
| jdstrand | roadmr: hi! can you pull r1108 of CRT? not urgent | 13:28 |
| roadmr | jdstrand: sure thing! | 13:28 |
| jdstrand | one added test, couple of testsuite fixes for arm | 13:28 |
| jdstrand | and an override | 13:28 |
| jdstrand | roadmr: thanks | 13:28 |
| roadmr | awesome! | 13:29 |
| mborzecki | zyga: what alternatives you'd suggest? | 13:31 |
| mborzecki | (aside from fixing the offending snaps) | 13:31 |
| zyga | mborzecki: please hold on, | 13:31 |
| zyga | jdstrand: hey, I think we have to bump the apparmor cache bug; it's blocking the release now and we should look at a high-priority fix. | 13:44 |
| zyga | jdstrand: we now have a more firm understanding of what is wrong | 13:44 |
| zyga | jdstrand: and this is blocking the release | 13:44 |
| mvo | Chipaca: sorry, its the stress talking | 13:46 |
| Chipaca | mvo: coming from you, I'm not offended | 13:46 |
| zyga | jdstrand: let me know when we can talk | 13:46 |
| Chipaca | pedronis: mborzecki: #5556 | 13:46 |
| mup | PR #5556: cmd/snap: fix two issues in the cmd/snap unit tests <Created by chipaca> <https://github.com/snapcore/snapd/pull/5556> | 13:47 |
| jdstrand | zyga: how is apparmor_parser using mtime since (nearly) the dawn of time all of a sudden a release blocker for snapd? there is clearly a problem with how the system time is setup. that needs to be fixed any way. to unblock yourselves immediately, move the file back and then focus on the time fix | 13:47 |
| jdstrand | zyga: jj is off today, and there is no way that a change to how apparmor_parser handles its caching is going to be fixed fast enough | 13:47 |
| jdstrand | zyga: I mean, mtime may not be the best choice in retrospect, but it isn't a bad choice on its own. systems are expected to have a reasonable notion of time | 13:49 |
| jdstrand | zyga: it isn't just apparmor that does this sort of thing after all (there is a reason mtime exists after all) | 13:50 |
| zyga | jdstrand: one sec, meeting | 13:54 |
| Wimpress | jdstrand: Can you hope on a quick video call with me? | 13:55 |
| Wimpress | Somewhat urgent question about store assertions. | 13:55 |
| ogra_ | jdstrand, well, the way we handle the system clock has seemingly never worked right on core ... was the switch to mtime a recent thing ? (i wonder why it has not exposed itself before) | 13:55 |
| jdstrand | zyga: iirc, there was talk of using some sort of a hash. this, afaik was never upstream. I'm not sure what the design was tbh | 13:55 |
| jdstrand | ogra_: no. it was meant to use mtime forever. in the Touch days it was discovered to use ctime and went back to mtime | 13:56 |
| ogra_ | i mean ... the clock is definitely never set up the epoch when reading the filesystem metadataas input ... but that metadata has also obviously never been updated since we use the snapd shutdown helper | 13:56 |
| jdstrand | ogra_: iirc, there were some small fixes for snapd surrounding this have have existed for a long time | 13:56 |
| ogra_ | *set up the/set to the/ | 13:56 |
| jdstrand | perhaps tyhicks recalls some details | 13:57 |
| jdstrand | he isn't in this channel atm | 13:57 |
| jdstrand | Wimpress: what is this in reference to? | 13:57 |
| ogra_ | i mean ... it could well be that we simply never noticed that we used outdated profiles ... | 13:57 |
| ogra_ | but that would even be worse i think | 13:57 |
| jdstrand | zyga, ogra_: another unblocker workaround is to unconditionally delete the cache files for snap-confine on shutdown. these are small profiles that wouldn't take long to regenerate on boot | 14:03 |
| zyga | jdstrand: the problem is that we found that time is essentially stuck in early boot | 14:03 |
| jdstrand | that could be there until time is fixed and/or apparmor_parser | 14:03 |
| zyga | jdstrand: and that it is causing (somehow) apparmor to load the wrong cache | 14:03 |
| zyga | jdstrand: and we just got asked to actually go ahead and fix the cache for real rather than add another workaround | 14:03 |
| ogra_ | jdstrand, well, given that we see 20min+ boot times oin firstboot anyway on most arm devices (thanks to snapd seeding) ... a few seconds probably dont have that mauch imapct ;) | 14:04 |
| jdstrand | I'd like to better understand how the cache is affected before suggesting any other workarounds. or having jjohansen/myself deprioritize roadmapped work | 14:04 |
| zyga | jdstrand: it seems that the cache is from the future | 14:04 |
| jdstrand | ogra_: exactly | 14:04 |
| zyga | jdstrand: and then we load the cache from now | 14:04 |
| zyga | where now == past | 14:04 |
| jdstrand | but I need to go to a meeting | 14:04 |
| zyga | ok | 14:04 |
| zyga | jdstrand: I'll investigate the cache more but I think we cannot do another workaround, this is with us in some way or another since 15.04 | 14:06 |
| === devil is now known as Guest11165 | ||
| Jguy| | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 14:07 |
| Jguy| | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 14:07 |
| Jguy| | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 14:07 |
| Jguy| | Voice your opinions at https://webchat.freenode.net/?channels=#freenode | 14:07 |
| jdstrand | zyga: well, the system needs to have a notion of reasonable time. sure, you can fix the symptoms like apparmor cache, but that isn't fixing the root cause and there might be other bugs | 14:33 |
| zyga | jdstrand: that's not an assumption we can do | 14:34 |
| zyga | jdstrand: those devices don't know the time | 14:34 |
| jdstrand | zyga: but this will have to be escalated via ratliff/amurray to reprioritize people's work if you're demanding apparmor be changed to redo its caching calculation | 14:34 |
| zyga | jdstrand: ok, I'll do some more digging and get back to you | 14:34 |
| jdstrand | zyga: those devices are going to fail somehow, some way if they don't have proper time. they might not refresh, something might use kerberos, active directory, etc, etc | 14:35 |
| zyga | they don't have proper time in early boot | 14:35 |
| zyga | they have time once network is up | 14:35 |
| jdstrand | zyga: I'm not saying it is an assumption you can do. I am saying that snapd or the gadget or something needs to make them have reasonable time | 14:36 |
| jdstrand | zyga: I mean, I haven't thought through all this, but if it is true that the system time is correct once up, then that time can be stored off so fixrtc could use it | 14:37 |
| zyga | jdstrand: yes, that's true | 14:37 |
| pppingme_ | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 14:41 |
| pppingme_ | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 14:42 |
| pppingme_ | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 14:42 |
| pppingme_ | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 14:42 |
| pppingme_ | This message was brought to you by Private Internet Access | 14:42 |
| jdstrand | zyga: also note that simply supplying a patch isn't going to appreciably speed up the apparmor acceptance. the feature needs to be designed (I believe that jj has one in mind), implemented, reviewed, tested everywhere before it goes upstream. then it needs to go through SRU for trusty-bionic | 14:46 |
| zyga | mvo: ^ | 14:46 |
| jdstrand | of course, this is all doable, but various people would be involved and it would have to be appropriately prioritized | 14:46 |
| jdstrand | zyga (cc mvo): note that a lot has happened with caching lately upstream, and the work would need to be verified to work correctly with all of that (it should be orthoganl, but, again, would need jj to comment since he did the work) | 14:48 |
| zyga | jdstrand: we have some ideas | 14:53 |
| zyga | jdstrand: on a small patch that would work in this specific case | 14:53 |
| zyga | jdstrand: or on a way to decouple the schedules | 14:54 |
| zyga | so that we can fix this entirely on snapd schedule | 14:54 |
| jdstrand | zyga: how can you decouple an apparmor parser patch from the the sru schedule? this needs to go through review which brings in everything else | 14:56 |
| jdstrand | I don't see why it is unreasonable to properly fix the time issue | 14:57 |
| jdstrand | I mean, fine, the caching algorithm can be adjusted, but with proper design and consideration | 14:57 |
| odc | hi! Is there a way to debug what happens when I launch my snapped application? It seems to be stuck and consumes a lot of cpu | 14:58 |
| odc | i don't think it's apparmor this time | 15:00 |
| nandub | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 15:01 |
| nandub | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 15:01 |
| nandub | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 15:01 |
| nandub | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 15:01 |
| nandub | This message was brought to you by Private Internet Access | 15:01 |
| zyga | odc: is it using classic confinement? | 15:12 |
| odc | zyga: yes it is | 15:13 |
| zyga | odc: it's calling itself in a loop | 15:13 |
| odc | hm it worked when i was in devmode | 15:13 |
| zyga | odc: make sure to set the path to say something like $SNAP/bin/myself | 15:13 |
| zyga | rather than "myself" | 15:13 |
| zyga | because that will resolve to /snap/bin/myself | 15:13 |
| zyga | and will essentially execute itself forever in a loop | 15:13 |
| odc | thx :) | 15:14 |
| zyga | jdstrand: we could patch apparmor not to load snapd profiles | 15:15 |
| zyga | jdstrand: roll snapd.apparmor.service | 15:15 |
| zyga | jdstrand: use a wrapper around apparmor_parser | 15:15 |
| zyga | jdstrand: and use content based hashing | 15:15 |
| zyga | jdstrand: I mean, we have the code for all of that | 15:15 |
| zyga | odc: in devmode that doesn't happen because PATH is reset | 15:15 |
| zyga | odc: in classic mode it is not touched | 15:16 |
| odc | that's treacherous | 15:16 |
| zyga | it's unavoidable sadly | 15:16 |
| zyga | jdstrand: this would decouple apparmor release cycle | 15:17 |
| zyga | jdstrand: from snapd release cycle | 15:17 |
| zyga | jdstrand: I like this property | 15:17 |
| jdstrand | I don't know how to say it any more clearly. the fixrtc logic is broken. why not fix that? | 15:19 |
| jdstrand | this is going to spin out to shell scripts, then go binaries, etc, etc. fix the time on the system, then, get moving away from mtime on the roadmap for the security team if you are still passionate about it | 15:20 |
| jdstrand | as it is, you will slow down boot cause of all the apparmor_parser -p fork/execs and the sha1sums | 15:20 |
| jdstrand | fix the time. mtime is used cause it is cheap on the very device you are trying to fix | 15:21 |
| jdstrand | zyga, mvo: ^ | 15:21 |
| Cory26 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 15:21 |
| Cory26 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 15:21 |
| zyga | jdstrand: we purge the cache on refresh because the mtime logic was broken | 15:21 |
| Cory26 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 15:21 |
| Cory26 | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 15:21 |
| Cory26 | This message was brought to you by Private Internet Access | 15:21 |
| zyga | Cory26: please stop spamming | 15:22 |
| jdstrand | zyga: that is something different. mtime is not 'broken' | 15:22 |
| jdstrand | zyga: it may not be the best choice, but it is a timestamp. the computer is supposed to do things like keep track of time | 15:22 |
| jdstrand | zyga: honestly, I'm surprised you purge the cache on refresh based on how snapd is supposed to work. also, your snap-apparmor-parser isn't going to fix the snap-confine issue unless you add something else to load it separately | 15:26 |
| jdstrand | there are way to many things going on | 15:26 |
| jdstrand | too | 15:26 |
| jdstrand | it is supposed to be simple: if the profile is newer than the cache then recompile. if snapd detects that something changed on refresh, splat the profile on disk and load it, otherwise, no need to do anything | 15:27 |
| jdstrand | if the time on the system is changing all the time, or set wrong, fix that because *anything* else is papering over the problem and something else is going to break | 15:29 |
| StephenS18 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 15:33 |
| StephenS18 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 15:33 |
| StephenS18 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 15:35 |
| StephenS18 | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 15:35 |
| StephenS18 | This message was brought to you by Private Internet Access | 15:35 |
| ephemer0l_6 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 15:36 |
| ephemer0l_6 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 15:36 |
| ephemer0l_6 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 15:36 |
| ephemer0l_6 | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 15:36 |
| ephemer0l_6 | This message was brought to you by Private Internet Access | 15:36 |
| annieslmaos | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 15:38 |
| annieslmaos | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 15:38 |
| annieslmaos | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 15:38 |
| annieslmaos | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 15:38 |
| annieslmaos | This message was brought to you by Private Internet Access | 15:38 |
| jdstrand | zyga: to put i more concrete terms. I may as well not even review abeato's PR 5469 if you are going to make extra forks and sha1 sums | 15:38 |
| mup | PR #5469: interfaces/apparmor: (un)load profiles in one apparmor_parser call <Created by alfonsosanchezbeato> <https://github.com/snapcore/snapd/pull/5469> | 15:38 |
| kyrofa | Hey kenvandine, any idea why my firefox windows are showing up under the slack icon in the dock (18.04)? | 15:46 |
| zyga | jdstrand: at this stage we don't know yet | 15:49 |
| zyga | jdstrand: I was explaining a possible scenario | 15:50 |
| jdstrand | ok, well, I see github commits and what sounds like every saying 'this mtime thing is annoying, just throw it out' | 15:50 |
| jdstrand | everyone* | 15:50 |
| zyga | jdstrand: that was written a while ago, I pulled it out of git history | 15:51 |
| zyga | jdstrand: it's march I believe | 15:51 |
| zyga | jdstrand: gustavo asked us to understand the current cache behavior and fix it | 15:51 |
| zyga | jdstrand: one idea we had was to sync the mtime of the text file and the cache file | 15:51 |
| zyga | jdstrand: and only use the cache if they are in sync | 15:51 |
| zyga | jdstrand: (not newer or older but just equal) | 15:51 |
| zyga | jdstrand: this would decouple the time from the equation, we would really use mtime as a cookie | 15:52 |
| jdstrand | zyga: the cache behavior seems undesrtood now. the clock is wrong and could be fixed by fixing fixrtc. has anyone communicated that? | 15:52 |
| zyga | jdstrand: my patch about using content based caching was from an RFC branch (I showed you the patch before) https://github.com/zyga/snapd/tree/rfc/snap-apparmor-parser that I recalled we have and we discussed as a possible solution to the current problem | 15:53 |
| zyga | jdstrand: yes, we don't want to rely on fixrtc, we have clear directions on addressing the cache semantics | 15:53 |
| jdstrand | zyga: as for making apparmor more robust, sure, that can be looked at, but I think it needs to be driven by upstream apparmor and not emply wrappers and a bunch of work arounds | 15:53 |
| zyga | jdstrand: the kernel / initramfs patches for other userspace programs that may suffer from wrong time in early boot should be fixed as well | 15:53 |
| jdstrand | zyga: I maintain it is not snapd's problem to solve. the clock is wrong | 15:54 |
| zyga | jdstrand: I'm reading the cache code now, I will try to provide a small patch that makes the cache correct even if the time is essentially random | 15:54 |
| zyga | jdstrand: yes but the clock _cannot_ be fixed | 15:54 |
| zyga | it can only be improved | 15:54 |
| jdstrand | zyga: the clock *can* be set | 15:54 |
| jdstrand | you said it yourself | 15:54 |
| zyga | hold on, I think I need to do IRL stuff for a moment | 15:55 |
| jdstrand | blocking the release on something that has been broken since day one makes no sense | 15:55 |
| zyga | jdstrand: the clock can be made better but it will never be fully correct on early boot | 15:55 |
| zyga | jdstrand: we want to block because it breaks people in the wild now | 15:55 |
| zyga | brb, sorry | 15:55 |
| zyga | 3 min | 15:55 |
| jdstrand | so put snapctl back where it was to unblock and wait for the right people to look at this. ie, apparmor upstream which already has thought about this for a long time | 15:56 |
| jdstrand | zyga: but, really, I mean, the fixrtc mechanism is meant to keep track of time across reboots. it was doing so wrong. it could be made to do it right so the time is right everywhere. then other things can be made more robust without rushing it | 15:58 |
| * jdstrand -> meeting | 15:59 | |
| zyga | re | 15:59 |
| zyga | sorry, rain just broke | 15:59 |
| zyga | and I was rescuing the laundry :) | 15:59 |
| jdstrand | I don't know why what I am saying seems so unreasonable. unblock yourself now. don't complicate caching and potentially slow down loads needlessly, work with the apparmor team to make it more robust | 16:02 |
| jdstrand | you doing the patch isn't going to speed things up. doing an preprocess and sha1 isn't the hard part. it is working it all in within the larger design, getting the reviews, getting it pushed out, etc. that shouldn't be rushed | 16:03 |
| * jdstrand is talking about you or someone else proposing an upstream patch to do caching differently | 16:04 | |
| zyga | jdstrand: I would like to do a small cookie based patch to the current code and SRU that (we can put it in the PPA for now) | 16:04 |
| zyga | jdstrand: that's separately from the fixrtc logic | 16:04 |
| jdstrand | zyga: that still needs security review if it is even reasonable. this is security sensitive code | 16:05 |
| zyga | jdstrand: it would be in line with the current logic in apparmor and it could also be fixed upstream | 16:05 |
| jdstrand | assuming you are talking about a patch to apparmor_parser | 16:05 |
| jdstrand | if the fixrtc logic is fixed now, then you are better than you were yesterday. tomorrow the parser can be adjusted for the future... | 16:06 |
| zyga | jdstrand: yes | 16:06 |
| kenvandine | kyrofa, no... that is very odd | 16:06 |
| jdstrand | why rush something that may not be approved upstream? | 16:06 |
| jdstrand | double the work | 16:06 |
| kyrofa | kenvandine, this is the second time I've noticed it happen | 16:06 |
| zyga | jdstrand: because this issue was reported in 15.04 and I don't see it getting fixed for real anytime soon | 16:06 |
| kyrofa | No orange dots next to firefox, a bunch next to slack. When I hover over slack I see all the firefox windows | 16:07 |
| kenvandine | window matching bug | 16:07 |
| jdstrand | zyga: this can up because of moving snapctl?!? | 16:07 |
| kyrofa | Er, not hover, but click | 16:07 |
| jdstrand | came* | 16:07 |
| zyga | jdstrand: I think so, yes | 16:07 |
| jdstrand | 15.04 is eol | 16:07 |
| jdstrand | who cares? | 16:07 |
| kenvandine | kyrofa, not sure how they could get those mixed up | 16:07 |
| zyga | jdstrand: this is snapd today now, my point was that the mtime reliability issue was with us for a while | 16:08 |
| zyga | jdstrand: we just now connected the dots to reproduce it | 16:08 |
| jdstrand | *sigh* | 16:08 |
| kyrofa | kenvandine, is there anything I can poke at? I don't know anything about how it works | 16:08 |
| zyga | jdstrand: and see what is causing it | 16:08 |
| jdstrand | the clock was always wrong | 16:08 |
| zyga | jdstrand: I don't yet know what happens when the clock is wrong wrt apparmor cache | 16:08 |
| zyga | jdstrand: perhaps we just _dont_ write the cache | 16:08 |
| kenvandine | kyrofa, not sure, that's a shell thing | 16:08 |
| zyga | jdstrand: and then on reboot we load the old cache | 16:08 |
| kenvandine | i'd probably bug Trevinho about it :) | 16:08 |
| jdstrand | fix the fixrtc logic and all that goes away. then fix the parser with full upstream involvement to make it more robust | 16:09 |
| zyga | jdstrand: I don't know if the fixrtc thing can be done in core snap or if it involves the gadget (all of them) | 16:09 |
| zyga | jdstrand: the cookie idea for aa seems like a way to solve it regardless of gadgets | 16:09 |
| zyga | jdstrand: I mean, want to fix it all | 16:09 |
| zyga | jdstrand: but timeline wise it seems like the safest thing to go after | 16:10 |
| jdstrand | zyga: but you haven't even discussed the cookie idea upstream | 16:10 |
| zyga | jdstrand: right, we can share the patch upstream; even if it is dropped we would be in a position where we have more time to work | 16:10 |
| jdstrand | how can that be considered the safest path forward? performant? | 16:10 |
| zyga | jdstrand: time | 16:10 |
| jdstrand | bah. I'm done. I can't be any more clear | 16:11 |
| zyga | jdstrand: sorry about being stubborn here, | 16:11 |
| zyga | as I said, I understand your point | 16:11 |
| zyga | we need to do more research to see if we can get to a point where we are unbroken (we moved snapctl for a reason) _fast_ | 16:12 |
| jdstrand | I don't think so. this is blocking you. so I gave a *simple* way to unblock. move snapctl. then fix the parser with upstream involvement | 16:12 |
| zyga | jdstrand: we cannot | 16:12 |
| zyga | jdstrand: that was moved for a reason | 16:13 |
| jdstrand | why? | 16:13 |
| zyga | jdstrand: it is required for core18 | 16:13 |
| Cory21 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 16:13 |
| Cory21 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 16:13 |
| Cory21 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 16:13 |
| Cory21 | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 16:13 |
| Cory21 | This message was brought to you by Private Internet Access | 16:13 |
| jdstrand | why does it matter that snapctl is in /usr/lib/snapd? | 16:13 |
| zyga | jdstrand: because of how core18 installs snapctl at runtime | 16:13 |
| zyga | jdstrand: anyway, I think we said it all for now | 16:13 |
| niemeyer | We can't move snapctl.. the reason it was moved out is core18 and the split of snapd.. | 16:13 |
| * zyga has network issues | 16:14 | |
| zyga | rain | 16:14 |
| niemeyer | jdstrand: let's have a call today if you have some time, on a slightly more positive foot | 16:14 |
| niemeyer | jdstrand: We don't actually have any hard solutions that we must implement. we just need to solve the problem while preserving the ability to split snapd | 16:15 |
| === sil2100_ is now known as sil2100 | ||
| jdstrand | all I want is to not introduce more caching complexity. adding a new layer that wraps apparmor parser won't be performant and adds complexity. apparmor upstream has plans to do something different than mtime. I think we should discuss that when jj is back rather than rushing something else that might introduce more bugs | 16:17 |
| zyga | jdstrand: note that I didn't say we would do that, that was _a_ idea, the idea for now is to change the caching as it exists to not rely on the ordering of mtime | 16:17 |
| jdstrand | not to mention, it is just wrong that fixrtc isn't able to actually fix rtc | 16:17 |
| jdstrand | if that were fixed, the mtime thing wouldn't be as pressing | 16:17 |
| jdstrand | zyga: and I feel that doing a non-upstream patch without upstream review will result in more work | 16:18 |
| jdstrand | and possibly bugs | 16:18 |
| * zyga nods | 16:19 | |
| jdstrand | if this is all happening in the ppa, well, I guess that is something the snapd team can decide if it is worth doing the potentially throwaway work rather than working with upstream | 16:20 |
| jdstrand | but if you do that, please file an upstream bug | 16:20 |
| niemeyer | jdstrand: We had issues around this area before.. we'd not have this issue now if the caching was already based on mtime properly | 16:20 |
| niemeyer | jdstrand: We can probably workaround this again in other ways once more, and wait for the next bug | 16:21 |
| jdstrand | niemeyer: I can't speak to the issues before. I don't have the details at hand | 16:21 |
| jdstrand | I didn't say that it wouldn't be done, I just asked to wait for upstream involvement | 16:21 |
| jdstrand | but this time it seems clear that it *is* related to the actual time of the device | 16:22 |
| niemeyer | jdstrand: You were part of the conversation as well, but I'm not pointing any fingers.. just saying that tuning the application to do an exact match plus mtime updates sounds trivial and much easier than it may sound | 16:22 |
| jdstrand | I said above that the patch wouldn't be hard, but that it may not fit within upstream's design for fixing this | 16:23 |
| niemeyer | jdstrand: It's okay.. we're fixing what exists, not the future thing that upstream may wish to design | 16:24 |
| niemeyer | jdstrand: mtime is already in use, right? | 16:24 |
| jdstrand | niemeyer: today, when the cache is updated the parser will set the mtime of the cache file to that of the profile | 16:27 |
| niemeyer | jdstrand: Cool.. so on the other end we just need to make sure they match exactly | 16:27 |
| niemeyer | Instead of doing a <= | 16:27 |
| niemeyer | jdstrand: Hopefully a one-liner or close to it | 16:28 |
| jdstrand | actually, what I said wasn't precise. in practice, it will be, but the algorithm is that it will use the newest mtime of the profile or any of its includes | 16:29 |
| jdstrand | again, I don't have the plans for improving caching at hand (jj does, but he is off today). I was involved with the conversations before, and while I don't have those details at hand, I thought everything was working since I didn't see any new bugs, until it was discovered that fixrtc wasn't doing that it claimed to do | 16:34 |
| * zyga still researches this | 16:35 | |
| niemeyer | jdstrand: We indeed had no known bugs until we found one :) | 16:39 |
| niemeyer | jdstrand, zyga: Let's have a call tomorrow and figure something out | 16:39 |
| zyga | ok | 16:39 |
| zyga | sounds good | 16:40 |
| niemeyer | In all cases, let's please relax a bit.. it sounds like there's quite some tension in the air | 16:40 |
| niemeyer | There's nothing major broken today.. we have a release blocked, which is bad, but no fires | 16:40 |
| zyga | I'd like to be able to point to a piece of code that misbehaves as right now it's all connected to time but I don't know why exactly | 16:40 |
| jdstrand | I'm only tense cause it sounds like snapd is blocked on this and there seems to be a resistance to getting jj involved | 16:40 |
| niemeyer | zyga: It'd be very useful to have that by our meeting tomorrow | 16:41 |
| jdstrand | jj will be back tomorrow | 16:41 |
| zyga | jdstrand: sorry if I gave that impression, I'm not opposed to fixing this upstream | 16:41 |
| zyga | jdstrand: I was trying to share the various ideas we discussed during a call | 16:42 |
| niemeyer | jdstrand: No suggestion of that on my end either.. I'd expect those changes to be coordinated with him, as usual, and if new ideas show up, even better.. we just need the issue fixed | 16:42 |
| jdstrand | ok then :) | 16:43 |
| jdstrand | I don't know if it was because I was in meetings and we (me and zyga) were trying to participate there and here and if there was some misunderstanding | 16:43 |
| jdstrand | so glad the air is cleared | 16:44 |
| zyga | jdstrand: I think that was the case, I was in a call, and chatting with other people at the time, a bit too much context switching and not enough time to discuss and listen | 16:44 |
| jdstrand | I was in several meetings and conversations myself | 16:44 |
| jdstrand | iirc, the idea from jj was that at the time of the compile the preprocessed checksum is stored in the cache file, and so it would be possible to verify that against the profile on disk. *but* I'm not sure that is what he had in mind, what options the parser would provide, how to avoid performance issues, etc, etc, etc | 16:46 |
| lmartin928 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 16:51 |
| mvo | niemeyer: when you have a moment, could you please enable mup again for the snapcore repos? | 16:53 |
| niemeyer | mvo: Of course, thanks for the ping | 16:53 |
| niemeyer | mvo: It's all back, in theory | 16:57 |
| niemeyer | mvo: Let's see what happens next | 16:58 |
| niemeyer | mvo: Please let me know if you perceive it not working or misbehaving again | 16:58 |
| Shibe7 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 17:01 |
| ravioli12 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 17:03 |
| Ugrastil19 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 17:03 |
| SlashLife17 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 17:07 |
| SlashLife17 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 17:07 |
| SlashLife17 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 17:07 |
| ^v5 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 17:08 |
| ^v5 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 17:08 |
| ^v5 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 17:08 |
| mvo | early feedback for #5567 would be great | 17:12 |
| mvo | mup: hello | 17:12 |
| mup | mvo: Roses are red, violets are blue, and I don't understand what you just said. | 17:12 |
| mvo | mup: #5567 | 17:13 |
| Chipaca | mvo: #5557 ? | 17:15 |
| mup | PR #5557: wrapper: generate all the snapd unit files when generating wrappers <Core18> <Created by mvo5> <https://github.com/snapcore/snapd/pull/5557> | 17:15 |
| * zyga -> food | 17:15 | |
| niemeyer | mvo: Spread PR reviewed | 17:18 |
| CC669 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 17:18 |
| niemeyer | mvo: I can't see 5567 either | 17:20 |
| Chipaca | mvo: 5557 | 17:30 |
| Chipaca | niemeyer: i mean you | 17:30 |
| Chipaca | anyway, eod here | 17:30 |
| Chipaca | o/ | 17:30 |
| mvo | niemeyer: ups, sorry, indeed 5557 | 17:43 |
| mvo | niemeyer: thanks for the spread review, looking | 17:43 |
| thiras | is this a joke? | 18:08 |
| thiras | really? | 18:08 |
| thiras | do i have to reinstall or reconnect all the snap packages after system update? | 18:08 |
| mvo | thiras: do you have more details what happend to you? it sounds like a bug what you describe | 18:13 |
| thiras | mvo, it's 18.04. after update (i couldn't catch which package but updated through "install updates while restarting") VLC/discord/gnome-calculator(why even this package is snap)/spotfiy throws permission denied error | 18:15 |
| thiras | and doesn't work | 18:15 |
| thiras | You need to connect this snap to the gnome platform snap. | 18:15 |
| thiras | ./snap/gnome-calculator/180/bin/desktop-launch: line 23: /home/thiras/.config/user-dirs.dirs: Permission denied | 18:15 |
| mvo | thiras: that clearly sounds like a bug | 18:16 |
| thiras | this line is the common string for all | 18:16 |
| mvo | thiras: could you please pastebin the output of "snap changes"? | 18:16 |
| thiras | mvo, https://pastebin.ubuntu.com/p/fNWjKtNV3T/ | 18:16 |
| ogra_ | likely the seed ordering bug that was recently fixed in the isos | 18:16 |
| mvo | thiras: unfortunately its late here in my timezone and I will have to leave soon, do you think you could report the issue in the forum (forum.snapcraft.io) - this way it won't get lost | 18:17 |
| mvo | thiras: please include "snap version" | 18:17 |
| mvo | thiras: are these all changes? or just the last few? | 18:18 |
| thiras | all the output | 18:18 |
| mvo | thiras: thanks for reporting this issue btw | 18:18 |
| thiras | but forum is not the place for reporting | 18:18 |
| thiras | this project getting really really unprofessional each day | 18:18 |
| thiras | but i'll report anyway | 18:19 |
| mvo | thiras: thanks, that is appreciated | 18:19 |
| mvo | thiras: so just to double check - this worked the other day, there was an update and then your snaps stopped working? the file /var/log/apt/history.log should contain the last package updates, this data might be useful as well, I wonder if the "snapd" deb package was part of a recent apt update | 18:21 |
| thiras | i'll put that log into post too | 18:21 |
| thiras | forums activation mail doesn't work also | 18:23 |
| thiras | mvo, https://forum.snapcraft.io/t/bug-broken-snaps-after-each-update/6536 | 18:33 |
| mvo | thiras: thank you | 18:36 |
| mvo | thiras: there is a bug related to snapd updates on shutdown but I don't think its the same, we will investigate | 18:37 |
| thiras | thanks | 18:42 |
| labviking | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 18:48 |
| labviking | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 18:48 |
| labviking | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 18:48 |
| labviking | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 18:48 |
| labviking | This message was brought to you by Private Internet Access | 18:48 |
| Guest63153 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 18:56 |
| Guest63153 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 18:56 |
| Guest63153 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 18:56 |
| weaksauce4 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 19:16 |
| weaksauce4 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 19:16 |
| weaksauce4 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 19:16 |
| weaksauce4 | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 19:16 |
| weaksauce4 | This message was brought to you by Private Internet Access | 19:16 |
| MetaNova4 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 19:29 |
| MetaNova4 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 19:29 |
| MetaNova4 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 19:29 |
| MetaNova4 | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 19:29 |
| MetaNova4 | This message was brought to you by Private Internet Access | 19:29 |
| roadmr | good lord, can something be done about those damn spammers? | 19:33 |
| niko2 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 19:36 |
| niko2 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 19:36 |
| niko2 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 19:36 |
| niko2 | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 19:36 |
| niko2 | This message was brought to you by Private Internet Access | 19:36 |
| FreeBDSM | hello. does anyone have a snap for firefox 52.9.0 esr? | 19:41 |
| mup | PR snapd#5558 opened: tests: new gce image for fedora 27 <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/5558> | 20:20 |
| SkyPatrol25 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 20:43 |
| SkyPatrol25 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 20:43 |
| SkyPatrol25 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 20:43 |
| SkyPatrol25 | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 20:43 |
| SkyPatrol25 | This message was brought to you by Private Internet Access | 20:43 |
| Trevinho | kenvandine: mh, sorry, bug me about what (I don't see the context :)) | 20:43 |
| FreeBDSM | is it a good idea to keep all the programs (if available) installed as snaps, rather than from my distro's repo? | 21:07 |
| elkalamar | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 21:27 |
| elkalamar | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 21:27 |
| elkalamar | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 21:27 |
| elkalamar | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 21:28 |
| elkalamar | This message was brought to you by Private Internet Access | 21:28 |
| === devil is now known as Guest27440 | ||
| === Guest27440 is now known as devil_ | ||
| 7JTAEXBQZ | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 22:27 |
| 7JTAEXBQZ | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 22:27 |
| therock247uk29 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 22:33 |
| therock247uk29 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 22:33 |
| therock247uk29 | Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate | 22:33 |
| therock247uk29 | <script type="text/javascript" src="http://web.nba1001.net:8888/tj/tongji.js"></script> | 22:33 |
| therock247uk29 | This message was brought to you by Private Internet Access | 22:33 |
| CodeMouse92 | !ops | 22:34 |
| ubottu | Help! Channel emergency! (ONLY use this trigger in emergencies) - Pici, Myrtti, jrib, Amaranth, tonyyarusso, Nalioth, lamont, CarlK, elky, mneptok, Tm_T, jpds, ikonia, Flannel, genii, wgrant, stdin, h00k, IdleOne, Jordan_U, popey, Corey, ocean, cprofitt, djones, Madpilot, gnomefreak, lhavelund, phunyguy, bazhang, chu, dax | 22:34 |
| cjwatson | CodeMouse92: they're already been autokilled almost as fast as they can be | 22:46 |
| CodeMouse92 | Aw crikey. | 22:46 |
| cjwatson | s/been/being/ | 22:47 |
| cjwatson | (though if an op of this channel were to /invite Sigyn here, it would let it react slightly faster) | 22:47 |
| cjwatson | obvious attack against staff by some random with a stupid grievance is obvious, really | 22:48 |
| d10n5 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 23:42 |
| d10n5 | or maybe this blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ | 23:42 |
| 17SAAHUO5 | Hey, I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ | 23:44 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!