=== braderhart is now known as Guest13679 === chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk === chihchun_afk is now known as chihchun [05:07] morning === chihchun is now known as chihchun_afk [06:07] interesting failure in unit tests: https://paste.ubuntu.com/p/PB3FhvyfT7/ === chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk === chihchun_afk is now known as chihchun [07:00] mvo: hey [07:02] mborzecki: hey, good morning [07:03] mvo: have you seen something like this? https://paste.ubuntu.com/p/PB3FhvyfT7/ [07:05] mborzecki: yes, that rings a bell - not sure though what is causing this, maybe some incorrect mocking, i.e. your device is already registered [07:07] mvo: happened once on travis, couldn't reproduce it locally, feels like some race between mock server being started and the actual code accessing it [07:07] mborzecki: yeah, that sounds likely === chihchun is now known as chihchun_afk === chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk [07:52] tests are super unhappy again [07:53] new failure in unit gccgo unit tests https://paste.ubuntu.com/p/NfYDvfwGdg/ [07:54] mborzecki: woah, strange. what distro/system is that? [07:54] Chipaca may be right that we're messing something up with fds in the tests [07:54] mvo: 2018-07-25 06:25:13 Error executing google:ubuntu-16.04-64:tests/unit/gccgo : [07:55] mvo: https://api.travis-ci.org/v3/job/407668444/log.txt full log [07:55] mvo: the auto import unit test failed there too [07:57] we didn't see this errors a lot before [07:57] wondering what changed [07:57] good good morning === chihchun_afk is now known as chihchun [08:01] mvo: hey, welcome back [08:01] I'm sorry for being late today, we were stargazing last night and ... well :) [08:01] we saw Mars, red and bright like a cherry on the sky :) [08:02] we also saw the ISS, I think, I never saw it before [08:02] it's very distinctive, quite a sight [08:04] it was also the last call as it's a rare sight over here and it's expected to be cloudy for the next week [08:08] zyga: hey hey [08:12] mvo: will you be publishing busybox-static for more arches? [08:12] mborzecki: yes, working on it as we speak [08:12] mvo: got it, great :) [08:13] mborzecki: its a bit of fiddling/clicking around in LP, but should be ready soon [08:13] embrace the API :) [08:14] cjwatson: *cough* I really should use it more [08:14] * mvo is sometimes a bit set in his ways [08:26] mvo: btw. i've seen apt-hooks test fail a couple of times on travis [08:27] mborzecki: yeah, this is what 5548 should fix [08:27] mborzecki: or rather, what do you mean with "fails" [08:27] mborzecki: do you have an example? [08:28] hrm, except that 5548 failed with the "MATCH" failure [08:28] :( [08:29] mvo: the output did not contain information about aws-cli snap [08:29] Hey guys, sorry to interrupt, just a stupid question. Is the $SNAPCRAFT_STAGE variable empty when building a "classic" snap? [08:29] mvo: unfortunately i didn't notice anyting specific in the debug log [08:30] mborzecki: ohhh [08:30] mborzecki: thats interessting, it might be a race with the update-catalog code [08:32] mborzecki: also grrrr @snapcraft, it insistend on having a linker in the base, I'm looking for a workaround now [08:36] I have a CMakeLists containing a $SNAPCRAFT_STAGE variable that works well when in a devmode (it takes the value the proper path) but remains empty when building it in classic [08:49] wondering if those connection refused errors may be caused by http server taking a longer while to start accepting on the socket, looking in stdlib there is no synchronization between the http.Server.Start() and actual http.Server.Serve(), so just calling http.NewServer() there's a chance that the if you do a request the server won't be waiting in Accept() yet [09:06] heh https://paste.ubuntu.com/p/Pw77jthXgQ/ [09:18] mborzecki: test-snapd-busybox-static should be ready on all arches now, might be easiest to just force push the base removal PR without the last commit [09:20] mvo: pushed and restarted the build [09:21] mborzecki: ta [09:30] mvo: hey [09:30] did you have a chance to look at the kernel->fixrtc->apparmor issue? [09:31] specifically I think we must do something about it before the release [09:31] we can either undo the snapctl change so that existing profiles remain compatible [09:31] we could change apparmor caching but that's a bigger (beyond snapd) work [09:32] I think we could also fix fixrtc (heh) [09:32] since it involves several subsytems it's going to be a coordination issue more than bugfixing issue === chihchun is now known as chihchun_afk [09:34] zyga: sure, let me check [09:34] mvo: there's a trivial tweak to fixrtc to use the more appropriate timestamp [09:35] mvo: and if we could somehow fix the shutdown process to write the last-write-time timestamp on shutdown/reboot we would not need any changes to apparmor [09:36] mvo: I can send you debdiffs [09:36] mvo: then those need review from ubuntu developers and need to be SRUd [09:36] mvo: the shutdown fix is more subtle, no idea how to fix that yet [09:37] mvo: we could (terrible terrible idea) hack something so that we write the last-write-time ourselves on core [09:37] zyga: we have our own shutdown C program [09:37] mvo: or perhaps instead of using last-write-time of the rootfs, instead look for mtime of a canary file in /writable [09:37] zyga: maybe that helps? [09:37] mvo: that would be an non-invasive way to fix it [09:37] mvo: yes, we could use it to touch /writable/.timestamp [09:38] and look at that from fixrtc [09:38] mvo: longer term we should ask the kernel team for opinion/fix (maybe just foundations) [09:38] zyga: lets fix fixrtc first, that seems a pretty obvoious one [09:38] mvo: do you have a pi at home? I didn't bring any here [09:38] mvo: note that fixing fixrtc alone won't solve the issue [09:38] zyga: I have a pi [09:38] but sure, let me start with that [09:39] excellent [09:39] zyga: let me look at our custom shutdown, maybe we can hook into that [09:39] mvo: can you boot your pi and just do a sanity check please [09:40] mvo: use "dumpe2fs -h" on the root filesystem [09:40] mvo: and pastebin the result [09:40] mvo: I'm also curious how we are special, my fedora system (using lvm and ext4) is not affected by that [09:40] mvo: so something works correct in this case (timestamps updated) but not on core [09:40] Chipaca: hi, did you get to anything debugging the connection refused issue? [09:41] mborzecki: I was able to reduce its incidence a lot, but not eliminate it, meaning that the changes I did were not finding the root cause, just making it less probable [09:41] Chipaca: tried this locally https://paste.ubuntu.com/p/Pw77jthXgQ/ [09:41] zyga: the remount -o ro - is that done by systemd? [09:42] Chipaca: but at times i got no panic and the http requests still failed with connection refused [09:42] Chipaca: stracing it now [09:42] mvo: I _think_ so, not sure if intrinsic part of systemd or just some service we have [09:42] mborzecki: Hmm. I'll add this to the changes I've done, let's see what gives [09:42] mvo: I just recall seeing that message [09:42] * zyga reboots [09:42] Chipaca: good morning! do you think we can merge 5531 (it got plenty of +1) or shall we ask for a gustavo review first? [09:46] so my fedora system is _not_ affected [09:46] trying with ubuntu now [09:46] mborzecki: cmd/snap/main_test.go:132: url.Hostname undefined (type *url.URL has no field or method Hostname) [09:46] mborzecki: i think i might be losing it [09:46] Chipaca: i'm using go 1.10.1 [09:47] mvo: I don't have a strong opinion either way; I think I described the pr in the standup on monday and he didn't flag it, fwiw [09:48] mborzecki: hah [09:48] Chipaca: url.Host instead? [09:48] mborzecki: so url.Host is host+port [09:48] yeah [09:48] Chipaca: ok, I leave it to you, we can either double check in the standup or just go ahead (easy enough to revert if needed) [09:50] mborzecki: ... value *os.PathError = &os.PathError{Op:"close", Path:"/tmp/check-4087160113880545930/34/var/lib/snapd/system-key.WWtDQKfmnsvk~", Err:0x9} ("close /tmp/check-4087160113880545930/34/var/lib/snapd/system-key.WWtDQKfmnsvk~: bad file descriptor") [09:51] Chipaca: right, seen this one too [09:51] mborzecki: WAT!!wat [09:51] that's wat^(wat^....^wat) wat times [09:52] Chipaca: close EBADFD - it got closed elsewhere, you have descriptor usage bugs? [09:52] zyga: it's in master :P [09:53] is golang protecting against close(-1) ? [09:53] zyga: this is runnig unit tests for cmd/snap on master [09:53] Chipaca: who would ever do that [09:54] zyga: I've got a PR that fails with one particular race every time, but the race is on master already [09:54] I think I described the issue yesterdya already [09:54] eh [09:54] new one: ... value *os.PathError = &os.PathError{Op:"read", Path:"/tmp/check-5611194952709039976/48/mountinfo", Err:0x9} ("read /tmp/check-5611194952709039976/48/mountinfo: bad file descriptor") [09:54] but, importantly, different [09:54] but, strangely, stll in auto_import [09:54] Chipaca: i see int quite often here [09:54] (where other bugs have been seen) [09:55] mborzecki: you see int? [09:55] Chipaca: yeah, int is all i see [09:55] so much int :) [09:55] isn't that an '80s ballad [09:56] jdstrand: hey, 4.18 kernel seems to be misbehaving wrt apparmor again: https://bugs.launchpad.net/snappy/+bug/1709155 [09:56] jdstrand: I recall you said something about that a few weeks ago [09:56] any bells? [09:56] Bug #1709155: Better error message for unsupported kernel [09:58] mborzecki: have you seen panics, with this change? [10:00] Chipaca: mmh, I think we EBADF errors somewhere else at some point, trying to remember where [10:00] Chipaca: twice [10:01] Chipaca: strace when aliases failes with connection refused [10:01] Chipaca: https://paste.ubuntu.com/p/NcVssQP7sY/ [10:02] * zyga cries a little wrt shell in fixrtc [10:03] pedronis: I recall (a long time ago) when we didn't realize some part of golang did dup(2) on a FD and we were leaking fds in the server [10:05] mborzecki: what're the openat … /json things? [10:06] zyga: my pi is unhappy right now, looking into it, but can't ssh into it right now [10:06] seems to be the AuthFile [10:06] that's okay [10:06] mvo: do you have a serial port adapter? [10:08] Chipaca: maybe because of successive connect attempts [10:13] Morning. [10:13] zyga: yes, it tells me the ip but no ssh, i may have used this to test disabling the ssh login :/ [10:13] mbeneto: what do you see if you run the tests under a 'ulimit -H -n 200'? [10:13] mborzecki: ^ [10:13] mbeneto: sorry :-) [10:13] mvo: ah, that bug :) [10:13] mvo: heh, well [10:14] Just want to confirm that only content snaps published by Canonical can be connected to by 3rd parties. [10:14] Wimpress: or snaps with an appropriate store side declaration [10:14] Yep. [10:14] In this case we have a new content snap we want 3rd parties to be able to use freely. [10:15] So publishing as Canonical is the way to go, right? [10:16] mborzecki: ok now i got a panic in httptest/server itself [10:16] Wimpress: to be clear Canonical is not special here in code, we setup a declaration either way [10:16] it's a policy question [10:17] Chipaca: in listener? [10:17] Wimpress: what content is that? will canonical maintain it? [10:17] Chipaca: err i mean when setting up listener? [10:17] mborzecki: https://pastebin.ubuntu.com/p/ryWkGQp3FD/ [10:24] zyga: dumpe2fs output https://paste.ubuntu.com/p/jzVHMqRJjP/ [10:25] Filesystem created: Wed Jan 10 21:18:38 2018 [10:25] Last mount time: Wed Jan 10 21:19:40 2018 [10:25] Last write time: Wed Jan 10 21:19:40 2018 [10:25] now reboot [10:25] and look again [10:25] if those don't change in a sensible way (they are already off IMO) it's confirmed [10:25] mvo: so in the early boot phase your device think it's still winter [10:26] mvo: I'm looking at vanilla ubuntu, fedora is certainly not affected somehow [10:26] mvo: classic ubuntu is also not affected [10:27] though let me triple check to be certain [10:27] I have a pi running raspbian as well, let me check that [10:28] zyga pedronis I've not actually been involved in assertions for content snaps. [10:28] ... value client.ConnectionError = client.ConnectionError{error:(*url.Error)(0xc82000a630)} ("cannot communicate with server: Post http://127.0.0.1:42000/v2/aliases: dial tcp 127.0.0.1:42000: getsockopt: connection reset by peer") [10:29] EVERYTHING IS TERRIBLE [10:29] Chipaca: stopped with dlv at a breakpoint in TestAliasesNoneFilterSnap [10:29] Chipaca: there don't seem to be many fds open [10:29] The content is the chromium-ffmpeg codecs. Will be published by Canonical since Canonical are the MPEG-LA license holder. [10:29] We have an initial browser vendor who will connect to the that snap. [10:29] mvo: interestingly raspbian _is_ affected [10:30] Filesystem created: Thu Jan 1 00:00:35 1970 [10:30] Last mount time: Sun Jul 22 19:02:57 2018 [10:30] Last write time: Thu Nov 3 17:16:45 2016 [10:30] but my dates are even more crazy than yours [10:30] last write time is equally bogus there [10:30] this machine is using a few snaps [10:30] I will remove them now to see if there is any relation to shutdown without snaps [10:31] Wimpress: you will need to ask jdstrand to setup the declaration [10:31] pedronis: Thanks, will do. [10:32] * zyga rebooted his raspbian pi [10:34] mvo: so last-write-time is unreliable in raspbian (somehow) but not on ubuntu (laptop) [10:34] I wonder if lack-or-presence of rtc is a factor here [10:35] I will look if I can capture the shutdown sequence somewhere [10:36] zyga: ta [10:37] zyga: fwiw, the downloaded base image I was using is from 10 Jan, so that might explain the date [10:37] zyga: I rebooted now but the last-write-time is still the same [10:37] yeah [10:37] you can replicate the issue with just loopback ext4 [10:38] just mount -o remount,ro [10:38] and then unmount [10:38] I think the proper fix for now is a canary file [10:38] we will need to patch shutdown scripts but that is ok [10:38] well, the shutdown _helper_ [10:38] and fixtrc in our ppa [10:39] mvo: the fixed fixrtc will restore clock based on the canary file [10:39] mvo: and that should be it really [10:39] then we pass this to foundations to untangle and fix properly with SRU candence [10:39] mvo: I think that's the best case scenario for now [10:41] mvo: I wonder if any other magic bugs can be attributed to devices waking up and thinking time is standing still [10:42] mvo: also it would be interesting to have a core device metric with (has-hardware-rtc, has-working-ntp) [10:44] zyga: it looks like this is somewhat specfic to core, on ubuntu my last-write-time looks vageuly ok [10:44] zyga: so maybe the best case is we fix this? [10:46] mvo: I agree but I also think there is something missing [10:46] is the remount ro specific to core? [10:46] is that a workaround for broken shutdown from 15.04 [10:46] and we just carry it? [10:46] zyga: I think that is what we need to find out [10:46] yeah, [10:51] zyga@pi3-1:~ $ journalctl --list-boots [10:51] -1 5a859aca24244b1aa150eadb4461c434 Wed 2018-07-25 10:49:49 UTC—Wed 2018-07-25 10:49:55 UTC [10:51] 0 a78a995e1b794fc3829e3ff69c90cc92 Thu 2016-11-03 17:16:45 UTC—Wed 2018-07-25 10:50:47 UTC [10:52] enabling persistent journal shows interesting dates [10:54] zyga, abeato has a patch for canary file support in fixrtc we havent merged [10:54] ogra_: oh, nice, do you have a link for that? [10:54] ... should consider using that one ;( [10:54] err [10:54] ;) [10:54] * abeato searching... [10:54] not off the top of my head, thats why i pinged abeato ;) [10:55] thanks abeato [10:56] hmm, interesting ... i'm doing a snap build, but snapcraft clean fails with permission denied (files of the build in prime/ are root only apparently) i wonder how thats possible given i didnt use sudo or anything [10:58] mvo: so interesting, there's a systemd unit [10:59] Jul 25 10:55:19 pi3-1 systemd[1]: Stopped Remount Root and Kernel File Systems. [10:59] it runs just before actual shutdown, inspecting it now [11:00] systemctl cat systemd-remount-fs.service [11:00] thats an upstream one AFAIK [11:01] that's just done on startup [11:01] * zyga keeps looking [11:01] (this is a non-issue) [11:02] Jul 25 10:55:19 pi3-1 systemd[1]: Failed to set timeout to 600s: Invalid argument [11:02] Jul 25 10:55:19 pi3-1 kernel: systemd-shutdow: 26 output lines suppressed due to ratelimiting [11:02] I need to find the rate limit sysctl toggle [11:03] printk_ratelimit [11:04] that's in sysctl.conf? [11:05] in sysctl.conf it is kernel.printk_ratelimit=X [11:05] (X is seconds, default is 5) [11:06] zyga, ogra_ this was the additional initrd script I used in some projects: https://paste.ubuntu.com/p/mNw7qKpbyx/ [11:06] ok rebooting [11:07] abeato: thanks [11:07] it looks for dates on some files, when the mount date was not really reliable [11:07] np [11:08] it sadly makes everything snap specific ... [11:08] so nothing for upstream [11:09] right, although it can be modfified easily [11:09] yeah [11:10] ogra_: fixrtc is not upstream [11:11] Chipaca: heh, it stopped reproducing now [11:12] mborzecki: on updated master? [11:12] ha [11:12] Chipaca: are you on 16.04? [11:12] zyga: yes [11:12] Chipaca: no, the same code, just stopped appearing [11:12] Chipaca: can you run: lsblk -a and find your rootfs ext4 [11:12] Chipaca: then please run sudo dumpe2fs -h /dev/sdaX [11:12] and look at the three timestamps and paste them here [11:12] mvo: maybe it depends on systemd version [11:13] and is fixed in bionic / fedora 28 [11:13] but not in xeial [11:13] *xenial [11:13] and not in my old raspbian [11:13] zyga: i can do that on encrypted and non-encrypted / [11:13] Chipaca: non-encrypted please [11:13] though should not matter as ext4 doesn't see the encryption layer [11:13] zyga: i mean, i have both, on separate machines (obviously) [11:13] and this is an ext4 superblock feature [11:14] zyga: http://paste.ubuntu.com/p/vqFY25f3zs/ [11:14] ha [11:14] Filesystem created: Sat Feb 24 17:42:06 2018 [11:14] Last mount time: Fri Jul 13 21:11:46 2018 [11:14] Last write time: Fri Jul 13 21:11:46 2018 [11:14] is this machine up since the 13th of July? [11:16] zyga: probably yes [11:16] hmmm, would you be ok rebooting it? [11:16] zyga, ubuntu upstream [11:16] zyga: up 11 days [11:16] ok [11:16] zyga: http://paste.ubuntu.com/p/3KghN9nj7v/ [11:16] zyga: that one's up 8 days [11:16] zyga: I don't recall knowing anything about https://bugs.launchpad.net/snappy/+bug/1709155/comments/4 [11:16] Bug #1709155: Better error message for unsupported kernel [11:16] so somehow classic xenial is not broken [11:16] jdstrand: thank you [11:17] zyga: I would be ok rebooting it if needed [11:17] unless you never rebooted before it should be consistent [11:18] zyga: i've rebooted lots of times [11:18] just not in the past few weeks [11:18] ok [11:19] jjohansen: hey, the bug description is wrong, but starting at comment #4, people are complaining about "cannot perform readlinkat() on the mount namespace file descriptor of the init process" in 4.18. this reminds me of https://bugs.launchpad.net/apparmor/+bug/1656121, but people aren't talking about complain mode [11:19] Bug #1656121: unexpected errno=13 and disconnected path when trying to open /proc/1/ns/mnt from a unshared mount namespace

[11:19] [11:19] jjohansen: I've asked for more info (in particular, denials). does the 4.18 have the sauce patches? [11:20] jjohansen: actually, seems most how are complaining are talking about mainline/upstream [11:20] s/how/who/ [11:21] * Chipaca ~> lunch [11:25] mvo: raspbian uses "fake-hwclock" [11:27] but it runs in mid-early userspace (not in intrd) [11:29] mborzecki: merged master insto the green cehck mark and didn't get the connection error this time (got others instead) [11:29] now running tests in a loop with the race detector on [11:29] we'll see [11:29] * Chipaca ~> really lunch [11:30] Chipaca: btw the advise tests do direct FD manipulations in goroutines [11:30] there might be something there [11:30] pedronis: ack [11:33] zyga, yeah, that wont help with what initially triggered the fixrtc creation ... ("mount time is in the future yadda yadda, boom") [11:33] mhm [11:34] zyga, though perhaps combining both might work [11:34] I'm trying to understand why last-write-time is bad on raspbian [11:34] and not on ubuntu [11:34] but that will indeed result in jumpy clock [11:34] is it caused by some systemd shutdown logic [11:34] s/jumpy/more jumpy/ [11:35] are we using systemd in the initrd in classic ubuntu? [11:36] AFAIK fedora does now [11:36] no [11:36] there were attempts from ... hmm, how was he called ... james something [11:36] he spent 6 months on it and gave up [11:36] i'd actually love if we could perhaps go the android approach one day ... [11:37] initrd as rootfs ... and just mount all subdirs from disk ... i.e. no rootfs pivoting at all [11:37] that would surely work with systemd ... [11:37] migrating todays initramfs logic will be a lot harder [11:38] (especially since there are tons of mudular bits that get dynamically added or removed ... every package can enhance the initrd) [11:39] Chipaca: I see some code similar to the one we had problem with here: ac0523dadf923f4dc6bb [11:40] ogra_: james hunt? [11:41] jdstrand, ah, yeah, him ! [11:41] ok, so somewhat closer now [11:41] I'm looking at source of part of systemd [11:42] mborzecki: hey, did you implement timers? if so, fyi, https://forum.snapcraft.io/t/system-options/87/12 [11:42] Chipaca: mborzecki: I think there's a chance the issues come from https://github.com/snapcore/snapd/pull/5122 [11:42] PR #5122: snap: add support for `snap advise-snap --from-apt` [11:42] s/the issues/some of the issues/ [11:43] mborzecki: but the reason I mention it is to ask if they are expected to work on 14.04? it seems it doesn't support 'systemctl list-timers' and though the timer file is generated, it doesn't ever seem to fire [11:44] mborzecki: it is fine (with me anyway) if it isn't supported, but seems like something that should be documented. maybe it is and I just don't know where [11:45] jdstrand: i'm aware of list-timers not working, but timers as such should likely work on 14.04, unless systemd is that old there [11:46] jdstrand: well, it's possible to enable a timer unit and start it [11:47] mborzecki: it has 229-4ubuntu21.2. I used 'timer: 0:00-24:00/24', but it never fired after reboot afaics. it worked as expected on 16.04 [11:48] mborzecki: I could start it. I thought I even enabled it... but it wouldn't fire hourly [11:49] jdstrand: journalctl -u *.timer ? [11:50] mborzecki: it's gone now. I guess I can create bug/topic. which do you prefer? [11:50] jdstrand: let's do a topic first [12:05] * zyga -> walk, see you at the standup [12:06] mborzecki: ok, it seems the problem was actually with the logging to rsyslog on 14.04 [12:07] phew :) [12:07] mborzecki: I created a timer that fires every minute and updates a file in SNAP_COMMON and echos to stdout. the file is updated, rsyslog is not. on 16.04, I see both [12:08] mborzecki: journalctl shows it. so this is configuration issue with logging [12:09] as it happens, the snap that I was using was specifically trying to generate a syslog entry [12:09] I could use logger, etc. but between that and list-timers, I thought there was an issue. but there isn't [12:10] well, at least we won't have to debug systemd [12:10] jdstrand: btw. there are some timer related properties if you do systemctl show foo.timer [12:10] mborzecki: I was surprised to see how long the timer file was for the pathological 00:00-24:00/1440 [12:10] jdstrand: but i don't recall if those are present in older releases [12:11] jdstrand: yes, our syntax is more expressive, we could do some optimization and be smarted about generated schedules [12:11] * jdstrand nods [12:12] ok, thanks! [12:13] hm something weird about how snaps run in amazon linux [12:16] zyga: /etc/os-release inside a snap should come from the core snap right? [12:29] Yes [12:29] But by accident [12:29] Because it is a slimlink [12:30] On typical etc [12:30] That goes to /lib [12:34] jdstrand, hi, I think this MP is waiting for your review: https://github.com/snapcore/snapd/pull/5469 [12:34] PR #5469: interfaces/apparmor: (un)load profiles in one apparmor_parser call [12:37] Chipaca: anything came up with the race detector? [12:38] mborzecki: just one minor thing in cmd_wait_test, so far [12:38] mvo, did you see this one? [12:38] https://travis-ci.org/snapcore/snapd/builds/407953474#L5654 [12:38] (cmd_wait_test uses a rather funny way of doing the check it does, fwiw...) [12:38] i might just rewrite it out of spite [12:40] mborzecki: infuriatingly, twice i've hit the 'connection refused' error with no race detection [12:40] heh [12:41] Chipaca: also funny how even a tiny change in code makes it disappear to pop up in another test [12:42] jdstrand: there are no sauce patches in ubuntu or upstream around mount [12:42] mborzecki: another one that I hit frequently is the 'queueing for later' (which is just another connection error) [12:42] Chipaca: yup, same here [12:43] the only patch missing upstream is af_unix [12:43] Hello, after running df, I am seeing this: /dev/loop1 640 640 0 100% /snap/gnuchess/9 Could you please explain what it is and how can I unmount it to free some space? [12:44] ok, interestingly the snaps with commands that use relative paths, eg. ../../../bin/sh fail on amazon linux, it's clearly doing something wrong with the paths and calling eg. /var/lib/snapd/bin/sh instead of /bin/sh [12:44] zyga: any clues ^^ ? [12:45] mojtaba: why would just unmounting it free some space? [12:45] mojtaba: but, to answer your question with an explanation, look at the output of "snap list --all gnuchess" [12:45]