[00:00] Chipaca: I have to reimage the Pi, and then I can try to relogin with the new key's. I appreciate your help, and especially not leaving me hanging. [00:03] newbsie: how long does reimaging take? [00:04] ok [00:04] I think I'm done for tonight [00:06] Just done, the big moment [00:06] Chipaca: can you look at https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-zesty-snappy-dev-image/zesty/amd64/s/snapd/20170511_000322_edb35@/log.gz [00:06] ... Panic: sync: unlock of unlocked mutex (PC=0x45BDAC) [00:07] this looks rather bad [00:07] zyga [00:07] zyga: sorry, what is known_hosts? [00:07] yes? [00:07] newbsie: it's a file where ssh remembers the hosts it saw and their keys [00:08] zyga: that one's for pedronis :-) [00:08] Chipaca: also here https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial-snappy-dev-image/xenial/i386/s/snapd/20170511_000455_63ba2@/log.gz [00:08] I'm now getting "Host key verification failed" [00:08] Yes, yes, yes! [00:09] newbsie: becausethey ip you are sshing had changed identity [00:09] newbsie: you reimaged, so the host key changed [00:09] It's working now. [00:09] newbsie: wooo [00:09] That only took a day, but the more you put in, the more you feel like you accomplished, although I stood on the shoulder of zyga and Chipaca. XD [00:10] *that's* why i'm so sore! [00:10] we ran out of machines [00:10] newbsie: let's make it suck less. Looking forward to the forum post. [00:10] Chipaca: I'm not that heavy. Although I do like burgers, steak and fatty food in general. [00:10] zyga: la la la can't hear you i'm already asleep [00:11] Chipaca: yes, good idea [00:11] * zyga afk [00:26] zyga: I'm going to log off too, but I really appreciate your help earlier. Will definitely make that post on the snapcraft forum. === chihchun_afk is now known as chihchun [05:45] PR snapcraft#1309 opened: meta: read and write the desktop file with utf-8 encoding [05:51] PR snapcraft#1310 opened: tests: use C.UTF-8 for the docker locale [06:27] PR snapd#3293 closed: spread.yaml: increase linode workers by 50% [06:28] PR snapd#2869 closed: interfaces/builtin: add online-accounts-service interface [07:04] zyga: ping [07:05] zyga: is https://paste.ubuntu.com/24553247/ expected? [07:05] that is with 2.25 on suse [07:14] morning [07:14] good morning mvo :) wrt 2.25 SRU and bug #1689332 if we don't promote the ubuntu-core snap shipping 2.25 (2109 for amd64, currently in beta) we are good to go right? [07:14] Bug #1689332: internal error with any command using ubuntu-core snap on classic [07:14] hey pstolowski [07:14] fgimenez: good morning. yes, I think so, we just freeze ubuntu-core from now [07:15] PR snapd#2941 closed: interfaces/builtin: add network-status interface [07:16] mvo: ok, shall i put a comment in the sru bug and mark it as verification-done? we only found this issue [07:16] fgimenez: yes, please do, I will revert beta ubuntu-core to stable so this issue will be gone [07:17] mvo: ok thanks! [07:17] fgimenez: hm, actually it won't be gone :/ we need to land 3289 first [07:19] mvo: do you know if the message shown with https://paste.ubuntu.com/24553247/ is expected with 2.25? this is on suse so with forced-devmode [07:19] mvo: aha ok, this won't be added to the current 2.25, will it? i mean, do we need to wait until 2.26? [07:21] PR snapd#3296 opened: image: fix go vet issue [07:21] morphis: that is something for zyga - but I got feedback on the sprint about this as well, quite a few people see this or a variation of it. iirc we added the message as a measure of debugging this, we always had this issue but silently ignored it. [07:21] fgimenez: correct, for 2.25 we can do little without quite a bit of extra effort (doing a point release, re-doing the verifications) [07:22] mvo: ack thanks [07:22] morphis: the message also looks slightly odd, lets wait for zyga [07:23] 3289 needs a second review fwiw, but I guess its a bit early still [07:43] mvo: still about 2.25, pls correct me if i'm wrong but even if snapd#3289 hasn't landed, if the current ubuntu-core stable is available in all the channels then if snapd still allows installing it it wouldn't break anything? sorry maybe i'm missing something [07:43] PR snapd#3289: daemon: do not allow to install ubuntu-core anymore [07:48] morphis: known issue, thank you for reminding me [07:49] pstolowski: hey, just a reminder, you didn't add the pre-attribute definers to the test [07:49] pstolowski: I added them already but I thought you would in your attribute branch [07:49] zyga: actually feels bad to have that, something we can easily fix for 2.25? [07:49] morphis: yes [07:49] morphis: the patch will remove two lines [07:50] I will be back shortly, I worked too late last night [07:51] fgimenez: lets try :) [07:51] zyga: if you have a fix in mind, would be great to get it for 2.26 [07:51] zyga: please ping when you are back so that we can talk details [07:52] zyga: just one last quick question: why do we have all the %dir /var/lib/snapd/.. lines in the spec file for snapd on suse? Isn't it enough to one for /var/lib/snapd and snapd will take care to create all sub dirs when necessary? [07:56] fgimenez: you are right, things work fine with ubuntu-core from stable it seems [07:59] fgimenez: so no blocker, all good. thanks for this info! [07:59] zyga, sorry about that, it didn't occur to me [08:00] mvo: np thank you! i've double checked that installing 1797 (current stable) from edge works fine too [08:02] morning all [08:02] re [08:02] hey Chipaca, good morning [08:03] mvo: http://paste.ubuntu.com/24553441/ i'll put a comment on the sru bug about this and mark it as verification-done, is that ok? [08:03] hey Chipaca [08:03] pstolowski: I made a remark in your PR before it was merged, no worries now but we need to be careful, tests will not always show us that stale code is in place [08:03] Chipaca: maybe interessting for you - shellcheck in zesty gives me the following output http://paste.ubuntu.com/24553330/ [08:03] mvo: yes, I'll prepare one shortly [08:03] fgimenez: yes [08:04] oooohhh [08:04] oooooooooohhhh [08:04] oh [08:04] sigh [08:04] read *with* -r misbehaves on 14.04 [08:04] zyga: great, please push it to the top of the todo, its the last blocker for the release (well, not blocker but really nice to have) [08:04] Chipaca: meh, then we can just turn this warning off :) [08:04] or was that only when combined with a timeout [08:04] mvo: no, this might be the reason for a bug we've got :-) [08:04] Chipaca: ohhhh [08:05] Chipaca: interessting [08:05] that's what I said :-) [08:05] lol [08:05] * mvo hugs Chipaca [08:05] * Chipaca hugs mvo right back [08:05] zyga: and very excited that you have a fix almost ready, great news :) [08:05] if my memory serves (for once), it only misbehaves when combined with a timeout [08:05] and we don't do that, so this should be good [08:05] I'll give it a try [08:05] mvo: thanks! [08:06] mvo: wrt the last two warnings, I'll add a disably thing [08:06] pstolowski: wooooh, 3119 landed! great job [08:07] Chipaca: thank you [08:07] mvo, yeah :} [08:07] pedronis: and good news for you as well, the new number of machines increased and a PR takes currently 30min for a full test run [08:07] (down from ~48min) [08:08] mvo: doing that now [08:11] mvo: https://github.com/snapcore/snapd/pull/3297 [08:11] PR snapd#3297: overlord/ifacestate: don't spam logs with harmless auto-connect messages [08:11] Son_Goku: did you test if the removal of the snapd package on fedora removes everything from /var/lib/snapd? [08:11] PR snapd#3297 opened: overlord/ifacestate: don't spam logs with harmless auto-connect messages [08:11] PR snapd#3298 opened: interfaces/builtin: ensure we don't register interfaces twice [08:11] mvo: I might be wrong, but I blocked snapd#3290 [08:11] PR snapd#3290: add support for `snap install foo --channel=3.4` [08:12] mvo: just 'cause i'm ornery [08:12] Chipaca: meh, branches [08:12] mvo: can you consider https://github.com/snapcore/snapd/pull/3282 as well? [08:12] PR snapd#3282: hooks: default timeout [08:13] mvo: basically missing a “and !strings.ContainsWotsit(mx.Channel, '/')” [08:13] Chipaca: I have a look after 2.26, maybe its not so un-ambigious as the doc made me believe [08:13] Chipaca: aha, ok [08:13] pedronis: did you see those panics? (are you awake? :-) ) [08:13] pedronis: did you see [08:13] PANIC: snapstate_test.go:4416: snapmgrTestSuite.TestEnsureRefreshesAlreadyRanInThisInterval [08:14] Chipaca: yeah, that is making sense [08:14] ... Panic: sync: unlock of unlocked mutex (PC=0x809A45A) [08:14] zyga: HAH! beat you to it [08:14] mvo: ^^ (I'd also recommend to hold the release until we know what that is) [08:14] zyga, Chipaca: where are those? === arubislander1 is now known as arubislander [08:14] mvo: in tests, randomly, very often [08:14] mvo: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial-snappy-dev-image/xenial/i386/s/snapd/20170511_000455_63ba2@/log.gz === arubislander is now known as Guest22573 [08:14] for instance here: https://github.com/snapcore/snapd/pull/3051 [08:14] PR snapd#3051: interfaces: add consoles interface [08:15] Chipaca: did you talk to pedronis already? [08:15] ah [08:15] I see :D [08:15] :) [08:15] * zyga is not awake yet [08:15] * Chipaca offers some really nice coffee [08:16] Chipaca: yes, I should get some [08:16] Chipaca: after working late I could not sleep [08:16] zyga: happens [08:16] Chipaca: so I went on reading, then my son woke up because he wasn't feeling well [08:16] Chipaca: so more stuff at 3AM [08:16] * zyga is in Z state today [08:17] (but as we saw with the ubuntu release name, after Z, there A) [08:17] mvo: wrt the channel, i think it's unambiguous wrt lack of / [08:17] Chipaca: cool [08:23] zyga, hey, where can I find a list of what should be enabled in the kernel for UC? [08:23] mvo: sorry for spamming here but: [08:23] /usr/lib/go-1.6/src/runtime/panic.go:443 [08:23] in gopanic [08:23] /usr/lib/go-1.6/src/math/rand/rand.go:64 [08:23] in Rand.Int63n [08:23] /usr/lib/go-1.6/src/math/rand/rand.go:203 [08:24] in Int63n [08:24] /tmp/autopkgtest.odhoEa/build.iw1/snapd/_build/src/github.com/snapcore/snapd/timeutil/schedule.go:106 [08:24] in randDur [08:24] mvo: does that say that Rand.Int63n panics? [08:24] abeato_: no idea [08:24] abeato_: sorry :/ [08:24] abeato_: I don't think anyone maintains a list [08:24] abeato_: we only have "clues" [08:24] zyga, nw, the thing is that I remember having seen some sort of "bits" files that gave some clues [08:25] zyga: DEBUG: host arch (kernel) is '1073741864' [08:25] Chipaca: yes? [08:25] Chipaca: that's seccomp's way of describing architectures [08:25] zyga: is that a pointer getting printed as a string? [08:25] (yuck but expected) [08:26] no, that's an opaque thing [08:26] yuck [08:26] yep [08:26] Chipaca: I think there are some flags inside there too [08:26] anyway [08:26] Chipaca: any insight into that trace above? [08:27] aha [08:27] if n <= 0 { [08:27] panic("invalid argument to Int63n") [08:27] } [08:27] so rand *can* panic [08:27] mvo: I think this is the bug: [08:27] zyga, of these which should be included? : http://paste.ubuntu.com/24553511/ [08:27] func randDur(dur time.Duration) time.Duration { [08:27] return time.Duration(rand.Int63n(int64(dur))) [08:27] } [08:27] mvo: you convert this to int64 which is signed [08:27] abeato_: I really don't know [08:27] zyga: the panics very much say it's about locking something twice [08:28] Chipaca: locking is another issue (perhaps) [08:28] zyga, ok ok :) [08:28] Chipaca: but this is IMO wrong too [08:28] ah [08:28] sorry [08:28] where's that coffee i talked about [08:30] mvo: and then this: [08:30] when := a.Sub(now) + randDur(b.Add(-5*time.Minute).Sub(a)) [08:30] here we can call randDur with a negative duration [08:30] when this happens we panic [08:30] * zyga -> coffee [08:31] looks like that needs to be b-a-5m only if b-a>5m [08:32] Son_Goku: ok, remoable doesn't really work for me with your snapd package in testing [08:32] mvo: zyga: i'll fix this :) [08:32] thank you! [08:33] * zyga thinks that with snapd architecture growing we could consider some better error resilience, [08:33] like not crashing the whole damn thing at once [08:40] mvo: btw. I saw there is a name change in the vendor release tarball for snapd, 2.24 has vendor.orig.tar.xz and 2.25 has vendor.tar.xz [08:40] mvo: is that intentional? [08:42] morphis: sort of, I have a script now (finally) that is doing all this, so pick a name that you like best and from that point on it will be always the same [08:42] zyga, Chipaca: thank you [08:43] a second review for 3297 would be great [08:43] * zyga looks [08:43] aww, drat [08:43] I cannot review my code [08:43] pstolowski: ^^ [08:44] mvo: something you suggested https://github.com/snapcore/snapd/pull/3298/files [08:44] PR snapd#3298: interfaces/builtin: ensure we don't register interfaces twice [08:47] mvo: great, I am fine with both names, just wondered that is has changed with 2.25 [08:51] zyga: thank you! [08:51] morphis: ok, I will leave the 2.25 name then [08:52] rharper, a safe bet is snap_core or snap_kernel from /proc/cmdline [08:52] morphis, zyga I think you have seen this before: http://paste.ubuntu.com/24553598/ . Was there a kernel patch for this? [08:55] abeato_: possible [08:55] it appeass here: https://bugs.launchpad.net/apparmor/+bug/1656121 [08:55] Bug #1656121: unexpected errno=13 and disconnected path when trying to open /proc/1/ns/mnt from a unshared mount namespace

[08:55] [08:55] PR snapd#3297 closed: overlord/ifacestate: don't spam logs with harmless auto-connect messages [08:56] abeato_: so we added support for /proc/*/ns/mnt with the single patch we imported into the 3.4 kernel, I fear we need a bit more than that [08:57] morphis, hmm, I see [08:58] PR snapd#3299 opened: timeutil: avoid panicing when the window is very small [08:58] zyga: mvo: ^ [08:59] mvo: what're you up to? [08:59] mvo: (want me to fix snapd/#3290 for you?) [08:59] snapd/3290 [08:59] snapd/#3290 [08:59] mup: OI! [08:59] Chipaca: Unknown commands are unknown. [08:59] snapd#3290 [08:59] PR snapd#3290: add support for `snap install foo --channel=3.4` [08:59] sigh... [08:59] it's going to be a long day [09:00] PR snapd#3296 closed: image: fix go vet issue [09:00] Chipaca: looking! [09:01] Chipaca: yay, thank you! [09:01] abeato_: that probably tells you that /proc/1/ns/mnt does not exist [09:01] abeato_: is this a pre 3.8 kernel? [09:01] Chipaca: we should really practice commit messages [09:01] Chipaca: you cannot return 0 AFAIR [09:02] zyga: why not? [09:02] Chipaca: that function panics on zero too [09:02] which function [09:02] ah, rand.Int64n [09:02] Chipaca: /usr/share/go-1.6/src/math/rand/rand.go on line 64 [09:02] 63n [09:02] * [09:02] yp [09:02] yep [09:02] <= then? [09:02] random of 0 is pretty non random :D [09:03] zyga, it does exist, it is 3.4 + ns patch, but maybe we miss something still [09:03] fo sho [09:03] well, you cannot return 0 :) [09:03] zyga: why can't i return 0? [09:03] abeato_: try stracing [09:03] Chipaca: ahh [09:03] zyga: the rand call is in the function i'm returning from [09:03] Chipaca: sorry, I misread that [09:03] Chipaca: yes, that's ok [09:03] Chipaca: I thought this returns the input to randint, [09:03] zyga: still need to change it to a <= [09:03] but it's not :) [09:04] adding a test [09:05] tested, fixed, pushed [09:05] * zyga looks at http://radio.garden [09:05] Chipaca: thank you! [09:06] Chipaca: ideally we would simply not allow windows smaller than 30min [09:06] Chipaca: ? [09:07] pedronis: look at the logs zyga pasted (with a double mutex lock panic) [09:07] that's refresh.schedule stuff [09:07] pedronis: good morning :-) [09:07] pedronis: sorry, i thought they were all the same things [09:07] ? [09:07] pedronis: but the one that's for you is the mutex double lock one [09:07] let me get the right one for you [09:08] pedronis: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-zesty-snappy-dev-image/zesty/amd64/s/snapd/20170511_000322_edb35@/log.gz === JanC is now known as Guest55043 === JanC_ is now known as JanC [09:08] not lock, unlock [09:08] w/e :-) [09:09] Chipaca: well we have some codes that can hit that [09:09] is anybody else getting a "los glaciares national park" google doodle, or is google being extra creepy with me? [09:09] ? [09:09] Chipaca: no [09:09] Chipaca: on google.co.uk? [09:10] yes [09:10] (also in the chrome startup page) [09:10] Chipaca: check out that radio thing, it's amazing :) [09:11] mvo: so just make the schedule parser return an error? [09:11] zyga, morphis http://paste.ubuntu.com/24553660/ It seems that /proc/1/ns/mnt is not a link, but looking at my local machine it seems it should be... [09:11] abeato_: aha, I know what's going on [09:11] Chipaca: either that or just enlarge it to 30min [09:11] abeato_: you found a bug :) [09:11] Chipaca: but I guess for now this fine [09:11] lol [09:11] abeato_: on older kernels it was not a link [09:11] abeato_: hmm hmm hmm [09:12] Chipaca: we can do a followup [09:12] abeato_: let me wake up and think about this [09:12] abeato_: interesting [09:12] Chipaca: I want 2.26 out :) so followup [09:12] abeato_: but if you can give me a way to run this locally to experimen [09:12] abeato_: I would appreciate that [09:12] abeato_: as a quick work-around do this: [09:12] zyga, hmm... maybe I can give you remote access to the board [09:13] abeato_: comment-out the call to sc_reassociate_with_pid1_mount_ns in snap-confine.c [09:13] abeato_: that's good too [09:13] abeato_: perhaps we can discuss this on a private channel [09:13] Chipaca: what's the question, the test does Unlock, and will not relock under an unexpected panic [09:13] zyga: you have any reference to resources who talk about ns/mnt being changed to a symlink= [09:13] zyga: this really means we're still missing patches [09:13] Chipaca: I mean around Ensure [09:13] morphis: no, it's just changed in a future kernel version [09:14] morphis: it's even documented in namespaces(7) [09:14] pedronis: i guess the question is "can this be fixed so it doesn't fail at random", with a side of "is this a bad test or can this actually happen" [09:15] Chipaca: it's a test trying to not get too fussy [09:15] Chipaca: there is nothing deep going on here [09:15] just read the code [09:21] PR snapd#3300 opened: cmd/snap-confine: don't fail on pre 3.8 kernel [09:23] mvo: should i fix the --channel one for you? [09:24] mvo: or do we bump it [09:24] Chipaca: isn't that fixed, I thought I pushed a fix [09:24] ah maybe i need to refresh [09:25] Chipaca: short story, we should in theory never use lock/unlock pairs without defer, in practice we trade that for readability sometimes under the assumption that what is enclosed shouldn't panic or we are in a test [09:25] mvo: doesn't look fixed [09:25] mvo: only one commit [09:26] ohhhhh [09:27] pedronis: this might actually be the same bug then :-) [09:27] pedronis: i mean, this panic happens because the ensure panics [09:27] and the defer then kicks in and panics again? [09:27] Chipaca: yes [09:27] and the panic is inside rand, called from timeutil/schedule.go [09:27] so it _is_ the same thing :-) [09:28] Chipaca: we lock defer unlock then unlock then panic, the deferred unlock is not pleased [09:28] whee [09:28] pedronis: then it's fixed already :-) (not on master yet but soon) [09:28] pedronis: thank you [09:29] Chipaca: anyway it's a panic on panic as you noticed, and at least with Unlock/Lock it will not deadlock [09:39] Chipaca: hrm, test failure in snap auto mount test, looks like a flaky test, I have a look [09:40] mvo: _did_ you push a fix to 3290? [09:40] mvo: (no you didn't :-) ) [09:40] Chipaca: can you tell me why this failed perhaps? https://travis-ci.org/snapcore/snapd/builds/231080836?utm_source=github_status&utm_medium=notification [09:40] * Chipaca looks [09:41] Chipaca: the non-debug output is short and lacks data, the debug output is enormous and full of store stuff [09:42] zyga: I don't know, but I do know that if that used MATCH instead of grep, we'd know :-) [09:42] now trolling the debug output [09:42] Chipaca: ups, sorry, forgoten push [09:42] mvo: :-) [09:45] zyga: I approved 3300 but it appears like there are still issues [09:45] thank you [09:45] I'm looking at that now [09:45]