[05:21] morning [06:12] PR snapd#7443 opened: timeutil: fix schedules with ambiguous nth weekday spans [06:15] pedronis: hey [06:15] gopkg is down? [06:22] mborzecki: from spread tests? I haven't seen that, I have see lots of things timing out though [06:23] pedronis: the unit test travis job failed because of this in 7443, though it's green now after i restarted it [07:05] morning [07:06] hey pstolowski [07:12] zyga: any updates on the pi system that was problematic yesterday? [07:12] sil2100: if you have time a review for https://github.com/snapcore/core18/pull/139 would be great (should be simple) [07:12] PR core18#139: hooks: add missing dosfstools to get fsck.fat [07:18] Good morning [07:18] hey zyga ! [07:18] Starting just now, sorry, today kids go to school at 8:45 [07:19] mvo: the reporter promised to try rebooting and report back [07:19] So no update yet [07:19] zyga: I see no new bugreport from that system in the error tracker since ~2 days [07:19] zyga: no idea if thats good or bad though :/ [07:20] I’ll reach out [07:20] Mvo we have another problem [07:20] Please check the mail from Jamie [07:30] anyoen feels like looking at https://github.com/snapcore/snapd/pull/7443 ? [07:30] PR #7443: timeutil: fix schedules with ambiguous nth weekday spans [07:31] keep in mind, there be dragons [07:47] re [07:48] PR snapd#7444 opened: overlord/snapstate/policy, etc: introduce policy, move canRemove to it [07:48] pedronis: ^ [07:51] Chipaca: thanks [07:52] pedronis: there's a bunch of things I'm itching to do to that code, but I've tried to leave it alone as much as possible :) [07:55] Chipaca: were the tests in snapstate that you are removing still passing? in general in this cases I prefer to do that removal as a 2nd step [07:56] pedronis: some of them were failing [07:56] but I could've made them pass with some tweaks [07:56] pedronis: I can put them back, fix them, and then remove them [07:56] Chipaca: would be better for reviewers I think [07:57] ok [08:00] Chipaca: I also I recommend having different file per types already, because the PR that decides we need that will be weird and hard to review [08:00] otherwise [08:00] hmm [08:00] pedronis: ok [08:00] git will not help there, splitting a file in a lot of files [08:01] pedronis: should I close that, force-push with these changes? [08:01] you can force push [08:01] k [08:01] I'm just giving high level comments [08:01] not really started reviewing in detail as such [08:01] heh [08:02] mvo: sure thing! [08:02] remember our 3K diffs [08:02] check this out https://github.com/apple/swift/pull/20315/files [08:02] PR apple/swift#20315: [String] Use a UTF-8 representation for native strings [08:02] hello sil2100 [08:02] how are you [08:02] PR snapd#7444 closed: overlord/snapstate/policy, etc: introduce policy, move canRemove to it [08:03] zyga: hello o/ Still a bit tired, but not bad - how about you? [08:03] sil2100: enjoying the sun today, looking at landing some branches and finally doing a release for opensuse [08:03] sil2100: mvo sent a small patch yesterday, I pinged you about it [08:04] sil2100: can you look at it today please? [08:05] zyga: yes, looking at it right now ;) [08:06] pedronis: WDYT of canremove_test? [08:06] PR core18#139 closed: hooks: add missing dosfstools to get fsck.fat [08:06] pedronis: organisation wise i mean [08:06] thank you sil2100 :) [08:07] now we need a regression test [08:07] mvo: I think it'd be easier to write snapd side [08:07] mvo: we can run it in nightly suite [08:07] zyga: yeah [08:07] and ensure nightly is ran on all relases [08:07] mvo: we should write that down somewhere on release acceptance [08:08] Chipaca: yes, I think organising tests by policy method makes sense [08:09] we can always have _test.go if there is really something very specific or helper to test [08:09] k [08:32] #7416 needs reviews (it's big) [08:32] PR #7416: seed/seedwriter: start of Writer and internal policy16/tree16 [08:46] Chipaca, pedronis 6404 is extremly confused in the cla check. it thinks it needs to check 196 emails and fails on a gazillion of them. I'm not sure its worth fixing the checker, I could create a new PR, point to the discussion of the old PR and get on with things? wdyt? [08:46] Chipaca: i.e. it looks like its not checking alternative emails in LP so it misses a lot of @canonical.com ones [08:50] pedronis: i broke 7444 so 7445 is up for your viewing pleasure [08:51] PR snapd#6705 closed: bootloader: little kernel support [08:51] PR snapd#7445 opened: overlord/snapstate/policy, etc: introduce policy, move canRemove to it [08:51] mvo: ...? [08:51] mvo: let me see [08:52] ah, super old pr? [08:52] mvo: there's a small change I'd like to you to try first [08:53] Chipaca: ok [08:54] PR snapd#7125 closed: snapstate: make progress reporting less granular [08:54] mborzecki: re [09:02] whoops [09:04] ? [09:05] pushed code with for-debug panics [09:06] not entirely convinced the panics aren't actually properly deserved tho :) [09:06] but not right now [09:13] degville: did I understand correctly that you were looking into https://forum.snapcraft.io/t/trouble-installing-snapd-on-rhel-8/13140 ? or, who was? [09:13] * Chipaca has bad memory [09:14] Chipaca: yes, that's the post I talked about yesterday. Although I've not had much luck. Our docs reference EPEL for 7, and things have changed slightly for 8. [09:15] degville: is the poster aware of your efforts? [09:18] Chipaca: maybe - I responded on the RHEL install page on docs category. [09:18] Chipaca: I had hoped it would be simply adding new instructions for the new repo config for EPEL, but that hasn't worked for me either. === JanC_ is now known as JanC [09:23] PR snapd#7446 opened: tests/mountinfo-tool: proper formatting of opt_fields [09:27] rogpeppe: hello, did you have any luck rebooting your board? [09:30] zyga: i'm waiting for someone to be around to power cycle when it fails to reboot [09:30] zyga: actually, i'll just try it anyway [09:30] rogpeppe: understood, thank you so much for helping us understand this issue [09:30] rogpeppe: we've fixed missing fsck.vfat already, we're going to implement a test that verifies the filesystem is really checked and repaired automatically next [09:32] zyga: ok, rebooting. keeping fingers crossed :) [09:32] mvo: ^ :) [09:41] rogpeppe: did it boot? [09:42] zyga: yes! [09:42] excellent!!! [09:42] rogpeppe: can you now try to refresh core18 snap [09:42] it should be successful now [09:43] zyga: i have to say i wasn't expecting that :) [09:43] and with some luck, it may unblock all refreshes [09:43] rogpeppe: thanks to the image you sent me I will investigate how uboot you have sees that [09:44] perhaps there's a discrepancy there [09:44] (and by "that", I mean the file that controls the boot process) [09:44] zyga: ok, refreshed, waiting for reboot now. more crossed fingers. :) [09:49] zyga: sadly it hasn't come back up after that [09:49] huh [09:49] huh [09:49] the plot thickens then [09:50] let's wait some more and in case nothing changes, reboot it manually [09:50] *clot [09:50] zyga: for the record, this was what the snap refresh command printed: http://paste.ubuntu.com/p/fPdkPjxdwY/ [09:51] once it recovers, let's look at "snap changes" and at journald [09:54] Chipaca: is it likely that core18 revision 1100 is corrupt but stays in cache? [09:54] Chipaca: and if so, is there a way we could verify that somehow from command line? [09:55] zyga: no, things are only moved if sha3 is ok [09:55] yes but if the card is corrputed [09:55] and the file on disk is just corrputed [09:55] is it going to be in cache after a failed refresh / reboot? [09:55] *corrupted [09:56] so, sudo find /var/lib/snapd/cache | xargs sudo snap info --verbose [09:56] will print the sha3 of all snaps in the cache [09:57] by re-reading them i mean [09:57] mmm [09:57] rogpeppe: ^^ [10:00] PR snapd#7447 opened: snapstate: do not allow removal of the snapd snap [10:00] now, the sha3 is also the filename [10:00] but because of Reasons, it's formatted differently :-| [10:02] Chipaca: \o/ for your cla check unshallow magic! [10:03] rogpeppe: did it recover after power-cycle? [10:04] zyga: i'm waiting for my dad to see the email asking him to power cycle... [10:04] understood [10:04] I'll check if we can enable watchdog from the bootloader [10:04] so that things like that don't require intervention [10:04] zyga: what do you mean by "watchdog" there? [10:06] mvo: :) [10:06] mvo: in store/cache.go, why is it using the hex digest instead of the EncodeDigest thing? [10:07] bah, i guess store/store.go is the one that actually uses one or t'other [10:08] Chipaca: I don't know why hex digest, sry [10:08] k [10:08] Chipaca: with the new unshallo I guess I can now remove the zyga@xenial-server from the whitelist, right? [10:08] Chipaca: hex might slightly more filename friendly? [10:09] pedronis: a tiny bit, but the inconsistency bites [10:09] mvo: dunno. Maybe? [10:09] mvo: feel free to try :-) [10:09] Chipaca: will do [10:09] pedronis: bah, we could also make snap info print out both variants [10:10] feels messy tho [10:10] that doesn't sound great [10:10] info is messy enough as is [10:11] * Chipaca covers info's ears [10:13] pedronis: EncodeDigest is a base64 url encoding, so it should be fine for filenames [10:13] it's got _ and - and alphanum [10:16] i'll look into switching it sometime [10:16] zyga: ok, looks like it was power-cycled. here's the snap changes output: http://paste.ubuntu.com/p/My5Cv4P443/ [10:18] zyga: for the record, it took two power cycles to come up again, as before [10:19] rogpeppe: there is one more angle [10:19] rogpeppe: we are testing two things at the same time [10:20] rogpeppe: if you recall, your kernel snap is refreshed and waiting to boot as well [10:20] rogpeppe: so it might be either of the two that cause issues [10:20] zyga: i've emailed you the ouput of `journalctl --system` [10:20] I'm reading it now [10:21] mborzecki: funny bug that one with snap unset & unsupported core options you found... subtle bug in transaction.Changes() [10:21] pstolowski: heh ;) [10:23] Sep 11 09:45:32 localhost snapd[690]: stateengine.go:150: state ensure error: devicemgr: snap "core18" has "refresh-snap" change in progress [10:23] Sep 11 10:09:25 localhost snapd[690]: handlers.go:460: Reported install problem for "core18" as 37ec03ba-d47c-11e9-a667-fa163e6cac46 OOPSID [10:23] mvo: ^ can you please look at that oops id? [10:24] zyga: can't you? [10:24] no [10:24] or if I do [10:24] I don't know I do [10:24] last time I only could look at the graph of errors [10:24] * zyga checks [10:26] Chipaca: perhaps I should ask, can you see those? [10:26] zyga: https://errors.ubuntu.com/oops/37ec03ba-d47c-11e9-a667-fa163e6cac46 [10:26] zyga: you need to log in [10:26] zyga: looks the same as the last one, no more extra data [10:26] oh, I see that one [10:26] I tried https://errors.ubuntu.com/?problem/37ec03ba-d47c-11e9-a667-fa163e6cac46 [10:27] zyga: I'm not even seeing the "has ... changes in progress" in the oops :( [10:27] mvo: that was in the journal [10:27] thank you for teaching me about the proper URL Chipaca [10:27] it's odd that there's "problem" and "oops" separately [10:27] ¯\_(ツ)_/¯ [10:27] zyga: problems are aggregations [10:27] yeah, on second thought it does make sense [10:27] but ... uuid, [10:27] look up [10:27] anyway :D [10:28] zyga: at the bottom of every problem there's a list of oopses [10:28] oopsen [10:28] oopsi [10:28] * Chipaca hides [10:28] I'll check what the state engine message is about [10:28] oppsinen? [10:28] I prefer oopsi [10:31] rogpeppe: I think we need to think about what may be going on now [10:32] rogpeppe: I will gather more ideas from the team and see if any of those are worth experimenting at your inconvenience [10:36] PR snapd#7448 opened: tests: remove mount_id and parent_id from mount-ns test data [10:36] zyga: thanks. let me know if you have any more ideas. [10:37] rogpeppe: thank you for working with us again, I'll try to replicate your setup locally to see if anything can be learned from this [10:41] pedronis: there's a question about snap create-user --force-managed on https://bugs.launchpad.net/snappy/+bug/1647256 that I'm not sure how to answer [10:41] Bug #1647256: snap create-user cannot add a new user to an existing ubuntu core device [10:57] zyga: put in my queue, I don't have an immediate answer [11:00] zyga: FYI I just tried fsck.vfat again and it's all clean [11:01] rogpeppe: mmm, that's useful, so it's not corruption that is causing the problem [11:01] rogpeppe: can you check what Chipaca suggested above: [11:02] zyga: what was that (I see quite a few messages from Chipaca) [11:02] ? [11:02] * Chipaca is a chatterbox [11:02] one sec :) [11:02] so, sudo find /var/lib/snapd/cache | xargs sudo snap info --verbose [11:02] rogpeppe: ^ [11:02] sudo find /var/lib/snapd/cache | xargs sudo snap info --verbose [11:03] wow, snap info is really slow! [11:05] with verbose, it's doing some expensive stuff yes [11:05] Chipaca, zyga: http://paste.ubuntu.com/p/vSpW6tmnJn/ [11:05] ok, gimme a bit to check those [11:06] Chipaca: can I ask the dummy question of unifying the way the hash is printed [11:06] Chipaca, zyga: FWIW, the info for the hydroctl service doesn't seem to be correct - it says that the hydroctl.hydroserver service is disabled/inactive, but it's definitely running currently [11:06] so that it is easier to eyeball-check [11:06] rogpeppe: oh, can you give us the output of systemctl status for that service unit? [11:07] rogpeppe: I think at some point it'd be good to report issues or we will be lost in the details [11:07] zyga: as in `systemctl status hydroctl.hydroserver` ? [11:07] I think that it ought to be systemctl status snap.hydroctl.hydroserver.service [11:08] zyga: ah, ok [11:08] snapd prepends "snap." to all units installed by snap packages [11:08] zyga: http://paste.ubuntu.com/p/ntMd8jXMFg/ [11:09] indeed, if snap service output disagrees, that's a bug! [11:09] can you collect that as well and report it via bugs.launchpad.net/snapd [11:10] Chipaca: can we look up snap revision for a blob for "snap info --verbose"? [11:10] zyga: without truncated logs FWIW http://paste.ubuntu.com/p/rjxwqpvFM5/ [11:10] thank you [11:10] zyga: sorry, i'm not exactly sure what's the bug here [11:10] rogpeppe: disagreement, snap service should not misrepresent facts [11:11] zyga: you mean "snap info", right? [11:11] zyga: rogpeppe all those hashes are coorect [11:12] Chipaca: thank you for checking that! [11:13] rogpeppe: perhaps I misunderstood, what did you mean when you said "FWIW, the info for the hydroctl service doesn't seem to be correct - it says that the hydroctl.hydroserver service is disabled/inactive, but it's definitely running currently" [11:13] how did you produce the information? [11:13] zyga: which information? [11:13] information for hydroctl service [11:14] zyga: it runs a web service that I can access [11:14] heh [11:14] aha [11:14] rogpeppe: sorry if that's confusing [11:14] you ran snap info on a file, not on an installed snap [11:14] it should not print that service info for files [11:14] i'll look into it [11:15] Chipaca: ok, i see [11:15] rogpeppe: if you run snap info --verbose hydroctl, you should see the right status there [11:16] ah, I understand now [11:16] thank you Chipaca [11:17] https://bugs.launchpad.net/snapd/+bug/1843567 [11:17] Bug #1843567: snap info reports incorrect service status === pedronis_ is now known as pedronis === ricab is now known as ricab|lunch [11:40] PR snapd#7449 opened: overlord/configstate: special-case json.RawMessage("null") in transaction Changes() [11:41] niemeyer: is gopkg.in in trouble? [11:41] 70 PRs [11:41] time to stop [11:41] Chipaca: looks ok to me [11:42] just got a string of failed tests because of 'Failed for "gopkg.in/retry.v1" (failed to clone repo)' because 'error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502' [11:42] eh, redoing [11:44] mborzecki: can you take a look at #7449? [11:44] PR #7449: overlord/configstate: special-case "null" in transaction Changes() [11:44] mborzecki: and i'm going to look at your schedule fix [11:44] pstolowski: I'm looking at that too [11:45] pstolowski: sure, looking [11:50] pstolowski: looking at your comment there [11:50] pstolowski: the decoder acts on the type you give, so if you give a map, you get a nil map [11:50] pstolowski: though one might discuss if "null" is a valid JSON representation of a map [11:50] pstolowski: I think, perhaps surprisingly so, that it is [11:51] zyga: yep, it's a bit ambigous [11:51] pstolowski: as a technicality, perhaps the len check could be merged into the original line, after err == nil [11:51] and the comment put up front, with slight rewording [11:51] so that it's reasier to read [11:51] I think again that the point is that the decoding is not unexpected [11:52] zyga: yes i had the len check merged but found it easier to read if the case is isolated with comment [11:52] ymmv [11:52] alternatively, it's the single condition, len > 0 [11:52] as err is otherwise unused [11:56] uh [11:56] we have another thing to fix in master [11:56] 'debian-sid-64' now has support. Please update this test [11:56] this is from - google:debian-sid-64:tests/main/system-usernames [11:57] something for jdstrand [11:57] we should open a priority pull request to make address that [12:03] pedronis: in snapstate, any reason to load the yaml and look at type instead of using snapst.Type()? (afaik no but you're better at tracking this) [12:04] Chipaca: depends where, in some places we might not have set snapst type yet [12:04] basically the one from the yaml might be safer to know it's there [12:04] pedronis: in places that call canRemove [12:04] :) [12:05] should be safe, last famous words [12:05] :-) [12:05] off to pick up the kids [12:05] pedronis: i can always make it fall back to loading the yaml if unset [12:06] might be easier to assume app if it's not [12:06] for remove [12:06] we might even have code like that anyway [12:06] because remove should work even with broken snaps [12:06] in theory [12:06] or at least it should try [12:06] pedronis: fair [12:07] I'll be tweaking the code in a followup :-) [12:07] going through this slowly [12:17] ooh, i think i see a bug [12:18] * Chipaca kicks off a core18 [12:18] Getting this error a lot since Monday, anyone else seeing something similar when running on spread/google? "- Start snap "lxd" (11824) services ([start snap.lxd.activate.service] failed with exit status 1: Job for snap.lxd.activate.service failed because the control process exited with error code." [12:19] sergiusens: what's in the journal when that happens? [12:21] Chipaca: I will have to add a "debug" stanza to see (I can only trigger the google provider through travis, me have no keys) [12:22] sergiusens: k [12:22] sergiusens: you should have a debug stanza, yes :-) [12:22] the logs of when things break can be quite valuable [12:26] Chipaca: does debug at the top level run fine (spread.yaml) or do I need debug-each (does that exists?) [12:27] sergiusens: debug at the toplevel should work fine [12:27] sergiusens: debug-each exists, indeed [12:28] ok, since this fails for the top level prepare I will add a debug, but just in case, will also add "debug-each" [12:37] we had also this related to lxd.activate: https://forum.snapcraft.io/t/snapd-pegged-cpu-updates-never-completing-on-core/13142 === ricab|lunch is now known as ricab [12:53] pedronis: mvo: we have a bug where we'll take "core" to satisfy a "base: core16" snap, but not check in canRemove [12:54] Chipaca: ok [12:54] Chipaca: "journalctl -xe" or more? [12:55] Chipaca: I don't think is very urgent, just keep track of it somehow [12:55] sergiusens: probably enough for now (you'll add more as you go i guess?) [12:55] pedronis: i'm in the right place to fix it now tho [12:55] :) [12:55] but, sure [12:55] Chipaca: if it's not too much extra lines/work [12:57] eep, 70 PRs :-| [13:00] Chipaca: and master being red [13:00] mmm, red [13:04] Chipaca: It's not in trouble but it's suffering.. there are spikes over the day that may put the machine over capacity for a moment [13:04] niemeyer: ack [13:05] Chipaca: We're already in the process of moving it inside Canonical DC [13:09] PR snapd#7450 opened: tests: debian sid now ships new seccomp, adjust tests [13:09] jdstrand: can you please look at https://github.com/snapcore/snapd/pull/7450 -- I must be missing something there [13:09] PR #7450: tests: debian sid now ships new seccomp, adjust tests [13:15] thank you, I'll check the other tests as well [13:15] I'll rewrite the commit message as well [13:20] jdstrand: fixed and pushed [13:20] thanks! [13:21] thank you for the urgent review :) [13:26] np === jdstrand_ is now known as jdstrand [13:41] mvo: we can always productively add _more_ PRs [13:51] Bug #1843589 opened: core snap stuck refreshing [13:52] ugh, mvo, Chipaca ^^^^ [13:53] yeah, reading [13:53] something's missing there though [13:53] bah, it's strange [13:53] stranger snaps [13:57] * zyga break for coffee and some rest [13:58]