[00:21] PR snapd#7427 opened: tests/overlord/snapstate: refactor for cleaner test failures [00:28] PR snapd#7428 opened: systemd, wrappers/services: add helpers for disabling multiple services in a snap [00:29] PR snapd#7429 opened: wrappers/services: add CurrentSnapServiceStates + unit tests [00:43] PR snapd#7430 opened: [RFC] overlord/snapstate: set snap-setup-task for first task in change [00:53] PR snapd#7431 opened: overlord/snapstate: don't re-enable and start disabled services on refresh, etc [05:18] morning [06:18] mvo: morning [06:40] PR snapd#7427 closed: tests/overlord/snapstate: refactor for cleaner test failures === pstolowski|afk is now known as pstolowski [07:00] morning [07:07] hey pstolowski [07:12] pstolowski: hey [07:16] nice, thunderstorm at 9am :/ [07:20] mhm. quiet here but very cloudy and raining [07:21] pstolowski: it's finished now [07:21] pstolowski: < 10 minutes total, heh [07:28] hey [07:28] oh boy [07:28] I was in #snapcraft [07:28] complaining that there is nobody there [07:29] and it's a quiet day :D [07:29] how are you guys? [07:29] mborzecki, pstolowski: really? it's sunny and almost 20 here [07:29] zyga: hey, good morning. good, good, but it sounds like you need a cup of coffee :) [07:29] mvo: haha :) [07:29] I guess so [07:29] I'm reviewing channels PR [07:29] mvo: I was on a bike ride already, should be more awake [07:32] zyga: the front is moving NW, so enjoy your weather [07:32] zyga: about that, this is what poland looked like on 03.09 https://www.wykop.pl/cdn/c3201142/comment_I9dTRiDkK6k1pkTXd1VyOL7XMeNwQ8PD.jpg [07:33] mborzecki: looks like cake [07:34] zyga: mhm, the bottom screen capture is from Lodz afaik [07:35] mborzecki: polska A i polska B ;-) [07:39] I should make that coffee [07:40] it was surprising because mvo didn't join #snapcraft [07:40] but was doing reviews [07:40] so I was thinking there's an IRC split [07:53] zyga: conflicts in https://github.com/snapcore/snapd/pull/7421 [07:53] PR #7421: cmd/snap-confine: unmount /writable from snap view [07:59] zyga, hi, do you know why doing something https://github.com/maas/maas/blob/master/Makefile#L770-L784 would break a try'd snap? [08:00] zyga, I ran that and now I get [08:00] $ maas [08:00] execv failed: No such file or directory [08:20] ackk: are you running the maas binary you think you are running? [08:25] Chipaca: do you think you could have a look at https://github.com/snapcore/spread/pull/69 ? should be easy but I want it to be as good as possible before giving it to Gustavo [08:25] PR spread#69: Manage pagination retrieving images from google backend [08:25] mvo: I do think I could [08:25] Chipaca: thank you! [08:26] Chipaca: (I hope this means you will actually do it and not just think about it ;) I guess I phrased my question a bit too vague) [08:26] :-p [08:26] maybe it was just the right amount of vague [08:27] heh [08:33] Trevinho: hello [08:33] Trevinho: I got a note that you experienced an interesting failure on your system last week [08:44] mborzecki: hey [08:44] I noticed yo talked to Trevinho last week [08:44] zyga: hm? [08:44] quick question: [08:44] the "mount core" task failed [08:44] the log has [08:44] Mount snap "core" (7396) [08:44] 2019-09-05T20:09:11+02:00 ERROR remove /snap/core/7396: device or resource busy [08:44] does this seem to imply to you that succeeded on the DO path [08:44] something later failed [08:44] and then we went to UNDO [08:45] and this is when this failed? [08:45] zyga: i think Trevinho posted snap change for that refresh, and indeed it was undoing something [08:45] zyga: let me check my history [08:46] Hold today at 17:13 CEST today at 20:08 CEST Remove snap "core" (7169) from the system [08:46] this is what seems to have failed [08:46] and then we got the other error [08:46] because the new core was already used by something else [08:46] or by some scanning utility that held open file descriptors to core [08:47] so the question is: do you think that it was the undo path that failed? [08:50] zyga: this is the snap change output https://pastebin.ubuntu.com/p/Yhy3hxGG8b/ [08:55] mborzecki: yes, I have that [08:57] zyga: hard to tell from the log, but i think the problem was 2019-09-05T19:38:54+02:00 ERROR cannot remove snap file "core", will retry in 3 mins: remove /snap/core/7169: device or resource busy [08:57] zyga: then it went with undo [08:57] zyga: and then errored out in undo of setup-snap, which got logged with default case handler in taskrunner [08:57] mmm [08:57] mhm [09:22] mborzecki: I've made a patch for that [09:25] Chipaca, wdym? I get the same error by running /snap/bin/maas, if that's what you mean [09:25] ackk: ah, good. Ok. What happens if you 'snap run --trace maas'? [09:25] strace* [09:26] $ snap run --strace maas [09:26] error: exit status 1 [09:26] Chipaca, ^ [09:26] PR snapd#7432 opened: boot,dirs,image: introduce boot.MakeBootable, use it in image instead of ad hoc code [09:32] Chipaca, fwiw the same happens with the maas service, it fails to start with "file not found" [09:33] mborzecki, mvo: https://github.com/snapcore/snapd/pull/7433 [09:33] PR #7433: systemd: detach rather than unmount .mount units [09:33] PR snapd#7433 opened: systemd: detach rather than unmount .mount units [09:33] ackk: what does snap pack --check-skeleton of the prime directory say, if anything? [09:33] i think 'snap try' already does that but just in case [09:38] Chipaca, nothing [09:38] ackk: if that check passes, that means the binary itself that the app points to exists. As that's probably a shell wrapper, look at it and see what it calls [09:38] ackk: or [09:38] ackk: tell you what, if it is a wrapper, add "set -x" to the top of it :-) [09:39] ackk: and see which command it is that fails [09:39] Chipaca, execv failed: No such file or directory [09:39] Chipaca: hi, could you review #7425, #7411 and #7197 (they are already marked for you, there's a couple more but they either need more work or are descendant from those) [09:39] Chipaca, I put a -x on the snapcraft wrapper [09:39] PR #7425: channel: introduce Resolve and ResolveLocked [09:39] PR #7411: cmd/model: output tweaks, add'l tests [09:39] PR #7197: usersession: track connections to session agent for exit on idle and peer credential checks [09:39] but it's not even run [09:40] pedronis: will do [09:40] pedronis: thanks [09:40] ackk: how can i reproduce this? [09:47] Chipaca, I guess by fetching the maas code, building the snap and using snap try [09:47] ackk: if you pack the snap, does it still break in this way? [09:47] Chipaca, I've run into this a lot of times in the past, recently thought it got fixed since I wasn't seeing it anymore [09:47] Chipaca, well if I remove and reinstall it even with try it gets fixed [09:48] Chipaca, it's not a snap filesystem issue for sure [09:48] Chipaca, if it matters I'm running it in a container, with bind-mounted home [09:48] Chipaca, so I can't really try a packed snap, because of the snapfuse issue... [09:48] ackk: what happens if you disable and then enable the maas snap? [09:51] Chipaca, starts working again it seems [09:51] ackk: does maas use layouts or content? [09:51] Chipaca, no [09:51] content interface* [09:52] ackk: it looks like the rsync somehow breaks the mount but not sure how [09:52] ackk: unless … [09:53] ackk: as well as doing that series of rsyncs you linked to, do you also somehow re-create the 'prime' directory? [09:53] Chipaca, no, just that rsync [09:53] hm [09:53] I run 'make sync-dev-snap' and that's it [09:53] (and snap restart maas) [09:55] ackk: would be interesting to look at 'findmnt' (maybe 'findmnt -D') for when it works vs when it doesn't [09:56] Chipaca, ok, lemme see if it happens again [09:56] I have the one when it works [09:59] Chipaca, of course now I ran sync like 30 times and it doesn't happen ;/ [09:59] :-) [10:03] * Chipaca takes a break [10:03] ogra: hey, any thoughts on https://bugs.launchpad.net/snapd/+bug/1843077 ? not sure it belongs to snapd, probably somewhere else (core?) [10:03] Bug #1843077: Data corruption when UC is not powered down cleanly [10:15] zyga: oh, nice one! [10:24] zyga: updated #7419 [10:24] PR #7419: cmd/snap-confine: add unit tests for sc_invocation, cleanup memory leaks in tests [10:24] pstolowski: can you take a look at that PR too? ^^ [10:28] pstolowski, the bug is way to vague ... "doesnt boot" ... :P ... (what did cause it exactly ... whats the actual error they see then etc etc) ... you can indeed tune some settings like having dirty pages flushed more often to disk and such but thats typically coming at a cost for other (generic) use-cases ... and it also only minimizes the risk. i'm not sure there is much that can be done about the journal [10:28] oh ... and on a sidenote ... i'm officially on vacation (til 23rd) ;) [10:29] (and "other filesystems" is a foundations question ;) ) [10:31] i only do run two flash based boards over here ... (the others from SD) ... neither of them has ever shown such a behaviour [10:31] (hmm ... both on core 16 though ... thats perhaps a datapoint) [10:35] mborzecki: i'll [10:36] pstolowski: thx! [10:36] ogra: i see, thanks, and sorry for disturbing! enjoy your vacation! [10:38] mborzecki: looking [10:39] pstolowski, i wouldnt need to answer, all fine you didnt disturb ;) [10:44] a second review for 7432 would be great, should be super straightforward [10:48] * pstolowski lucnh [10:48] * pstolowski lunch [11:16] * zyga debugs something [11:16] mvo: sorry for not doing more reviews :/ [11:18] zyga: no worries [11:31] PR snapd#7426 closed: cmd/snap-update-ns: clarify sharing comment [11:32] PR snapd#7434 opened: seed/seedwriter: implement WriteMeta and tree16 corresponding code [11:40] pedronis: in #7425, why 'locked' instead of 'pinned'? seems confusing [11:40] PR #7425: channel: introduce Resolve and ResolveLocked [11:41] Chipaca: because in Toronto we discussed we don't want to use pinned, I'm exploring other options [11:41] we'll have some clarity hopefully in Paris [11:41] I thought it was going to be like progress.NewTaskProgressAdapterLocked [11:42] it's the toronto water, it clogs the brain [11:59] mborzecki: I patched snap-image to work with recovery, but there was an inconsistency in the way the utility was handling file copying so I had to change it a little bit [12:00] cmatsuoka: cool, are the changes in your branch? [12:00] mborzecki: not yet, still working on that [12:01] cmatsuoka: ok, ping me when you push it, i'll take a look, perpahs somethins that could be fix in the RFC PR [12:01] mborzecki: the main problem was related to how file organization is always used (source/target with different paths) [12:02] mborzecki: I couldn't use the standard Write() methods to get files from the gadget to the prepared area, and then from prepared to the final image, because it would want to organize files twice [12:03] mborzecki: and I also can't copy from gadget to image and prepared area to image in two different passes, because the writer was designed to do it in a single pass [12:04] mborzecki: so I added an extra parameter to Write() to tell if file organization is to be used or not [12:05] cmatsuoka: so you used Write() to some staging location and then Write() again from the staging to actual destination? [12:05] re [12:06] mborzecki: I tried numerous approaches and that seemed to be the one that didn't require major rewrites, but if you have any better idea I'll gladly accept it [12:08] mborzecki: my first attempt was to copy from gadget to image in addition to the regular copy, but the writer wants to create the image from staging in a single pass [12:13] cmatsuoka: what is the prepared area, why do you need two places/copies? [12:13] is actually not super clear === ricab is now known as ricab|lunch [12:31] pedronis: the "prepared area" is the directory written by snap prepare-image. snap image get bits from there, and bits directly from the gadget [12:32] pedronis: however, ubuntu-seed needs data from both at the same time (recovery from prepare-image, bootloader from gadget) [12:32] pedronis: because prepare-image doesn't add content files [12:33] cachio: https://paste.ubuntu.com/p/3pJzxxVT45/ so far we assumed it's only on arch? [12:34] mborzecki, yes [12:34] cmatsuoka: but we are not going to call prepare-image at runtime [12:34] pedronis: yes, but that's for the initial image generation [12:34] mborzecki, all the failures which I saw were on arch [12:35] cmatsuoka: there is no strict reason btw not to move things from prepare-image to just snap-image if it makes sense [12:36] cmatsuoka: also I just encapsulated the relevant code I think, the one that does gadget bits in prepare-image [12:36] cmatsuoka: you should probably read https://github.com/snapcore/snapd/pull/7432 [12:36] PR #7432: boot,dirs,image: introduce boot.MakeBootable, use it in image instead of ad hoc code [12:36] pedronis: I'll check that, thanks [12:37] cmatsuoka: to be clear, we haven't yet exactly decided what prepare-image/ubuntu-image do for core 20 images [12:38] if doing less or doing more is more convenient we ca tweak/decide [12:41] pedronis: yes, the snap-image investigation made me understand the issues there to see what kind of tweaks could be useful [12:42] PR snapd#7435 opened: tests: explicitly restore after using LXD [12:42] good morning y'all [12:43] hey ijohnson :) [12:43] hey zyga o/ [12:47] WAT? https://www.irccloud.com/pastebin/rRkpecF0/ [12:49] pedronis: so from your comment, is #7428 unnecessary then? should I close that PR or might it still be useful to have those methods for a different use case [12:50] PR #7428: systemd, wrappers/services: add helpers for disabling multiple services in a snap [12:52] ijohnson: we tend not to add unused helpers, it code base is big/sometimes confusing enough :) [12:52] s/it code/the code/ [12:52] ack, I'll close the PR then [12:55] PR snapd#7428 closed: systemd, wrappers/services: add helpers for disabling multiple services in a snap [13:01] ah, my bad [13:01] local bug [13:19] PR snapd#7436 opened: many: make per-snap mount namespace MS_SHARED [13:37] PR snapcraft#2702 closed: Release changelog for 3.8 === ricab|lunch is now known as ricab [13:45] * zyga wants to go for lunch, ttyl [14:11] pstolowski: noticed a little bug with unset, it's possible to unset an option that is not supported by core/system and otherwise would raise an error when tried with snap set [14:12] mborzecki: uh, nice catch, thank you! will fix [14:15] PR snapd#7395 closed: cmd/snap-confine: apply global mount namespace initialization for confined and classic snaps [14:23] wow, some heavy rain here [14:30] PR snapd#7432 closed: boot,dirs,image: introduce boot.MakeBootable, use it in image instead of ad hoc code [14:31] PR snapd#7419 closed: cmd/snap-confine: add unit tests for sc_invocation, cleanup memory leaks in tests [14:35] grr regex... am I missing something obvious here? https://pastebin.ubuntu.com/p/rFh2SzQG4z/ [14:36] I have a feeling this has something to do with gocheck automatically adding the end of the line marker to the regex in ErrorMatches [14:37] ijohnson: does it help if you prefix the regexp with ".* "? [14:38] ijohnson: yep, this, and possibly .* at the end. you also have ] there, needs escaping [14:38] ah actually it does if I add .* to the start and then \n.* at the end too [14:38] went with [14:38] c.Assert(err, ErrorMatches, ".*is-enabled snap.hello-snap.svc1.service] failed with exit status 1: whoops\n.*") [14:39] pstolowski: I thought I would have to escape ] but it didn't seem to be necessary for some reason [14:40] second look at https://github.com/snapcore/spread/pull/69 would be great, hopefully good now [14:40] PR spread#69: Manage pagination retrieving images from google backend [14:47] ijohnson: interesting, maybe it's smart enough if opening [ is missing; some regex implementations would probably freak out [14:47] yeah true, do you think I should escape it anyways in case go ever changes behavior? seems unlikely I guess [14:48] ijohnson: i'd escape it just to avoid confusion / make it clear [14:48] ack [14:48] zyga: hey, hello... I saw mborzecki said wrote a patch, is that enough or do you need some more debugging data? [14:49] fyi I'm still affected, so... I can test new versions and/or provide more debugging [14:49] I suppose I still won't reboot for a while... Too much precious stuff in my /tmp xD [15:11] Trevinho: hey, I think one thing might be useful [15:11] Trevinho: please install lsof and run lsof on the paths that failed [15:11] Trevinho: like /snap/core/1234 (fix the number) [15:11] Trevinho: lastly, please report a bug, this will aid in our tracking and allow us to store such logs easily [15:11] Trevinho: if you don't want to just say so and I'll report it [15:13] zyga: ok, as per lsof, there's nothing coming for the new version of the core [15:13] and the old one? [15:14] zyga: https://pastebin.ubuntu.com/p/jdbv6dRS79/ [15:14] Trevinho: is core rev 7270 the one that failed to be removed? [15:14] yep [15:15] my current one [15:27] Trevinho: understood, thank you [15:36] Trevinho: let me know about that bug number if you have one [15:41] zyga: I also now tried to close all the app using current snap and it still failed, but without hanging at least... [15:41] https://pastebin.ubuntu.com/p/9H3w9bSgMZ/ [15:41] so I suppose that in such case core should not block if removal of an old version failed [15:42] I mean, is that a fatal error? :o [15:42] Trevinho: what does lsof say now>? [15:42] nothing [15:42] lsof isn't perfect though, we'll see [15:42] as it wasn't saying anything of the failed-to-remove version before [15:42] yeah, that's the problem [15:42] PR snapd#7437 opened: wrappers/services.go: add disabled svc list arg to AddSnapServices [15:43] zyga: I forgot, what's the command to get debug info from a change? [15:43] snap changes [15:43] snap change NNN or snap tasks NNN [15:44] * cachio lunch [15:51] Bug #1843286 opened: Snap failing to remove old core doesn't update and keeps spinnig the CPU [15:51] zyga: https://bugs.launchpad.net/snappy/+bug/1843286 [15:51] Bug #1843286: Snap failing to remove old core doesn't update and keeps spinnig the CPU [15:51] Trevinho: thank you very much [15:54] I have the most of lack of luck recently [15:58] PR snapcraft#2707 opened: manifest: sort package and snap lists for consistency === pstolowski is now known as pstolowski|afk [16:34] PR snapd#7433 closed: systemd: detach rather than unmount .mount units [17:05] * zyga EODs [17:29] zyga: do you know how i might find out why my ubuntu-core-running raspberry pi is crashing? i'm now running on a not-much-used pi 3 and it's crashing regularly about twice a day (hard crash - it doesn't auto-reboot) [17:30] rogpeppe: enable persistent journal and look at the logs there; look at the serial output for hints [17:30] no other ideas from the top of my head [18:35] zyga: thanks. looks like i can enable persistent journal with `mkdir /var/log/journal`. does that seem right to you? [18:36] yes [18:36] zyga: sadly i can't look at serial output. i'm 500 miles away from the actual device now... [18:37] zyga: i've now enabled persistant journal; hopefully i might see something after the next crash. [18:37] zyga: BTW the other odd thing is that it only boots up about 50% of the time. [18:39] zyga: BTW thanks a lot for your interaction on this issue. it's really nice to have some feedback. [18:40] I'm really happy to help [18:40] it's hard to help everyone that comes around [18:40] but it's also genuinely interesting to see what is the cause of this problem === Aavar_ is now known as Aavar [19:20] * ijohnson relocates [20:22] hi there, observing something interesting with the organize keyword, and wanted to run it by someone. When using this keyword on a part using the 'python' plugin it doesn't seem to do anything? Is this expected behaviour? [20:34] PR snapd#7407 closed: tests: run failing tests on ubuntu eoan due to is now set as unstable [20:48] zyga: FWIW i was ssh'd into the system when i saw this broadcast message: [20:48] reboot scheduled to update the system [20:48] The system is going down for reboot at Mon 2019-09-09 19:36:00 UTC! [20:49] zyga: so perhaps it's not crashing after all, but just automatically rebooting and shutting down and not restarting [20:49] zyga: why would it be rebooting automatically to update the system? [20:50] zyga: that certainly might explain why the system is going down so regularly! [23:23] PR snapcraft#2708 opened: tools: make environment-setup container name and image configurable