mupPR snapd#7427 opened: tests/overlord/snapstate: refactor for cleaner test failures <Simple 😃> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7427>00:21
mupPR snapd#7428 opened: systemd, wrappers/services: add helpers for disabling multiple services in a snap <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7428>00:28
mupPR snapd#7429 opened: wrappers/services: add CurrentSnapServiceStates + unit tests <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7429>00:29
mupPR snapd#7430 opened: [RFC] overlord/snapstate: set snap-setup-task for first task in change <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7430>00:43
mupPR snapd#7431 opened: overlord/snapstate: don't re-enable and start disabled services on refresh, etc <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7431>00:53
mborzeckimvo: morning06:18
mupPR snapd#7427 closed: tests/overlord/snapstate: refactor for cleaner test failures <Simple 😃> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7427>06:40
=== pstolowski|afk is now known as pstolowski
mvohey pstolowski07:07
mborzeckipstolowski: hey07:12
mborzeckinice, thunderstorm at 9am :/07:16
pstolowskimhm. quiet here but very cloudy and raining07:20
mborzeckipstolowski: it's finished now07:21
mborzeckipstolowski: < 10 minutes total, heh07:21
zygaoh boy07:28
zygaI was in #snapcraft07:28
zygacomplaining that there is nobody there07:28
zygaand it's a quiet day :D07:29
zygahow are you guys?07:29
zygamborzecki, pstolowski: really? it's sunny and almost 20 here07:29
mvozyga: hey, good morning. good, good, but it sounds like you need a cup of coffee :)07:29
zygamvo: haha :)07:29
zygaI guess so07:29
zygaI'm reviewing channels PR07:29
zygamvo: I was on a bike ride already, should be more awake07:29
mborzeckizyga: the front is moving NW, so enjoy your weather07:32
mborzeckizyga: about that, this is what poland looked like on 03.09 https://www.wykop.pl/cdn/c3201142/comment_I9dTRiDkK6k1pkTXd1VyOL7XMeNwQ8PD.jpg07:32
zygamborzecki: looks like cake07:33
mborzeckizyga: mhm, the bottom screen capture is from Lodz afaik07:34
zygamborzecki: polska A i polska B ;-)07:35
zygaI should make that coffee07:39
zygait was surprising because mvo didn't join #snapcraft07:40
zygabut was doing reviews07:40
zygaso I was thinking there's an IRC split07:40
mborzeckizyga: conflicts in https://github.com/snapcore/snapd/pull/742107:53
mupPR #7421: cmd/snap-confine: unmount /writable from snap view <Created by zyga> <https://github.com/snapcore/snapd/pull/7421>07:53
ackkzyga, hi, do you know why doing something https://github.com/maas/maas/blob/master/Makefile#L770-L784 would break a try'd snap?07:59
ackkzyga,  I ran that and now I get08:00
ackk$ maas08:00
ackkexecv failed: No such file or directory08:00
Chipacaackk: are you running the maas binary you think you are running?08:20
mvoChipaca: do you think you could have a look at https://github.com/snapcore/spread/pull/69 ? should be easy but I want it to be as good as possible before giving it to Gustavo08:25
mupPR spread#69: Manage pagination retrieving images from google backend <Created by sergiocazzolato> <https://github.com/snapcore/spread/pull/69>08:25
Chipacamvo: I do think I could08:25
mvoChipaca: thank you!08:25
mvoChipaca: (I hope this means you will actually do it and not just think about it ;) I guess I phrased my question a bit too vague)08:26
Chipacamaybe it was just the right amount of vague08:26
zygaTrevinho: hello08:33
zygaTrevinho: I got a note that you experienced an interesting failure on your system last week08:33
zygamborzecki: hey08:44
zygaI noticed yo talked to Trevinho last week08:44
mborzeckizyga: hm?08:44
zygaquick question:08:44
zygathe "mount core" task failed08:44
zygathe log has08:44
zygaMount snap "core" (7396)08:44
zyga2019-09-05T20:09:11+02:00 ERROR remove /snap/core/7396: device or resource busy08:44
zygadoes this seem to imply to you that succeeded on the DO path08:44
zygasomething later failed08:44
zygaand then we went to UNDO08:44
zygaand this is when this failed?08:45
mborzeckizyga: i think Trevinho posted snap change for that refresh, and indeed it was undoing something08:45
mborzeckizyga: let me check my history08:45
zygaHold    today at 17:13 CEST  today at 20:08 CEST  Remove snap "core" (7169) from the system08:46
zygathis is what seems to have failed08:46
zygaand then we got the other error08:46
zygabecause the new core was already used by something else08:46
zygaor by some scanning utility that held open file descriptors to core08:46
zygaso the question is: do you think that it was the undo path that failed?08:47
mborzeckizyga: this is the snap change output https://pastebin.ubuntu.com/p/Yhy3hxGG8b/08:50
zygamborzecki: yes, I have that08:55
mborzeckizyga: hard to tell from the log, but i think the problem was 2019-09-05T19:38:54+02:00 ERROR cannot remove snap file "core", will retry in 3 mins: remove /snap/core/7169: device or resource busy08:57
mborzeckizyga:  then it went with undo08:57
mborzeckizyga:  and then errored out in undo of setup-snap, which got logged with default case handler in taskrunner08:57
zygamborzecki: I've made a patch for that09:22
ackkChipaca, wdym? I get the same error by running /snap/bin/maas, if that's what you mean09:25
Chipacaackk: ah, good. Ok. What happens if you 'snap run --trace maas'?09:25
ackk$ snap run --strace maas09:26
ackkerror: exit status 109:26
ackkChipaca, ^09:26
mupPR snapd#7432 opened: boot,dirs,image: introduce boot.MakeBootable, use it in image instead of ad hoc code <Created by pedronis> <https://github.com/snapcore/snapd/pull/7432>09:26
ackkChipaca, fwiw the same happens with the maas service, it fails to start with "file not found"09:32
zygamborzecki, mvo: https://github.com/snapcore/snapd/pull/743309:33
mupPR #7433: systemd: detach rather than unmount .mount units <Bug> <Created by zyga> <https://github.com/snapcore/snapd/pull/7433>09:33
mupPR snapd#7433 opened: systemd: detach rather than unmount .mount units <Bug> <Created by zyga> <https://github.com/snapcore/snapd/pull/7433>09:33
Chipacaackk: what does snap pack --check-skeleton of the prime directory say, if anything?09:33
Chipacai think 'snap try' already does that but just in case09:33
ackkChipaca, nothing09:38
Chipacaackk: if that check passes, that means the binary itself that the app points to exists. As that's probably a shell wrapper, look at it and see what it calls09:38
Chipacaackk: or09:38
Chipacaackk: tell you what, if it is a wrapper, add "set -x" to the top of it :-)09:38
Chipacaackk: and see which command it is that fails09:39
ackkChipaca, execv failed: No such file or directory09:39
pedronisChipaca: hi, could you review #7425, #7411 and #7197 (they are already marked for you, there's a couple more but they either need more work or are descendant from those)09:39
ackkChipaca, I put a -x on the snapcraft wrapper09:39
mupPR #7425: channel: introduce Resolve and ResolveLocked <Created by pedronis> <https://github.com/snapcore/snapd/pull/7425>09:39
mupPR #7411: cmd/model: output tweaks, add'l tests <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7411>09:39
mupPR #7197: usersession: track connections to session agent for exit on idle and peer credential checks <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/7197>09:39
ackkbut it's not even run09:39
Chipacapedronis: will do09:40
Chipacapedronis: thanks09:40
Chipacaackk: how can i reproduce this?09:40
ackkChipaca, I guess by fetching the maas code, building the snap and using snap try09:47
Chipacaackk: if you pack the snap, does it still break in this way?09:47
ackkChipaca, I've run into this a lot of times in the past, recently thought it got fixed since I wasn't seeing it anymore09:47
ackkChipaca, well if I remove and reinstall it even with try it gets fixed09:47
ackkChipaca, it's not a snap filesystem issue for sure09:48
ackkChipaca, if it matters I'm running it in a container, with bind-mounted home09:48
ackkChipaca, so I can't really try a packed snap, because of the snapfuse issue...09:48
Chipacaackk: what happens if you disable and then enable the maas snap?09:48
ackkChipaca, starts working again it seems09:51
Chipacaackk: does maas use layouts or content?09:51
ackkChipaca, no09:51
Chipacacontent interface*09:51
Chipacaackk: it looks like the rsync somehow breaks the mount but not sure how09:52
Chipacaackk: unless …09:52
Chipacaackk: as well as doing that series of rsyncs you linked to, do you also somehow re-create the 'prime' directory?09:53
ackkChipaca, no, just that rsync09:53
ackkI run 'make sync-dev-snap' and that's it09:53
ackk(and snap restart maas)09:53
Chipacaackk: would be interesting to look at 'findmnt' (maybe 'findmnt -D') for when it works vs when it doesn't09:55
ackkChipaca, ok, lemme see if it happens again09:56
ackkI have the one when it works09:56
ackkChipaca, of course now I ran sync like 30 times and it doesn't happen ;/09:59
* Chipaca takes a break10:03
pstolowskiogra: hey, any thoughts on https://bugs.launchpad.net/snapd/+bug/1843077 ? not sure it belongs to snapd, probably somewhere else (core?)10:03
mupBug #1843077: Data corruption when UC is not powered down cleanly  <snapd:New> <https://launchpad.net/bugs/1843077>10:03
mvozyga: oh, nice one!10:15
mborzeckizyga: updated #741910:24
mupPR #7419: cmd/snap-confine: add unit tests for sc_invocation, cleanup memory leaks in tests <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7419>10:24
mborzeckipstolowski:  can you take a look at that PR too? ^^10:24
ograpstolowski, the bug is way to vague ... "doesnt boot" ... :P ... (what did cause it exactly ... whats the actual error they see then etc etc) ... you can indeed tune some settings like having dirty pages flushed more often to disk and such but thats typically coming at a cost for other (generic) use-cases ... and it also only minimizes the risk. i'm not sure there is much that can be done about the journal10:28
ograoh ... and on a sidenote ... i'm officially on vacation (til 23rd) ;)10:28
ogra(and "other filesystems" is a foundations question ;) )10:29
ograi only do run two flash based boards over here ... (the others from SD) ... neither of them has ever shown such a behaviour10:31
ogra(hmm ... both on core 16 though ... thats perhaps a datapoint)10:31
pstolowskimborzecki: i'll10:35
mborzeckipstolowski: thx!10:36
pstolowskiogra: i see, thanks, and sorry for disturbing! enjoy your vacation!10:36
zygamborzecki: looking10:38
ograpstolowski, i wouldnt need to answer, all fine you didnt disturb  ;)10:39
mvoa second review for 7432 would be great, should be super straightforward10:44
* pstolowski lucnh10:48
* pstolowski lunch10:48
* zyga debugs something11:16
zygamvo: sorry for not doing more reviews :/11:16
mvozyga: no worries11:18
mupPR snapd#7426 closed: cmd/snap-update-ns: clarify sharing comment <Simple 😃> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/7426>11:31
mupPR snapd#7434 opened: seed/seedwriter:  implement WriteMeta and tree16 corresponding code <Created by pedronis> <https://github.com/snapcore/snapd/pull/7434>11:32
Chipacapedronis: in #7425, why 'locked' instead of 'pinned'? seems confusing11:40
mupPR #7425: channel: introduce Resolve and ResolveLocked <Created by pedronis> <https://github.com/snapcore/snapd/pull/7425>11:40
pedronisChipaca: because in Toronto we discussed we don't want to use pinned, I'm exploring other options11:41
pedroniswe'll have some clarity hopefully in Paris11:41
ChipacaI thought it was going to be like progress.NewTaskProgressAdapterLocked11:41
Chipacait's the toronto water, it clogs the brain11:42
cmatsuokamborzecki: I patched snap-image to work with recovery, but there was an inconsistency in the way the utility was handling file copying so I had to change it a little bit11:59
mborzeckicmatsuoka: cool, are the changes in your branch?12:00
cmatsuokamborzecki: not yet, still working on that12:00
mborzeckicmatsuoka: ok, ping me when you push it, i'll take a look, perpahs somethins that could be fix in the RFC PR12:01
cmatsuokamborzecki: the main problem was related to how file organization is always used (source/target with different paths)12:01
cmatsuokamborzecki: I couldn't use the standard Write() methods to get files from the gadget to the prepared area, and then from prepared to the final image, because it would want to organize files twice12:02
cmatsuokamborzecki: and I also can't copy from gadget to image and prepared area to image in two different passes, because the writer was designed to do it in a single pass12:03
cmatsuokamborzecki: so I added an extra parameter to Write() to tell if file organization is to be used or not12:04
mborzeckicmatsuoka: so you used Write() to some staging location and then Write() again from the staging to actual destination?12:05
cmatsuokamborzecki: I tried numerous approaches and that seemed to be the one that didn't require major rewrites, but if you have any better idea I'll gladly accept it12:06
cmatsuokamborzecki: my first attempt was to copy from gadget to image in addition to the regular copy, but the writer wants to create the image from staging in a single pass12:08
pedroniscmatsuoka: what is the prepared area, why do you need two places/copies?12:13
pedronisis actually not super clear12:13
=== ricab is now known as ricab|lunch
cmatsuokapedronis: the "prepared area" is the directory written by snap prepare-image. snap image get bits from there, and bits directly from the gadget12:31
cmatsuokapedronis: however, ubuntu-seed needs data from both at the same time (recovery from prepare-image, bootloader from gadget)12:32
cmatsuokapedronis: because prepare-image doesn't add content files12:32
mborzeckicachio: https://paste.ubuntu.com/p/3pJzxxVT45/ so far we assumed it's only on arch?12:33
cachiomborzecki, yes12:34
pedroniscmatsuoka:  but we are not going to call prepare-image at runtime12:34
cmatsuokapedronis: yes, but that's for the initial image generation12:34
cachiomborzecki, all the failures which I saw were on arch12:34
pedroniscmatsuoka: there is no strict reason btw not to move things from prepare-image to just snap-image if it makes sense12:35
pedroniscmatsuoka: also I just encapsulated the relevant code I think, the one that does gadget bits in prepare-image12:36
pedroniscmatsuoka: you should probably read https://github.com/snapcore/snapd/pull/743212:36
mupPR #7432: boot,dirs,image: introduce boot.MakeBootable, use it in image instead of ad hoc code <Created by pedronis> <https://github.com/snapcore/snapd/pull/7432>12:36
cmatsuokapedronis: I'll check that, thanks12:36
pedroniscmatsuoka: to be clear, we haven't yet exactly decided what prepare-image/ubuntu-image do for core 20 images12:37
pedronisif doing less or doing more is more convenient we ca tweak/decide12:38
cmatsuokapedronis: yes, the snap-image investigation made me understand the issues there to see what kind of tweaks could be useful12:41
mupPR snapd#7435 opened: tests: explicitly restore after using LXD <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/7435>12:42
ijohnsongood morning y'all12:42
zygahey ijohnson :)12:43
ijohnsonhey zyga o/12:43
zygaWAT? https://www.irccloud.com/pastebin/rRkpecF0/12:47
ijohnsonpedronis: so from your comment, is #7428 unnecessary then? should I close that PR or might it still be useful to have those methods for a different use case12:49
mupPR #7428: systemd, wrappers/services: add helpers for disabling multiple services in a snap <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7428>12:50
pedronisijohnson: we tend not to add unused helpers, it code base is big/sometimes confusing enough :)12:52
pedroniss/it code/the code/12:52
ijohnsonack, I'll close the PR then12:52
mupPR snapd#7428 closed: systemd, wrappers/services: add helpers for disabling multiple services in a snap <Created by anonymouse64> <Closed by anonymouse64> <https://github.com/snapcore/snapd/pull/7428>12:55
zygaah, my bad13:01
zygalocal bug13:01
mupPR snapd#7436 opened: many: make per-snap mount namespace MS_SHARED <Created by zyga> <https://github.com/snapcore/snapd/pull/7436>13:19
mupPR snapcraft#2702 closed: Release changelog for 3.8 <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2702>13:37
=== ricab|lunch is now known as ricab
* zyga wants to go for lunch, ttyl13:45
mborzeckipstolowski: noticed a little bug with unset, it's possible to unset an option that is not supported by core/system and otherwise would raise an error when tried with snap set14:11
pstolowskimborzecki: uh, nice catch, thank you! will fix14:12
mupPR snapd#7395 closed: cmd/snap-confine: apply global mount namespace initialization for confined and classic snaps <Created by bboozzoo> <Closed by bboozzoo> <https://github.com/snapcore/snapd/pull/7395>14:15
mborzeckiwow, some heavy rain here14:23
mupPR snapd#7432 closed: boot,dirs,image: introduce boot.MakeBootable, use it in image instead of ad hoc code <Created by pedronis> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/7432>14:30
mupPR snapd#7419 closed: cmd/snap-confine: add unit tests for sc_invocation, cleanup memory leaks in tests <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/7419>14:31
ijohnsongrr regex... am I missing something obvious here? https://pastebin.ubuntu.com/p/rFh2SzQG4z/14:35
ijohnsonI have a feeling this has something to do with gocheck automatically adding the end of the line marker to the regex in ErrorMatches14:36
mvoijohnson: does it help if you prefix the regexp with ".* "?14:37
pstolowskiijohnson: yep, this, and possibly .* at the end. you also have ] there, needs escaping14:38
ijohnsonah actually it does if I add .* to the start and then \n.* at the end too14:38
ijohnsonwent with14:38
ijohnsonc.Assert(err, ErrorMatches, ".*is-enabled snap.hello-snap.svc1.service] failed with exit status 1: whoops\n.*")14:38
ijohnsonpstolowski: I thought I would have to escape ] but it didn't seem to be necessary for some reason14:39
mvosecond look at https://github.com/snapcore/spread/pull/69 would be great, hopefully good now14:40
mupPR spread#69: Manage pagination retrieving images from google backend <Created by sergiocazzolato> <https://github.com/snapcore/spread/pull/69>14:40
pstolowskiijohnson: interesting, maybe it's smart enough if opening [ is missing; some regex implementations would probably freak out14:47
ijohnsonyeah true, do you think I should escape it anyways in case go ever changes behavior? seems unlikely I guess14:47
pstolowskiijohnson: i'd escape it just to avoid confusion / make it clear14:48
Trevinhozyga: hey, hello... I saw mborzecki said wrote a patch, is that enough or do you need some more debugging data?14:48
Trevinhofyi I'm still affected, so... I can test new versions and/or provide more debugging14:49
TrevinhoI suppose I still won't reboot for a while... Too much precious stuff in my /tmp xD14:49
zygaTrevinho: hey, I think one thing might be useful15:11
zygaTrevinho: please install lsof and run lsof on the paths that failed15:11
zygaTrevinho: like /snap/core/1234 (fix the number)15:11
zygaTrevinho: lastly, please report a bug,  this will aid in our tracking and allow us to store such logs easily15:11
zygaTrevinho: if you don't want to just say so and I'll report it15:11
Trevinhozyga: ok, as per lsof, there's nothing coming for the new version of the core15:13
zygaand the old one?15:13
Trevinhozyga: https://pastebin.ubuntu.com/p/jdbv6dRS79/15:14
zygaTrevinho: is core rev 7270 the one that failed to be removed?15:14
Trevinhomy current one15:15
zygaTrevinho: understood, thank you15:27
zygaTrevinho: let me know about that bug number if you have one15:36
Trevinhozyga: I also now tried to close all the app using current snap and it still failed, but without hanging at least...15:41
Trevinhoso I suppose that in such case core should not block if removal of an old version failed15:41
TrevinhoI mean, is that a fatal error? :o15:42
zygaTrevinho: what does lsof say now>?15:42
zygalsof isn't perfect though, we'll see15:42
Trevinhoas it wasn't saying anything of the failed-to-remove version before15:42
Trevinhoyeah, that's the problem15:42
mupPR snapd#7437 opened:  wrappers/services.go: add disabled svc list arg to AddSnapServices <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7437>15:42
Trevinhozyga: I forgot, what's the command to get debug info from a change?15:43
zygasnap changes15:43
zygasnap change NNN or snap tasks NNN15:43
* cachio lunch15:44
mupBug #1843286 opened: Snap failing to remove old core doesn't update and keeps spinnig the CPU <Snappy:New> <https://launchpad.net/bugs/1843286>15:51
Trevinhozyga: https://bugs.launchpad.net/snappy/+bug/184328615:51
mupBug #1843286: Snap failing to remove old core doesn't update and keeps spinnig the CPU <Snappy:New> <https://launchpad.net/bugs/1843286>15:51
zygaTrevinho: thank you very much15:51
zygaI have the most of lack of luck recently15:54
mupPR snapcraft#2707 opened: manifest: sort package and snap lists for consistency <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/2707>15:58
=== pstolowski is now known as pstolowski|afk
mupPR snapd#7433 closed: systemd: detach rather than unmount .mount units <Bug> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/7433>16:34
* zyga EODs17:05
rogpeppezyga: do you know how i might find out why my ubuntu-core-running raspberry pi is crashing? i'm now running on a not-much-used pi 3 and it's crashing regularly about twice a day (hard crash - it doesn't auto-reboot)17:29
zygarogpeppe: enable persistent journal and look at the logs there; look at the serial output for hints17:30
zygano other ideas from the top of my head17:30
rogpeppezyga: thanks. looks like i can enable persistent journal with `mkdir /var/log/journal`. does that seem right to you?18:35
rogpeppezyga: sadly i can't look at serial output. i'm 500 miles away from the actual device now...18:36
rogpeppezyga: i've now enabled persistant journal; hopefully i might see something after the next crash.18:37
rogpeppezyga: BTW the other odd thing is that it only boots up about 50% of the time.18:37
rogpeppezyga: BTW thanks a lot for your interaction on this issue. it's really nice to have some feedback.18:39
zygaI'm really happy to help18:40
zygait's hard to help everyone that comes around18:40
zygabut it's also genuinely interesting to see what is the cause of this problem18:40
=== Aavar_ is now known as Aavar
* ijohnson relocates19:20
ec0hi there, observing something interesting with the organize keyword, and wanted to run it by someone. When using this keyword on a part using the 'python' plugin it doesn't seem to do anything? Is this expected behaviour?20:22
mupPR snapd#7407 closed: tests: run failing tests on ubuntu eoan due to is now set as unstable <Created by sergiocazzolato> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7407>20:34
rogpeppezyga: FWIW i was ssh'd into the system when i saw this broadcast message:20:48
rogpeppereboot scheduled to update the system20:48
rogpeppeThe system is going down for reboot at Mon 2019-09-09 19:36:00 UTC!20:48
rogpeppezyga: so perhaps it's not crashing after all, but just automatically rebooting and shutting down and not restarting20:49
rogpeppezyga: why would it be rebooting automatically to update the system?20:49
rogpeppezyga: that certainly might explain why the system is going down so regularly!20:50
mupPR snapcraft#2708 opened: tools: make environment-setup container name and image configurable <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/2708>23:23

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!