=== Guest41378 is now known as jero [05:17] morning [05:51] PR snapcraft#2161 closed: many: automatically detect dependency changes [06:38] hey everyone [06:38] I will have an unusual Friday with some school stuff interfering today [06:38] I will attempt to minimise it though [06:38] zyga: good morning [06:39] hey [06:39] I'm sorry, I collapsed yesterday, I was sleeping since ~~19:00 [06:39] zyga: no worries [06:39] I saw your ping about managing to run apps on top of your PR, that's fantastic [06:39] I will share what I have on system slots [06:40] though it's not fully working yet (tests) [06:40] (tests are still choppy) [06:40] zyga: thanks, its small steps on my part but it is nice to see (some) things coming together [06:41] zyga: and thanks for working on the interfaces part! [06:43] zyga: restarted the build in 5325 [06:44] I noticed someone did, thank you [06:44] that branch is cursed :) [06:46] hey mborzecki, good morning [06:46] mvo: hey [06:47] i'll restart the build in 5293 a couple of times, just to rule out the luck factor [06:47] #5293 [06:47] PR #5293: tests: fix interfaces-calendar-service test when gvfsd-metadata loks the xdg dirctory [06:48] if that's green, then we're down to just one notoriously flaky test - econnreset :P i'm sure pstolowski|afk is super happy about that === pstolowski|afk is now known as pstolowski [07:09] mornings [07:11] mborzecki: i'm tempted to relax retry check in econnreset test to just for retry on either download or assertions, instead of expecting retries on download only [07:11] *to just check* [07:22] PR snapd#5325 closed: interfaces: add Repository.AllInterfaces [07:47] mvo: I struggle with the number of places that need to know about "system", [07:47] I need to take my dog out but after that let's chat [08:15] hm that slack carshing xwayland is so weird [08:15] there's a backtrace posted in the forums, given that slack is classic, it's probably doing some LD_* magic [08:16] but none of the symbols seem to point back to /var/lib/snapd/snap/.. [08:16] unless the backtrace is wrong, or at least the library locations [08:20] zyga: sounds good, just ping me when you are back and have time (I was in a different meeting just now) [08:25] PR snapd#5330 opened: snapstate: support restarting snapd from the snapd snap on core18 [08:28] mvo: re, [08:30] mvo: so I'm pondering this [08:30] the interface repository aspect itself is clean [08:30] but overlord is more subtle, [08:30] mvo: hi, I remembered something about setup-profiles phase2 for core that we need to consider I think [08:30] we can either teach all of interface manager to skip/ignore system snap [08:30] and risk leaks to other managers perhaps [08:31] or return virtual system snap from snap manager's APIs [08:32] mvo: I was also considering actually installing a real snap that shows up in /snap but I think that is more complex actually (e.g. revert is messy now) [08:32] pedronis: oh, what is that? [08:32] mvo: that it was also stopping going forward until we were restarted, that has relevance to autoconnect for example [08:33] PR core18#27 opened: run-snapd-from-snap: start snapd after seeding [08:33] mvo: zyga: we even the question of when we run autoconnect for things in system, either that needs to happen in the new snapd [08:34] *either way [08:34] zyga: hm, would it help if snapstate would have a fake system snap? [08:34] that's a good point [08:34] pedronis: could it run.. twice? [08:34] it doens't need to run twice [08:34] is already in the right place [08:34] mvo: yes, it would, I'm doing that now [08:34] is the waiting for restart that is missing now [08:34] mvo: that part actually works now, I'm just moving things around to have only one instance of snap.Info in memory [08:34] and to coordinate the implicit interfaces there [08:35] mvo: zyga: are we going to filter it out of snap list ? [08:35] or not [08:35] pedronis: aha, ok. I remember (vaguely). should we do that in link-snap now? [08:35] pedronis: currently it doesn't show up in snap list [08:35] it's only virtually returned by Get [08:35] zyga: by hacking ? [08:35] and ignored by Set [08:35] interesting [08:35] that is, it is only a snap you can get the "state" of but it's not really anywhere else [08:36] bit unclear how that works with reloading profiles though [08:36] reloading profiles? [08:36] on startup with system key changes? [08:36] yes [08:37] the interface manager has explicit code to load the virtual system snap [08:37] so that works ok [08:37] anyway, it's not fully baked yet [08:37] you can cheat in snapsWithSecurityProfiles [08:37] so we may abandon that totally [08:37] oh, yes, that's a good idea [08:37] I could remove the special code from interface manager then [08:37] zyga: it sounds messy though overall from afar [08:37] tbh [08:37] pedronis: I agree [08:37] pedronis: it's an experiment still, we may end up doing something totally different [08:38] mvo: where we put depends exactly when we run autoconnect logic for "system" interfaces [08:38] in theory it should happen in autoconnect for "core" or "snapd" [08:39] * mvo nods [08:39] mvo: we can add waiting to link-snap (but is annoying if I remember) or auto-connect or yet something else (but then it's hard across upgrade) [08:40] mvo: I think we need in auto-connect itself because upgrades, if it's in link-snap is the old snapd that needs to wait there [08:40] and it will not [08:41] mmh, well, not it will still wait in setup-profiles [08:41] so maybe doing something different is ok [08:41] but we need to think [08:41] pedronis: right. my gut feeling is that waiting in auto-connect - because that is close to why its needed. but I have not thought deeply about it yet [08:44] mvo: which brings back the annoying part of that story that is about detecting rollbacks or missing reboots [08:44] (I hoped we could live without but seems not) [08:45] mvo: ah, but now we don't need to wait for a reboot, just a restart [08:45] bit easier [08:48] pedronis: yeah, let me have a quick look [08:49] mvo: to notice the issue, you need to do something like install a snap that needs a new auto-connected system interface, notice that is not connect, refresh snapd to one that has it, if snapd doesn't do auto-connect with the right new version, the connection is still not there [08:49] sil2100: hey, sorry that I didn't get to #24 for core18 earlier. I added some thoughts, hope they are helpful [08:49] mvo: admittedly all a bit of a corner case [08:49] PR #24: Renamed .bzrignore to .gitignore. Added .coverage. Removed the ./ in front of ignored paths [08:51] mvo: which also reminds that we need to teach about the "snapd" snap to snapstate.go byKind [08:51] it should sort first [08:54] pedronis: I guess http://paste.ubuntu.com/p/YbV92FPgd8/ is too naive for the wait? [08:54] pedronis: let me look at the sorting [08:55] mvo: it's not unreasonable, the issue is more the rollback checks, we don't need them for snapd, but we need them for core [08:55] that one still fully reboots [08:56] I mean core as base [08:56] pedronis: aha, so we need all of GuardCoreSetupProfilesPhase2 back it seems [08:56] some parts of it [08:56] yes, sorry [08:56] pedronis: and teach it about bases [08:56] pedronis: thank you [08:57] mvo: really only core [08:57] as a base [08:57] for now [08:57] undering the assumption that base themselves [08:57] don't have slots [08:57] (we might change that at some point, but don't think we want to go there now) [08:58] pedronis: +1 [09:03] PR snapd#5331 opened: snapstate: sort "snapd" first [09:22] mvo: ok, I have something that vaguely works [09:22] I can share it now, let me commit [09:23] it still doesn't pass unit tests [09:26] zyga: ok [09:31] PR snapd#5332 opened: tests: relax check for download start, match either download or assertion retry attempt [09:36] pedronis: do you think #5314 is close to landing now? [09:37] PR #5314: many: rename snap.Info.Name() to snap.Info.InstanceName(), leave parallel-install TODOs [09:37] mvo: ok let me get rid of junk from history and push it [09:38] mborzecki: probably, but I need to look at it again [09:38] Chipaca: +1 to #5326 with some final nitpicks [09:38] pedronis: ok, take your time, better catch all the places now [09:39] PR #5326: api/snapctl: allow -h and --help for regular users [09:39] sorry [09:39] Chipaca: I meant #5327 [09:39] PR #5327: store: switch store.SnapInfo to use the new v2/info endpoint [09:39] mborzecki: of course we have some pending PRs adding more Name() and RealName uses [09:41] mvo: I have one idea I lijke [09:41] "snap info system" [09:41] could do what "snap version" says, e.g. synthesise nice description, summary, [09:41] use os-release [09:41] etc [09:42] it sounds nice and would be ... cool :) [09:44] pedronis: i guess i could nag people to add a todo note while they're at it, otoh wouldn't want to disrupt PRs too much, i can always merge this locally, since it's a rename the the code will fail to build (well maybe aside from RealName, this may easily fall under the radar) [09:45] mborzecki: any further tought btw whether we should put InstanceKey in Info/SnapState/SnapSetup or SideInfo ? [09:46] mvo: command-not-found broke in bionic [09:46] it's broken, but not as you know it https://www.irccloud.com/pastebin/Za3Og5z0/ [09:46] also, I wonder why it used "sudo snap install" twice [09:49] zyga: hm, hm, what version of snapd do you have? edge? [09:49] I'm running master [09:49] + my magic changes [09:52] zyga: this indicats some mismatch between snapd and snap [09:52] pedronis: i'm leaning towards side-info, other option i looked at is keeping it in snapstate as a separate field, but i'm not convinced, feels like it's shifting the cost of tracking it to other places, besides we already repeat a bunch of data in side-info for every revision, instance keys are relatively short (at least for now) so there wouldn't be that much overhead [09:53] mvo: ah, that's possible [09:53] I only have ./snap [09:53] heh formatting heredoc in task.yaml is awkward, especially if it's nested in some for loop/if block [10:05] mborzecki: it still feels a bit dirty, because everything so far in SideInfo is revision/store related [10:08] error: e-book-client-error-quark[2] Conflicting UIDs found in added contacts [10:08] wtf [10:13] mvo: https://github.com/snapcore/snapd/compare/master...zyga:rfc/virtual-system-snap?expand=1 [10:13] mvo: pull this and have a look [10:13] I will now look at what tests are unhappy [10:13] towards making it unit test happy [10:13] and then looking at spread happy [10:13] pedronis: fair point, let's assume then that it's in SnapState (and thus in Info & SnapSetup), a bit more work but will probably pay off in the long run [10:14] all suggestions welcome [10:14] mborzecki: given that we have Name() in at least a couple of those it should be well hidden, the question is how many fuctions take SideInfo directly, which is something I'm looking into [10:15] now [10:15] #5293 is green for the 5th time now, killing gvfsd seems to work, needs a 2nd review and we could land it [10:15] PR #5293: tests: fix interfaces-calendar-service test when gvfsd-metadata loks the xdg dirctory [10:16] zyga: thanks, I check after lunch [10:16] mborzecki: not a tone, some Mock functions but we have options tehre [10:16] pedronis: heh, at least a couple of apis in snapstate will need adjustments, eg InstallPath which I touched recently [10:16] mborzecki: I know, I couple functions taking *SideInfo, will really need to take *SideInfo, instanceKey string [10:16] pedronis: those apis which take a name should be fine though [10:17] name as in snap name [10:17] yep [10:18] mborzecki: the ReadInfo ones are the mostly annoying, though usually we call them through other helpers [10:18] mborzecki: also not sure the one that opens the file (vs deals with an installed snap) needs the instance key [10:19] pedronis: iirc readinfo takes a name, it should be fine then [10:19] ah, yes [10:19] mborzecki: so maybe you could quickly check how many places doing ReadInfoFromSnapFile will need to stick a instanceKey in the result [10:19] that seems the biggest sticking point [10:21] pedronis: yeah, i'm already setting it explicitly https://github.com/bboozzoo/snapd/blob/bboozzoo/parallel-snap-install-from-snap/snap/info.go#L933 so a simple adjustment [10:23] mborzecki: ?, I'm talking about *SnapFile [10:23] understood about just ReadInfo [10:24] it's used not a lot [10:24] we should be able to survice [10:24] survive [10:25] yup, a handfull of places [10:25] same OpenSnapFile [10:25] with the latter we have a name or SnapSetup around [10:26] mborzecki: so, yes I would try to add InstanceKey to Info SnapState and SnapSetup and see how it goes [10:26] seems cleaner long term [10:26] pedronis: what's the revert plan for instances [10:26] CE will test reverting core / snapd [10:26] zyga ? [10:26] how do we plan that after installing instanced snaps revert won't cause issues on the system [10:27] it's kind of hard [10:27] right but we need _a_ plan [10:27] CE has a test that shows that revert must work [10:27] well on core devices nothing should do that [10:27] we've been burned by that before [10:27] with the former we seem to have the name in the caller (or don't care as in case of snap try) [10:27] unless is done explciitly [10:27] pedronis: as long as they don't use instances (I doubt they do) it would work, right? [10:27] yes [10:27] perfect [10:27] we should coordinate with CE to let them know in case [10:28] it might also make sense [10:28] to have as experimental feature for a while [10:28] which means you really need to opt in [10:28] I agree [10:28] to play with it [10:28] experimental.parallel-installation [10:29] or somesuch [10:29] maybe use the word "instance" somehow [10:29] people will get LL vs L wrong [10:29] and instance is the word we use in various messages [10:29] mvo_: so one thing I was worried about yesterday [10:29] that I mentioned at the standup [10:29] but then it slipped my mind [10:30] was core transition with system in place [10:30] mborzecki: I will do another pass over InstanceName after lunch at this point [10:30] we need to do something about it [10:30] pedronis: great, thanks [10:30] we also need to look at the core snap and the existing real core-support plug there [10:30] I will park this branch and look after other work now, let's sync when you are back [10:31] meanwhile, i'm updating wife's laptop to f28 to check that xwayland thing, can't believe is coming down like this [10:33] that guy on arch, for whom the build failed had this in makepkg.conf LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,now", notice how it's missing -z before now [10:38] zyga: that kill -9 gvfsd -> https://i.imgur.com/Iu0soAq.jpg [10:40] pedronis: disconnect hooks conflict are more fun than autoconnect.. do you have a moment for HO? [11:11] pstolowski: nowish [11:11] pedronis: ok [11:12] pedronis: in the standup ho [11:14] * Chipaca lunches [11:28] Chipaca: https://raymii.org/s/blog/snap_install_mosaic_the_first_graphical_webbrowser_on_Ubuntu.html [11:28] :) [11:30] popey: :) [11:37] ah drat i was going to have lunch [11:37] * Chipaca is having one of those days [11:37] * Chipaca goes === pstolowski is now known as pstolowski|lunch [11:46] mborzecki: Sent some notes on #5314.. have a morning full of meetings from now on, so won't be able to complete it before my afternoon [11:46] PR #5314: many: rename snap.Info.Name() to snap.Info.InstanceName(), leave parallel-install TODOs [11:53] niemeyer: thanks, want to chat about this before/after the standup? [11:54] mborzecki: Sure, but can't before my afternoon.. have 4 meetings in sequence now [11:59] mborzecki: niemeyer: I should probably be there too, otherwise we might start to push in different directions just because of misaligments/misunderstanding [11:59] mvo_, niemeyer https://github.com/snapcore/snapd/pull/5309 is ready for another round [11:59] PR #5309: overlord/configstate: add watchdog options [12:03] mvo_: do you want to sync before the standup? [12:03] I will be absent later today for school stuff [12:03] zyga: sure we can sync [12:03] pedronis: fwiw, we can go down the introduce StoreName() (and make it the same as InstanceName()) route, either having TODOs and introducing the change later, or adding it right now works for me [12:03] zyga: when you want [12:03] ok, let's jump into the hangout [12:04] zyga: ok, I need one minute or so for tea [12:04] mborzecki: I'm fine with a do nothing StoreName personally [12:05] mborzecki: it seems also safer in the sense that new PRs might need to add places that need it actually [12:07] pedronis: ok, let do this then, i'll put aside the spread and snapstate stuff now and will focus on this [12:15] hi! what's the best way of using Ubuntu Core for an organisation. create a dummy user and add everyone's keys, or is there a way of registering an SSO account to a group with members? :-) [12:38] pedronis: I've pushed a slightly more channel-y test [12:39] pedronis: this'll all fall apart when we stop passing architecture to the store though :) === pstolowski|lunch is now known as pstolowski [12:40] Nafallo: create a user for the project, have it register the name of the snaps, and give devs developer access to the snap [12:40] Nafallo: then as the team evolves it's easier to manage than if you added the keys like you propose [12:40] Chipaca: thanks, this and the old one still miss the HasLen check though [12:41] pedronis: i'll just change the check to do a big deepequals [12:41] bah [12:41] better errors this way [12:41] hmm [12:42] * Chipaca fixes [12:43] pedronis: done [12:44] Chipaca: thanks, just need to be green now === mup_ is now known as mup [12:54] Chipaca: I'm not sure we'll be doing snaps, rather than using Ubuntu Core for container hosts. last step in registration requires an Ubuntu SSO e-mail to grab the SSH keys etc... :-) [12:56] Nafallo: ah! I'd misunderstood sorry [13:12] PR snapd#5332 closed: tests: econnreset - relax check for download start, match either download or assertion retry attempt [13:12] Nafallo: if i were in your place I'd check with mwhudson [13:13] Nafallo: but it's something like 1am for him so probably not on irc [13:18] alright. thanks for hilighting him :-) [13:44] PR snapd#5327 closed: store: switch store.SnapInfo to use the new v2/info endpoint [13:44] pedronis: ^ [13:45] \o/ [13:47] * zyga goes to school [13:49] Chipaca: nice! [13:49] PR snapd#5333 opened: tests: show status of the partial test-snapd-huge snap in econnreset test [13:49] cachio: ^- not sure if this is helpful but it might [13:50] cachio: or did you had a chance to look at this already via some debug shell? [13:50] mvo_, yes I did [13:51] mvo_, I reproduced the error with this https://github.com/snapcore/snapd/pull/5329/files [13:51] PR #5329: DON'T REVIEW: tests: Adding debug information to know why econnreset is failing [13:51] and the file was empty [13:52] the partial [13:56] zyga, do you think we can merge this one #5293 ? [13:56] PR #5293: tests: fix interfaces-calendar-service test when gvfsd-metadata loks the xdg dirctory [13:58] cachio: oh, interessting, it was empty, hm, hm, I though we had code that errors if the file is empty [13:58] pedronis: niemeyer: pushed StoreName to #5314 [13:58] PR #5314: many: rename snap.Info.Name() to snap.Info.InstanceName(), leave parallel-install TODOs [13:59] cachio: aha, nice, I see your branch already has most of the info I also am keen to see. let me check the log [14:01] cachio: if you have a log captured from the run from 5329 I would love to have a look. if not I will just wait for it to fail in spread :) [14:05] mvo_, let me check [14:09] mvo_, https://paste.ubuntu.com/p/Bq8Z2H7X7G/ [14:09] this is hte last one which I could reproduce from my machine [14:09] there is a reboot message at the end [14:43] niemeyer, today morning I run spread -gc and deleted 465 machines [14:43] Wow.. what was that about? [14:43] all of them created this morning [14:44] That sounds suspect [14:44] most of them created at 4am my time [14:45] niemeyer, I realized the number when I saw the log after all of them were deleted [14:46] cachio: What's the cause? Did you check what those machines were doing? [14:46] niemeyer, no, when I realized the big number all the machines were already deleted [14:47] cachio: Can you have a pass through Travis and see why that huge number was left behind? [14:47] niemeyer, sure [14:48] cachio: Thanks! [14:48] cachio: We just need to make sure we're not missing anything.. that high number of machines going past 2h definitely means something [14:50] niemeyer, yes, it could be caused with aprox 8 builds canceled [14:50] niemeyer, I launch 54 machines on each build [14:51] we launch [14:54] cachio: *cough* I think I found the reason for the econnect issue - the network ist just too fast. the debug log from my latest pr shows the test-snapd-huge.snap is there (no partial). so I think it got all downloaded in the time it took the script to do the iptable --block commmand [14:56] mvo_, yes, that could be a reason, but I tested and it takes like 5/10 seconds to download [14:56] I wonder what we should do: a) make test-snapd-huge bigger b) retry the test if its too fast c) add artificial throttling into snapd via some config option (I like (c)) [14:57] cachio: the log indicates its the case but let me double check, maybe we need to log slightly more (i.e. when a download finished) [14:58] Throttle vía Linux [14:58] * zyga sends regards from school [14:58] cachio: hm, you are right, someting is fishy [14:59] cachio: after 1s it had downloaded 33Mb which is fast but not fast enough unless it picks up speed very quickly afterwards [14:59] mvo_, yesterday I tested the same because I had the same theory [14:59] What is inside the snap? [15:00] so I made many downloads and allways it was taking like 10 secdonds to complete the downlaod [15:00] cachio: yeah, then maybe the blocking is not working correctly [15:00] as in, what takes up the space [15:01] cachio: because the debug output indicates that the file is fully there [15:01] mvo_, mmmm [15:02] mvo_, this explains why it is not retrying [15:02] cachio: exactly [15:03] cachio: so I think we should add code that checks if the file is fully there [15:03] -rw-r--r-- 1 test test 0 Jun 15 04:00 test-snapd-huge_1.snap.partial [15:03] but after the test we see this [15:04] mvo_: wouldn't chaning the sleep 1 to sleep .01 or sth be enough? [15:04] Chipaca: yeah, that might be worth a shoot [15:04] also that task should have a debug entry that cats the error log [15:04] I'm adding logging to store/Download fwiw [15:05] as i was looking into this as well right now [15:05] Chipaca: cool, let me update my PR to shorten the sleep and to give better errors [15:05] fwtw, I tried sleep = .1 in my PR (the one i talked about in the standup) and it didn't help [15:06] I think checking quicker is the right approach; my worry is that throttling will bite us in nasty ways [15:06] pstolowski: it seems to be baby steps, maybe it just takes too long for the kernel to apply the rule [15:07] pstolowski: its definitely confusing though [15:07] tell me ;) [15:07] :) [15:08] can we rule our caching out? I think that's the one thing i haven't investigated [15:08] mvo_: another thing we can do is add a rate limiter to store/download, for use in tests [15:09] mvo_, Chipaca something weird is that it is just happening in some systems [15:09] not in all of them [15:10] Chipaca: yeah, I was thinking about this [15:10] cachio: 'snap instal bofh' <- know the TRUE reasons for things [15:10] for example, the errors you are seeing are because.... We had to turn off that service to comply with the CDA Bill. [15:10] mvo_: first, logs :) [15:11] otherwise it's all guesswork [15:12] Chipaca: I added some more bits in the tests but yeah, probably debug logs++ [15:12]