[05:17] <mborzecki> morning
[05:51] <mup> PR snapcraft#2161 closed: many: automatically detect dependency changes <Created by kyrofa> <Closed by kyrofa> <https://github.com/snapcore/snapcraft/pull/2161>
[06:38] <zyga> hey everyone
[06:38] <zyga> I will have an unusual Friday with some school stuff interfering today
[06:38] <zyga> I will attempt to minimise it though
[06:38] <mvo> zyga: good morning
[06:39] <zyga> hey
[06:39] <zyga> I'm sorry, I collapsed yesterday, I was sleeping since ~~19:00
[06:39] <mvo> zyga: no worries
[06:39] <zyga> I saw your ping about managing to run apps on top of your PR, that's fantastic
[06:39] <zyga> I will share what I have on system slots
[06:40] <zyga> though it's not fully working yet (tests)
[06:40] <zyga> (tests are still choppy)
[06:40] <mvo> zyga: thanks, its small steps on my part but it is nice to see (some) things coming together
[06:41] <mvo> zyga: and thanks for working on the interfaces part!
[06:43] <mborzecki> zyga: restarted the build in 5325
[06:44] <zyga> I noticed someone did, thank you
[06:44] <zyga> that branch is cursed :)
[06:46] <mvo> hey mborzecki, good morning
[06:46] <mborzecki> mvo: hey
[06:47] <mborzecki> i'll restart the build in 5293 a couple of times, just to rule out the luck factor
[06:47] <mborzecki> #5293
[06:47] <mup> PR #5293: tests: fix interfaces-calendar-service test when gvfsd-metadata loks the xdg dirctory <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/5293>
[06:48] <mborzecki> if that's green, then we're down to just one notoriously flaky test - econnreset :P i'm sure pstolowski|afk  is super happy about that
[07:09] <pstolowski> mornings
[07:11] <pstolowski> mborzecki: i'm tempted to relax retry check in econnreset test to just for retry on either download or assertions, instead of expecting retries on download only
[07:11] <pstolowski> *to just check*
[07:22] <mup> PR snapd#5325 closed: interfaces: add Repository.AllInterfaces <Simple> <Created by zyga> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/5325>
[07:47] <zyga> mvo: I struggle with the number of places that need to know about "system",
[07:47] <zyga> I need to take my dog out but after that let's chat
[08:15] <mborzecki> hm that slack carshing xwayland is so weird
[08:15] <mborzecki> there's a backtrace posted in the forums, given that slack is classic, it's probably doing some LD_* magic
[08:16] <mborzecki> but none of the symbols seem to point back to /var/lib/snapd/snap/..
[08:16] <mborzecki> unless the backtrace is wrong, or at least the library locations
[08:20] <mvo> zyga: sounds good, just ping me when you are back and have time (I was in a different meeting just now)
[08:25] <mup> PR snapd#5330 opened: snapstate: support restarting snapd from the snapd snap on core18 <Created by mvo5> <https://github.com/snapcore/snapd/pull/5330>
[08:28] <zyga> mvo: re,
[08:30] <zyga> mvo: so I'm pondering this
[08:30] <zyga> the interface repository aspect itself is clean
[08:30] <zyga> but overlord is more subtle,
[08:30] <pedronis> mvo: hi, I remembered something about setup-profiles phase2 for core that we need to consider I think
[08:30] <zyga> we can either teach all of interface manager to skip/ignore system snap
[08:30] <zyga> and risk leaks to other managers perhaps
[08:31] <zyga> or return virtual system snap from snap manager's APIs
[08:32] <zyga> mvo: I was also considering actually installing a real snap that shows up in /snap but I think that is more complex actually (e.g. revert is messy now)
[08:32] <mvo> pedronis: oh, what is that?
[08:32] <pedronis> mvo: that it was also stopping going forward until we were restarted, that has relevance to autoconnect for example
[08:33] <mup> PR core18#27 opened: run-snapd-from-snap: start snapd after seeding <Created by mvo5> <https://github.com/snapcore/core18/pull/27>
[08:33] <pedronis> mvo: zyga:  we even the question of when we run autoconnect for things in system, either that needs to happen in the new snapd
[08:34] <pedronis> *either way
[08:34] <mvo> zyga: hm, would it help if snapstate would have a fake system snap?
[08:34] <zyga> that's a good point
[08:34] <zyga> pedronis: could it run.. twice?
[08:34] <pedronis> it doens't need to run twice
[08:34] <pedronis> is already in the right place
[08:34] <zyga> mvo: yes, it would, I'm doing that now
[08:34] <pedronis> is the waiting for restart that is missing now
[08:34] <zyga> mvo: that part actually works now, I'm just moving things around to have only one instance of snap.Info in memory
[08:34] <zyga> and to coordinate the implicit interfaces there
[08:35] <pedronis> mvo: zyga: are we going to filter it out of snap list ?
[08:35] <pedronis> or not
[08:35] <mvo> pedronis: aha, ok. I remember (vaguely). should we do that in link-snap now?
[08:35] <zyga> pedronis: currently it doesn't show up in snap list
[08:35] <zyga> it's only virtually returned by Get
[08:35] <pedronis> zyga: by hacking ?
[08:35] <zyga> and ignored by Set
[08:35] <pedronis> interesting
[08:35] <zyga> that is, it is only a snap you can get the "state" of but it's not really anywhere else
[08:36] <pedronis> bit unclear how that works with reloading profiles though
[08:36] <zyga> reloading profiles?
[08:36] <zyga> on startup with system key changes?
[08:36] <pedronis> yes
[08:37] <zyga> the interface manager has explicit code to load the virtual system snap
[08:37] <zyga> so that works ok
[08:37] <zyga> anyway, it's not fully baked yet
[08:37] <pedronis> you can cheat in  snapsWithSecurityProfiles
[08:37] <zyga> so we may abandon that totally
[08:37] <zyga> oh, yes, that's a good idea
[08:37] <zyga> I could remove the special code from interface manager then
[08:37] <pedronis> zyga: it sounds messy though overall from afar
[08:37] <pedronis> tbh
[08:37] <zyga> pedronis: I agree
[08:37] <zyga> pedronis: it's an experiment still, we may end up doing something totally different
[08:38] <pedronis> mvo: where we put depends exactly when we run autoconnect logic for "system" interfaces
[08:38] <pedronis> in theory it should happen in autoconnect for "core" or "snapd"
[08:39]  * mvo nods
[08:39] <pedronis> mvo: we can add waiting to link-snap (but is annoying if I remember) or auto-connect or yet something else (but then it's hard across upgrade)
[08:40] <pedronis> mvo: I think we need in auto-connect itself because upgrades, if it's in link-snap is the old snapd that needs to wait there
[08:40] <pedronis> and it will not
[08:41] <pedronis> mmh, well, not it will still wait in setup-profiles
[08:41] <pedronis> so maybe doing something different is ok
[08:41] <pedronis> but we need to think
[08:41] <mvo> pedronis: right. my gut feeling is that waiting in auto-connect - because that is close to why its needed. but I have not thought deeply about it yet
[08:44] <pedronis> mvo: which brings back the annoying part of that story that is about detecting rollbacks or missing reboots
[08:44] <pedronis> (I hoped we could live without but seems not)
[08:45] <pedronis> mvo: ah, but now we don't need to wait for a reboot, just a restart
[08:45] <pedronis> bit easier
[08:48] <mvo> pedronis: yeah, let me have a quick look
[08:49] <pedronis> mvo: to notice the issue, you need to do something like install a snap that needs a new auto-connected system interface, notice that is not connect, refresh snapd to one that has it, if snapd doesn't do auto-connect with the right new version, the connection is still not there
[08:49] <mvo> sil2100: hey, sorry that I didn't get to #24 for core18 earlier. I added some thoughts, hope they are helpful
[08:49] <pedronis> mvo: admittedly all a bit of a corner case
[08:49] <mup> PR #24: Renamed .bzrignore to .gitignore. Added .coverage. Removed the ./ in front of ignored paths <Created by elopio> <Closed by elopio> <https://github.com/snapcore/snapd/pull/24>
[08:51] <pedronis> mvo: which also reminds that we need to teach about the "snapd" snap to snapstate.go byKind
[08:51] <pedronis> it should sort first
[08:54] <mvo> pedronis: I guess http://paste.ubuntu.com/p/YbV92FPgd8/ is too naive for the wait?
[08:54] <mvo> pedronis: let me look at the sorting
[08:55] <pedronis> mvo: it's not unreasonable,  the issue is more  the rollback checks,  we don't need them for snapd, but we need them for core
[08:55] <pedronis> that one still fully reboots
[08:56] <pedronis> I mean  core as base
[08:56] <mvo> pedronis: aha, so we need all of GuardCoreSetupProfilesPhase2 back it seems
[08:56] <pedronis> some parts of it
[08:56] <pedronis> yes, sorry
[08:56] <mvo> pedronis: and teach it about bases
[08:56] <mvo> pedronis: thank you
[08:57] <pedronis> mvo: really only core
[08:57] <pedronis> as a base
[08:57] <pedronis> for now
[08:57] <pedronis> undering the assumption that base themselves
[08:57] <pedronis> don't have slots
[08:57] <pedronis> (we might change that at some point, but don't think we want to go there now)
[08:58] <mvo> pedronis: +1
[09:03] <mup> PR snapd#5331 opened: snapstate: sort "snapd" first <Created by mvo5> <https://github.com/snapcore/snapd/pull/5331>
[09:22] <zyga> mvo: ok, I have something that vaguely works
[09:22] <zyga> I can share it now, let me commit
[09:23] <zyga> it still doesn't pass unit tests
[09:26] <mvo> zyga: ok
[09:31] <mup> PR snapd#5332 opened: tests: relax check for download start, match either download or assertion retry attempt <Created by stolowski> <https://github.com/snapcore/snapd/pull/5332>
[09:36] <mborzecki> pedronis: do you think #5314 is close to landing now?
[09:37] <mup> PR #5314: many: rename snap.Info.Name() to snap.Info.InstanceName(), leave parallel-install TODOs <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/5314>
[09:37] <zyga> mvo: ok let me get rid of junk from history and push it
[09:38] <pedronis> mborzecki: probably, but I need to look at it again
[09:38] <pedronis> Chipaca: +1 to #5326 with some final nitpicks
[09:38] <mborzecki> pedronis: ok, take your time, better catch all the places now
[09:39] <mup> PR #5326: api/snapctl: allow -h and --help for regular users <Created by stolowski> <https://github.com/snapcore/snapd/pull/5326>
[09:39] <pedronis> sorry
[09:39] <pedronis> Chipaca: I meant #5327
[09:39] <mup> PR #5327: store: switch store.SnapInfo to use the new v2/info endpoint <Created by chipaca> <https://github.com/snapcore/snapd/pull/5327>
[09:39] <pedronis> mborzecki: of course we have some pending PRs adding more Name() and RealName uses
[09:41] <zyga> mvo: I have one idea I lijke
[09:41] <zyga> "snap info system"
[09:41] <zyga> could do what "snap version" says, e.g. synthesise nice description, summary,
[09:41] <zyga> use os-release
[09:41] <zyga> etc
[09:42] <zyga> it sounds nice and would be ... cool :)
[09:44] <mborzecki> pedronis: i guess i could nag people to add a todo note while they're at it, otoh wouldn't want to disrupt PRs too much, i can always merge this locally, since it's a rename the the code will fail to build (well maybe aside from RealName, this may easily fall under the radar)
[09:45] <pedronis> mborzecki: any further tought btw whether we should put InstanceKey in Info/SnapState/SnapSetup or SideInfo ?
[09:46] <zyga> mvo: command-not-found broke in bionic
[09:46] <zyga> it's broken, but not as you know it https://www.irccloud.com/pastebin/Za3Og5z0/
[09:46] <zyga> also, I wonder why it used "sudo snap install" twice
[09:49] <mvo> zyga: hm, hm, what version of snapd do you have? edge?
[09:49] <zyga> I'm running master
[09:49] <zyga> + my magic changes
[09:52] <mvo> zyga: this indicats some mismatch between snapd and snap
[09:52] <mborzecki> pedronis: i'm leaning towards side-info, other option i looked at is keeping it in snapstate as a separate field, but i'm not convinced, feels like it's shifting the cost of tracking it to other places, besides we already repeat a bunch of data in side-info for every revision, instance keys are relatively short (at least for now) so there wouldn't be that much overhead
[09:53] <zyga> mvo: ah, that's possible
[09:53] <zyga> I only have ./snap
[09:53] <mborzecki> heh formatting heredoc in task.yaml is awkward, especially if it's nested in some for loop/if block
[10:05] <pedronis> mborzecki: it still feels a bit dirty, because everything so far in SideInfo is revision/store related
[10:08] <Chipaca> error: e-book-client-error-quark[2] Conflicting UIDs found in added contacts
[10:08] <Chipaca> wtf
[10:13] <zyga> mvo: https://github.com/snapcore/snapd/compare/master...zyga:rfc/virtual-system-snap?expand=1
[10:13] <zyga> mvo: pull this and have a look
[10:13] <zyga> I will now look at what tests are unhappy
[10:13] <zyga> towards making it unit test happy
[10:13] <zyga> and then looking at spread happy
[10:13] <mborzecki> pedronis: fair point, let's assume then that it's in SnapState (and thus in Info & SnapSetup), a bit more work but will probably pay off in the long run
[10:14] <zyga> all suggestions welcome
[10:14] <pedronis> mborzecki: given that we have Name() in at least a couple of those it should be well hidden, the question is how many fuctions take SideInfo directly, which is something I'm looking into
[10:15] <pedronis> now
[10:15] <mborzecki> #5293 is green for the 5th time now, killing gvfsd seems to work, needs a 2nd review and we could land it
[10:15] <mup> PR #5293: tests: fix interfaces-calendar-service test when gvfsd-metadata loks the xdg dirctory <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/5293>
[10:16] <mvo_> zyga: thanks, I check after lunch
[10:16] <pedronis> mborzecki: not a tone, some Mock functions but we have options tehre
[10:16] <mborzecki> pedronis: heh, at least a couple of apis in snapstate will need adjustments, eg InstallPath which I touched recently
[10:16] <pedronis> mborzecki: I know,  I couple functions taking  *SideInfo,  will really need to take  *SideInfo, instanceKey string
[10:16] <mborzecki> pedronis: those apis which take a name should be fine though
[10:17] <mborzecki> name as in snap name
[10:17] <pedronis> yep
[10:18] <pedronis> mborzecki: the ReadInfo ones  are the mostly annoying, though usually we call them through other helpers
[10:18] <pedronis> mborzecki: also not sure the one that opens  the file (vs deals with an installed snap) needs the instance key
[10:19] <mborzecki> pedronis: iirc readinfo takes a name, it should be fine then
[10:19] <pedronis> ah, yes
[10:19] <pedronis> mborzecki: so maybe you could quickly check how many places doing  ReadInfoFromSnapFile  will need to stick a instanceKey in the result
[10:19] <pedronis> that seems the biggest sticking point
[10:21] <mborzecki> pedronis: yeah, i'm already setting it explicitly https://github.com/bboozzoo/snapd/blob/bboozzoo/parallel-snap-install-from-snap/snap/info.go#L933 so a simple adjustment
[10:23] <pedronis> mborzecki: ?, I'm talking about *SnapFile
[10:23] <pedronis> understood about just ReadInfo
[10:24] <pedronis> it's used not a lot
[10:24] <pedronis> we should be able to survice
[10:24] <pedronis> survive
[10:25] <mborzecki> yup, a handfull of places
[10:25] <pedronis> same OpenSnapFile
[10:25] <pedronis> with the latter we have a name or SnapSetup around
[10:26] <pedronis> mborzecki: so, yes I would try  to add InstanceKey to Info SnapState and SnapSetup and see how it goes
[10:26] <pedronis> seems cleaner long term
[10:26] <zyga> pedronis: what's the revert plan for instances
[10:26] <zyga> CE will test reverting core / snapd
[10:26] <pedronis> zyga ?
[10:26] <zyga> how do we plan that after installing instanced snaps revert won't cause issues on the system
[10:27] <pedronis> it's kind of hard
[10:27] <zyga> right but we need _a_ plan
[10:27] <zyga> CE has a test that shows that revert must work
[10:27] <pedronis> well on core devices nothing should do that
[10:27] <zyga> we've been burned by that before
[10:27] <mborzecki> with the former we seem to have the name in the caller (or don't care as in case of snap try)
[10:27] <pedronis> unless is done explciitly
[10:27] <zyga> pedronis: as long as they don't use instances (I doubt they do) it would work, right?
[10:27] <pedronis> yes
[10:27] <zyga> perfect
[10:27] <zyga> we should coordinate with CE to let them know in case
[10:28] <pedronis> it might also make sense
[10:28] <pedronis> to have as experimental feature for a while
[10:28] <pedronis> which means you really need to opt in
[10:28] <zyga> I agree
[10:28] <pedronis> to play with it
[10:28] <mborzecki> experimental.parallel-installation
[10:29] <mborzecki> or somesuch
[10:29] <zyga> maybe use the word "instance" somehow
[10:29] <zyga> people will get LL vs L wrong
[10:29] <zyga> and instance is the word we use in various messages
[10:29] <zyga> mvo_: so one thing I was worried about yesterday
[10:29] <zyga> that I mentioned at the standup
[10:29] <zyga> but then it slipped my mind
[10:30] <zyga> was core transition with system in place
[10:30] <pedronis> mborzecki: I will do another pass over InstanceName after lunch at this point
[10:30] <zyga> we need to do something about it
[10:30] <mborzecki> pedronis: great, thanks
[10:30] <zyga> we also need to look at the core snap and the existing real core-support plug there
[10:30] <zyga> I will park this branch and look after other work now, let's sync when you are back
[10:31] <mborzecki> meanwhile, i'm updating wife's laptop to f28 to check that xwayland thing, can't believe is coming down like this
[10:33] <mborzecki> that guy on arch, for whom the build failed had this in makepkg.conf LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,now", notice how it's missing -z before now
[10:38] <mborzecki> zyga: that kill -9 gvfsd -> https://i.imgur.com/Iu0soAq.jpg
[10:40] <pstolowski> pedronis: disconnect hooks conflict are more fun than autoconnect.. do you have a moment for HO?
[11:11] <pedronis> pstolowski: nowish
[11:11] <pstolowski> pedronis: ok
[11:12] <pstolowski> pedronis: in the standup ho
[11:14]  * Chipaca lunches
[11:28] <popey> Chipaca: https://raymii.org/s/blog/snap_install_mosaic_the_first_graphical_webbrowser_on_Ubuntu.html
[11:28] <popey> :)
[11:30] <Chipaca> popey: :)
[11:37] <Chipaca> ah drat i was going to have lunch
[11:37]  * Chipaca is having one of those days
[11:37]  * Chipaca goes
[11:46] <niemeyer> mborzecki: Sent some notes on #5314.. have a morning full of meetings from now on, so won't be able to complete it before my afternoon
[11:46] <mup> PR #5314: many: rename snap.Info.Name() to snap.Info.InstanceName(), leave parallel-install TODOs <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/5314>
[11:53] <mborzecki> niemeyer: thanks, want to chat about this before/after the standup?
[11:54] <niemeyer> mborzecki: Sure, but can't before my afternoon.. have 4 meetings in sequence now
[11:59] <pedronis> mborzecki: niemeyer: I should  probably be there too, otherwise we might start to push in different directions just because of misaligments/misunderstanding
[11:59] <abeato> mvo_, niemeyer https://github.com/snapcore/snapd/pull/5309 is ready for another round
[11:59] <mup> PR #5309: overlord/configstate: add watchdog options <Created by alfonsosanchezbeato> <https://github.com/snapcore/snapd/pull/5309>
[12:03] <zyga> mvo_: do you want to sync before the standup?
[12:03] <zyga> I will be absent later today for school stuff
[12:03] <mvo_> zyga: sure we can sync
[12:03] <mborzecki> pedronis: fwiw, we can go down the introduce StoreName() (and make it the same as InstanceName()) route, either having TODOs and introducing the change later, or adding it right now works for me
[12:03] <mvo_> zyga: when you want
[12:03] <zyga> ok, let's jump into the hangout
[12:04] <mvo_> zyga: ok, I need one minute or so for tea
[12:04] <pedronis> mborzecki: I'm fine with a do nothing StoreName personally
[12:05] <pedronis> mborzecki: it seems also safer in the sense that new PRs might need to add places that need it actually
[12:07] <mborzecki> pedronis: ok, let do this then, i'll put aside the spread and snapstate stuff now and will focus on this
[12:15] <Nafallo> hi! what's the best way of using Ubuntu Core for an organisation. create a dummy user and add everyone's keys, or is there a way of registering an SSO account to a group with members? :-)
[12:38] <Chipaca> pedronis: I've pushed a slightly more channel-y test
[12:39] <Chipaca> pedronis: this'll all fall apart when we stop passing architecture to the store though :)
[12:40] <Chipaca> Nafallo: create a user for the project, have it register the name of the snaps, and give devs developer access to the snap
[12:40] <Chipaca> Nafallo: then as the team evolves it's easier to manage than if you  added the keys like you propose
[12:40] <pedronis> Chipaca: thanks,  this and the old one still miss the HasLen check though
[12:41] <Chipaca> pedronis: i'll just change the check to do a big deepequals
[12:41] <Chipaca> bah
[12:41] <Chipaca> better errors this way
[12:41] <Chipaca> hmm
[12:42]  * Chipaca fixes
[12:43] <Chipaca> pedronis: done
[12:44] <pedronis> Chipaca: thanks, just need to be green now
[12:54] <Nafallo> Chipaca: I'm not sure we'll be doing snaps, rather than using Ubuntu Core for container hosts. last step in registration requires an Ubuntu SSO e-mail to grab the SSH keys etc... :-)
[12:56] <Chipaca> Nafallo: ah! I'd misunderstood sorry
[13:12] <mup> PR snapd#5332 closed: tests: econnreset - relax check for download start, match either download or assertion retry attempt <Created by stolowski> <Closed by stolowski> <https://github.com/snapcore/snapd/pull/5332>
[13:12] <Chipaca> Nafallo: if i were in your place I'd check with mwhudson
[13:13] <Chipaca> Nafallo: but it's something like 1am for him so probably not on irc
[13:18] <Nafallo> alright. thanks for hilighting him :-)
[13:44] <mup> PR snapd#5327 closed: store: switch store.SnapInfo to use the new v2/info endpoint <Created by chipaca> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/5327>
[13:44] <Chipaca> pedronis: ^
[13:45] <pedronis> \o/
[13:47]  * zyga goes to school
[13:49] <mvo_> Chipaca: nice!
[13:49] <mup> PR snapd#5333 opened: tests: show status of the partial test-snapd-huge snap in econnreset test <Created by mvo5> <https://github.com/snapcore/snapd/pull/5333>
[13:49] <mvo_> cachio: ^- not sure if this is helpful but it might
[13:50] <mvo_> cachio: or did you had a chance to look at this already via some debug shell?
[13:50] <cachio> mvo_, yes I did
[13:51] <cachio> mvo_, I reproduced the error with this https://github.com/snapcore/snapd/pull/5329/files
[13:51] <mup> PR #5329: DON'T REVIEW: tests: Adding debug information to know why econnreset is failing <Blocked> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/5329>
[13:51] <cachio> and the file was empty
[13:52] <cachio> the partial
[13:56] <cachio> zyga, do you think we can merge this one #5293 ?
[13:56] <mup> PR #5293: tests: fix interfaces-calendar-service test when gvfsd-metadata loks the xdg dirctory <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/5293>
[13:58] <mvo_> cachio: oh, interessting, it was empty, hm, hm, I though we had code that errors if the file is empty
[13:58] <mborzecki> pedronis: niemeyer: pushed StoreName to #5314
[13:58] <mup> PR #5314: many: rename snap.Info.Name() to snap.Info.InstanceName(), leave parallel-install TODOs <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/5314>
[13:59] <mvo_> cachio: aha, nice, I see your branch already has most of the info I also am keen to see. let me check the log
[14:01] <mvo_> cachio: if you have a log captured from the run from 5329 I would love to have a look. if not I will just wait for it to fail in spread :)
[14:05] <cachio> mvo_, let me check
[14:09] <cachio> mvo_, https://paste.ubuntu.com/p/Bq8Z2H7X7G/
[14:09] <cachio> this is hte last one which I could reproduce from my machine
[14:09] <cachio> there is a reboot message at the end
[14:43] <cachio> niemeyer, today morning I run spread -gc and deleted 465 machines
[14:43] <niemeyer> Wow.. what was that about?
[14:43] <cachio> all of them created this morning
[14:44] <niemeyer> That sounds suspect
[14:44] <cachio> most of them created at 4am my time
[14:45] <cachio> niemeyer, I realized the number when I saw the log after all of them were deleted
[14:46] <niemeyer> cachio: What's the cause?  Did you check what those machines were doing?
[14:46] <cachio> niemeyer, no, when I realized the big number all the machines were already deleted
[14:47] <niemeyer> cachio: Can you have a pass through Travis and see why that huge number was left behind?
[14:47] <cachio> niemeyer, sure
[14:48] <niemeyer> cachio: Thanks!
[14:48] <niemeyer> cachio: We just need to make sure we're not missing anything.. that high number of machines going past 2h definitely means something
[14:50] <cachio> niemeyer, yes, it could be caused with aprox 8 builds canceled
[14:50] <cachio> niemeyer, I launch 54 machines on each build
[14:51] <cachio> we launch
[14:54] <mvo_> cachio: *cough* I think I found the reason for the econnect issue - the network ist just too fast. the debug log from my latest pr shows the test-snapd-huge.snap is there (no partial). so I think it got all downloaded in the time it took the script to do the iptable --block commmand
[14:56] <cachio> mvo_, yes, that could be a reason, but I tested and it takes like 5/10 seconds to download
[14:56] <mvo_> I wonder what we should do: a) make test-snapd-huge bigger b) retry the test if its too fast c) add artificial throttling into snapd via some config option (I like (c))
[14:57] <mvo_> cachio: the log indicates its the case but let me double check, maybe we need to log slightly more (i.e. when a download finished)
[14:58] <zyga> Throttle vía Linux
[14:58]  * zyga sends regards from school
[14:58] <mvo_> cachio: hm, you are right, someting is fishy
[14:59] <mvo_> cachio: after 1s it had downloaded 33Mb which is fast but not fast enough unless it picks up speed very quickly afterwards
[14:59] <cachio> mvo_, yesterday I tested the same because I had the same theory
[14:59] <zyga> What is inside the snap?
[15:00] <cachio> so I made many downloads and allways it was taking like 10 secdonds to complete the downlaod
[15:00] <mvo_> cachio: yeah, then maybe the blocking is not working correctly
[15:00] <zyga> as in, what takes up the space
[15:01] <mvo_> cachio: because the debug output indicates that the file is fully there
[15:01] <cachio> mvo_, mmmm
[15:02] <cachio> mvo_, this explains why it is not retrying
[15:02] <mvo_> cachio: exactly
[15:03] <mvo_> cachio: so I think we should add code that checks if the file is fully there
[15:03] <cachio> -rw-r--r--   1 test test     0 Jun 15 04:00 test-snapd-huge_1.snap.partial
[15:03] <cachio> but after the test we see this
[15:04] <Chipaca> mvo_: wouldn't chaning the sleep 1 to sleep .01 or sth be enough?
[15:04] <mvo_> Chipaca: yeah, that might be worth a shoot
[15:04] <Chipaca> also that task should have a debug entry that cats the error log
[15:04] <Chipaca> I'm adding logging to store/Download fwiw
[15:05] <Chipaca> as i was looking into this as well right now
[15:05] <mvo_> Chipaca: cool, let me update my PR to shorten the sleep and to give better errors
[15:05] <pstolowski> fwtw, I tried sleep = .1 in my PR (the one i talked about in the standup) and it didn't help
[15:06] <Chipaca> I think checking quicker is the right approach; my worry is that throttling will bite us in nasty ways
[15:06] <mvo_> pstolowski: it seems to be baby steps, maybe it just takes too long for the kernel to apply the rule
[15:07] <mvo_> pstolowski: its definitely confusing though
[15:07] <pstolowski> tell me ;)
[15:07] <cachio> :)
[15:08] <pstolowski> can we rule our caching out? I think that's the one thing i haven't investigated
[15:08] <Chipaca> mvo_: another thing we can do is add a rate limiter to store/download, for use in tests
[15:09] <cachio> mvo_, Chipaca something weird is that it is just happening in some systems
[15:09] <cachio> not in all of them
[15:10] <mvo_> Chipaca: yeah, I was thinking about this
[15:10] <Chipaca> cachio: 'snap instal bofh' <- know the TRUE reasons for things
[15:10] <Chipaca> for example, the errors you are seeing are because.... We had to turn off that service to comply with the CDA Bill.
[15:10] <Chipaca> mvo_: first, logs :)
[15:11] <Chipaca> otherwise it's all guesswork
[15:12] <mvo_> Chipaca: I added some more bits in the tests but yeah, probably debug logs++
[15:12] <cachio> Chipaca, hehe
[15:12] <mvo_> Chipaca: I added some more debug and an explicit check if the download finished with dates, lets see what the outcome is
[15:13] <pstolowski> pedronis: i think i need to bring installTask back into checkConnectConflicts, it seems to be the only way to be able to reason about and avoid self-conflicts
[15:13] <pstolowski> pedronis: (for disconnect-interfaces)
[15:14] <pedronis> pstolowski: why would you have self conflicts ?
[15:14] <pedronis> I'm missing something
[15:14] <pedronis> pstolowski: what are you conflicting on?
[15:15] <pedronis> pstolowski: you probably need to add a flag like "auto" to the disconnects you create
[15:15] <pedronis> though
[15:15] <pedronis> maybe that's the missing bit
[15:15] <pstolowski> pedronis: for auto-connect we don't have them because we consider auto-connect after we linked the snap; but as I examine all non ready tasks in disconnected-interfaces i encounter own tasks for unlink-snap for example, as they are pending
[15:15] <pedronis> ah
[15:16] <pedronis> mmh
[15:17] <pedronis> no, don't understand
[15:18] <pedronis> pstolowski: I see, the problem is that the old code was naive
[15:19] <pstolowski> pedronis: we consider snap-removal, so we have disconnect-interfaces in the tasks and unlink-snap as well, we find conflict with ourselves
[15:19] <pedronis> though it might work because of the other conflict logic
[15:20] <pedronis> pstolowski: I see, you need to pass in your snapName  (no point passing in a task)
[15:20] <pstolowski> yes, or that
[15:20] <pedronis> pstolowski: yea, pleas pass just a name
[15:21] <pstolowski> ok
[15:21] <pedronis> and then just before the last if k == "unlink-snap" ....
[15:22] <pedronis> you need to check   if snapName is that name and continue?
[15:22] <pstolowski> yes that's the idea
[15:24] <pedronis> pstolowski: call it disconnectingSnap ?
[15:25] <pstolowski> sounds good
[15:34] <pedronis> mborzecki: did the last pass over #5314
[15:34] <mup> PR #5314: many: rename snap.Info.Name() to snap.Info.InstanceName(), leave parallel-install TODOs <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/5314>
[15:35] <cachio> niemeyer, well
[15:35] <cachio> for each build errored we leave 54 machines alive
[15:36] <pedronis> mborzecki: mostly that we don't need some todos, a we could use StoreName even a bit less... otoh I have  a question about app name shortcuts... also couple of places the todo is deeper than what to pick I think
[15:36] <cachio> and for each cancelled too
[15:36] <cachio> niemeyer, in the last 4 hours we left 108 machines
[15:37] <cachio> because build error
[15:38] <mborzecki> pedronis: thanks, i'll take a look
[15:38] <cachio> niemeyer, I don't have so much information about the executions on this morning because most of the builds were executed again and the logs are gone
[15:39] <pedronis> mborzecki: also there's a conflict with John merge I think
[15:44] <Chipaca> *shocked*
[15:47]  * cachio lunch
[15:51] <niemeyer> cachio: Ack, thanks
[15:51] <niemeyer> Lunch as well
[16:09] <Chipaca> mvo_: cachio: pushed some more debug to #5333
[16:09] <mup> PR #5333: tests: show status of the partial test-snapd-huge snap in econnreset test <Created by mvo5> <https://github.com/snapcore/snapd/pull/5333>
[16:09] <cachio> Chipaca, nice, thanks
[16:10] <mvo_> Chipaca: *hug* thank you
[16:10] <Chipaca> mvo_: :-)
[16:10] <mvo_> a review for 5324 would be really nice
[16:10] <Chipaca> mvo_: on it
[16:11] <mvo_> Chipaca: thank you again!
[16:31]  * Chipaca thinks popeycore is a genre of music, played by bands that dress up as pirates and out-nerd each other on performing authentic interpretations of sea shanties
[16:32] <Chipaca> either that, or really small popes
[16:32]  * Chipaca agrees
[16:50] <mup> PR core18#28 opened: Fix the UID/GID mapping <Created by sil2100> <https://github.com/snapcore/core18/pull/28>
[16:50]  * mvo_ hugs sil2100 
[17:22] <zyga> re
[17:27]  * zyga wonders how things are
[17:28] <mborzecki> zyga: things are 30 minutes till the match starts :)
[17:29] <zyga> mborzecki: I'm not a football fan rctually
[17:31]  * sil2100 hugs mvo_ back
[17:31] <sil2100> mvo_: I looked at your PR and I still need to test it, guess I'm too burned out today to not miss anything important
[17:31] <sil2100> So I'll review it properly on Monday morning
[17:31] <sil2100> o/
[17:36] <mup> PR snapd#5293 closed: tests: fix interfaces-calendar-service test when gvfsd-metadata loks the xdg dirctory <Created by sergiocazzolato> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/5293>
[17:37] <Chipaca> mvo_: hm
[17:37] <Chipaca> mvo_: [ "filename" -gt 1 ] does not seem like a good test for 'was the file downloaded'
[17:39]  * Chipaca lets the test go ahead just in case he's missing something
[17:39]  * Chipaca expects to see a 'integer expression expected' error
[17:39] <cachio> Chipaca, I also added your debug info to the other branch
[17:39] <Chipaca> cachio: heh, ok
[17:39] <cachio> so if I can reproduce the error it will help
[17:40] <Chipaca> cachio: including the debug: entry in task.yaml?
[17:40] <cachio> Chipaca, yes
[17:40] <Chipaca> yes part of the problem with this thing is that it wasn't printing anything useful on error
[17:40] <Chipaca> so unless you were running with -debug you were SOL
[17:40] <Chipaca> this should help
[17:40] <cachio> yes agree
[17:40] <cachio> the problem is that now it is not failing :)
[17:41] <cachio> it works properly when you add debug info
[17:41] <Chipaca> mbuahaha
[17:41] <Chipaca> proper heisenbug
[17:41] <Chipaca> it'll be doing crystal meth next
[18:07] <mvo_> Chipaca: did I forgot a -count ?
[18:08] <Chipaca> mvo_: what were you wanting that test to test
[18:08] <Chipaca> mvo_: and what command takes -count
[18:08] <mvo_> Chipaca: checking if test-snapd-huge is actually there
[18:08] <mvo_> Chipaca: instead of .partial
[18:08] <Chipaca> mvo_: I changed the test to [ -n "$(find ...)" ]
[18:09] <mvo_> Chipaca: aha, nice
[18:09] <Chipaca> mvo_: I suspected you were thinking [ "$(find ... | wc -l)" -gt 0 ]
[18:09] <Chipaca> but that seemes contrived for the -gt 0 case :)
[18:09] <Chipaca> and you'd written -gt 1 which made it worse :)
[18:10] <mvo_> Chipaca: :(
[18:10] <Chipaca> that si: it was [ "$(find test-snapd-hug_*.snap)" -gt 1 ]
[18:10] <Chipaca> which has a bug because bash would expand that * sometimes
[18:10] <mvo_> Chipaca: a clear sign I need to call it a week ;)
[18:10] <Chipaca> i mean
[18:10] <Chipaca> find -name <that>
[18:10] <Chipaca> heh
[18:11] <Chipaca> mvo_: yes :)
[18:11] <mvo_> Chipaca: thanks for fixing it
[18:11] <Chipaca> mvo_: worse thing was that it was working
[18:11] <Chipaca> :)
[18:11] <mvo_> Chipaca: it was?
[18:11] <Chipaca> because the [ was returning 2, but it was in an if, so the if was never true
[18:11] <Chipaca> 2 == WTF
[18:12] <mvo_> uff
[18:12] <Chipaca> :)
[18:12] <Chipaca> mvo_: gotta love shell
[18:12] <Chipaca> on fridays
[18:12] <mvo_> Chipaca: I do - with passion
[18:12] <Chipaca> post beer o'clock
[18:12] <mvo_> (and fire)
[18:12] <Chipaca> :-D
[18:12] <mvo_> Chipaca: :) beer o'clock - can't wait to see results from the PR
[18:14] <Chipaca> mvo_: same
[18:15] <Chipaca> mvo_: it failed before with errors in unrelated tests, of course
[18:15] <Chipaca> anyway, off to put pizzas in the oven
[18:15] <Chipaca> full friday mode here
[19:10]  * zyga hugs Chipaca’s rational Friday mode
[19:11] <zyga> Well
[19:11] <zyga> I meant to say I hug chipaca because of the pizza really
[19:35] <mup> PR snapd#5334 opened: tests: show executed tests on current system when a test fails <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/5334>
[20:20] <mup> PR snapd#5333 closed: tests: show status of the partial test-snapd-huge snap in econnreset test <Created by mvo5> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/5333>