[07:18] good morning [07:18] * zyga welcomes everyone after the busiest break ever :) [07:53] brb [08:02] re [08:07] hello mvo :) [08:13] hey zyga [08:13] how are you doing? [08:13] zyga: I think what you had last week (infection) got me sunday and today :/ [08:13] zyga: but otherwise doing well! [08:13] ouch, good luck with that [08:13] zyga: how are you? === pstolowski|afk is now known as pstolowski [08:13] morning! [08:13] hey pstolowski [08:13] mvo: tired [08:13] mvo: they are still at the hospital [08:14] maybe if all goes good they will be back on Tuesday [08:14] but I was told the exact same thing a week ago [08:14] zyga: uh, good luck! [08:17] sil2100: hey, are you guys in the process of building new cosmic images or something? or any other reason why our snapd sru to comsic got rejected? [08:22] * zyga goes to make tea, picks up ancient PR https://github.com/snapcore/snapd/pull/6111 [08:22] PR #6111: packaging/opensuse: move most logic to snapd.mk [08:22] mvo: the message i saw implied that vorlon wanted the version currently in -proposed to migrate to updates rather than superseding it and waiting another 7 days [08:22] mwhudson: aha, misread the message then [08:23] mwhudson: yeah, thats fine [08:23] mwhudson: I missed that cosmic got accepted [08:23] mvo: well, or i did :) [09:05] mvo: it got rejected? What was the rejection message? [09:36] mvo: hey, i've run console-conf twice and it consistently fails on network_wifi_static test - http://paste.ubuntu.com/p/fGwfvvSymX/ - know? must be a flaky test since i had no issue with console-conf during all the previous testing [09:37] *known [09:40] * dot-tobias says hi [09:40] o/ [10:01] hey Chipaca [10:01] it's good to be back :) [10:01] zyga: :-) you're feeling better, then? [10:01] zyga: less contagious? [10:01] Chipaca: yes [10:01] Chipaca: though the expanded family is not reunited yet, they are still at the hospital for a few more days [10:02] but there's good progress so hopefully by the middle of the week we'll be all home [10:16] Chipaca: can I just give you (and team) a massive <3 for having the date in snap info now. It really improves my workflow. [10:16] mvo: ok, hmmm [10:17] pstolowski: thanks! now that sergio is back its probably best to check with him, I have not run this test myself on the pi, only in a VM [10:17] mvo: ok [10:17] mvo: so, since there's 2.37.1+18.10 in cosmic-proposed ready to go, I guess Steve wanted that to be released, right? [10:17] mvo: I remember you mentioned .1 didn't have any blocking issues? [10:17] popey: that one goes all the way up to sabdfl [10:18] It's ace [10:18] sil2100: sorry for the slow reply, was in a meeting [10:18] sil2100: there is one small issue if people have symlinks in /var/lib/jenkins [10:18] sil2100: but I think we can fix this by removing "/var/lib/jenkins" from the core snap [10:19] sil2100: I slightly prefer .2 as it fixes this known regression but its really a corner case [10:20] eeek, so the core18 'fix' for jenkins that added /var/lib/jenkins actually caused a regression? [10:21] sil2100: it was not really a fix actually [10:21] sil2100: we misunderstood a subset of the bug :( [10:27] hey pedronis :) [11:07] * Chipaca takes a break [11:13] PR snapd#6488 opened: interfaces/network-manager: no peer label check for hostname1 [11:35] mvo: do you have a moment to re-review https://github.com/snapcore/snapd/pull/6486 ? that would unblock another pr [11:35] PR #6486: interfaces/hotplug: renamed RequestedSlotSpec to ProposedSlot, removed Specification [12:18] cachio: hey [12:18] pstolowski, hey [12:19] If someone from the Snappy HWE team would be so kind to have a look at this: https://forum.snapcraft.io/t/apparmor-denials-for-dbus-objectmanager-network-manager/9885/4?u=tobias → any ETA for network-manager 1.10 track on ARM? [12:19] cachio: how was your holidays? [12:19] pstolowski, awesome!!! thanks for asking [12:19] hey cachio ! welcome back [12:20] mvo, thanks [12:20] I am in the jungle again :( [12:21] mvo, I am creating images to run beta validation for 37.2 [12:23] cachio: great! [12:25] cachio: i've been running console-conf validation tests this morning, and network_wifi_static test was failing consistently; is this a known issue? full log - http://paste.ubuntu.com/p/fGwfvvSymX/ [12:25] pstolowski, checking [12:26] pstolowski, it was passing during last executions [12:26] pstolowski, which board is that? [12:27] pi2 pi3? [12:27] cachio: pi3 [12:28] cachio: it's possible i did something wrong. the testing guide doesn't tell to pre-configure pi3 before tests, but i found it was necessary (manual setup of console-conf on first boot, so that test scripts can connect) [12:29] pstolowski, yes, initial setup is required [12:29] it is for 37.2? [12:30] niemeyer, hey [12:30] niemeyer, I see a permissions error creating new images https://paste.ubuntu.com/p/zsPBGfszkw/ [12:30] cachio: Hey [12:31] niemeyer, all the images scripts have failed due to this permission error [12:31] ogra: curious that you'd have 6 hardlink-count-1 files in the cache; has anything strange happened recently? [12:32] ogra: like, snapd crashing mid install? [12:32] or … dunno :-) [12:32] cachio: Are you sure you have the right credentials? Just reminding of the previous issue we discussed [12:32] cachio: I'm checking the project [12:33] niemeyer, well, those scripts are using the sa that you created a time ago [12:33] the new one [12:33] image-baker [12:33] cachio: I know they should be.. just raising the fact last time it was a credentials issue. [12:33] cachio: Anyway, checking [12:34] ogra: ah, figured it out, you've done a 'snap remove' :-) [12:35] cachio: ok. i'm pretty positive console-conf works and it was just something wrong with the test. i've been using console-conf manually a lot last week during beta-validation [12:37] where should I put a de-logger? (that converts the debug http snapd logs to something more legible) test-snapd-tools? [12:37] pstolowski, it is probably there is a problem with the test, I'll check it later today [12:37] pstolowski, it is for 2.37.2 right? [12:38] cachio: Done, please test now [12:39] cachio: Something must have changed in the code [12:39] cachio: Either way, I just added the permission.. doesn't seem problematic [12:39] cachio: right [12:39] niemeyer, what I saw reviewing the logs is that the image is created but the permission error is raised [12:40] niemeyer, no code changes in our side but perhaps the google library I am using changed [12:40] cachio: That's what I mean too [12:41] niemeyer, :) [12:41] niemeyer, thanks for the update [12:41] cachio: Please let me know if it works [12:42] niemeyer, can I run it now? [12:42] niemeyer, in progress [12:44] hey niemeyer :) [12:44] hey cachio :) [12:44] zyga: Heya! [12:44] zyga: How're things there? [12:45] niemeyer: great, I just got a call from my wife that they are coming home today :) [12:45] zyga: Fantastic news [12:47] zyga: \o/ [12:48] cachio: thank you for the validation, we did a lot of those already, give me a sec, I will send you our trello link [12:48] mvo, THANKS [12:49] mvo: I've built the early version of refresh-app-awareness [12:50] zyga@yantra:~/snapd/cmd/snapd> snap refresh gnome-calculator [12:50] error: cannot refresh "gnome-calculator": snap "gnome-calculator" has running apps or hooks [12:50] niemeyer: ^ will be ready for review by the next sprint [12:51] cachio: see /msg - I sent you the link and also added you to the card [12:51] zyga: nice! [12:51] zyga: really nice [12:52] zyga: Sweet! [12:53] zyga: How's the strategy for not delaying forever going? [12:53] niemeyer: it's not, I just started on this todat [12:53] (I'm back after last week) [12:53] zyga: Ack [12:54] mvo, nice, it is ok if I finish the missing ones? [12:54] niemeyer: I think we will do what was planned in cape town [12:54] niemeyer, ERROR: (gcloud.compute.images.create) Could not fetch resource: [12:54] zyga, hey [12:54] cachio: sure [12:54] cachio: thank you! [12:55] all of the code so far is on https://github.com/snapcore/snapd/compare/master...zyga:feature/refresh-app-awareness?expand=1 [12:55] mvo, I'll update the card [12:55] and review the results [12:55] cachio: ? [12:56] niemeyer, it is failing but now with that permission error [12:56] cachio: That's not a permission error [12:56] cachio: great, thank you! [12:56] cachio: The permission error is what comes below it [12:57] Chipaca: hey [12:57] zyga: 'sup [12:57] do you have a minute for a silly question [12:57] so I added an extra check in Update and UpdateMany [12:57] when I run snap refresh gnome-calculator [12:57] niemeyer, right now it failed because I need to clean up the temp images [12:58] I get this delay of about 4 seconds [12:58] niemeyer, sorry, there was a name overlap [12:58] and then it does stuff and actually says that the snap cannot be refreshed [12:58] cachio: Coo [12:58] l [12:58] Chipaca: so either I did something silly and I really do the refresh check and then fail [12:58] or something else is at play [12:58] any ideas [12:58] ? [12:59] Chipaca: https://github.com/snapcore/snapd/compare/master...zyga:feature/refresh-app-awareness?expand=1#diff-4f584006240926759b614bfd921c5fc0R1147 [12:59] * Chipaca looks [13:01] zyga: I'm not sure where the 4 seconds are coming from, but you're checking all snaps instead of just those that have a refresh? [13:01] Chipaca: I'm checking those that have the refresh, I think [13:01] zyga: you get the candidates after doing the soft refresh check [13:01] right [13:01] so it shouldn't be slow, right? [13:02] zyga: log stuff before, during, and after the loop? [13:02] is UpdateMany hit on one snap name? [13:02] or is that always routed to Update? [13:02] zyga: one snap goes to Update [13:02] Update takes many more options [13:02] snap refresh <1 snap> [13:02] always goes through Update [13:03] 2019/02/11 14:03:42.831902 snapstate.go:1174: before soft check [13:03] 2019/02/11 14:03:42.832542 snapstate.go:1176: failed soft check [13:04] so the check itself is fast, must be something prior to that [13:04] I'll dig [13:04] might be assertion refreshes [13:04] zyga: sorry, silly question [13:04] if you have a lot of snaps [13:04] zyga: why check before looking for candidates? [13:04] you'd be erroring out even for snaps that don't have a refresh [13:05] ah, good point [13:05] well [13:05] that sounds wrong :) [13:05] hmm [13:05] especially for auto-refresh [13:05] zyga: also, also [13:05] zyga: why is softwarerefreshcheck not cehcking all snaps in a batch? [13:05] if you are running and you are asking to refresh something is it better to get nothing to do or cannot because running? [13:06] Chipaca: why would it? [13:06] zyga: the things it's doing look like they'd be a lot more efficient checking for N snaps at once instead of 1 at a time in a loop [13:06] which things specifically? [13:06] I could move the get-from-state part outside of the function [13:07] but other than that? [13:07] Chipaca: as for timing, Update is called after the delay, so something else is slow [13:07] Chipaca: zyga@yantra:~/snapd/cmd/snapd> SNAPD_DEBUG=1 snap refresh gnome-calculator [13:07] 2019/02/11 14:06:22.606869 cmd_linux.go:70: DEBUG: re-exec not supported on distro "opensuse-tumbleweed" yet [13:07] 2019/02/11 14:06:27.672286 error.go:100: DEBUG: error: cannot refresh "gnome-calculator": snap "gnome-calculator" has running apps or hooks [13:07] error: cannot refresh "gnome-calculator": snap "gnome-calculator" has running apps or hooks [13:08] Chipaca: the check is entirely per-snap anyway [13:08] zyga: the logs in snapd are the ones you want, not the ones in snap :-) [13:08] Chipaca, nothing has happened but my install is really old so it could well bee that at some point something did happen [13:08] definitely nothing recently though [13:08] aha [13:09] it checks for a zillion assertions first [13:09] zyga: as pedronis siad [13:09] said* [13:10] ogra: but it should be cleaned up on any install/refresh [13:10] hmm, weird then [13:11] ogra: can you turn on debug and install something you don't have? (may i suggest 'icdiff' :) [13:12] Chipaca: one more question [13:12] zyga: shoot [13:13] Chipaca: when refreshing many I may get three cases: some can be refreshed and some cannot or all can or cannot be refreshed [13:13] when all cannot be refreshed it can be a synchronous error [13:13] but when some can be refreshed [13:13] I would like to refresh those [13:13] and still do something sane with the ones I cannot [13:13] Chipaca, not right now but i'll try that later [13:13] any advice on that? [13:14] zyga: it depends on what was requested [13:14] snap refresh gnome-calculator core # refreshing core in the background but failing the gnome-calculator refresh in a way that the user sees [13:14] we have various code that does similar things right now just with logs [13:14] but has comments to move to warning [13:15] I see [13:15] zyga: if it's a bare 'snap refresh', I'd suggest to tell the user and update what can be done [13:15] should that be a warning? [13:15] Chipaca: tell via warning or is there another mechanism? [13:15] hold on [13:15] if it's "snap refresh foo bar baz", i'd say fail the whole thing [13:16] auto-refresh, use warnings [13:16] ok [13:16] I'll try that :) [13:16] yes [13:16] that's three different things [13:16] but first, walk the dog [13:16] 'snap refresh': print message, update what's updatable [13:16] time is running quickly today [13:16] 'snap refresh foo bar baz' -> error [13:16] auto-refresh -> warning [13:16] Chipaca: print message -> warning, right? [13:16] or if not, how do I print? [13:16] zyga: no [13:16] logger or something smarter? [13:17] niemeyer, https://paste.ubuntu.com/p/B6KnsSfsxG/ [13:17] niemeyer, same error [13:17] zyga: abuse the fact that the log of tasks get printed to the user? [13:17] not sure [13:17] compute.globalOperations.get permission [13:17] might need something adhoc [13:17] cachio: The image admin service now has that permission [13:17] haha, ok [13:17] Chipaca, pedronis: thank you, that is super useful [13:17] Chipaca: we don't have good support for that case [13:17] cachio: I suspect something very similar to the issue we've observed last time is taking place [13:18] Chipaca: inform but proceed [13:18] pedronis: zyga: we could use warnings for the inform-but-proceed but it'd suck a bit [13:18] Chipaca: agreed [13:18] unless we tweaked refresh for this case [13:18] but if we're tweaking, we could tweak to jfdi [13:18] * pedronis quick errands [13:18] Oh my [13:19] And now GCE just did something silly and destroyed the role due to the lack of atomicity of their operations [13:19] ARGH [13:20] niemeyer, :( [13:21] Extremely frustrating [13:21] cachio: We'll need to recreate the role again from the ground up.. but I'll have lunch first === ricab is now known as ricab|lunch [13:21] niemeyer, sure, np [13:21] thanks for helping [13:43] PR snapcraft#2466 closed: ruby plugin: support new download URL [13:45] cachio: i think our nested vm images will need updating and checking, i found out last week while working on hotplug tests that 16.04 image couldn't be found anymore; i'm using SPREAD_NESTED_SYSTEM=bionic with google-nested:ubuntu-18.04-64 which works but upgrades grub and some other packages on every run [13:46] pstolowski, thanks for the headsup [13:47] I'll do it ones the issue with gce account is fixed, I hope today [13:47] once [13:51] mvo: should we revert the core18 change for /var/lib/jenkins then? [13:51] mvo: I'm asking because I'd like to spin the first .2 core18 candidate images [13:52] mvo: is it safe to use the current core18 + snapd 2.37.1? [13:59] sil2100: I think reverting adding /var/lib/jenkins is the right move [14:02] mvo: let me do that then, I'll try to fast-track the new core18 to stable ASAP [14:10] sil2100: \o/ [14:24] cachio: I've recreated the role.. can you please give it a shot when you have a moment? I'll likely have missed something [14:31] niemeyer, hey, ready [14:31] niemeyer, do we need a new sa.json file? [14:33] zyga: mvo: the two PRs I was mentioning were #6111 and #6410 [14:33] PR #6111: packaging/opensuse: move most logic to snapd.mk [14:33] PR #6410: release-tools: add debian-package-builder [14:35] cachio: No, I've just lost the roles [14:35] cachio: Sorry, the permissions of the role actually [14:35] niemeyer, nice [14:35] niemeyer, I'll try again in that case [14:35] cachio: Google replaced the permissions instead of updating, due to some kind of error in the console [14:35] cachio: So I had to come up with the permissions again, which of course will likely be wrong [14:36] pedronis: ok [14:36] niemeyer, :( [14:36] niemeyer, I am running a task now [14:36] Thanks [14:37] I'll let you know if it goes well [14:37] mvo: we also need to remove /var/lib/jenkins from core, right? [14:38] mvo: can we though? will it break 2.37.1 using things? [14:38] pedronis: yeah, sil2100 is working on removing /var/lib/jenkins [14:38] mvo: from core18, there is also core [14:38] pedronis: it should not break, we need to double check but its marked as "optional" in the snap-confine [14:39] pedronis: we need to double check if that "optional" checks both sides of the bind mount [14:39] yes === ricab|lunch is now known as ricab [14:43] cachio: hey, one question, is it possible to re-run individual beta validation tests with the scripts? or only manually without help of the scripts (SPREAD_EXTERNAL_ADDRESS=...) ? [14:45] mvo: seems either side can be missing afaict [14:47] pedronis: ok, if thats the case we should be good by just removing it from the core snap with no ill effects [14:52] pstolowski, yes [14:52] pstolowski, SPREAD_TESTS=external:ubuntu-core-16-arm-64:tests/main/interfaces-hostname-control [14:53] will run just that test [14:53] pstolowski, SPREAD_TESTS=external:ubuntu-core-16-arm-64:tests/main/interfaces-hostname-control,external:ubuntu-core-16-arm-64:tests/main/snap-logs [14:53] will run both [14:54] pstolowski, it is possible to skip tests too with SKIP_TESTS=tests/main/revert-devmode [15:01] Hey snappy people. I get a "Revision 410 is not approved." when trying to release microk8s on arm64. Why? How can I see what is wrong? [15:04] kjackal: why - it's still pending automated review. what's wrong? nothing, you just need to wait a bit for it to finish review :) [15:04] Ah, ok, where did you see that? [15:04] (I was afraid i broke something in the builders) [15:05] kjackal: in the dashboard.snapcraft.io page for microk8s; you'll notice (on the left side) that revision 410 has a clock next to it, indicating it's awaiting review [15:05] kjackal: if you click on it, you'll see "automated review not yet completed" and "task xxxxxx awaiting execution" [15:06] Ahh dashboard! I was only looking at the https://snapcraft.io/microk8s/releases [15:06] thank you roadmr [15:06] pedronis: so, wrt macaroon expiry, indeed the refresh endpoint is ok and the download endpoint 401's [15:06] another thing to remember (if I'm right) is that automated tests are sequentrial, so if you have two builds ready for testing they will be done one-at-a-time [15:06] --r [15:07] kjackal: np. Usually if something breaks we let you know about it; if you have no news, it usually means things are going well [15:07] no news is good news! [15:07] ... unless Trump has gone quiet :-o [15:08] if you wish upon a star... [15:12] pedronis: I addressed the remodel feedback now [15:19] niemeyer, https://paste.ubuntu.com/p/mc3SttFRwv/ [15:19] this permission is missing [15:20] cachio: Added [15:20] niemeyer, tx [15:26] mvo: thanks, I will again [15:26] Chipaca: ok, can we do something only snapd side or do we need store changes? [15:28] ta [15:33] niemeyer, images are being created now [15:33] niemeyer, thanks for the support [15:33] cachio: Nice, phew [15:50] pedronis: there's nothing to tell snapd the macaroon is expired until suddenly download 401s [15:50] hmmm [15:51] pedronis: we could add code to just retry without the macaroon? [15:51] PR snapd#6486 closed: interfaces/hotplug: renamed RequestedSlotSpec to ProposedSlot, removed Specification [15:52] * Chipaca tries that [15:53] pedronis: #5962 updated [15:53] PR #5962: ifacestate/hotplug: hotplug handlers [15:56] mwhudson, mvo: yes, cosmic-proposed version released now [15:56] Chipaca: can't we know we don't need to send the macaroon upfront ? do we produce an error early if it's just "snap refresh one-snap" [15:56] * zyga waits for the final paperwork to go home from hospital [15:57] I may be back to work some more later tonight but I think I will really be back tomorrow, this is a good time as any to EOD [15:58] pedronis: wrt sending the macaroon, we can check private==false, and maybe there's code to check payment [15:58] not sure we have all the bits for the latter but we might [15:58] Chipaca: there'a Paid flag [15:58] pedronis: where? [15:58] not sure if we keep around or not [15:58] Chipaca: on Info for sure [15:59] pedronis: ah, details_v2 sets it from prices [15:59] ok [15:59] zyga: yeah, take family time [16:00] pedronis: is this what we want to do? [16:00] Chipaca: possibly, but best if to discuss this in Malta with the store? [16:00] Chipaca: does the test I mentioned btw passes or fail (on staging) ? [16:02] pedronis: I don't know [16:02] Chipaca: could you try it ? [16:02] vorlon: thank you, I will do a .2 soonish then [16:02] pedronis: I don't know :-) [16:02] pedronis: i don't know how to run spread against staging, do you know where that's documented / explained / described? [16:02] mvo: will do the PR to remove /var/lib/jenkins from core [16:03] pedronis: thank you, please check with sil2100 to avoid duplication [16:03] mvo: sorry [16:03] mvo: I meant to ask will you do? [16:03] Chipaca: not sure it's documented, cachio should be able to tell you [16:04] sil2100: hey, are you on the /var/lib/jenkins dir removal from core/core18 PR? asking because pedronis is keen to get rid of it and we want to avoid duplication of effort [16:05] cachio: hiya. I'd like to run a spread test against the staging store. How do I do that? [16:06] cachio: one of the ones that use SPREAD_STORE_USER and _PASSWORD [16:06] Chipaca, you need to check a PR? [16:06] cachio: master [16:07] Chipaca, we ran today against master [16:07] Chipaca, this is the log [16:08] Chipaca, https://travis-ci.org/snapcore/spread-cron/builds/491457033 [16:08] pedronis: is system-files meant to be used to give you access to things in /sys? [16:08] cachio: thanks! [16:08] mvo: I reverted the change in core18, I guess the packages are now in testing [16:09] Chipaca, this is the config we use to run it https://github.com/snapcore/spread-cron/blob/snapd-staging-store/run-checks [16:09] Chipaca: in which sense? it can be used for that, but a proper interface would be better [16:09]