[06:11] <zyga> o/
[06:48] <mup> PR snapd#7224 opened: xdgopenproxy: update test API to match upstream <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/7224>
[06:51] <Aavar> Good morning
[06:53] <zyga> o/
=== pstolowski|afk is now known as pstolowski
[07:09] <pstolowski> morning
[07:12] <zyga> hey pawel
[07:12] <zyga> I'm looking at https://github.com/snapcore/snapd/pull/7222/files
[07:12] <zyga> I have some doubts about that code, digging now
[07:12] <mup> PR #7222: tests: show just the last log as part of the debug output when check journal logs <Simple 😃> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7222>
[07:20] <pstolowski> zyga: good, ty, i was confused about it
[07:25] <pstolowski> zyga: Sergio said it's "log=$(get_journalctl_log "$@")", is this because of -x flag when we run scripts?
[07:28] <zyga> if that's -x then the commit message and explanation are confusing
[07:28] <zyga> I thought he meant the actual log being printed over and over
[07:29] <zyga> network is super slow
[07:35] <zyga> pstolowski: do you think this will pass? https://www.irccloud.com/pastebin/kg4wlBTJ/
[07:38] <pstolowski> zyga: hmm, no retry-loop? i'm skeptical, although i've never paid attention to the logs of passed tests where we actually retried, perhaps not needed anymore? worth trying out
[07:39] <zyga> I think the whole retry loop is a sign of broken stuff elsewhere
[07:41] <zyga> eh
[07:41] <zyga> everything fails
[07:41] <zyga> this is so depressing
[07:41] <zyga> how can we change anything where everything is always broken
[07:46] <pstolowski> zyga: maybe the retry loop was added in the early days of journal log checks and before we understood it better and flush/sync were added
[07:54] <zyga> yeah
[07:54] <zyga> pstolowski: 215/350 passed
[07:54] <zyga> without the loop
[08:05] <mup> PR snapd#7225 opened: tests: don't repeat checks <Created by zyga> <https://github.com/snapcore/snapd/pull/7225>
[08:07] <zyga> pstolowski: I opened https://github.com/snapcore/snapd/pull/7225 -- it passed on xenial locally, let's see what happens across a broader set of distributions
[08:07] <mup> PR #7225: tests: don't repeat checks <Created by zyga> <https://github.com/snapcore/snapd/pull/7225>
[08:10] <pstolowski> zyga: we should also re-run it multiple times if it passes
[08:11] <zyga> I think it will fail
[08:11] <zyga> but not on this
[08:11] <pstolowski> uhm
[08:12] <zyga> hmm, some drunk folks with chainsaws are going to cut a tree next to our house
[08:12] <zyga> "official woodcutters"
[08:12] <zyga> how I hate our countryside attitude
[08:13] <zyga> before getting to work, get f*** drunk
[08:13] <zyga> complain to anyone and you'll have hell
[08:49] <zyga> pstolowski: https://github.com/snapcore/snapd/pull/7219#pullrequestreview-273016244
[08:49] <mup> PR #7219: devicestate/firstboot: check for missing bases early <Needs Samuele review> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7219>
[08:53] <pstolowski> zyga: ty
[08:57] <pstolowski> zyga: anything you want me to review?
[08:58] <zyga> pstolowski: nothing, sorry; I need to re-think my approach
[08:58] <zyga> mount stuff is stuck in recursive issues
[08:58] <zyga> cgroup stuff is new and too early
[08:58] <zyga> I would love help with making tests more robust
[08:58] <zyga> that helps everyone
[08:59] <pstolowski> yeah, i'm contemplating where to start with this
[09:00] <zyga> pstolowski: I have a simple suggestion
[09:00] <zyga> pstolowski: read main top to bottom
[09:00] <zyga> garden cruft
[09:00] <zyga> some is so obvious it pains to read
[09:00] <zyga> one branch per cruft
[09:00] <zyga> at a deeper level we neet better tooling
[09:00] <zyga> *need
[09:01] <zyga> but even with better tooling, we really need better test quality
[09:01] <zyga> and we need to maintain existing tests
[09:07] <zyga> they are commencing to cut the tree next to our house
[09:07] <zyga> I'll go and pay attention now
[09:07] <zyga> pstolowski: one more advice, it's good to review everything to bottom
[09:07] <zyga> it helps our colleagues to progress
[09:07] <zyga> even if you don't feel comfortable with something, just say so in the review
[09:11] <pstolowski> sure, +1
[09:11] <pstolowski> zyga: good luck with those trees..
[09:12] <pstolowski> only test that failed in one of the PRs:
[09:12] <pstolowski> prepare of google:ubuntu-16.04-64:tests/main/
[09:12] <pstolowski> Fetching snap "ubuntu-core"
[09:12] <pstolowski> error: stream error: stream ID 1; PROTOCOL_ERROR
[09:13] <zyga> yeah
[09:13] <zyga> it's great to have all the moving parts
[09:13] <zyga> something always fails
[09:13] <zyga> eh :/
[09:32] <zyga> trees keep falling
[09:38] <pstolowski> zyga: trees as master tree or tress as physical trees ? :}
[09:38] <pstolowski> zyga: i though that already landed https://github.com/snapcore/snapd/pull/7202 ; should we try to land this asap?
[09:38] <mup> PR #7202: tests: sync journal log before start the test <Simple 😃> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7202>
[09:38] <zyga> pstolowski: physical trees :)
[09:39] <pstolowski> ah, falling, not failing. right ;)
[09:39] <zyga> restarted :/
[09:44] <mup> PR snapd#7226 opened: tests: mountinfo-tool fail if there are no matches <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/7226>
[09:48] <pstolowski> +1 with suggestion ^
[10:10] <pstolowski> zyga: how far is https://github.com/snapcore/snapd/pull/7168 from being landable?
[10:10] <mup> PR #7168: tests: measure testbed for leaking mountinfo entries <Created by zyga> <https://github.com/snapcore/snapd/pull/7168>
[10:15] <zyga> pstolowski: far, blocked by spread bug
[10:15] <zyga> Though we may try to scope it to fewer systems
[10:15] <zyga> We may also disable tests, such as lxd, where the bugs matter
[10:16] <zyga> One of the two trees is down
[10:16] <zyga> Looking at the more risky one
[10:16] <zyga> It leans heavily towards another house
[10:35] <zyga> Second tree down
[10:35] <zyga> It was about 180 years old
[10:36] <zyga> I’ll get back to work soon
[10:36] <pstolowski> zyga: sounds sad :(
[10:59] <zyga> drama over
[11:00] <pstolowski> zyga: just got protocol error on recently restarted PR; this is ~4th occurence of protocol error that i see today
[11:00] <zyga> ok, back to work
[11:01] <zyga> pstolowski: I saw two more
[11:16]  * pstolowski lunch
[12:18] <zyga> everything ultimately fails on protocol error this week
[12:21] <pstolowski> :(
[12:22] <pstolowski> and we are no closer to fixing this, which is super annoying
[12:30] <zyga> pstolowski: it seems there's an outage
[12:31] <zyga> pstolowski: I'm just doing reviews now
[12:31] <zyga> pstolowski: https://github.com/snapcore/snapd/pull/7225
[12:31] <zyga> ha
[12:31] <zyga> this passed
[12:31] <zyga> ;D
[12:31] <mup> PR #7225: tests: don't repeat checks <Created by zyga> <https://github.com/snapcore/snapd/pull/7225>
=== ricab is now known as ricab|lunch
[12:32] <pstolowski> zyga: nice.. i'd land it once situation is under control and stable though
[12:32] <zyga> yeah, no rush on that one
[12:32] <zyga> but it reaffirms my feeling that that loop was papering over something else being bogus
[12:36] <zyga> either I have more network woes or github is slow
[12:37] <zyga> https://status.snapcraft.io/ indicates outage
[12:37] <zyga> please refrain from restarting anything
[12:47] <tomwardill> ah, zyga beat me to it. Store is having issues atm, we're starting to shed some search load to try and get out of it.
[12:47] <tomwardill> so expect intermittent service at best I'm afraid
[12:48] <DreyMIX> hello! api.snapcraft.io is down?
[12:48] <popey_> yes, see status.snapcraft.io for details
[12:49] <DreyMIX> ok thx, I was updating ubuntu to version 19. What do I do now?
[12:52] <popey_> DreyMIX: you're mid way through an Ubuntu desktop upgrade?
[12:53] <DreyMIX> Yes, I launched the terminal update via the "do-release-upgrade" command
[12:55] <tomwardill> problem identified, should start seeing store improvement soonish
[12:56] <popey_> DreyMIX: what state is it in right now? is there an error message or something? (maybe paste it to paste.ubuntu.com) ?
[12:59] <DreyMIX> popey_Unfortunately I had a previous error in the sendmail package during the installation, so I had to finish the process. Now I have uninstalled sendmail. How can I restore the update?
[13:00] <popey_> DreyMIX: so has the upgrade crashed out?
[13:00] <DreyMIX> yes
[13:01] <popey_> Ok, can you look in /etc/apt/sources.list and see if it's retained the new release codename of 'disco' (for 19.04)
[13:01] <popey_> (sometimes the upgrade rolls back, but I suspect it hasn't here)
[13:01] <DreyMIX> ok i check
[13:03] <DreyMIX> https://paste.ubuntu.com/p/wNrvNBRRC7/
[13:03] <DreyMIX> However if I try to re-run the "do-release-upgrade" command it tells me that there is no new release.
[13:03] <popey_> hang on
[13:04] <popey_> ok, good, it's partially upgraded.
[13:04] <popey_> try this: sudo dpkg --configure -a
[13:04] <popey_> if it just immediately finishes, chances are your upgrade is done.
[13:04] <popey_> it's more likely however to being processing a bunch of packages as the upgrade isn't finished
[13:05] <DreyMIX> ok done.  He gave me no output
[13:05] <popey_> ok, good
[13:05] <popey_> what desktop did you use before the upgrade, GNOME? KDE?
[13:05] <DreyMIX> gnome
[13:05] <popey_> ok, there's an extra command I'd run just to make sure...
[13:05] <DreyMIX> ubuntu 18.10
[13:05] <popey_> sudo apt install ubuntu-desktop^
[13:05] <popey_> note the ^ on the end
[13:05] <DreyMIX> ok
[13:06] <popey_> this *may* install some extra packages (which may have been missing) or it might do nothing but spit a lot of text out
[13:06] <DreyMIX> 0 updated, 0 installed, 0 to remove and 0 not updated.
[13:07] <DreyMIX> Would everything seem ok?
[13:18] <tomwardill> store is operating normally, let us know if you see anything weird (er than normal)
[13:22] <zyga> cachio: so, two things I was trying to say over HO
[13:23] <zyga> cachio: 1) try the http 2.0 off switch idea
[13:23] <zyga> cachio: 2) try checking the error tracker for a spike in errors from snapd, snapd reports failed installs, we can see when this started perhaps
[13:24] <zyga> pstolowski, ijohnson, degville, cachio: I cannot join the video call in any meaningful way now
[13:24] <zyga> have a productive afternoon and then a great weekend and let's come back to fight this next week
[13:25] <zyga> roadmr: ^ not sure if you can see the error tracker data
[13:25] <zyga> but it's a data point for us
[13:25] <degville> zyga: thanks for letting us know - you too.
[13:26] <roadmr> zyga: I can give the error tracker a try, where does it live?
[13:29] <zyga> roadmr: all I know is the user side: https://errors.ubuntu.com/
[13:29] <roadmr> zyga: oh that tracker! sure, I'll have a look. Thanks!
[13:31] <cachio> zyga, so, to disable http2 should be enough with GODEBUG=http2client=0
[13:31] <cachio> zyga, where I need to add that
[13:32] <cachio> to the systemd local config for snapd?
[13:35] <zyga> cachio: yeah, I think so
[13:35] <zyga> in the override file
[13:35] <zyga> we set SNAPD_DEBUG in the same way AFAIR
[13:36] <cachio> zyga, nice, I'll try it
[13:36] <zyga> thank you!
[13:36] <zyga> :)
=== ricab|lunch is now known as ricab
[14:01]  * zyga goes to the roof to tweak the antenna some more
[14:31] <mup> PR snapd#7227 opened: tests: Enable verbose http2 <⛔ Blocked> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7227>
[14:49] <mup> PR core18#136 opened: Remove the 60-unminimize motd, identify system as Ubuntu Core 18 <Created by sil2100> <https://github.com/snapcore/core18/pull/136>
[15:08] <ijohnson> hmm so I am hacking on snapd same as I used to, but now it seems there is an API mismatch between the "snap" command and the "snapd" daemon
[15:08] <ijohnson> $ snap install docker
[15:08] <ijohnson> error: cannot install "": cannot install snap with empty name
[15:09] <mup> PR snapd#7228 opened: add audio-playback/record and pulseaudio spread tests <Created by jdstrand> <https://github.com/snapcore/snapd/pull/7228>
[15:10] <ijohnson> even if I disable re-exec for the "snap" command with SNAP_REEXEC, I still can't get it to work: https://pastebin.canonical.com/p/Pv3h4gWFKx/
[15:12]  * cachio lunch
[15:26] <pstolowski> ijohnson: interesting, i had the same a few weeks ago.. tl;dr i've removed and refreshed all the vendor stuff in my tree, not sure what really changed.. try that maybe
[15:27] <pstolowski> ijohnson: what i noticed practically every command that required snap argument stopped working when this happened
[15:27] <ijohnson> right that's exactly what I'm seeing now
[15:28] <pstolowski> ijohnson: right.. remove all vendor packages and get deps againb
[15:28] <ijohnson> yeah okay that did it updating the vendor directory
[15:28] <ijohnson> thanks pstolowski!
[15:29] <pstolowski> ijohnson: cool!
[15:30] <pstolowski> ijohnson: i'm now wondering if it's not a real issue
[15:31] <ijohnson> something with gorilla/context? that was the dep that got removed for me
[15:33] <pstolowski> ijohnson: commit be4fc4d117c255cadd697a93f9f94e49c708f2c3
[15:34] <ijohnson> interesting
[15:35] <ijohnson> this is a situation where it would be nice if we were able to use go modules and not have govendor and go itself be out of sync with respect to the dependencies :-(
[15:38] <pstolowski> ijohnson: i don't get why didn't it break the build for me
[15:38] <ijohnson> yeah that's the most troubling part of this all is why the build works but is subtly broken
[15:39] <pstolowski> ijohnson: and until you hit it now i was suspecting some kind of messup on my VM
[15:40] <ijohnson> yeah I think I'll investigate this a bit, but this got me thinking it would be nice to have logging of the HTTP request/responses from the "snap" binary and snapd
[15:41] <ijohnson> we log HTTP stuff with SNAPD_HTTP_DEBUG to the store from snapd, but nothing on the "snap" client and only the response code/time/etc. on the snapd side with SNAPD_DEBUG
[15:53] <mup> PR snapd#5644 closed: interfaces: add audio-playback/audio-record and make pulseaudio manually connect <Created by jdstrand> <Merged by jdstrand> <https://github.com/snapcore/snapd/pull/5644>
[15:59] <ogra> stgraber, i'm trying to run debootstrap inside an lxc container (on top of ubuntu-core but that shouldnt matter) .... does that only work in privileged containers ?
[15:59] <ogra> root@bionic:~/cubox-classic# debootstrap bionic foobar
[15:59] <ogra> mknod: /root/cubox-classic/foobar/test-dev-null: Operation not permitted
[15:59] <ogra> E: Cannot install into target '/root/cubox-classic/foobar' mounted with noexec or nodev
[15:59] <ogra> (is there any doc for this ?)
[16:00] <stgraber> ogra: yes, only privileged containers can mknod stuff. It's in theory possible with LXD edge + 5.0 kernel to get our syscall interception to make this work for unpriv containers but it's still pretty untested for this scenario
[16:05] <ogra> thanks btw ... :)
=== pstolowski is now known as pstolowski|afk
[17:25]  * zyga crashed again, 
[17:25] <zyga> I need a day of sleep
[18:23] <mup> PR snapcraft#2514 closed: test: autopkgtest beta <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2514>
[18:26] <mup> PR snapcraft#2658 opened: candidate testing <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2658>
[19:31] <ijohnson> hey cachio, I've seen google:debian-sid-64:tests/main/sbuild:{any,normal} both fail repeatedly in spread, do you think that's related to the protocol errors we've been seeing?
[19:31] <ijohnson> I remember seeing a PR from you split up that test into 2 different ones
[19:31] <cachio> ijohnson, hey, I was researching this today
[19:32] <cachio> the command is getting stuck
[19:32] <ijohnson> I don't know enough about the debian build task to know what's going on, but yeah I see
[19:32] <ijohnson> E: Build failure (dpkg-buildpackage died)
[19:32] <ijohnson> as the reason for it dying
[19:32] <cachio> so far I could not reproduce it and the lgos dont provide usefull information
[19:32] <ijohnson> yeah I don't see much useful info other than just that
[19:32] <ijohnson> hmm
[19:32] <cachio> ijohnson, the problem is that it is not showing any output
[19:33] <ijohnson> ah is it supposed to show more output?
[19:33] <cachio> I ran that manually and all the sbuild output is printed on the screen
[19:35] <cachio> ijohnson, I just started a job with -show-output to see the output of the test
[19:35] <ijohnson> Ah okay, nice
[19:35] <ijohnson> I just was gonna see if you knew what was up with that test, but it seems you're on top of things
[19:35] <ijohnson> thanks
[19:36] <cachio> ijohnson, perhaps it is consuming to much cpu
[19:36] <cachio> trying to be :)
[19:36] <cachio> but now working arout the protocol error
[19:38] <ijohnson> :-) yeah it would be good to figure out what we can do about the protocol error
[20:08]  * juliank tried and failed to use the code snap and the go snap together to build go apps in visual studio code
[20:09] <juliank> When /snap/bin/code invokes /snap/bin/go nothing happens
[20:09] <juliank> I'd have hoped this would work, given both are classic snaps
[20:10] <ijohnson> juliank: what do you mean nothing happens, can you open the terminal in vscode and use the go command there?
[20:10] <ijohnson> I've used the go snap inside vscode as a snap and it did work, although I do know of one issue with vscode and other snaps specifically
[20:11] <ijohnson> the bug I know about is https://bugs.launchpad.net/snapd/+bug/1835805
[20:11] <mup> Bug #1835805: strict snap run from classic snap can't write to filesystem <snapd:New> <https://launchpad.net/bugs/1835805>
[20:11] <juliank> ijohnson: it works from the terminal
[20:12] <juliank> ijohnson: if run on save by code, nothing happens
[20:12] <ijohnson> hmm, do you mean one of the go extensions doesn't work?
[20:13] <juliank> If I snap remove go, apt install golang it works fine
[20:13] <ijohnson> do you see any denials in the system journal with `journalctl -e --no-pager | grep DENIED` ?
[20:14] <juliank> let me reinstall the snap and check
[20:14] <juliank> As I see quite a few snap-confine file_inherit denials
[20:14] <ijohnson> the journal should still be around even if you removed the snap
[20:14] <ijohnson> interesting
[20:15] <ijohnson> that's the same kind of denial I saw in my bug report
[20:15] <juliank> AVC apparmor="DENIED" operation="file_inherit" profile="/usr/lib/snapd/snap-confine" pid=31372 comm="snap-confine" family="unix" sock_type="stream" protocol=0 requested_mask="send receive" denied_mask="send receive" addr=none peer_addr=none
[20:15] <ijohnson> could you comment on the bug report I linked to above with your log and reproducing instructions?
[20:16] <juliank> huh it works from snap run --shell
[20:21] <juliank> Let's start with a fresh code
[20:26]  * cachio afk
[22:44] <mup> PR core-build#52 opened: initramfs: get snap parameters from grubenv <Created by cmatsuoka> <https://github.com/snapcore/core-build/pull/52>