| zyga | o/ | 06:11 |
|---|---|---|
| mup | PR snapd#7224 opened: xdgopenproxy: update test API to match upstream <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/7224> | 06:48 |
| Aavar | Good morning | 06:51 |
| zyga | o/ | 06:53 |
| === pstolowski|afk is now known as pstolowski | ||
| pstolowski | morning | 07:09 |
| zyga | hey pawel | 07:12 |
| zyga | I'm looking at https://github.com/snapcore/snapd/pull/7222/files | 07:12 |
| zyga | I have some doubts about that code, digging now | 07:12 |
| mup | PR #7222: tests: show just the last log as part of the debug output when check journal logs <Simple 😃> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7222> | 07:12 |
| pstolowski | zyga: good, ty, i was confused about it | 07:20 |
| pstolowski | zyga: Sergio said it's "log=$(get_journalctl_log "$@")", is this because of -x flag when we run scripts? | 07:25 |
| zyga | if that's -x then the commit message and explanation are confusing | 07:28 |
| zyga | I thought he meant the actual log being printed over and over | 07:28 |
| zyga | network is super slow | 07:29 |
| zyga | pstolowski: do you think this will pass? https://www.irccloud.com/pastebin/kg4wlBTJ/ | 07:35 |
| pstolowski | zyga: hmm, no retry-loop? i'm skeptical, although i've never paid attention to the logs of passed tests where we actually retried, perhaps not needed anymore? worth trying out | 07:38 |
| zyga | I think the whole retry loop is a sign of broken stuff elsewhere | 07:39 |
| zyga | eh | 07:41 |
| zyga | everything fails | 07:41 |
| zyga | this is so depressing | 07:41 |
| zyga | how can we change anything where everything is always broken | 07:41 |
| pstolowski | zyga: maybe the retry loop was added in the early days of journal log checks and before we understood it better and flush/sync were added | 07:46 |
| zyga | yeah | 07:54 |
| zyga | pstolowski: 215/350 passed | 07:54 |
| zyga | without the loop | 07:54 |
| mup | PR snapd#7225 opened: tests: don't repeat checks <Created by zyga> <https://github.com/snapcore/snapd/pull/7225> | 08:05 |
| zyga | pstolowski: I opened https://github.com/snapcore/snapd/pull/7225 -- it passed on xenial locally, let's see what happens across a broader set of distributions | 08:07 |
| mup | PR #7225: tests: don't repeat checks <Created by zyga> <https://github.com/snapcore/snapd/pull/7225> | 08:07 |
| pstolowski | zyga: we should also re-run it multiple times if it passes | 08:10 |
| zyga | I think it will fail | 08:11 |
| zyga | but not on this | 08:11 |
| pstolowski | uhm | 08:11 |
| zyga | hmm, some drunk folks with chainsaws are going to cut a tree next to our house | 08:12 |
| zyga | "official woodcutters" | 08:12 |
| zyga | how I hate our countryside attitude | 08:12 |
| zyga | before getting to work, get f*** drunk | 08:13 |
| zyga | complain to anyone and you'll have hell | 08:13 |
| zyga | pstolowski: https://github.com/snapcore/snapd/pull/7219#pullrequestreview-273016244 | 08:49 |
| mup | PR #7219: devicestate/firstboot: check for missing bases early <Needs Samuele review> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7219> | 08:49 |
| pstolowski | zyga: ty | 08:53 |
| pstolowski | zyga: anything you want me to review? | 08:57 |
| zyga | pstolowski: nothing, sorry; I need to re-think my approach | 08:58 |
| zyga | mount stuff is stuck in recursive issues | 08:58 |
| zyga | cgroup stuff is new and too early | 08:58 |
| zyga | I would love help with making tests more robust | 08:58 |
| zyga | that helps everyone | 08:58 |
| pstolowski | yeah, i'm contemplating where to start with this | 08:59 |
| zyga | pstolowski: I have a simple suggestion | 09:00 |
| zyga | pstolowski: read main top to bottom | 09:00 |
| zyga | garden cruft | 09:00 |
| zyga | some is so obvious it pains to read | 09:00 |
| zyga | one branch per cruft | 09:00 |
| zyga | at a deeper level we neet better tooling | 09:00 |
| zyga | *need | 09:00 |
| zyga | but even with better tooling, we really need better test quality | 09:01 |
| zyga | and we need to maintain existing tests | 09:01 |
| zyga | they are commencing to cut the tree next to our house | 09:07 |
| zyga | I'll go and pay attention now | 09:07 |
| zyga | pstolowski: one more advice, it's good to review everything to bottom | 09:07 |
| zyga | it helps our colleagues to progress | 09:07 |
| zyga | even if you don't feel comfortable with something, just say so in the review | 09:07 |
| pstolowski | sure, +1 | 09:11 |
| pstolowski | zyga: good luck with those trees.. | 09:11 |
| pstolowski | only test that failed in one of the PRs: | 09:12 |
| pstolowski | prepare of google:ubuntu-16.04-64:tests/main/ | 09:12 |
| pstolowski | Fetching snap "ubuntu-core" | 09:12 |
| pstolowski | error: stream error: stream ID 1; PROTOCOL_ERROR | 09:12 |
| zyga | yeah | 09:13 |
| zyga | it's great to have all the moving parts | 09:13 |
| zyga | something always fails | 09:13 |
| zyga | eh :/ | 09:13 |
| zyga | trees keep falling | 09:32 |
| pstolowski | zyga: trees as master tree or tress as physical trees ? :} | 09:38 |
| pstolowski | zyga: i though that already landed https://github.com/snapcore/snapd/pull/7202 ; should we try to land this asap? | 09:38 |
| mup | PR #7202: tests: sync journal log before start the test <Simple 😃> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7202> | 09:38 |
| zyga | pstolowski: physical trees :) | 09:38 |
| pstolowski | ah, falling, not failing. right ;) | 09:39 |
| zyga | restarted :/ | 09:39 |
| mup | PR snapd#7226 opened: tests: mountinfo-tool fail if there are no matches <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/7226> | 09:44 |
| pstolowski | +1 with suggestion ^ | 09:48 |
| pstolowski | zyga: how far is https://github.com/snapcore/snapd/pull/7168 from being landable? | 10:10 |
| mup | PR #7168: tests: measure testbed for leaking mountinfo entries <Created by zyga> <https://github.com/snapcore/snapd/pull/7168> | 10:10 |
| zyga | pstolowski: far, blocked by spread bug | 10:15 |
| zyga | Though we may try to scope it to fewer systems | 10:15 |
| zyga | We may also disable tests, such as lxd, where the bugs matter | 10:15 |
| zyga | One of the two trees is down | 10:16 |
| zyga | Looking at the more risky one | 10:16 |
| zyga | It leans heavily towards another house | 10:16 |
| zyga | Second tree down | 10:35 |
| zyga | It was about 180 years old | 10:35 |
| zyga | I’ll get back to work soon | 10:36 |
| pstolowski | zyga: sounds sad :( | 10:36 |
| zyga | drama over | 10:59 |
| pstolowski | zyga: just got protocol error on recently restarted PR; this is ~4th occurence of protocol error that i see today | 11:00 |
| zyga | ok, back to work | 11:00 |
| zyga | pstolowski: I saw two more | 11:01 |
| * pstolowski lunch | 11:16 | |
| zyga | everything ultimately fails on protocol error this week | 12:18 |
| pstolowski | :( | 12:21 |
| pstolowski | and we are no closer to fixing this, which is super annoying | 12:22 |
| zyga | pstolowski: it seems there's an outage | 12:30 |
| zyga | pstolowski: I'm just doing reviews now | 12:31 |
| zyga | pstolowski: https://github.com/snapcore/snapd/pull/7225 | 12:31 |
| zyga | ha | 12:31 |
| zyga | this passed | 12:31 |
| zyga | ;D | 12:31 |
| mup | PR #7225: tests: don't repeat checks <Created by zyga> <https://github.com/snapcore/snapd/pull/7225> | 12:31 |
| === ricab is now known as ricab|lunch | ||
| pstolowski | zyga: nice.. i'd land it once situation is under control and stable though | 12:32 |
| zyga | yeah, no rush on that one | 12:32 |
| zyga | but it reaffirms my feeling that that loop was papering over something else being bogus | 12:32 |
| zyga | either I have more network woes or github is slow | 12:36 |
| zyga | https://status.snapcraft.io/ indicates outage | 12:37 |
| zyga | please refrain from restarting anything | 12:37 |
| tomwardill | ah, zyga beat me to it. Store is having issues atm, we're starting to shed some search load to try and get out of it. | 12:47 |
| tomwardill | so expect intermittent service at best I'm afraid | 12:47 |
| DreyMIX | hello! api.snapcraft.io is down? | 12:48 |
| popey_ | yes, see status.snapcraft.io for details | 12:48 |
| DreyMIX | ok thx, I was updating ubuntu to version 19. What do I do now? | 12:49 |
| popey_ | DreyMIX: you're mid way through an Ubuntu desktop upgrade? | 12:52 |
| DreyMIX | Yes, I launched the terminal update via the "do-release-upgrade" command | 12:53 |
| tomwardill | problem identified, should start seeing store improvement soonish | 12:55 |
| popey_ | DreyMIX: what state is it in right now? is there an error message or something? (maybe paste it to paste.ubuntu.com) ? | 12:56 |
| DreyMIX | popey_Unfortunately I had a previous error in the sendmail package during the installation, so I had to finish the process. Now I have uninstalled sendmail. How can I restore the update? | 12:59 |
| popey_ | DreyMIX: so has the upgrade crashed out? | 13:00 |
| DreyMIX | yes | 13:00 |
| popey_ | Ok, can you look in /etc/apt/sources.list and see if it's retained the new release codename of 'disco' (for 19.04) | 13:01 |
| popey_ | (sometimes the upgrade rolls back, but I suspect it hasn't here) | 13:01 |
| DreyMIX | ok i check | 13:01 |
| DreyMIX | https://paste.ubuntu.com/p/wNrvNBRRC7/ | 13:03 |
| DreyMIX | However if I try to re-run the "do-release-upgrade" command it tells me that there is no new release. | 13:03 |
| popey_ | hang on | 13:03 |
| popey_ | ok, good, it's partially upgraded. | 13:04 |
| popey_ | try this: sudo dpkg --configure -a | 13:04 |
| popey_ | if it just immediately finishes, chances are your upgrade is done. | 13:04 |
| popey_ | it's more likely however to being processing a bunch of packages as the upgrade isn't finished | 13:04 |
| DreyMIX | ok done. He gave me no output | 13:05 |
| popey_ | ok, good | 13:05 |
| popey_ | what desktop did you use before the upgrade, GNOME? KDE? | 13:05 |
| DreyMIX | gnome | 13:05 |
| popey_ | ok, there's an extra command I'd run just to make sure... | 13:05 |
| DreyMIX | ubuntu 18.10 | 13:05 |
| popey_ | sudo apt install ubuntu-desktop^ | 13:05 |
| popey_ | note the ^ on the end | 13:05 |
| DreyMIX | ok | 13:05 |
| popey_ | this *may* install some extra packages (which may have been missing) or it might do nothing but spit a lot of text out | 13:06 |
| DreyMIX | 0 updated, 0 installed, 0 to remove and 0 not updated. | 13:06 |
| DreyMIX | Would everything seem ok? | 13:07 |
| tomwardill | store is operating normally, let us know if you see anything weird (er than normal) | 13:18 |
| zyga | cachio: so, two things I was trying to say over HO | 13:22 |
| zyga | cachio: 1) try the http 2.0 off switch idea | 13:23 |
| zyga | cachio: 2) try checking the error tracker for a spike in errors from snapd, snapd reports failed installs, we can see when this started perhaps | 13:23 |
| zyga | pstolowski, ijohnson, degville, cachio: I cannot join the video call in any meaningful way now | 13:24 |
| zyga | have a productive afternoon and then a great weekend and let's come back to fight this next week | 13:24 |
| zyga | roadmr: ^ not sure if you can see the error tracker data | 13:25 |
| zyga | but it's a data point for us | 13:25 |
| degville | zyga: thanks for letting us know - you too. | 13:25 |
| roadmr | zyga: I can give the error tracker a try, where does it live? | 13:26 |
| zyga | roadmr: all I know is the user side: https://errors.ubuntu.com/ | 13:29 |
| roadmr | zyga: oh that tracker! sure, I'll have a look. Thanks! | 13:29 |
| cachio | zyga, so, to disable http2 should be enough with GODEBUG=http2client=0 | 13:31 |
| cachio | zyga, where I need to add that | 13:31 |
| cachio | to the systemd local config for snapd? | 13:32 |
| zyga | cachio: yeah, I think so | 13:35 |
| zyga | in the override file | 13:35 |
| zyga | we set SNAPD_DEBUG in the same way AFAIR | 13:35 |
| cachio | zyga, nice, I'll try it | 13:36 |
| zyga | thank you! | 13:36 |
| zyga | :) | 13:36 |
| === ricab|lunch is now known as ricab | ||
| * zyga goes to the roof to tweak the antenna some more | 14:01 | |
| mup | PR snapd#7227 opened: tests: Enable verbose http2 <â›” Blocked> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7227> | 14:31 |
| mup | PR core18#136 opened: Remove the 60-unminimize motd, identify system as Ubuntu Core 18 <Created by sil2100> <https://github.com/snapcore/core18/pull/136> | 14:49 |
| ijohnson | hmm so I am hacking on snapd same as I used to, but now it seems there is an API mismatch between the "snap" command and the "snapd" daemon | 15:08 |
| ijohnson | $ snap install docker | 15:08 |
| ijohnson | error: cannot install "": cannot install snap with empty name | 15:08 |
| mup | PR snapd#7228 opened: add audio-playback/record and pulseaudio spread tests <Created by jdstrand> <https://github.com/snapcore/snapd/pull/7228> | 15:09 |
| ijohnson | even if I disable re-exec for the "snap" command with SNAP_REEXEC, I still can't get it to work: https://pastebin.canonical.com/p/Pv3h4gWFKx/ | 15:10 |
| * cachio lunch | 15:12 | |
| pstolowski | ijohnson: interesting, i had the same a few weeks ago.. tl;dr i've removed and refreshed all the vendor stuff in my tree, not sure what really changed.. try that maybe | 15:26 |
| pstolowski | ijohnson: what i noticed practically every command that required snap argument stopped working when this happened | 15:27 |
| ijohnson | right that's exactly what I'm seeing now | 15:27 |
| pstolowski | ijohnson: right.. remove all vendor packages and get deps againb | 15:28 |
| ijohnson | yeah okay that did it updating the vendor directory | 15:28 |
| ijohnson | thanks pstolowski! | 15:28 |
| pstolowski | ijohnson: cool! | 15:29 |
| pstolowski | ijohnson: i'm now wondering if it's not a real issue | 15:30 |
| ijohnson | something with gorilla/context? that was the dep that got removed for me | 15:31 |
| pstolowski | ijohnson: commit be4fc4d117c255cadd697a93f9f94e49c708f2c3 | 15:33 |
| ijohnson | interesting | 15:34 |
| ijohnson | this is a situation where it would be nice if we were able to use go modules and not have govendor and go itself be out of sync with respect to the dependencies :-( | 15:35 |
| pstolowski | ijohnson: i don't get why didn't it break the build for me | 15:38 |
| ijohnson | yeah that's the most troubling part of this all is why the build works but is subtly broken | 15:38 |
| pstolowski | ijohnson: and until you hit it now i was suspecting some kind of messup on my VM | 15:39 |
| ijohnson | yeah I think I'll investigate this a bit, but this got me thinking it would be nice to have logging of the HTTP request/responses from the "snap" binary and snapd | 15:40 |
| ijohnson | we log HTTP stuff with SNAPD_HTTP_DEBUG to the store from snapd, but nothing on the "snap" client and only the response code/time/etc. on the snapd side with SNAPD_DEBUG | 15:41 |
| mup | PR snapd#5644 closed: interfaces: add audio-playback/audio-record and make pulseaudio manually connect <Created by jdstrand> <Merged by jdstrand> <https://github.com/snapcore/snapd/pull/5644> | 15:53 |
| ogra | stgraber, i'm trying to run debootstrap inside an lxc container (on top of ubuntu-core but that shouldnt matter) .... does that only work in privileged containers ? | 15:59 |
| ogra | root@bionic:~/cubox-classic# debootstrap bionic foobar | 15:59 |
| ogra | mknod: /root/cubox-classic/foobar/test-dev-null: Operation not permitted | 15:59 |
| ogra | E: Cannot install into target '/root/cubox-classic/foobar' mounted with noexec or nodev | 15:59 |
| ogra | (is there any doc for this ?) | 15:59 |
| stgraber | ogra: yes, only privileged containers can mknod stuff. It's in theory possible with LXD edge + 5.0 kernel to get our syscall interception to make this work for unpriv containers but it's still pretty untested for this scenario | 16:00 |
| ogra | thanks btw ... :) | 16:05 |
| === pstolowski is now known as pstolowski|afk | ||
| * zyga crashed again, | 17:25 | |
| zyga | I need a day of sleep | 17:25 |
| mup | PR snapcraft#2514 closed: test: autopkgtest beta <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2514> | 18:23 |
| mup | PR snapcraft#2658 opened: candidate testing <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2658> | 18:26 |
| ijohnson | hey cachio, I've seen google:debian-sid-64:tests/main/sbuild:{any,normal} both fail repeatedly in spread, do you think that's related to the protocol errors we've been seeing? | 19:31 |
| ijohnson | I remember seeing a PR from you split up that test into 2 different ones | 19:31 |
| cachio | ijohnson, hey, I was researching this today | 19:31 |
| cachio | the command is getting stuck | 19:32 |
| ijohnson | I don't know enough about the debian build task to know what's going on, but yeah I see | 19:32 |
| ijohnson | E: Build failure (dpkg-buildpackage died) | 19:32 |
| ijohnson | as the reason for it dying | 19:32 |
| cachio | so far I could not reproduce it and the lgos dont provide usefull information | 19:32 |
| ijohnson | yeah I don't see much useful info other than just that | 19:32 |
| ijohnson | hmm | 19:32 |
| cachio | ijohnson, the problem is that it is not showing any output | 19:32 |
| ijohnson | ah is it supposed to show more output? | 19:33 |
| cachio | I ran that manually and all the sbuild output is printed on the screen | 19:33 |
| cachio | ijohnson, I just started a job with -show-output to see the output of the test | 19:35 |
| ijohnson | Ah okay, nice | 19:35 |
| ijohnson | I just was gonna see if you knew what was up with that test, but it seems you're on top of things | 19:35 |
| ijohnson | thanks | 19:35 |
| cachio | ijohnson, perhaps it is consuming to much cpu | 19:36 |
| cachio | trying to be :) | 19:36 |
| cachio | but now working arout the protocol error | 19:36 |
| ijohnson | :-) yeah it would be good to figure out what we can do about the protocol error | 19:38 |
| * juliank tried and failed to use the code snap and the go snap together to build go apps in visual studio code | 20:08 | |
| juliank | When /snap/bin/code invokes /snap/bin/go nothing happens | 20:09 |
| juliank | I'd have hoped this would work, given both are classic snaps | 20:09 |
| ijohnson | juliank: what do you mean nothing happens, can you open the terminal in vscode and use the go command there? | 20:10 |
| ijohnson | I've used the go snap inside vscode as a snap and it did work, although I do know of one issue with vscode and other snaps specifically | 20:10 |
| ijohnson | the bug I know about is https://bugs.launchpad.net/snapd/+bug/1835805 | 20:11 |
| mup | Bug #1835805: strict snap run from classic snap can't write to filesystem <snapd:New> <https://launchpad.net/bugs/1835805> | 20:11 |
| juliank | ijohnson: it works from the terminal | 20:11 |
| juliank | ijohnson: if run on save by code, nothing happens | 20:12 |
| ijohnson | hmm, do you mean one of the go extensions doesn't work? | 20:12 |
| juliank | If I snap remove go, apt install golang it works fine | 20:13 |
| ijohnson | do you see any denials in the system journal with `journalctl -e --no-pager | grep DENIED` ? | 20:13 |
| juliank | let me reinstall the snap and check | 20:14 |
| juliank | As I see quite a few snap-confine file_inherit denials | 20:14 |
| ijohnson | the journal should still be around even if you removed the snap | 20:14 |
| ijohnson | interesting | 20:14 |
| ijohnson | that's the same kind of denial I saw in my bug report | 20:15 |
| juliank | AVC apparmor="DENIED" operation="file_inherit" profile="/usr/lib/snapd/snap-confine" pid=31372 comm="snap-confine" family="unix" sock_type="stream" protocol=0 requested_mask="send receive" denied_mask="send receive" addr=none peer_addr=none | 20:15 |
| ijohnson | could you comment on the bug report I linked to above with your log and reproducing instructions? | 20:15 |
| juliank | huh it works from snap run --shell | 20:16 |
| juliank | Let's start with a fresh code | 20:21 |
| * cachio afk | 20:26 | |
| mup | PR core-build#52 opened: initramfs: get snap parameters from grubenv <Created by cmatsuoka> <https://github.com/snapcore/core-build/pull/52> | 22:44 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!