| === epod is now known as luk3yx | ||
| === chihchun_afk is now known as chihchun | ||
| === chihchun is now known as chihchun_afk | ||
| ijohnson | plars: canceled the job | 04:15 |
|---|---|---|
| plars | ijohnson: thanks, feel free to resubmit, just keep that section of the yaml from the example | 04:15 |
| mborzecki | morning | 05:17 |
| zyga | Good morning | 05:46 |
| mborzecki | zyga: hey | 06:09 |
| mborzecki | mvo: hi | 06:10 |
| mborzecki | mvo: zyga: i'll be out for a while in the morning, sent a message in the forum | 06:11 |
| mvo | hey mborzecki | 06:13 |
| mvo | mborzecki: no worries, thanks for letting us know | 06:13 |
| mvo | zyga: good morning to you as well | 06:13 |
| zyga | Hey mvo | 06:16 |
| zyga | Mvo some bad news from last evening | 06:17 |
| zyga | Mvo: snapd broke in adt | 06:17 |
| zyga | There is a bug with the details | 06:17 |
| zyga | I’m with the dog outside so no links, sorry | 06:17 |
| zyga | Just sort by snapd but numbers | 06:17 |
| mvo | zyga: oh, interessting | 06:21 |
| mvo | zyga: reading now, we need to investigate | 06:23 |
| mvo | having snap changes output would be good | 06:24 |
| zyga | mvo: I can look back home | 06:26 |
| mvo | zyga: it really looks a bit mysterious, no trace of core there | 06:29 |
| mvo | zyga: that sounds more like a bug in out testsuite | 06:30 |
| dot-tobias | good morning! | 06:35 |
| zyga | mvo: note that it happens outside of the test suite | 06:38 |
| zyga | It affects docker, for example | 06:39 |
| zyga | Hey dot-tobias :-) | 06:41 |
| dot-tobias | Hi zyga 😊 | 06:41 |
| mvo | zyga: oh, hmmmmm | 06:42 |
| === chrisccoulson_ is now known as chrisccoulson | ||
| zyga | I'm back home; I'll make coffee and see what I can find | 06:47 |
| zyga | back from breakfast | 07:11 |
| === pstolowski|afk is now known as pstolowski | ||
| pstolowski | morning | 07:12 |
| zyga | good morning pawel! | 07:12 |
| mvo | hey pstolowski | 07:15 |
| mvo | 6706 needs a review | 07:15 |
| zyga | mvo: doing | 07:18 |
| mvo | ta | 07:19 |
| zyga | done | 07:20 |
| mvo | ta | 07:23 |
| mvo | zyga: the indent in 6706 was added to make it easier to read, is that bad in some way? | 07:44 |
| zyga | it looks like there is an extra space there | 07:44 |
| zyga | perhaps tabs spaces? | 07:45 |
| mvo | zyga: it looks like its just a strange formating of the diff, let me double check with side-by-side view | 07:45 |
| mvo | zyga: it looks ok here (unless I miss something) the first "ifneq" is unchagned, maybe this is why it looks strange | 07:46 |
| zyga | https://usercontent.irccloud-cdn.com/file/sqaMmc5M/Screenshot%202019-04-11%20at%2009.47.03.png | 07:47 |
| zyga | does the line with the + has a space up front? | 07:47 |
| zyga | oh | 07:48 |
| zyga | mvo: the logic is wrong | 07:48 |
| zyga | no? | 07:48 |
| zyga | you want ifeq, not ifneq | 07:48 |
| zyga | because when you have two ifneq lines , one will always match | 07:48 |
| zyga | ah, I see, they *are* nested | 07:48 |
| zyga | mvo: let me suggest something | 07:48 |
| mvo | zyga: hm, let me look. if thats so then the test is also broken | 07:49 |
| mvo | zyga: I can change it to use "dpkg-architecture -qDEB_TARGET_ARCH_BITS" | 07:52 |
| zyga | mvo: added one more comment | 07:52 |
| zyga | mvo: +1 on that idea | 07:52 |
| zyga | mvo: on both the test and the build rules | 07:52 |
| mvo | zyga: ok, let me look at this | 07:53 |
| zyga | mvo: we have one more bug report https://bugs.launchpad.net/snapd/+bug/1824242 | 07:56 |
| zyga | perhaps something that warrants an SRU? | 07:57 |
| mup | Bug #1824242: snapd can't be purged with latest AWS Xenial AMI <cloud-images:New> <snapd:New> <https://launchpad.net/bugs/1824242> | 07:57 |
| mvo | zyga: yeah, if that is missing from 2.38 we definitely need to add the cleanup in 2.38.1 | 08:00 |
| mvo | zyga: 2.38.1 will also include 6706 | 08:00 |
| zyga | sorry for the busy morning | 08:00 |
| zyga | I am looking at the adt issue now | 08:00 |
| mvo | zyga: thanks, +1 | 08:02 |
| mvo | yeah, 2.38 has the right "rm -rf" call | 08:03 |
| zyga | let's update the bug to reflect that 2.38 fixes it | 08:03 |
| mvo | zyga: I just did that | 08:04 |
| pedronis | mvo: hi, I added this card: https://trello.com/c/tx49EL3F/221-add-black-box-test-testing-memory-mappings-mmap-sizes-and-max-resident-memory-against-decided-limits-budget-for-snapd-and-snap-i | 08:09 |
| willcooke | Hi gang. We're suddenly seeing this bug: https://bugs.launchpad.net/ubuntu/+source/gnome-initial-setup/+bug/1824188 | 08:12 |
| mup | Bug #1824188: Software tab is empty on clean 19.04 install <amd64> <apport-bug> <disco> <rls-dd-incoming> <gnome-initial-setup (Ubuntu):Confirmed> <https://launchpad.net/bugs/1824188> | 08:12 |
| willcooke | GNOME Initial Setup is getting a "connection reset" from snapd when trying to get the list of software for promotion at the end of set up during first login | 08:12 |
| willcooke | Could you take a look? | 08:12 |
| willcooke | or lend a hand | 08:12 |
| zyga | hey willcooke | 08:13 |
| seb128 | " Failed to get featured snaps: Failed to read from snapd: Error receiving data: Connection reset by peer" | 08:13 |
| zyga | I think we will have to, there are some more 19.04 reports coming in yesterday | 08:13 |
| zyga | I'll prep a VM as soon as I'm done looking at ADT issues | 08:13 |
| willcooke | thanks zyga | 08:13 |
| seb128 | I had snap list failing also yesterday, but disk was full so maybe a side effect of that | 08:13 |
| willcooke | I'll add some more logs to the bug | 08:13 |
| seb128 | thx zyga | 08:13 |
| zyga | thank you! | 08:13 |
| mvo | zyga: I updated 6706 - anything new on the ADT issue? | 08:38 |
| zyga | mvo: not yet, pulling many things at the same time makes my link slow; just constructed adt image | 08:40 |
| mvo | zyga: ok | 08:41 |
| mvo | zyga: it was specifically cosmic, right? | 08:41 |
| zyga | yes | 08:41 |
| zyga | I just spawned the test | 08:41 |
| zyga | we'll know shortly | 08:41 |
| mvo | ta | 08:42 |
| mborzecki | re | 08:47 |
| mvo | hey mborzecki - welcome back | 08:49 |
| mvo | mborzecki: want to look at 6706? should be easy | 08:49 |
| zyga | eh | 08:51 |
| zyga | I love hand-holding adt | 08:51 |
| zyga | nothing works in that thing | 08:51 |
| zyga | obviously it cannot talk to the network | 08:51 |
| * zyga is debugging | 08:51 | |
| mborzecki | mvo: sure, looking | 08:52 |
| mborzecki | ah that's the PIE thing | 08:52 |
| zyga | mvo: do you know how to spawn adt in qemu with bridged network or something that is not just broken? | 08:54 |
| zyga | I used | 08:55 |
| zyga | autopkgtest -s -U snapd_2.38+18.10.dsc -- qemu ./autopkgtest-cosmic-amd64.img | 08:55 |
| zyga | this doesn't work | 08:55 |
| mvo | zyga: oh, thats strange, that should work | 08:55 |
| zyga | let me grab some logs | 08:56 |
| mborzecki | mvo: posted some comments there | 09:08 |
| mborzecki | mvo: basically, i think it would make sense to include s-c in the test and make sure it's built with PIE, because it's C, so it's bad and all that :) | 09:08 |
| mborzecki | zyga: is snap confine poking $SNAP_DATA now? see a new denial in the selinux branch | 09:48 |
| zyga | mborzecki: details please | 09:48 |
| mborzecki | type=AVC msg=audit(1554975937.636:129): avc: denied { getattr } for pid=1099 comm="snap-confine" path="/var/snap/test-snapd-service/x1" dev="vda1" ino=393657 scontext=system_u:system_r:snappy_confine_t:s0 tcontext=system_u:object_r:snappy_var_t:s0 tclass=dir permissive=1 | 09:48 |
| mborzecki | zyga: ^ | 09:48 |
| mborzecki | ha, maybe it's the cwd changes in snap-confine | 09:50 |
| zyga | mborzecki: so, snap-confine always did that | 09:51 |
| mvo | mborzecki: sounds good, go for it | 09:51 |
| zyga | getattr? | 09:51 |
| mvo | mborzecki: or in a separate PR, no strong opinion | 09:51 |
| zyga | mborzecki: perhaps it is a combination of two things: | 09:51 |
| zyga | this being a service | 09:51 |
| mborzecki | zyga: yes, fstat probably | 09:51 |
| zyga | so HOME is set to $SNAP_DATA | 09:51 |
| zyga | and the cwd changes | 09:51 |
| zyga | since apparmor does not mediate fstat at all | 09:52 |
| zyga | it was not showing up | 09:52 |
| mborzecki | zyga: i see we set WorkingDirectory for generated services to $SNAP_DATA | 09:52 |
| zyga | why did it not show up in the selinux test? | 09:52 |
| mborzecki | so that's probably it | 09:52 |
| zyga | mborzecki: I agree | 09:52 |
| zyga | mborzecki: is the selinux test capable of seeing the denials now? | 09:53 |
| zyga | mborzecki: was it one test or a restore-time check in all tests? | 09:53 |
| mborzecki | zyga: intersting how this test will be able to detect this :) | 09:53 |
| mborzecki | zyga: it's the selinux-clean test, specificaly targeted to catch any denials that may come up | 09:54 |
| zyga | I see | 09:54 |
| zyga | perhaps we should move the check to post-restore stage | 09:54 |
| zyga | like we do with apparmor, I believe | 09:54 |
| mborzecki | zyga: yeah, seems like we'll be able to update the policy proactively now, rather than waiting for bug reports in rhbz | 09:55 |
| zyga | +1 | 09:55 |
| zyga | popey: on the 19.04 opengl issue, as you know I asked for some help on twitter and got a very useful response | 09:59 |
| popey | Excellent | 10:00 |
| popey | Good to hear twitter isn't just for nazis :D | 10:00 |
| zyga | popey: I will look at what the situation is inside a virtual 19.04 system in the context of a separate issue that was raised by willcooke today; after that I should be able to make some progress towards understanding the cause of the opengl regression | 10:00 |
| popey | Thanks for letting me know! | 10:00 |
| zyga | popey: arguably that was even more useful than insta-ordering a gtx on amazon | 10:01 |
| zyga | because I got feedback from more diverse collection of systems | 10:02 |
| popey | Wellll.... | 10:02 |
| popey | I mean, okay, but if we had some QA on these GTX systems *before* I found the issue, that argument doesn't hold. | 10:02 |
| zyga | We don't do QA like that I'm afraid | 10:08 |
| zyga | but we also know about the shortcomings of the solution we have now | 10:08 |
| zyga | and we have the improvements in the roadmap | 10:08 |
| zyga | for this cycle (though arguably we will be late) | 10:08 |
| zyga | I'm saying that the wheels started spinning already | 10:09 |
| zyga | mborzecki: some failures from autopkgtest in qemu: | 10:10 |
| zyga | https://www.irccloud.com/pastebin/vUlozfth/ | 10:10 |
| zyga | perhaps insufficient mocking? | 10:10 |
| zyga | running more tests | 10:11 |
| mborzecki | hm maybe | 10:12 |
| mborzecki | btw. there's no 18.04-32 in the spread suite? | 10:12 |
| mborzecki | zyga: the nvidia card is in an older system i have here, i can look into 19.04 later on if i manage to install it on a usb stick | 10:15 |
| zyga | mborzecki: I have useful data already, I need to process it first | 10:15 |
| mborzecki | zyga: ah, that's fine then | 10:15 |
| mborzecki | zyga: has anything changed with the libs again? | 10:16 |
| zyga | not sure yet | 10:16 |
| pstolowski | pedronis: hey, toughts on snap debug timings --ensure=... output: https://paste.ubuntu.com/p/dXZqyDxVG6/ ? | 10:17 |
| pedronis | pstolowski: thx, need to think a bit. I will look at the PR producing the data today | 10:19 |
| zyga | brb | 10:30 |
| zyga | Chipaca: hey, are you looking at the 19.04 empty software tab bug? | 10:41 |
| Chipaca | zyga: ye | 10:41 |
| zyga | great, thanks! | 10:41 |
| Chipaca | why is journalctl still broken on 19.04? | 10:48 |
| Chipaca | stuff is going to /var/log/syslog skipping journalctl entirely | 10:49 |
| Chipaca | this is a breaking change :-( | 10:49 |
| zyga | woah | 10:49 |
| zyga | that's odd | 10:49 |
| mborzecki | btw. 19.04 beta ok to use for install or should i rather try a daily one? | 10:50 |
| zyga | I downloaded daily but did not install it yet | 10:50 |
| mvo | zyga: anything new on the ADT issue? I ran it in qemu 3 times and no failure :/ | 10:50 |
| zyga | same here, more iterations | 10:50 |
| zyga | I updated the bug report a moment ago | 10:50 |
| Chipaca | willcooke: why is journalctl not showing logs on 19.04, do you know? | 10:52 |
| willcooke | Chipaca, we had this problem before when we were trying to work out why it wasnt installing some snaps. I don't think we ever got to the bottom of it. It seems that it was just snapd that wasn't logging properly, even with the logging turned right up. Was it that it was just taking a while to get flushed to the log? | 10:55 |
| Chipaca | willcooke: the logs are in /var/log/syslog, immediately, but journalctl always reports empty | 10:56 |
| willcooke | seb128, any ideas? ^ | 10:56 |
| mup | PR snapd#6706 closed: ubuntu: disable -buildmode=pie on armhf to fix memory issue <⚠ Critical> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6706> | 11:04 |
| mvo | I cherry picked 6706 now for 2.38 | 11:05 |
| zyga | mvo: will you also tackle the PIE of snap-confine? | 11:05 |
| mvo | zyga: not right now, also not as part of 2.38.1 - no time today. but feel free to grab it (or maybe mborzecki) | 11:06 |
| seb128 | Chipaca, willcooke, I guess snapd doesn't use the proper logging api/could integrate to the journal? a problem on the snapd side for sure | 11:11 |
| zyga | seb128: how does that explain difference between 19.04 and prior releases? | 11:11 |
| Chipaca | seb128: ? how can it be "a problem on the snapd side for sure" when the exact whatever-it-is-snapd-does works on anything other than 19.04? | 11:11 |
| seb128 | Chipaca, like you are sure it works on fedora 30 or other distros with the same systemd stack? | 11:12 |
| mvo | ddstreet: re failing autopkgtest - do you have a machine around that you can ssh into that has the failure? I would love to see the "snap changes" output of it | 11:13 |
| seb128 | zyga, I don't know what snapd is doing, maybe it's relying on old syslog and should integrate better with systemd journal? | 11:13 |
| seb128 | well, I guess it could be an issue on the systemd side | 11:13 |
| mvo | ddstreet: fwiw, it looks like things started to fail at 2019-04-03 | 11:13 |
| zyga | seb128: I see, perhaps the question is: how should snapd log? | 11:13 |
| seb128 | it should integrate to journalctl | 11:13 |
| seb128 | imho | 11:13 |
| seb128 | it could be that it's doing things in a "legacy" way and that regressed in the journal side so a bug | 11:14 |
| seb128 | anyway, I think that needs looking at by someone who understand the snapd logging code and what is done exactly | 11:14 |
| ddstreet | mvo, let me see if i can repro it in a system on the canonical vpn | 11:14 |
| Chipaca | snapd, and all snaps that have daemons, rely on the behaviour that a service printing to stdout/stderr ends up in the journal | 11:14 |
| Chipaca | seb128: ^ | 11:15 |
| seb128 | Chipaca, snapd re-exec itself? could that confuse the systemd units or something | 11:15 |
| seb128 | Chipaca, zyga, in any case it's an issue specific to snapd so unsure what it's doing differently but imho the best person to debug that is someone who knows what snapd is doing exactly | 11:16 |
| seb128 | or maybe talk to foundation if you believe the issue is on the journal side | 11:16 |
| seb128 | but they are probably going to need a testcase/details on what snapd is doing and how it's different from other services that work fine | 11:17 |
| Chipaca | gah | 11:17 |
| Chipaca | ok, the problem is different | 11:17 |
| Chipaca | seb128: 'journalctl -u snapd' says no entries | 11:17 |
| Chipaca | seb128: 'sudo journalctl -u snapd' works | 11:18 |
| Chipaca | so, it is a behaviour change in journalctl, but not as bad as i thought it was | 11:18 |
| seb128 | ah, so logging is working, good :) | 11:19 |
| zyga | Chipaca: perhaps it's just permission difference | 11:20 |
| zyga | Chipaca: on many distributions journalctl is locked down | 11:20 |
| Chipaca | zyga: yes, but you get an error when you try | 11:20 |
| Chipaca | not a 'no entries' | 11:20 |
| zyga | Chipaca: is it possible that you just reach the per-user instance of journalctl ? | 11:20 |
| Chipaca | ¯\_(ツ)_/¯ i have no idea | 11:21 |
| Chipaca | i'm trying to debug wacky interactions with snapd elsewhere in this | 11:21 |
| seb128 | journalctl -u snapd works on disco here, without sudo | 11:21 |
| Chipaca | not figure out why systemd changed the rules again | 11:21 |
| seb128 | but it's not a new install | 11:21 |
| seb128 | so maybe some permission is not correctly set or something | 11:21 |
| seb128 | Chipaca, does it work for other units? like gdm or upower? | 11:22 |
| Chipaca | seb128: no | 11:22 |
| seb128 | Chipaca, and "systemctl --system -u snapd"? | 11:23 |
| Chipaca | insufficient permissions | 11:23 |
| seb128 | k | 11:23 |
| seb128 | so that's the issue | 11:23 |
| seb128 | probably worth talking to xnox about and/or reporting on launchpad against systemd | 11:23 |
| xnox | que?! | 11:24 |
| seb128 | xnox, is it normal that journalctl on disco new installs require sudo to access the system logs? | 11:24 |
| xnox | what do you expect for "systemctl --system -u snapd" to do? there is no verb, maybe "journalctl -u snapd" ? | 11:24 |
| seb128 | (it doesn't on my upgraded system) | 11:24 |
| seb128 | xnox, | 11:25 |
| seb128 | <Chipaca> seb128: 'journalctl -u snapd' says no entries | 11:25 |
| seb128 | seb128: 'sudo journalctl -u snapd' works | 11:25 |
| xnox | seb128, it depends which groups the person is in. and what "new install" is - container, desktop, server? | 11:25 |
| seb128 | xnox, I think people see the problem on disco desktop/ubiquity install | 11:25 |
| xnox | one does need to be like in sudo group, or in adm, or somewhere like that. | 11:25 |
| xnox | hmmm | 11:25 |
| xnox | on desktop, the first user should be able to read all the logs i think. | 11:25 |
| seb128 | what group is that you need? | 11:26 |
| xnox | checking | 11:26 |
| xnox | i believe it was `adm` group, but that's from memory, looking at code | 11:26 |
| seb128 | Chipaca, what groups is your user in? does it include adm? | 11:26 |
| xnox | and $ sudo getfacl /var/log/journal/ | 11:28 |
| xnox | this has possibly regressed.... because i don't see us configuring adm group | 11:30 |
| seb128 | Chipaca, ^ | 11:30 |
| Chipaca | user is in adm cdrom sudo dip plugdev lpadmin sambashare | 11:31 |
| Chipaca | seb128: https://pastebin.ubuntu.com/p/6rg6tGVMJY/ | 11:33 |
| mvo | zyga: fun observation - docker.io deb package has a shell defer in debian/tests/common | 11:34 |
| zyga | mvo: oh, would you mind pasting it? | 11:34 |
| zyga | I wonder how they implemented it | 11:34 |
| mvo | zyga: common | 11:34 |
| Chipaca | systemd-journal systemd-timesync systemd-network systemd-resolve systemd-coredump | 11:35 |
| mvo | zyga: http://paste.ubuntu.com/p/dw3kKtzPJq/ | 11:35 |
| Chipaca | huh, xenial also has 'em | 11:35 |
| zyga | mvo: interesting, thank you! | 11:35 |
| seb128 | Chipaca, xnox, on my upgraded system there is a "group:adm:r-x" extra line compared to that | 11:35 |
| seb128 | Chipaca, can you file it on https://bugs.launchpad.net/ubuntu/+source/systemd/+filebug ? | 11:35 |
| seb128 | I guess something for xnox to poke at | 11:36 |
| xnox | Chipaca, and like doing $ sudo setfacl -nm g:adm:rx,d:g:adm:rx /var/log/journal/ | 11:36 |
| xnox | doesn't seem to help | 11:36 |
| xnox | oh, maybe i need to restart journald | 11:36 |
| kjackal | Hi jdstrand how is it going? I am officialy appointed to strict confinement of microk8s! Going through the instructions/patches gathered, i am facing problems. I am now focusing on kube-proxy. Can you spare some time to get me upto speed with your work? | 11:36 |
| Chipaca | xnox: do you need the bug? | 11:37 |
| xnox | Chipaca, yes, please | 11:37 |
| Chipaca | what's the name for the user that gets created on install? | 11:37 |
| kjackal | jdstrand: I see you have pushed the interfaces since early November but I wonder if i am missing something | 11:37 |
| Chipaca | 'default user'? | 11:37 |
| seb128 | that should be good enough | 11:38 |
| seb128 | I don't think we have a defined wording, default/first/... | 11:38 |
| xnox | Chipaca, can you fix it with: | 11:39 |
| kjackal | jdstrand: this is how kubeproxy is failing: https://pastebin.ubuntu.com/p/cN3sK5gpX8/ although I request for iptables proxy setup it falls back to userspace | 11:39 |
| xnox | $ sudo setfacl -R -nm g:adm:rx,d:g:adm:rx /var/log/journal | 11:39 |
| xnox | ? | 11:39 |
| mvo | mwhudson: did anything in the go snap changed around 2019-04-03? | 11:41 |
| seb128 | willcooke, ^ in case you didn't follow, snapd logging works, journal just regressed and the default user doesn't have the permission to read it without using sudo (workaround, 'sudo journalctl -u snapd' if you need some log) | 11:41 |
| mvo | ddstreet: I ran autopkgtest a couple of times on my machine for cosmic no luck to reproduce the issue so far. trying with -smp 1 now just to see if that makes a difference | 11:42 |
| mwhudson | mvo: there was a point release around then i think? | 11:43 |
| mvo | ddstreet: also on our side nothing should change, snapd bundles all go deps so it must be some package or other change, I wish there was a way to query what packages changed | 11:43 |
| mwhudson | mvo: ah no that was more recent, there were updates on 2019-03-14 and 2019-04-08 | 11:44 |
| mvo | mwhudson: its probably just coincidence, I'm just looking at some autopkgtest issue and it seems to have started around this time (and we use go snap during adt to build spread - but probably unreleated, sorry, I'm stabing a bit in the dark right now) | 11:44 |
| mvo | mwhudson: cool, thank you | 11:45 |
| * mvo rules that out then | 11:45 | |
| Chipaca | xnox: yes | 11:48 |
| === ricab is now known as ricab|lunch | ||
| Chipaca | xnox: seb128: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1824342 | 11:51 |
| mup | Bug #1824342: in 19.04, default user cannot access system journal <systemd (Ubuntu):New> <https://launchpad.net/bugs/1824342> | 11:51 |
| Chipaca | hth | 11:51 |
| seb128 | Chipaca, thx! | 11:51 |
| xnox | Chipaca, seb128 - this smells like a regression in systemd-tmpfiles utility implementation... bisecting. | 12:08 |
| Chipaca | popey: willcooke: figured out #1824188 | 12:15 |
| mup | Bug #1824188: Software tab is empty on clean 19.04 install <amd64> <apport-bug> <disco> <rls-dd-incoming> <gnome-initial-setup (Ubuntu):Confirmed> <snapd (Ubuntu):Invalid> <https://launchpad.net/bugs/1824188> | 12:15 |
| Chipaca | interesting how 19.04 lets you log in before seed.loaded is done | 12:19 |
| Chipaca | mvo: is that expected? | 12:20 |
| seb128 | it's a bit unfortunate that snapd is not ready by the time the graphical session starts | 12:21 |
| Chipaca | seb128: well, the service that waits for seeded does have a WantedBy=multi-user.target | 12:22 |
| Chipaca | seb128: so it shouldn't | 12:22 |
| Chipaca | ah but it doesn't have an Before/After thing? | 12:24 |
| Chipaca | mvo: WDYT of having snapd.seeded.service have Before=multi-user.target ? (it alreayd has WantedBy=) | 12:32 |
| Chipaca | seb128: that would slow down first boot considerably, though, which i'm sure would make xnox happy | 12:33 |
| xnox | Chipaca, allowing login is unrelated to reaching multi-user.target. | 12:34 |
| willcooke | seb128, could we have g-i-s wait until snapd is done seeding? | 12:34 |
| Chipaca | xnox: oh? oh. | 12:34 |
| xnox | there was a separate job that removes the nologin flag | 12:34 |
| seb128 | willcooke, and have your desktop sitting there doing nothing for like 3 min or slow systems? | 12:34 |
| xnox | it's quite early, in like either logind or getty starts | 12:35 |
| seb128 | willcooke, I would prefer not | 12:35 |
| willcooke | seb128, well, the desktop could be up and working, just that g-i-s doesn't start right away; | 12:35 |
| seb128 | Chipaca, did snapd change/is that a core18/seeding change side effect? | 12:35 |
| willcooke | s/;/? | 12:35 |
| cachio | mvo: hey I need to take my son to the doctor, I'll try to be back for standup | 12:35 |
| Chipaca | seb128: no | 12:35 |
| Chipaca | seb128: sorry, no to the first one | 12:35 |
| seb128 | we only started having those "things are not ready under seeding is done which takes a while" this cycle it looks like :/ | 12:36 |
| Chipaca | seb128: to the second one, maybe? as i say, it's racy, so maybe before you were always not hitting the restart | 12:36 |
| Chipaca | seb128: snapd has always restarted, on ubuntu, after first install of core | 12:36 |
| seb128 | willcooke, I think it's still suck as an user experience, boot the system, have it clean, start some apps, start working and then the welcome thing pops up | 12:36 |
| willcooke | yeah | 12:36 |
| Chipaca | two things | 12:37 |
| Chipaca | 1. retrying would be fine, nothing wrong with it | 12:37 |
| Chipaca | 2. i don't know what changes with the "these things are installed", but the things _aren't installed_ unless seeding is done, so that logic is bogus | 12:37 |
| seb128 | yes, still depends of how long it takes | 12:37 |
| seb128 | I think the issue is mainly that it takes that long to snapd to be "ready" | 12:38 |
| seb128 | any way we work around that from the desktop will be suboptimal | 12:38 |
| seb128 | either we need to delay the welcome screen | 12:38 |
| Chipaca | it's always going to take non-zero time to seed | 12:38 |
| seb128 | or have it to display spinner for $minutes on the software page | 12:38 |
| Chipaca | so the race will always be there if you don't handle it | 12:38 |
| seb128 | yes | 12:38 |
| seb128 | but seconds it's fine | 12:38 |
| seb128 | desktop takes like 10 seconds+ to load | 12:39 |
| seb128 | you probably have 30 seconds before an user hit that wizard page | 12:39 |
| Chipaca | well, that's part of the problem | 12:39 |
| Chipaca | it does not hit the query when it hits that wizard page | 12:39 |
| Chipaca | it hits it at the beginning | 12:39 |
| seb128 | that we can easily change | 12:40 |
| Chipaca | seb128: if you wait on first page of the wizard until seeding is done (watch it in a terminal), the last page is still blank (with the error in syslog) | 12:40 |
| Chipaca | seb128: could you 'busy' the next button on the before-last page of the wizard? | 12:40 |
| seb128 | Chipaca, that we should fix yes | 12:41 |
| Chipaca | until snapd is sod? | 12:41 |
| Chipaca | and then do the query | 12:41 |
| seb128 | it has been an issue for no user in bionic though | 12:41 |
| seb128 | so it's an indications things used to be ready much earlier from the snapd side | 12:41 |
| Chipaca | i could reproduce what i've done here in bionic and tell you exactly why it's not been an issue | 12:41 |
| seb128 | it was reliably ready before the desktop load and that stopped being true | 12:41 |
| seb128 | (our code didn't change) | 12:41 |
| Chipaca | seb128: it could be earlier _or later_ | 12:42 |
| Chipaca | to be clear the 'restart' is not the last thing that happens for seeding | 12:42 |
| Chipaca | more like the first thing | 12:42 |
| Chipaca | seb128: tbh, it's probably 19.04 being so amazingly fast at starting up | 12:43 |
| Chipaca | not sure what changed but you get to log in very fast | 12:43 |
| Chipaca | and that's awesome | 12:43 |
| Chipaca | but it catches snapd with its pants down | 12:43 |
| seb128 | I see | 12:43 |
| seb128 | Chipaca, willcooke, let's fix g-i-s to do the query when it loads the tab (if that's doable without refactoring the code a lot, we need to check) and see if that's enough | 12:44 |
| seb128 | then we can try to do the "retry if not ready and spin" | 12:44 |
| seb128 | that should be good enough, unless it spins then for 3 minutes | 12:44 |
| willcooke | seb128, sounds like a plan, thanks. Shall I ask ken to look? | 12:44 |
| seb128 | willcooke, wfm, we can also ask Andy | 12:45 |
| willcooke | seb128, oki, let's move to -desktop and work from there | 12:45 |
| willcooke | thanks Chipaca | 12:45 |
| seb128 | Chipaca, thx | 12:45 |
| Chipaca | seb128: in this kvm it took snapd 95 seconds from startup to seeded, and the query from g-i-s arrived at 30s | 12:46 |
| Chipaca | so it is a full minute later | 12:46 |
| seb128 | :( | 12:46 |
| seb128 | that's a sucky user experience still :/ | 12:46 |
| seb128 | willcooke, ^ | 12:46 |
| willcooke | ouch | 12:47 |
| willcooke | perhaps if it's not ready we just have to skip that page | 12:47 |
| willcooke | while being less than ideal, no page is better than a blank page | 12:47 |
| seb128 | willcooke, well, then we are going to skip it for most users :/ | 12:47 |
| mvo | cachio: good luck | 12:48 |
| seb128 | we maybe just need to kill it | 12:48 |
| seb128 | and do the hint on the launcher icon for gnome-software mp_t recommended | 12:48 |
| willcooke | lets talk in -desktop | 12:48 |
| seb128 | but that's not for disco at this point | 12:48 |
| seb128 | k | 12:48 |
| willcooke | yeah | 12:48 |
| Chipaca | willcooke: FWIW, http://paste.ubuntu.com/p/8rpHjxBmkq/ | 12:50 |
| Chipaca | although that's more for us than for you, it might shed some light on it all | 12:50 |
| pedronis | Chipaca: 23.382s - Make snap "core" (6673) available to the system , that is interesting and weird | 12:51 |
| Chipaca | pedronis: even more | 12:52 |
| Chipaca | Apr 11 12:03:23 ubuntu snapd[620]: taskrunner.go:426: DEBUG: Running task 7 on Do: Make snap "core" (6673) available to the system | 12:52 |
| Chipaca | Apr 11 12:03:46 ubuntu snapd[620]: task.go:337: DEBUG: 2019-04-11T12:03:46+01:00 INFO Requested daemon restart. | 12:52 |
| mup | PR #11: Publish coverage reports to coveralls <Created by elopio> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/11> | 12:52 |
| Chipaca | pedronis: ^ that's the majority of that time right there | 12:52 |
| Chipaca | pedronis: (those lines are consecutive in the logs) | 12:53 |
| * Chipaca hugs elopio | 12:54 | |
| willcooke | thanks Chipaca | 12:54 |
| Chipaca | willcooke: that's the output of 'sudo snap debug timings 1', which gives you the per-task times of the seed change | 12:54 |
| willcooke | very handy, thx | 12:55 |
| pedronis | newer snapd will show even more info | 12:55 |
| pedronis | Chipaca: anyway sounds we need to dig there at some point, because link-snap is not supposed to a particularly slow one | 12:56 |
| pedronis | especially for core that doesn't have services or apps | 12:56 |
| pedronis | Chipaca: standup? | 13:01 |
| Chipaca | omw | 13:01 |
| === ricab|lunch is now known as ricab | ||
| pedronis | mborzecki: mvo: btw I did a PR to simplify prepare-image (not high prio): https://github.com/snapcore/snapd/pull/6696 | 13:44 |
| mup | PR #6696: image: simplify prefer local logic and fixes <Created by pedronis> <https://github.com/snapcore/snapd/pull/6696> | 13:44 |
| mvo | pedronis: thanks | 13:45 |
| mborzecki | pedronis: thx, will review | 13:48 |
| Chipaca | mvo: the change that creates .../aux also changed the postrm rm of cache to -r | 13:59 |
| Chipaca | mvo: but if you have a newer snapd and use an older postrm, it'll likely break like that | 13:59 |
| Chipaca | mvo: that is a895e537c55a350af30250e5bedc1b16e0c095ab, #6034, which is in 2.38 | 14:02 |
| mup | PR #6034: many: save media info when installing, show it when listing <Squash-merge> <Created by chipaca> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/6034> | 14:02 |
| jdstrand | kjackal: hey, did you connect the kubernetes-support interface? the snap isn't allowed to load modules itself, but kubernetes-support tells snapd to load those ip_* modules. you also need to plugs and connect firewall-control (for the nf_* modules, though that should autoload) | 14:21 |
| jdstrand | kjackal: additional, like in journal logs for security policy violations | 14:22 |
| kjackal | ah I probably missed that | 14:23 |
| zyga | jdstrand: hey | 14:24 |
| zyga | jdstrand: having fun with getline | 14:24 |
| zyga | it's a peculiar beast | 14:24 |
| pstolowski | Chipaca: i can reproduce the "make snap core available" taking 23s issue on disco install; interestingly it looks all fine if i start with a clean state on my 18.04 test box, does that match your observations? | 14:27 |
| jdstrand | kjackal: yeah, when you do an unasserted install, you have to manually connect all the interfaces. once in the store we can issue a snap declaration that autoconnects them | 14:27 |
| Chipaca | pstolowski: i have not tested starting without a seed, if that's what you mean | 14:28 |
| jdstrand | zyga: interesting. note, this isn't a regression (the previous behavior was the same afaics), so a followup pr would be fine (unless you think otherwise) | 14:28 |
| zyga | yes | 14:28 |
| zyga | kernel plays ball so it's okay | 14:28 |
| zyga | but I want to fix it anyway | 14:28 |
| * jdstrand nods | 14:29 | |
| zyga | I may split this depending on the size | 14:29 |
| zyga | I started by adding sc_error support for mountinfo to really know what's wrong | 14:29 |
| zyga | parsing with scanf is only good for programming interviews and programming contents | 14:29 |
| cachio | Chipaca: hi | 14:34 |
| cachio | I think the issue with the resume for the .partial file is under some conditions | 14:35 |
| cachio | I could reproduce it many times but on a clean environment it doesn't happen | 14:35 |
| cachio | Chipaca: then, could you please take a look to this one #6694 ? thanks | 14:36 |
| mup | PR #6694: tests: improve how snaps are cached <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/6694> | 14:36 |
| Chipaca | cachio: i need to see the conditions :-) | 14:38 |
| cachio | mvo: is it ok if we start testing snapd on 19.04 on travis? | 14:44 |
| mvo | cachio: yes please | 14:52 |
| mvo | Chipaca: yeah, this was exactly the problem, old snpad is purged but fails | 14:52 |
| pstolowski | Chipaca: removing snapd on disco, removing state, unmounting everything, reinstalling snapd and letting it seed all the snaps from /var/lib/snapd/seed works fine (no 23s slowness). it's something during install only | 14:55 |
| pedronis | pstolowski: I did a pass over #6704 , some questions there | 15:00 |
| mup | PR #6704: overlord/devicestate,snapstate: measurements around ensure and related tasks <Created by stolowski> <https://github.com/snapcore/snapd/pull/6704> | 15:00 |
| pstolowski | pedronis: ty | 15:01 |
| mup | PR snapd#6708 opened: packaging/ubuntu: enable PIE hardening flags <⛔ Blocked> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6708> | 15:17 |
| mvo | zyga, Chipaca I cherry picked 6668 into upstream/2.38 which should hopefully make travis on 2.38 happy again | 15:21 |
| zyga | mvo: ack, thank you | 15:21 |
| zyga | jdstrand: I decided to split the getline fix into a new branch because it still needs some work to be properly propsable | 15:23 |
| zyga | I just pushed the changes you asked for (comments) and will merge when green | 15:23 |
| zyga | mvo: any idea why it only sometimes failed? | 15:24 |
| pstolowski | Chipaca, pedronis the "Make snap core..." slowness comes from regeneration of fc-cache. fc-cache-v6+fc-cache-v7 take at least 17s when i start with clean fontcache | 15:24 |
| zyga | mvo: is it because the aux directory is only sometimes created? | 15:24 |
| zyga | pstolowski: aaaah | 15:24 |
| pedronis | pstolowski: ah, so obvious but unpleasant | 15:24 |
| Chipaca | the installer could do that before the reboot, with some care | 15:25 |
| zyga | Chipaca: the installer might come with that cache | 15:26 |
| zyga | pre-baked | 15:26 |
| pstolowski | Chipaca: for me it happened before the reboot, i didn't reboot immediately though as i was investigating it | 15:26 |
| Chipaca | pstolowski: wat | 15:26 |
| pstolowski | Chipaca: i'd need to double check to be sure | 15:26 |
| Chipaca | pstolowski: the snapd 'inside' isn't running before the reboot | 15:26 |
| mvo | zyga: yes, I think its a race but again, no time to debug in detail | 15:27 |
| pstolowski | Chipaca: ah yes you're right, i forgot i'm running the snap from the live cd | 15:30 |
| jdstrand | zyga: thanks, I saw. sounds go to me | 15:30 |
| pstolowski | pedronis: we could maybe run fc-cache-v6 and v7 at the same time instead of in sequence, that could win a little bit? | 15:37 |
| zyga | pstolowski: ha, nice idea, especially in parallel | 15:40 |
| zyga | unless fc-cache itself is multi-threaded | 15:40 |
| pedronis | well 10s vs 20s is still a lot | 15:41 |
| pedronis | is good we understand the problem | 15:41 |
| pedronis | but we should probably talk with the installer people | 15:41 |
| pedronis | what's the best path forward | 15:41 |
| pstolowski | pedronis: shall i talk to them (i don't remember who the installer people are though) | 15:45 |
| pstolowski | ? | 15:45 |
| seb128 | pstolowski, you want foundations' team for the installer | 15:46 |
| pstolowski | seb128: ah, thanks! | 15:46 |
| pstolowski | i'm going to add timings around this area anyway, might be useful to have complete picture of first boot experience | 15:49 |
| seb128 | indeed | 15:50 |
| * cachio lunch | 15:58 | |
| === pstolowski is now known as pstolowski|afk | ||
| zyga | mborzecki: could you have another look at https://github.com/snapcore/snapd/pull/6643 | 16:03 |
| mup | PR #6643: tests: deny ioctl - TIOCSTI with garbage in high bits <Created by zyga> <https://github.com/snapcore/snapd/pull/6643> | 16:03 |
| zyga | not sure if it requires more changes | 16:03 |
| pedronis | pstolowski|afk: yes, do add timings around there | 16:21 |
| mvo | hrm, opensuse 15 is failing in travis at least in 2.38 - is that known? | 16:25 |
| mvo | does anyone has the curl command handy to revert a snap using the api? | 16:34 |
| mup | PR snapd#6709 opened: release: 2.38.1 <Created by mvo5> <https://github.com/snapcore/snapd/pull/6709> | 16:43 |
| cachio | mvo: sru for 2.38 is still needed? | 16:48 |
| cachio | I see you created sru for 2.38.1 | 16:48 |
| mvo | cachio: yes please | 16:51 |
| mvo | cachio: well, its a good question actually | 16:51 |
| mvo | cachio: yes | 16:51 |
| mvo | cachio: lets do 2.38 as its already in the queue | 16:51 |
| cachio | ok | 16:52 |
| mvo | cachio: it will only be a problem for people with low-mem arm devices that use an old 4.4 kernel before 4.4.78 - all ubuntu kernels are fine | 16:52 |
| cachio | mvo: nice, I'll start right now | 16:53 |
| cachio | thanks | 16:53 |
| mvo | thank you | 16:53 |
| cachio | mvo: http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#snapd | 16:56 |
| cachio | tests didn't run for xenial | 16:56 |
| mvo | cachio: aha, I see, poking | 16:57 |
| mvo | cachio: I raised in in #ubuntu-release | 17:00 |
| mvo | cachio: there is no golang-1.10 for powerpc it seems | 17:00 |
| cachio | mvo: ahh | 17:00 |
| cachio | ok | 17:00 |
| cachio | I'm running the other validations | 17:02 |
| cachio | thanks | 17:02 |
| mup | PR snapd#6605 closed: cmd/libsnap,osutil: fix parsing of mountinfo <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6605> | 17:24 |
| mup | PR snapd#6710 opened: tests: run spread tests on ubuntu 19.04 <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/6710> | 17:26 |
| mvo | cachio: 2.38.1 is in beta now, please validate | 17:59 |
| cachio | mvo: sure | 18:03 |
| mup | PR snapcraft#2532 opened: catkin stage-snaps test: limit to amd64, arm64, and armhf <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2532> | 18:21 |
| * zyga EODs | 18:57 | |
| mup | PR snapcraft#2532 closed: catkin stage-snaps test: limit to amd64, arm64, and armhf <Created by kyrofa> <Closed by sergiusens> <https://github.com/snapcore/snapcraft/pull/2532> | 20:03 |
| mup | PR snapcraft#2533 opened: tests: classic confinement spread tests for and maven <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2533> | 20:33 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
