=== chihchun_afk is now known as chihchun === chihchun is now known as chihchun_afk [04:52] Good morning [04:52] School day today so I will be a bit preoccupied in the next few hours [05:02] morning [05:17] https://github.com/snapcore/snapd/pull/5753/ is an easy win === chihchun_afk is now known as chihchun [06:11] mvo: hi, think we could land this? https://github.com/snapcore/snapd/pull/5753 [06:11] mborzecki: looking [06:11] btw. this one is easy win too https://github.com/snapcore/snapd/pull/5752 [06:11] mborzecki: yes! [06:12] mborzecki: 5752 failed in tests, because of grub? [06:12] mborzecki: in any case, approved and can land as soon as tests are happy [06:12] mborzecki: thank you [06:17] mvo: thanks [06:17] ok, i'm off to school [06:18] mborzecki: good luck === chihchun is now known as chihchun_afk [06:44] hey, daughter handled, now going to our son's new school a bit further away [06:45] I'll be operational around 10:30 [06:46] mborzecki: request: can you do a test PR and merge zyga/fix/trespassing-v2 into your instance work to see if you can dump the workarounds? please? [06:46] with that, I'm off to the bus stop === pstolowski|afk is now known as pstolowski [07:17] mornings [07:26] New school looks familiar === chihchun_afk is now known as chihchun [08:15] a second review for 5736 would be great [08:15] with that I can do a new 2.35.1 release [08:27] I’m stuck but I’ll review it now mvo [08:28] Ah I already did [08:33] Hey John [08:34] zyga: 'sup [08:41] School [08:41] hey Chipaca ! [08:41] New school, new class [08:41] mvo: o/! [08:42] Want out but $REALITY === chihchun is now known as chihchun_afk [09:10] re [09:11] zyga: let me check your branch [09:11] zyga: with mount ns mapping branch right? [09:12] Yes [09:12] With all of your instance features [09:13] School is over [09:13] I’ll be home in 15 minutes [09:22] zyga: dropped the workarounds and running with your branch now [09:25] is there anything wrong with the release process since this night? [09:26] I've got a " [09:26] Status [09:26] Manual review pending [09:26] gah. for the newly uploaded nightly. [09:26] "Automated review not yet completed", "Task 058f17f1-e946-43d1-a783-0aa4ca04acb0 failed" [09:27] also, good morning. [09:29] ^ sparkiegeek [09:29] mvo: is snap-failure expected to be used everywhere? [09:31] mborzecki: snap-failure? sorry, do you have some more context for me? [09:31] thresh, popey: good morning - can you share the name of your snap so I can take a look? [09:32] mvo: got a report from arch user, apparently snapd exiter unexpectedly and systemd tried to run snap-failure [09:32] sparkiegeek: vlc [09:32] mvo: iirc this would only make sense if reexec is used [09:32] mborzecki: oh, interessting [09:32] mborzecki: yeah, it should not run on classic [09:32] mvo: we have this in snapd.service.in OnFailure=snapd.failure.service [09:33] mvo: so it's tried everywhere [09:33] mborzecki: yeah [09:33] popey: thanks, looking [09:33] mvo: i'll try to ask why it failed in the first place anyway :P [09:33] mborzecki: please do [09:33] Morning all [09:33] mborzecki: I added an item to my todo to fix this [09:34] mvo: Just did some edits on the UC16/18 doc [09:34] niemeyer: good morning! [09:34] niemeyer: yay, thank you [09:34] mvo: Tried to make it a bit more high-level.. not quite sure about what's the audience for the doc, and the audience may also drift :) [09:35] niemeyer: yeah, looks good, thanks for streamlining it [09:36] mvo: np, let's see if we can think of other interesting deltas [09:36] mvo: Thanks for putting it up as well === chihchun_afk is now known as chihchun [09:39] zyga: getting some unexpected apparmor denials [09:40] * mvo nods [09:41] zyga: https://paste.ubuntu.com/p/Kdk8yjbFC9/ [09:42] anyone noticed that google:opensuse-42.3-64:tests/main/appstream-id is failing often? not jus the test, but afaict that particular combo [09:45] mborzecki: yes [09:47] mborzecki: looking [09:47] mborzecki: I'm at my desk, all fuss is over today [09:47] zyga: hehe, no mandatory ice cream? [09:48] mborzecki: but google:ubuntu-14.04-64:tests/main/appstream-id just now [09:48] *but also* [09:48] mborzecki: ice cream? no, but we did get a sandwich on the way home [09:48] pstolowski: I haven't seen appstream-id failures, is that new? [09:48] pstolowski: heh [09:49] [Mon Sep 3 09:28:38 2018] audit: type=1400 audit(1535966919.486:60): apparmor="DENIED" operation="capable" profile="snap-update-ns.test-snapd-layout" pid=17922 comm="3" capability=2 capname="dac_read_search" [09:49] [Mon Sep 3 09:28:39 2018] audit: type=1400 audit(1535966920.206:61): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="snap-update-ns.test-snapd-layout_foo" name="/tmp/.snap/snap/" pid=18214 comm="3" srcname="/snap/" flags="rw, rbind" [09:49] those two? [09:49] zyga: it does a round trip to the store, so maybe somewhere there [09:49] zyga: yes [09:49] so the first one is interesting dac_read_search is just taversing unreadable directories [09:49] zyga: i started noticing it today [09:49] we should add that to the profile, looks like an obviously missing thing [09:49] the second one is interesting [09:49] this looks like attempt to consturct a layout over all of /snap! [09:50] *construct [09:50] tests are very unreliable today, its quite annoying [09:50] mborzecki: can you share the mount profile, I suspect it is trying to create a directory, hits /snap/* and notices that this affects the host filesystem [09:50] zyga: the layout test? [09:50] mborzecki: then decides to make a mimic over /snap to avoid showing that on the host [09:51] mborzecki: yes, the one for layout_foo [09:51] zyga: ok, let me rerun this with spread [09:51] mborzecki: with --debug please [09:51] mborzecki: but it looks okay and expected-ish [09:51] mborzecki: it depends if the internal namespace has the right directories to begin with [09:51] mborzecki: I suspect we need to allow it to create instance directories [09:52] mborzecki: look at cmd/snap-update-ns/main.go:152 please [09:52] mborzecki: around the call to AddUnrestrictedPrefixes [09:52] mborzecki: I suspect that needs to include the layout variant of that [09:52] what is snapName for instances there? [09:52] is it "test-snapd-layout" or "test-snapd-layout_foo"? [09:53] zyga: _foo [09:53] test-snapd-layout_foo [09:55] mhm [09:55] interesting [09:55] perhaps we need the instance-less version there as well [09:55] that would explain why it is trying to construct a mimic [09:55] mborzecki: once it fails discard the mount namespace [09:55] mborzecki: and set SNAP_DEBUG=1 [09:55] and run test-snapd-shell.sh /bin/true [09:55] or something like that [09:55] ack [09:57] thank you! [09:57] I'm preparing the PR for review [09:57] I'll rework comments to explain the restricted mode [09:57] and drop some unused code [09:59] uh, we got some meeting [10:02] oh? [10:02] now? [10:03] ah. I see [10:04] the session is full [10:04] :P [10:04] what is that? [10:04] some kind of joke [10:06] haha [10:06] and it doesn't work in ff [10:06] because it's '2010' [10:06] meeting has ended [10:06] what kind of crap is that? [10:06] why do we have to have 2000 different meeting system [10:07] ok [10:07] I'll ignore that [10:43] zyga: https://paste.ubuntu.com/p/RdyjBJ9nM7/ [10:43] looking [10:44] zyga: https://paste.ubuntu.com/p/4MXzzC7Tc4/ [10:46] yeah [10:46] we need more logging :() [10:46] we need to log when we decide to make a mimic and log the reason for that [10:47] main.go:196: DEBUG: * mount (/snap/test-snapd-layout/x1/bin-very-weird-place /bin/very-weird-place none rbind,rw,x-snapd.origin=layout 0 0) [10:47] this is exactly what I said before [10:47] it's very interesting [10:47] because we are crating a mimic for /bin/ [10:48] (which is expected) but then because /snap/test-snapd-layout is not in the trusted path we go on to create a mimic for /snap [10:48] but this is unexpected and in fact, not allowed by appamor [10:48] so we fail as you saw [10:48] thanks [10:48] this will require a little bit of integration [11:06] still need a second review for 5736 [11:12] doing now [11:14] +1 [11:14] mvo: approved [11:17] zyga: ta, 5754 is another simple one where it would be great to get a review [11:21] sparkiegeek, so what was it? [11:22] thresh: I re-triggered the automated review check, which then passed successfully. We've been tweaking timeouts and retry behaviour in the store, and looks like this got caught up in the transition [11:22] Anyone able to help me out with an lxc issue - getting nothing from #lxc or #lxcontainers. Pretty sure it's related to user and group mappings but I just can't get my head round it? [11:25] mvo: that report is pretty old [11:27] zyga: it seems to be still valid though [11:27] yes, I'm worried that it will stay this way [11:27] zyga: it works with this diff now https://paste.ubuntu.com/p/KYVCj2vNZc/ [11:27] sparkiegeek, allright, thanks a lot! [11:27] zyga: aha, yes, that is unfortunate [11:27] zyga: we could try to pester sergio about it [11:27] mborzecki: excellent! [11:27] mborzecki: that's very reassuring :) [11:28] mborzecki: I'm almost done documenting new things, I removed all the cruft I found and I'll open a PR shortly [11:28] zyga: great, ping me for the review :P [11:28] I will :) [11:29] zyga: btw. could you do another pass on https://github.com/snapcore/snapd/pull/5713 ? [11:29] Chipaca: any concerns about 5754? iirc you added this originally [11:30] ouch, yes but it's pretty long so I'll do it in my review part of the day [11:30] mvo: looking [11:30] zyga: ack, thanks [11:31] mvo: snapcraft is using that exact (w/--dirty) command, though [11:32] mvo: does snapcraft still remove snapcraft.yaml if it's in snap/snapcraft.yaml? [11:32] (yes I can see how that might be inconvenient for us though) [11:32] sigh [11:32] dunno [11:33] mvo: I don't have strong opinions on this; the command that works out the version might have been committed by me, but the whole core teem collaborated on its details [11:33] it's the perfect shed [11:35] Chipaca: thanks, just wanted to double check with you, I remove -dirty for now and will see what needs to be done to get it fixed in snapcraft [11:35] Chipaca: I think snapcraft is running the git describe --dirty either at a different time or in a different checkout :/ [11:36] Chipaca: anyway, no worries, just wanted to double check that I don't step on toes or something :) [11:37] mvo: i've got 20 of them, step all you want [11:37] * Chipaca reveals why his typing is often poor [11:44] zyga, mvo: I'm awayish today, but I'll get back to you for some dragonboard logs/debugging for the console-conf segfault tomorrow [11:44] ok [11:46] hmm, got a meeting at standup time [11:50] zyga: pushed everything to bboozzoo/parallel-install-snap-namespace-mapping-with-trespassing-fix branch in my fork [11:50] ok [11:51] note that I will not present the trespassing fix as-is, I'm making it review-friendly now [11:51] I pushed to my branch and the last thing on my todo list is that tmpfs checker code [11:51] you know, the one we changed during the live review [11:51] I think it may still be wrong === paperManu_ is now known as paperManu [12:16] mvo: hi, have you tried to upload snapd with the snapd type, yet? [12:16] google:opensuse-42.3-64:tests/main/appstream-id failed again [12:17] mborzecki, it is failing in different systems [12:18] cachio: heh, must be mu luck then :) [12:18] mborzecki, I also saw some timeouts [12:18] pedronis: are you aware of any troubles store side? [12:19] not atm [12:19] mborzecki, pedronis https://paste.ubuntu.com/p/MkVNvYRb4H/ [12:19] I saw some of these [12:23] fwiw, searching does not seem to work locally either [12:24] wrz 03 14:23:47 galeon snapd[1848]: retry.go:52: DEBUG: The retry loop for https://api.snapcraft.io/v2/snaps/refresh finished after 4 retries, elapsed time=40.002221771s, status: Post https://api.snapcraft.io/v2/snaps/refresh: net/http: request canceled (Client.Timeout exceeded while awaiting headers) [12:24] it's hitting the retry loop [12:28] mborzecki, I see error no space left on device on amazon linux [12:29] mborzecki: store seems to be dying [12:29] Chipaca: death by 1000 requests [12:29] mborzecki, it is really weird because it is the only image I didnt update last week [12:29] mborzecki: I … hope it's more [12:29] but what do I know [12:29] * Chipaca goes to see a dog about lunch [12:31] pedronis: not yet, thats a nice idea [12:31] sil2100: thank you [12:37] $ snap find foo [12:37] error: unable to contact snap store [12:37] is the store down ? [12:38] ah, third try gets a response ... probably just a hiccup [12:39] heh ... and there is a forum post about it the second i ask ... [12:43] * zyga finishes writing a very long test [12:43] mborzecki: the test for whole thing was missing (unit test) [12:43] hm? [12:44] OK. Someone put me out of my misery - how come one is reporting lxd version 3.4 and one is reporting 3.0.1? https://paste.ubuntu.com/p/hjvZSZvRHw/ [12:45] Gargoyle: can you echo $PATH on both? [12:49] mborzecki: https://paste.ubuntu.com/p/N67TwFTJVy/ [12:49] Gargoyle, my guess would be you additionally have the deb installed on the first system [12:50] /snap/bin is only appended to the system PATH, so the deb would take precedence [12:50] Gargoyle: or what ogra wrote ^^ [12:50] Awww nuts... Good spot [12:51] must have sneaked in before a change was made to ansible scripts. :/ [12:58] mborzecki: https://github.com/zyga/snapd/blob/fix/trespassing-v2/cmd/snap-update-ns/change_test.go#L1795 [12:58] mborzecki: cachio: yes, store issues, https://forum.snapcraft.io/t/problem-request-failures-to-the-store/7177 [12:58] tadam :) [12:59] mborzecki: question wrt pushing this [12:59] mborzecki: one big or split? [12:59] oh [12:59] standup! [13:00] and now I can't re-install due to ^^^ [13:00] Time for a brew! :D [13:04] Hi, does ubuntu core base images include tools for connecting to wireless networks? Or do you need a device with ethernet to download the wireless tools? [13:06] Chipaca: hello [13:06] standup? [13:07] zyga: mvo: i've got a conflicting meeting [13:07] zyga: mvo: uk travel provider thing [13:07] thanks! [13:10] Chipaca: ok, enjoy, thanks [13:11] enjoyment was not in the cards [13:14] it comes with the process [13:19] hello! Just to say the store team are aware of some problems with long and failing requests, and we're looking at it, updates will be here: https://forum.snapcraft.io/t/problem-request-failures-to-the-store/7177 when we have them [13:28] (that probably explains why snap info is slow, thanks!) [13:32] Gargoyle: sudo snap info /var/lib/snapd/cache/* -> maybe you still have the .snap locally [13:33] Gargoyle: actually you might need to work a bit harder than that because the * won't expand [13:33] Gargoyle: sudo find /var/lib/snapd/cache -type f -print0 | xargs -0 sudo snap info [13:51] Brb, just a quick walk and lunch [13:53] Will need to step out to run an errand.. will be back later today. [14:07] hey! I'm having snapcraft issues under CI (I guess something impacting the docker image). It seems to have started on Friday: https://travis-ci.org/ubuntu/yaru/builds [14:07] for instance: https://travis-ci.org/ubuntu/yaru/builds/423938949 [14:08] (running "snapcraft" on the project works fine on up to date cosmic) [14:09] diddledan: is that building using snapcraft from upstream git? Looks like it is [14:09] didrocks: i have the same issue with the "josm" snap it seems [14:09] ello [14:09] s/diddledan/didrocks/ [14:09] it's using whatever the snapcraft upstream docker image is pointing at :) [14:09] didrocks: change your nick, foo! :-p [14:10] I demand priority [14:10] diddledan: I'm having it since 1998 ;) [14:10] bah [14:10] :D [14:10] like an good old wine :p [14:10] precedent, shmecedent! [14:11] ;) [14:11] have poked ev, as the snapcraft guys are on vacation today. [14:11] _all_ the snapcraft guys?! [14:11] thx! === chihchun is now known as chihchun_afk [14:12] yes, they're on a beach somewhere drinking mohitos [14:12] or something [14:12] bass turds! [14:12] (poo from a fish!) [14:12] * Chipaca expects a plugin: mojito [14:12] hah [14:12] getting flashbacks to seattle [14:12] haha [14:14] what happens in seattle stays unsaid. except Wimpress's blocked credit cards [14:16] https://youtu.be/zcqxCeYkkhk?t=7 [14:23] fwiw, 5606 needs a second review [14:24] (I mean, it technically has two but it changed quite a bit since the initial review from Chipaca) [14:27] mvo: looking [14:28] re [14:28] mvo: pedronis should probably confirm his concerns were addressed [14:28] mvo: i can then re-review if he doesn't have time to do a full pass [14:29] Chipaca: thanks, sounds reasonable [14:30] pedronis: does this mean you're re-reviewing, or leaving the road open for me to do so? [14:30] I'm leaving the road open [14:30] my original review was mostly, don't abuse context [14:30] ok [14:31] pedronis: it's the way context dresses [14:31] * Chipaca runs as far away as he can [14:31] Chipaca: bad John, bad John [15:16] Hi, the store should be operational again, let us know if you have any other (relevant) problems :) [15:29] thank you tomwardill :) [15:29] what was the issue? [15:30] we're not entirely sure, something triggered a bit of a death spiral, then the retry built up [15:30] looking for the cause now [15:30] *retry traffic [15:30] "death spiral" always a catchy turn of phrase, that :) [15:34] I'm trying to re-install lxd snap (after removing old apt version). But I have lxd.service, lxd.socket and lxd-container.service in /etc/systemd/system which are all symlinked to /dev/null. What's the best way to fix this? [15:35] * cachio lunch [15:35] That is, I am assuming that is why service lxd start/restart just returns an error about it being masked. [15:36] er, I had issues with lxd recently too and discovered the same thing... but I think it's normal [15:37] I think the snap one might be called snap.lxd or something [15:39] Yeah, 'snap services' lists lxd.daemon, and I have a snap.lxd.daemon service in systemctl [15:39] Gargoyle: the ones in lxd will have different names [15:39] Gargoyle: snap...service iirc [15:40] Gargoyle: snap.lxd.daemon.service, in this case [15:40] MattJ: that :-) [15:40] Gargoyle: however, 'service lxd' is not the thing to manipulate snap services [15:41] Gargoyle: 'snap start/stop/restart/logs' are snap commands to manipulate services that are snapped [15:41] and 'snap services', heh [15:42] in all of them, you can use a snap name to ask for everything in a snap, or add a service name to ask about just that one [15:43] OK. So possibly still got some "old apt" / snap version mix-ups then. Now getting - Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: connection refused [15:44] I assumed this was because lxd wasn't running. [15:45] * Chipaca takes a break [15:46] sparkiegeek, it seems it's the same issue again with vlc #555 [15:46] Gargoyle: _what_ prints that error? [15:47] lxc list [15:51] Gargoyle: and what's "which lxc"? [15:52] hmm. /usr/bin/lxc [15:52] Gargoyle: yeah [15:52] I'm just going to take this AWS instance out back and shoot it in the head! [15:52] Gargoyle: lxc and lxd might come from different debs [15:53] nvm. I'll rebuild a clean node tomorrow without the snap/apt conflicting version. [16:03] mvo, hey, I have noticed that there are a few very basic utils that are not included in core18: vi, ping... are those going to be add back? [16:09] abeato: we can add things back on a case-by-case basis, iirc edge has vim-tiny already [16:11] mvo, ping is definitely essential ... especially since you cant snap it (it is suid) [16:12] mvo, if you need to make room, drop bash !!! ;) [16:12] ogra: ok, let chat tomorrow, I need to run now but thats easy enough [16:12] mvo, perhaps a forum thread would be good to collect a list [16:13] (i'm sure there will be more and adding it throughout the cycle would be hit and miss for users in the end) [16:20] ppisati, on my pi3 systems the wlan hangs quite frequently in recent times (i sadly dont know when it started) it seems to be dying with "brcmf_sdio_hostmail: Unknown mailbox data content: 0x40012" [16:21] ppisati, i found https://github.com/raspberrypi/linux/issues/1342 and https://github.com/raspberrypi/linux/issues/2453#issuecomment-396063259 upstream ... bth seemingly unsolved though :( [16:22] (it hung three times today and 5 times yesterday ... to give you a figure) [16:34] pedronis: finally opened a PR with common snap directories for install/remove of instance-keyed snaps https://github.com/snapcore/snapd/pull/5758 iirc i had a spread test to check if dirs are created, i'll try to dig it up and push it there as well (but that's for tomorrow) [17:02] hrm! [17:04] I may have found and edge case to an edge case [17:04] * zyga runs hacked spread test to examine [17:09] * Chipaca EODs [17:20] * zyga found interesting things [17:20] snapd folks; question here about running postgresql which doesn't like to run as root; I said I'd get one of ya'll to pop in and comment on the possibility of running the daemon under a non-root account: https://forum.snapcraft.io/t/snapping-a-rails-app/7179/2 [17:29] diddledan: https://forum.snapcraft.io/t/multiple-users-and-groups-in-snaps/1461/3?u=chipaca probably? [17:39] thanks Chipaca . I've posted the link on the rails thread [17:39] diddledan: I don't know that status of that though :-) [17:39] also, I don't know what i'm doing here post-eod [17:39] looks like it's not ready yet [17:39] procrastinating from home chores i'm sure [17:59] zyga, how are we going to deal with the lack of extrausers in Fedora? [17:59] it's not like there's a port of this functionality using sssd? [18:00] not sure, is it relevant? [18:00] from what I can tell, it's kinda needed for supporting system users in snaps? [18:00] for services and such [18:00] I'm not sure that's the case [18:01] in any case we don't have support for additional users yet [18:01] oh okay === Son_Goku is now known as Conan_Kudo === Conan_Kudo is now known as Son_Goku [18:39] aggghhhh... deb version of lxd/lxc is installed in the AWS default 18.04 image! [18:39] That explains a lot. How can I find all it's packages and completely purge them so as not to interfere with the snap verison? [18:50] * zyga considers taking a bike [18:51] or [18:51] watching archer and having supper [18:51] or figuring out clashing symlinks [19:19] * Gargoyle does a little dance \o/ \\o o// [19:20] Containers launch properly using the snap version! [19:33] Now I just need to figure out why when the script is run by the team-city agent, it doesn't think lxc is installed.