[00:07] <zyga_> kyrofa: ack, I will try tomorrow
[00:08] <zyga_> jdstrand: ack
[00:13] <zyga> pedronis: ack
[05:06] <mup> Bug #1684014 changed: LibreOffice in snap won't print <libreoffice (Ubuntu):New> <https://launchpad.net/bugs/1684014>
[07:05] <zyga> Good morning
[07:15] <pstolowski> morning zyga!
[07:15] <zyga> Hey hey
[07:15] <zyga> It is Friday and everything looks terrible :-)
[07:18] <pstolowski> zyga: no kidding
[07:20] <zyga> pstolowski: fedora builds are broken
[07:20] <zyga> pstolowski: travis slots last night were all gone
[07:20] <zyga> pstolowski: and I didn't manage to release snapd
[07:21] <pstolowski> gosh
[07:22] <pstolowski> zyga: what's going on with fedora this time?
[07:25] <pedronis> pstolowski: afaict go build itself (wich is mostly a wrapper process) dies with exit 2 just before doing the steps of building snapd main itself (after having built all other packages)
[07:26] <pedronis> without printing any useful info
[07:26] <pedronis> fun :/
[07:26]  * pedronis -> afk
[07:30] <pedronis> pstolowski: zyga:   I tried to strace it,   it was late so maybe I missed something, but afaict no smoking gun there, just after it writes sig.o to signal.a  (the internal pack, internal meaning I discovered is code inside go build not an external exec) it just starts unlinking stuff from the work dir and then you get the exit 2
[07:30] <pedronis> so it's a sorf of unexplained semi orderly death
[07:30]  * pedronis really afk
[07:31] <pstolowski> were there any updates to fedora?
[07:32] <pedronis> I don't know,   golang claims to be from Mar 5  (so we should have been using it already?  but I don't know how updates percolates there)
[07:32] <zyga> hmm
[07:32] <pedronis> rpmbuild seems the same
[07:33] <pedronis> but maybe some lib changed
[07:33] <zyga> hey pedronis, thank you for the analysis
[07:34] <pedronis> zyga: for details, what I did sofar   is set the few GO env vars and rerun  the "go build" line   in a -debug spread session,   with -n  (to see how far we go, as I said very far)  and then with strace -f
[07:36] <pedronis> it's also very consistent
[07:36] <pedronis> dies the same way each time
[07:37] <zyga> pedronis:  do you remember what you set?
[07:37] <pedronis> I also double checked the content of the work dir ,  as I said signal.a  looks right  but   ... /cmd/snapd/_obj is not yet created
[07:37] <pedronis> zyga: it's in the log
[07:38] <pedronis> just GOPATH  and GOFLAGS
[07:38] <pedronis> I think
[07:38] <zyga> ok
[07:38] <pedronis> after of course cd-ing into /root/rpmbuild/....
[07:39]  * zyga goes to reproduce this with fresh head
[07:43] <zyga> so, the list of updates is on https://bodhi.fedoraproject.org/updates/?releases=F27
[07:43] <zyga> I'll see if there's something recently that could explain it
[07:49] <pstolowski> zyga: looking into this too
[07:50] <zyga> if we don't find anything quickly let's split focus and not all try to solve this
[07:51] <zyga> I went back 10 days and there's nothing that would indicate any relationship
[07:51] <zyga> the closest I saw was a clang update to ship a versioned compiler-version binary
[08:05] <zyga> I noticed that on 18.04 high-dpi regressed as compared to 17.10 (for snaps)
[08:23] <pstolowski> zyga: fwiw I got that error when building signal.a; then I executed /var/tmp/rpm-tmp.5TbNku manually and it succeeded
[08:23] <pstolowski> zyga: (fedora on linode)
[08:23] <zyga> interesting, I wonder what that tells us
[08:24] <pstolowski> exactly
[08:34] <pstolowski> zyga: i've rebuilt manually like that 3 times, worked
[08:35] <zyga> pstolowski: I'm doing a mock build to see how that works
[08:35] <pstolowski> zyga: do you know what creates and calls /var/tmp/rpm-tmp.5TbNku ?
[08:36] <zyga> I think it's part of some RPM macro but I don't know which
[08:36] <zyga> it could also be a recovery mechanism
[08:36] <zyga> when rpm fails, it writes "reproduce me" shell
[08:37] <pstolowski> zyga: yeah, ok, it's assembled from snapd.spec
[08:38] <zyga> Breakfast break
[08:39] <zyga> After that I will mock build
[08:39] <zyga> And try koji next
[08:41] <pedronis> pstolowski: that is interesting because if  I run the go build part alone again and again is still fails
[08:46] <pstolowski> pedronis: wow, interesting
[08:47] <pstolowski> pedronis, zyga and i'v just run the build via rpmbuild --with testkeys -bs and it failed
[08:47] <zyga> yes, I got that too
[08:48] <pedronis> so repeating rpmbuild  fails (different state each time),     repeating go build   fails (afaict)
[08:48] <pedronis> but repeating the tmp script works?
[08:48] <pedronis> fascinating
[08:48] <pstolowski> pedronis: i'm just going to try tmp script again
[08:49] <pedronis> is it succedding in the sense of exit 0
[08:49] <pedronis> or in the sense it build a snapd binary?
[08:51] <zyga> I tried to remove the spread + linode|google magic out of the equation
[08:51] <zyga> I built an source tarball and srpm
[08:51] <zyga> now building it in mock
[08:51] <zyga> this is essentially how all of fedora is normally built
[08:51] <zyga> iff this fails
[08:52] <zyga> we can give this srpm to people and ask for help
[08:52] <pstolowski> pedronis: yes, exit 0, no error around signal.a, the execution ends with https://pastebin.ubuntu.com/p/gtHJGPbPmV/
[08:52] <zyga> + mock is nicer than rpmbuild directly
[08:52] <pstolowski> pedronis: so, tmp script just succeeded again
[09:02] <zyga> so
[09:02] <zyga> ah, I didn't use test keys so maybe that's why
[09:02] <zyga> the source rpm + mock fail on pack but in a different spot
[09:02] <zyga> BUILDSTDERR: /usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/golang.org/x/net/context/ctxhttp.a -trimpath $WORK -shared -goversion go1.9.4 -p golang.org/x/net/context/ctxhttp -complete -installsuffix shared -buildid ab9e3e1669ce2ef40183ac44ed64b157e02355b9 -D _/usr/share/gocode/src/golang.org/x/net/context/ctxhttp -I $WORK -I /usr/share/gocode/pkg/linux_amd64_shared -pack ./ctxhttp.go
[09:03] <pedronis> pstolowski: how are you running it again,  I get  failed to create symbolic link because the link alreday exists
[09:05] <zyga> Pharaoh_Atem: hey
[09:05] <zyga> Pharaoh_Atem: so I have a srpm I can reproducibly fail to build in mock
[09:06] <pedronis> pstolowski: are you rm -rf something before running it again?
[09:06] <pstolowski> pedronis: I get that symink error too but that doesn't interrupt the build
[09:07] <pedronis> heh
[09:07] <pedronis> it definitely does here
[09:07] <pedronis> pstolowski: ah, are you running without -e ?
[09:07] <pedronis> that's why it finishes
[09:07] <pedronis> it ignores errors
[09:07] <pedronis> nothing magic
[09:07] <pstolowski> ah
[09:08] <pstolowski> dammit
[09:08] <pedronis> indeed
[09:08] <pedronis> too bad
[09:09] <pedronis> zyga: I have no clue what dimension changed (our code,  fedora bit) though ... it stopped working quite randomly yesterday afaict
[09:09] <zyga> yes, I'm totally puzzled
[09:09] <zyga> I will try with -testkeys now
[09:09] <zyga> as my failure looks the same but in different spot
[09:10] <pedronis> it seems a go build bug
[09:10] <pedronis> but unclear why it appears now
[09:10] <pedronis> what it triggers it
[09:10] <pedronis> s/what it/what7
[09:10]  * pedronis -> off
[09:12] <pedronis> zyga: one thing to try would also be  to copy over the BUILD directory to bionic and see if that go build command fails there too with 1.9,  fwiw it failed for me also when I didn't pass some of the ldflags stuff
[09:13] <pedronis> anyway as said, from the strace it seems to fail in the boring bits of driving the build
[09:29] <zyga> on my host it consistently fails elsewhere
[09:30] <zyga> maybe it depends number of CPUs?
[09:31] <zyga> oh, I'll try koi first
[09:31] <zyga> koji
[09:48] <pstolowski> fwiw it fails also with go1.9.5 and 1.10.1 on fedora
[10:01] <Caelum> zyga: my btrfs died, I submitted that request from the console on a half-broken system, just reinstalled now and picking up the pieces, on ext4 this time
[10:02] <pedronis> even just this fails:   cd /root/rpmbuild/BUILD/snapd-1337.2.31.1  ;  cd cmd/snapd;  go build -v -x .   with or without -a
[10:02] <pedronis> it just exit 2 before building main
[10:02] <zyga> Caelum: ouch
[10:02] <zyga> sorry to hear that
[10:02] <zyga> Caelum: I made some changes since your patches
[10:02] <Caelum> excellent
[10:03] <zyga> pedronis: I submitted a koji build https://koji.fedoraproject.org/koji/taskinfo?taskID=26057356
[10:03] <Caelum> so .32 is out
[10:03] <pedronis> pstolowski: would be interesting to copy over the BUILD to an ubuntu system and see if it fails too,   as I show there it seems to fail even without fancy options or anything
[10:04] <pstolowski> pedronis: yes, i'm going to try that next
[10:04] <pedronis> it just refuses to build cmd/snapd(/maing.go) on fedora
[10:04] <zyga> the log is a little bit more useful
[10:04] <zyga> https://koji.fedoraproject.org/koji/getfile?taskID=26057357&volume=DEFAULT&name=build.log&offset=-4000
[10:04] <zyga> (tail of the build log)
[10:04] <zyga> tail feature == good thing to copy
[10:05] <zyga> does this cp $WORK/b150/_pkg_.a /builddir/.cache/go-build/92/92ebdb22b8c38b43994a36a864adc877e4e85517bc1f1a7c1b22ee610a1af86d-d # internal
[10:05] <zyga> error: Bad exit status from /var/tmp/rpm-tmp.MIqip3 (%build)
[10:05] <zyga> say it is actually cp that failed?
[10:05] <pedronis> that's a completely different kind of failure
[10:05] <pedronis> though
[10:05] <pedronis> we don't know
[10:05] <pedronis> usually the last thing printed might have work or not
[10:06] <pedronis> as I said in the other failure
[10:06] <pedronis> the pack works
[10:06] <pedronis> is the next step that is not taken
[10:06] <zyga> this is in koji, the fedora build service
[10:06] <pedronis> I understand
[10:06] <pedronis> just saying that looks like a failure
[10:06] <zyga> if this is kernel related they are (perhaps) on a different version of that
[10:06] <pedronis> but a different failure
[10:07] <pedronis> it's also an exit 1 vs exit 2
[10:07] <zyga> note that the same SRPM was failing on pack on my host
[10:07] <pedronis> I note that on linode the failure mode is very deterministic
[10:08] <pedronis> as I said it seems to refuse to go to the last bit
[10:08] <pedronis> and build cmd/snapd itself
[10:08] <pedronis> (the pack itself works)
[10:08] <pedronis> indeed it fails like that, exit 2 refusing to build main even in builds that don't have a pack bit
[10:09] <pedronis> late
[10:10] <pedronis> zyga: as I said the failure in that  log, looks very different, it seems to be failing building one the prereq packages
[10:10] <pedronis> on snapd itself
[10:10] <pedronis> s/on snapd/not snapd/
[10:11] <pedronis> what   golang is that btw?
[10:11] <pedronis> 1.10?
[10:11] <zyga> I switched from F28 to F27 and now it's the same error I saw on my host
[10:11] <zyga> let me check
[10:11] <zyga> https://koji.fedoraproject.org/koji/getfile?taskID=26057448&volume=DEFAULT&name=build.log&offset=-4000 (the F27 build tail)
[10:12] <zyga> DEBUG util.py:439:   golang                              x86_64 1.9.4-2.fc27            build 622 k
[10:12] <zyga> 1.9.4
[10:12] <pedronis> on F27 ?
[10:12] <pedronis> on F28 ?
[10:13] <zyga> this is on F27 now
[10:13] <zyga> the prior build was on F28 and this is why it was different, it was also on using more recent golang
[10:14] <pedronis> it just feels like they brok all go building
[10:14] <pedronis> it's very strange
[10:14] <pedronis> also the errors are all different
[10:14] <zyga> I'll ask around
[10:18] <zyga> I asked on fedora-devel
[10:19] <zyga> not sure what to do now
[10:19] <zyga> I feel we should disable fedora for the time being
[10:19] <zyga> this would unblock the release
[10:22] <pedronis> zyga: interesting, not sure if related but  I see,    /usr/share/gocode/src/%{go_import_path}  like that in the filesystem
[10:22] <pedronis> like some package is really broken
[10:22] <zyga> oh
[10:22] <zyga> that looks like RPM macro that didn't expand
[10:22] <zyga> that's very interesting, let me look
[10:23] <zyga> pedronis: I'll disable fedora now
[10:25] <zyga> let's see if that passes
[10:26] <mup> PR snapd#4962 opened: spread.yaml: switch Fedora 27 tests to manual <Created by zyga> <https://github.com/snapcore/snapd/pull/4962>
[10:45] <zyga> I’m debugging this in fedora-devel
[10:52] <pedronis> zyga: pstolowski: so if I copy over to ubuntu the  gocode dir and the rpmbuild dir and set GOPATH to those, even old go 1.6 fails to build cmd/snapd with exit 2
[10:53] <zyga> cool find
[10:53] <zyga> but does that tree include effectively golang 1.9 source?
[10:53] <pedronis> no
[10:54] <pedronis> gocode has only our deps
[10:54] <pedronis> goroot is the 1.6 one
[10:54] <pstolowski> pedronis: and that gocode dir contains that suspicious unexpanded macro?
[10:54] <pedronis> yes
[10:54] <pedronis> but yes, it would be better to try this with ubuntu go 1.9
[10:54] <pedronis> less moving parts
[10:55] <pedronis> but is interesting nevertheless
[10:55] <pedronis> it really seems is some issue with src code  (ours or the one brought in by dep packages)
[10:56] <pedronis> and to be clear, this is just a plain:   go build -a -v -x .
[10:56] <pedronis> nothing fancy
[10:56] <pedronis> from cmd/snapd
[10:56] <zyga> let's try building older tree
[10:56] <pedronis> anyway  the %{}  thing for sure cannot bring any good
[10:57] <zyga> # cd /home/zyga/go/.cache/govendor/gopkg.in/yaml.v2; git reset --hard 86f5ed62f8a0ee96bd888d2efdfd6d4fb100a4eb
[10:57] <zyga> fatal: Could not parse object '86f5ed62f8a0ee96bd888d2efdfd6d4fb100a4eb'.
[10:57] <zyga> mmmm
[10:57] <zyga> ./get-deps.sh failed
[10:57] <zyga> worked after I removed ~/go/.cache
[10:58] <zyga> pedronis: "go build ./cmd/snapd" works
[10:58] <pedronis> ?
[10:58] <pedronis> where, in which sense?
[10:58] <pedronis> you mean with our usual vendor dir?
[10:58] <zyga> yes
[10:58] <zyga> in a checkout on F27
[10:58] <zyga> not in the whole build machinery
[10:59] <pedronis> so it seems some packages  are broken
[10:59] <pedronis> and render this %{} nonsense
[10:59] <zyga> I ran "go build -a -v -x" next and it printed a wall of text and exited with 130
[10:59] <thresh> good morning
[10:59] <pedronis> and that breaks things
[10:59] <pedronis> (strangely in a very non visible way)
[10:59] <zyga> hey thresh
[10:59] <zyga> pedronis: go build -a -v -x in cmd/snapd also works
[10:59] <zyga> from our checkout
[10:59] <zyga> ls
[11:02] <zyga> pedronis: https://github.com/snapcore/snapd/pull/4962
[11:02] <mup> PR #4962: spread.yaml: switch Fedora 27 tests to manual <Created by zyga> <https://github.com/snapcore/snapd/pull/4962>
[11:02] <zyga> let's land this
[11:03] <pstolowski> +1
[11:11] <pedronis> zyga: pstolowski:  yes definitely switching the share/gocode stuff with out usual deps makes thing work again
[11:13] <pedronis> even using them not through vendor to be clear, but just in the GOPATH
[11:14] <pedronis> zyga: pstolowski:  so  yesterday one of the deps packages was updated?   or something changed in the rpm go stuff that affects one happens when they are installed?
[11:14] <pedronis> I mean in fedora
[11:15] <zyga> pedronis: I looked at the list of changes published in bodhi and I dind't notice anything golang at all
[11:16] <zyga> pedronis: let's merge my PR and unblock master
[11:18] <pstolowski> and i've just removed that bogus /usr/share/gocode/%{go_import* path from fs but that didn't make a difference, must be something else in gocode dif
[11:18] <pstolowski> *dir
[11:20] <pstolowski> lunch break
[11:23] <pedronis> pstolowski: yea, I tried the same,  I don't get a missing package error, but even without the %{} nonsese, it still exit 2
[11:23] <pedronis> even go 1.6
[11:28] <zyga> pedronis: I think it may be yaml related
[11:28] <zyga> https://zero.crans.org/?3e5975e22418a02e#Xv5Juf3UYkZ0FgzYq9Uu2h9AFG4FKFMWqbITXDLVZno=
[11:28] <pedronis> I was thinking the same
[11:28] <pedronis> a bit unclear exactly what
[11:28] <zyga> this is something fedora developer was helping me with
[11:29] <zyga> I'll revert the yaml version bump and build
[11:32] <pedronis> but the run that bumped it was green ??
[11:33] <zyga> reverted, no change
[11:33] <zyga> so back to where we started :/
[11:34] <vidal72[m]> zyga: I fixed trailing " and there is no more backend_test.go failure in https://github.com/snapcore/snapd/pull/4944
[11:34] <zyga> pedronis: shall we merge 4962 or do you want to resolve this before master can move
[11:34] <mup> PR #4944: interfaces: add /var/lib/snapd/snap to @{INSTALL_DIR} <Created by Erick555> <https://github.com/snapcore/snapd/pull/4944>
[11:34] <pedronis> zyga: I should be off, I'm not taking decisions
[11:34] <zyga> ok
[11:35] <pedronis> zyga: anyway it's not a bug in our code afaict
[11:35] <zyga> yes, that is clear, I thin
[11:35] <zyga> *think
[11:53] <mup> PR snapd#4962 closed: spread.yaml: switch Fedora 27 tests to manual <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4962>
[12:11] <thresh> woah, my forum post is flagged as spam
[12:11] <thresh> verynice
[12:11] <thresh> > Multiple community members flagged this post before it was hidden
[12:11] <thresh> what
[12:12] <thresh> seriously?
[12:12]  * zyga -> walk
[12:15] <thresh> how is https://forum.snapcraft.io/t/snapcraft-stage-packages-inconsistent-behaviour/4746 a spam ?!
[12:15] <thresh> and/or advertisement
[12:15] <thresh> is there a saner platform to collaborate with devs?
[12:23] <jdstrand> niemeyer: hi! quick 'snap switch' question. I have canonical-livepatch on a system. it was tracking stable, r26. yesterady I did 'sudo snap refresh --edge canonical-livepatch'. this refreshed to r38 and marked the snap as starting to track edge (based on snap info). this is all fine
[12:24] <jdstrand> niemeyer: then I did 'sudo snap switch --channel=stable canonical-livepatch'. this did not refresh the snap (good) and marked the snap as tracking stable (as seen with snap info. also good)
[12:25] <jdstrand> niemeyer: so at this point, I thought that everything was all set for the snap to only refresh the next time stable refreshed, however, at some point later in the day, the snap refreshed automatically in the background and downgraded to stable. is this expected behavior?
[12:25] <jdstrand> s/stable refreshed/stable's revision changed/
[12:27] <vidal72[m]> thresh_: someone doesn't like you :)
[12:27] <vidal72[m]> thresh:  ^
[12:28] <jdstrand> niemeyer: here is a little more detail:
[12:28] <jdstrand> $ sudo snap changes canonical-livepatch
[12:28] <jdstrand> ID   Status  Spawn                 Ready                 Summary
[12:28] <jdstrand> 551  Done    2018-03-29T13:17:24Z  2018-03-29T13:17:31Z  Refresh "canonical-livepatch" snap from "edge" channel
[12:28] <jdstrand> 552  Done    2018-03-29T13:18:04Z  2018-03-29T13:18:04Z  Switch "canonical-livepatch" snap to stable
[12:28] <jdstrand> 553  Done    2018-03-29T20:49:59Z  2018-03-29T20:50:02Z  Auto-refresh snap "canonical-livepatch"
[12:28] <jdstrand> 554  Done    2018-03-30T12:20:37Z  2018-03-30T12:20:40Z  Refresh "canonical-livepatch" snap from "candidate" channel
[12:28] <jdstrand> niemeyer: it is '553' that did the downgrade back to stable
[12:29] <jdstrand> niemeyer: if it is operating as designed, that's fine (though, I wonder about the utility of 'switch' then), I just want to make sure it is working correctly
[12:31] <pedronis> jdstrand: no, that's the correct behavior, it will refresh to whatever is on stable
[12:31] <pedronis> s/correct/expected/
[12:31] <pedronis> remember there's no order in revisions
[12:34] <pedronis> jdstrand: switch is more useful if say candidate and revision have the same revision, but I also think we changed refresh to work more like switch in that case
[12:34] <pedronis> so it might not be as useful as it was
[12:35] <thresh> vidal72[m], but I'm amazing and nice
[12:36] <vidal72[m]> :)
[12:39] <zyga> Which post is that
[12:45] <thresh> I linked it half and hour ago here
[12:45] <thresh> afraid to post it twice now lol
[12:46] <jdstrand> niemeyer: nm, pedronis answered
[12:47] <jdstrand> pedronis: sure, I realize there is no order with revisions, but if I was on r26/stable then refreshed to r38/edge, then snap switched to stable, I didn't expect to go to r26/stable. I expected to go to !26/stable
[12:48] <jdstrand> pedronis: I'm not sure of the utility of snap switch then tbh. it ends up being a delayed refresh
[12:49] <jdstrand> pedronis: where I want a delayed refresh to a new thing in the switched to channel
[12:51]  * zyga is back from his walk
[12:53] <zyga> popey: what blocks sublime-text from release?
[12:54] <zyga> thresh: I didn't see the link, can you please re-post?
[13:00] <mup> PR snapd#4955 closed: cmd/snap-confine: make /run/media an alias of /media <Critical> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4955>
[13:00] <popey> zyga: actually realised that we should store rename s-t3 to s-t, otherwise the hundreds of people with st3 won't get updates
[13:01] <zyga> can we even do this?
[13:01] <zyga> I don't think we can
[13:01] <popey> I believe the store people can rename a snap yes
[13:01] <popey> it keeps the id
[13:01] <zyga> ah, in that case, all for it
[13:01] <zyga> can I help somehow
[13:01] <zyga> I'd love to just use s-t from snaps already
[13:01] <popey> are store people working today?
[13:02] <popey> yeah, sorry, been on vacation the last few days so it's been back burner
[13:03] <pstolowski> zyga: standup, or skip?
[13:08] <zyga> oh
[13:09] <zyga> sorry
[13:09] <popey> thresh: dunno why the forum is marking your posts as spam. I have unflagged them as such. Maybe niemeyer knows
[13:09] <zyga> pstolowski: let's skip maybe
[13:09] <zyga> pstolowski: google signed me out
[13:09] <zyga> pstolowski: is it just you and me?
[13:10] <zyga> ok, found my token
[13:14] <zyga> thank you popey
[13:18] <popey> wtf. i just installed a snap and as soon as the apparmor connections were made the session exploded
[13:18] <popey> i didnt even launch the snap, it ended immediately after install finished
[13:18] <jdstrand> popey: eom?
[13:18] <popey> eom?
[13:18] <zyga> popey: probably udev
[13:19] <zyga> or oom
[13:19] <zyga> (out of memory)
[13:19] <popey> unlikely
[13:19] <popey> it has 64GB
[13:19] <zyga> but I saw people reporting bugs about touchpad issues when installing a snap
[13:19] <zyga> woah, nice machine!
[13:19] <popey> just logged back in and I get a "xorg crashed" bug reporter
[13:19] <zyga> and it feels like udevadm trigger is not good
[13:19] <zyga> popey: can you ssh in
[13:19] <zyga> and run journalctl -f
[13:19] <zyga> and reproduce?
[13:19] <popey> i logged back in, on a desktop
[13:20] <popey> I'm on core beta
[13:20] <zyga> popey: is that a wayland session
[13:20] <zyga> when wayland crashes it takes the session with it
[13:20] <popey> hah, not on your life
[13:21] <zyga> well, review your log
[13:21] <zyga> there's gotta be a backtrace somewhere :)
[13:21] <popey> juat waiting for whoopsie to do its thing
[13:28] <popey> https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers/+bug/1760104  whoospie says it's an nvidia driver bug, so who knows
[13:29] <zyga> popey: *bam*
[13:30] <zyga> maybe it's our bug
[13:30] <zyga> I got 404, it's marked as security
[13:30] <popey> I wlll try and reproduce it
[13:30] <zyga> popey: look at your journal first
[13:30] <popey> given I am building a new rev of the snap
[13:30] <zyga> there is likely a backtrace there already
[13:31] <popey> will that be already attached to the bug?
[13:31] <zyga> I cannot see the bug
[13:31]  * zyga is super happy today
[13:31] <popey> https://www.irccloud.com/pastebin/Ic5TwB7Y/
[13:31] <popey> that's all i see in journal
[13:31] <zyga>  no no, that's -f
[13:32] <zyga> that's "follow"
[13:32] <zyga> run journalctl
[13:32] <zyga> and page it
[13:32] <zyga> until you see something that's clearly a backtrace
[13:32] <zyga> it should be from this boot only
[13:32] <zyga> so not _too_ long perhaps
[13:32]  * zyga is super happy because his T470 has a working modem now :)
[13:32] <zyga> it's internal, antennas work and I have 100GB of data plan to use
[13:33] <popey> wow
[13:33] <zyga> so I can now do what I love doing
[13:33] <zyga> work outside :)
[13:33] <popey> work anywhere
[13:33] <zyga> the modem I got before was not on the whitelist (curse you lenovo)
[13:33] <zyga> but this one is flawless :)
[13:34] <zyga> I'm slowly getting used to the T series, X was way smaller
[13:34] <zyga> but this is not huge either, it's the okay size for work
[13:34] <zyga> and it's not heavy which is what I was worried about
[13:39] <popey> https://paste.ubuntu.com/p/3FbykZXbx2/
[13:39] <popey> not a lot interesting in there?
[13:41] <thresh> many thanks popey
[13:41]  * thresh wanders off to friday evening endeavours
[13:42] <popey> \o/
[13:42] <zyga> Mar 30 14:15:58 hal compiz[23168]: XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":1"
[13:42] <zyga> but this is after X died
[13:42] <zyga> Mar 30 14:15:34 hal audit[1042]: AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.dosbox-x.dosbox-x" pid=1042 comm="apparmor_parser"
[13:42] <zyga> Mar 30 14:15:35 hal systemd[22791]: Starting Notification regarding a crash report...
[13:42] <zyga> Mar 30 14:15:36 hal update-notifier-crash[1108]: Xorg
[13:42] <zyga> we have the same mouse :)
[13:42] <popey> hah
[13:43] <jdstrand> popey: haha 'eom'. I meant oom, yes :)
[13:43] <popey> "End of memory" :D
[13:43] <zyga> hmmm
[13:43] <jdstrand> End Of Memory. hehe
[13:43] <zyga> Mar 30 14:15:33 hal systemd-udevd[689]: Process '/usr/lib/snapd/snap-device-helper change snap_vidcutter_vidcutter /devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card0 226:0' failed with exit code 2.
[13:43] <zyga> Mar 30 14:15:33 hal systemd-udevd[689]: Process '/usr/lib/snapd/snap-device-helper change snap_vlc_vlc /devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card0 226:0' failed with exit code 2.
[13:43] <zyga> Mar 30 14:15:33 hal systemd-udevd[689]: Process '/usr/lib/snapd/snap-device-helper change snap_writefull_writefull /devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card0 226:0' failed with exit code 2.
[13:43] <zyga> Mar 30 14:15:33 hal systemd-udevd[689]: Process '/usr/lib/snapd/snap-device-helper change snap_zzt_zzt /devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card0 226:0' failed with exit code 2.
[13:43] <zyga> jdstrand: ^ does this ring a bell
[13:43] <zyga> remember the nvidia udev bug?
[13:43] <zyga> is this back in some way?
[13:44] <zyga> then X crashes on udev
[13:44] <jdstrand> zyga: what size is it?
[13:44] <zyga> size?
[13:45] <zyga> the mouse?
[13:45] <jdstrand> zyga: your system. I was reading backscroll
[13:46] <jdstrand> zyga: but, yes, it does ring a bell. let me look
[13:46] <zyga> jdstrand: it's popey's system, we just share the same mouse :)
[13:46] <zyga> I bought my yesterday, wanted something that had a working wheel for games
[13:46] <popey> I get it on weekends
[13:47] <zyga> pstolowski: there is a broken gopkg.in/yaml.v2 upload 10 days ago
[13:47] <zyga> it broke us
[13:47] <zyga> pstolowski: people in fedora-devel are fixing it
[13:47] <pstolowski> \o/
[13:47] <pstolowski> zyga: thanks for info
[13:48]  * zyga painfully goes through secure code and makes all the functions into methods
[13:52] <jdstrand> zyga: this is what I was thinking of: https://github.com/snapcore/snapd/pull/4022/files
[13:52] <mup> PR #4022:  interfaces/opengl: don't udev tag nvidia devices and use snap-confine instead (2.28) <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4022>
[13:52] <zyga> yeah
[13:52] <jdstrand> actually this: https://github.com/snapcore/snapd/pull/3938
[13:52] <mup> PR #3938: interfaces/opengl: don't udev tag nvidia devices and use snap-confine instead <Created by jdstrand> <Merged by zyga> <https://github.com/snapcore/snapd/pull/3938>
[13:53] <zyga> popey: ls /dev/nvidia* /dev/nv*
[13:53] <zyga> maybe with -l
[13:53] <zyga> but it appears we are tagging something
[13:53] <zyga> and then failing
[13:53] <popey> http://paste.ubuntu.com/p/PRpFNNqc29/
[13:55] <jdstrand> no new devices there
[13:56] <jdstrand> popey: ls -lR /dev/dr[im]
[13:56] <popey> http://paste.ubuntu.com/p/PBx2T3XCTh/
[13:57] <zyga> jdstrand: are you sure?
[13:57] <zyga> ah
[13:57] <zyga> nvme
[13:57]  * zyga is a moron
[13:57] <jdstrand> popey: udevadm info /dev/dri/card0
[13:57] <popey> http://paste.ubuntu.com/p/C875ZW9MPB/
[13:59] <zyga> well, it's tagged for sure
[13:59] <zyga> popey: is this a dual-gpu machine?
[13:59] <popey> nope
[13:59] <jdstrand> and it's not the same bug since there is udev information on it
[14:00] <zyga> yes
[14:00] <zyga> that's a very good point
[14:00] <zyga> jdstrand: I kind of feel we should hash the udev tag, it's a leak of all the snaps on popeys's system
[14:01] <jdstrand> there are other leaks of that
[14:02] <zyga> I also wonder if there's something we could do to make the list of tags shorter
[14:02] <zyga> especially for udev
[14:02] <zyga> we could tag "capability" more than "snap"
[14:02] <zyga> but that's unrelated to the bug
[14:02] <jdstrand> I think we should consider hashes when we have fine-grained network mediation and/or inode labelling since we'll need to hash for both
[14:02] <zyga> jdstrand: can we run the device helper
[14:02] <zyga> to see the error?
[14:03] <jdstrand> zyga: sure, go ahead
[14:04] <zyga> I mean, I cannot :)
[14:05] <zyga> let me come up with the command to run
[14:05] <jdstrand> it's in the logs
[14:05] <jdstrand> /usr/lib/snapd/snap-device-helper change snap_zzt_zzt
[14:05] <jdstrand>               /devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card0 226:0
[14:06] <jdstrand> popey: do you have zzt installed?
[14:06] <zyga> yeah, I see now
[14:06] <zyga> usr/lib/snapd/snap-device-helper change snap_writefull_writefull /devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card0 226:0
[14:06] <popey> of course I do :D
[14:06] <zyga> popey: can you please run: /usr/lib/snapd/snap-device-helper change snap_writefull_writefull /devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card0 226:0
[14:06] <zyga> as root
[14:06] <jdstrand> popey: snap list shows it?
[14:06] <popey> jdstrand: yes
[14:06] <jdstrand> ok
[14:07] <popey>  /usr/lib/snapd/snap-device-helper: 31: /usr/lib/snapd/snap-device-helper: cannot create /sys/fs/cgroup/devices/snap.writefull.writefull/devices.allow: Directory nonexistent
[14:07] <jdstrand> interesting
[14:07] <jdstrand> I have issues like this here without nvidia
[14:07] <jdstrand> Mar 30 06:33:32 localhost systemd-udevd[23812]: Process '/usr/lib/snapd/snap-device-helper change snap_test-policy-app-consumer_opengl /devices/pci0000:00/0000:00:02.0/drm/card0 226:0' failed with exit code 2.
[14:08] <jdstrand> /usr/lib/snapd/snap-device-helper: 31: /usr/lib/snapd/snap-device-helper: cannot create /sys/fs/cgroup/devices/snap.test-policy-app-consumer.opengl/devices.allow: Directory nonexistent
[14:08] <jdstrand> oh right
[14:09] <jdstrand> so, the cgroup isn't created unless the snap is started
[14:09] <jdstrand> so the 'change' operation fails
[14:09] <zyga> oh
[14:09] <zyga> interesting
[14:09] <zyga> I doubt this crashes x
[14:09] <zyga> but we should fix it
[14:09] <jdstrand> we should probably just check that and exit 0
[14:09] <zyga> we should bail
[14:09] <jdstrand> that would not crash x
[14:09] <zyga> if there's no group at all
[14:10] <jdstrand> it is a spurious log entry
[14:10] <jdstrand> it's harmless
[14:11] <zyga> I agree
[14:12] <jdstrand> popey: you naming your system 'hal' when looking at udev events feels really weird
[14:12] <popey> hah :)
[14:12] <zyga> popey: the private bug you reported, does it contain a backtrace?
[14:13] <popey> zyga: added you as a subscriber, can you see it now? https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers/+bug/1760104
[14:16] <jdstrand> this has nothing to do with snappy afaics:
[14:16] <jdstrand> SegvAnalysis:
[14:16] <jdstrand>  Segfault happened at: 0x7f6b00000008: Cannot access memory at address 0x7f6b00000008
[14:16] <jdstrand>  PC (0x7f6b00000008) not located in a known VMA region (needed executable region)!
[14:16] <jdstrand>  Stack memory exhausted (SP below stack segment)
[14:16] <jdstrand> SegvReason: executing unknown VMA
[14:16] <jdstrand> Signal: 11
[14:17] <zyga> popey: I can
[14:17] <zyga> I agree with jdstrand
[14:17] <mborzecki> hellos
[14:17] <mborzecki> zyga: 2.32.2 coming later today?
[14:17] <zyga> hey hey
[14:17] <jdstrand> seems like a 'normal' crasher
[14:18] <zyga> yes, slowly
[14:18] <zyga> well, let me check PRs
[14:18] <jdstrand> popey: depending on how adventurous your are feeling, you could revert to the previous core, remove the current core, then refresh core and see if it triggers it
[14:19] <mup> PR snapd#4949 closed: tests: fix quoting issues in econnreset test <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4949>
[14:19] <popey> It wasnt core that triggered it
[14:20] <jdstrand> popey: you could remove/install then. the core idea is that it wuld stress test all the security systems
[14:20] <popey> hm, can't reproduce it with the same snap again
[14:20] <jdstrand> I suspect it is poor timing
[14:20] <jdstrand> right
[14:21] <jdstrand> I would see random stuff like that with intel until the drivers settled down
[14:21] <jdstrand> it seemed to be triggerable by clicking on a tab in chromium
[14:22] <zyga> mborzecki: hey
[14:22] <jdstrand> but most of the time I could click on tabs fine. it is probably memory that is usually set right but occasionaly gets overwritten with garbage
[14:22] <zyga> mborzecki: I have a small request if you can spare a moment
[14:22] <jdstrand> popey: good luck! (I wish I could be of more help)
[14:23] <popey> :)
[14:23] <popey> Thanks for the time chaps :)
[14:23] <zyga> mborzecki: 4963 CC pedronis
[14:23] <zyga> this is the nicer variant of the guardian
[14:24] <zyga> I'm just wondering if I should prefer the short "sec" or the longer "secure" for variable name :)
[14:24] <mup> PR snapd#4963 opened: cmd/snap-update-ns: convert Secure* family of functions into methods <Created by zyga> <https://github.com/snapcore/snapd/pull/4963>
[14:24] <zyga> jdstrand: ^ nothing to worry about, just small code tweak to avoid a global in an upcoming fix
[14:35] <zyga> https://src.fedoraproject.org/rpms/golang-gopkg-yaml/pull-request/1#request_diff fixes snapd in fedora :)
[14:43] <mup> PR snapcraft#1995 closed: options: introduce Project and ProjectInfo <enhancement> <Created by kalikiana> <Merged by kyrofa> <https://github.com/snapcore/snapcraft/pull/1995>
[14:47] <zyga> jdstrand: how do you feel about https://github.com/snapcore/snapd/pull/4399
[14:47] <mup> PR #4399: rewrite snappy-app-dev <Created by jdstrand> <https://github.com/snapcore/snapd/pull/4399>
[14:48] <jdstrand> zyga: it's in backlog to pickup. I will not get to it for a while yet
[14:48] <zyga> ok
[14:48] <jdstrand> I have to some Ubuntu stuff (apparmor, ufw), finish some policy updates PRs (snapd) and move to snap/usn notifications
[14:49] <jdstrand> I'll try to get to it after those things
[14:50] <zyga> that's all right, I'm not pushing for it, I'm just going through PRs to see where we are in general
[14:50]  * jdstrand nods
[14:57] <zyga> eh
[14:57] <zyga> + MATCH 'Machine is not enabled'
[14:57] <zyga> + canonical-livepatch status
[14:57] <zyga> 2018/03/30 14:42:51 cannot use livepatch: your kernel "4.13.0-1011-gcp" is not eligible for livepatch updates
[14:57] <zyga> error: pattern not found, got:
[14:57] <zyga> that sucks for sure
[14:57] <zyga> now I have to disable that test to unbreak master
[14:57] <zyga> why does everything break
[14:57] <zyga> and how did this ever pass?
[14:57] <zyga> (on GCE)
[15:05] <mup> PR snapd#4964 opened: tests: adjust canonical-livepatch test on GCE <Created by zyga> <https://github.com/snapcore/snapd/pull/4964>
[15:05]  * zyga wonders how this even landed
[15:06] <zyga> pstolowski, jdstrand: ^ (because you're the only pair of reviewers today)
[15:06] <cmars> zyga: might be that gce changed their kernel since that test was written?
[15:06] <zyga> I strongly doubt that
[15:07] <cmars> just an idea, dk
[15:07] <zyga> I'll check with cachio next week
[15:07] <pstolowski> zyga: kk
[15:10] <zyga> the livepatch test failed all master PRs
[15:10] <zyga> not a great release day either
[15:13] <pstolowski> +1
[15:13] <cmars> oh, we did just release a new version of the lp client to stable.. it might be doing a better job of checking compatibility than the older version.
[15:14] <zyga> ahhh
[15:14] <zyga> thank you for explaining that! that's useful
[15:18] <jdstrand> zyga: you're keeping me busy today. +1
[15:19] <zyga> sorry :)
[15:27]  * zyga -> dinner
[15:35] <mup> PR snapd#4965 opened: ifacestate: injectTasks helper <Blocked> <Created by stolowski> <https://github.com/snapcore/snapd/pull/4965>
[15:36] <Pharaoh_Atem> zyga: ???
[15:38] <popey> is nvidia on 18.04 still busted?
[15:38] <popey> (yes)
[15:38] <popey> i am on core from beta, and can't launch ohmygiraffe
[15:46] <zyga> popey: ish, we have a fix if stars align I will release it to beta today
[15:46] <zyga> Pharaoh_Atem: all good now
[15:46] <popey> ok, thanks
[15:47] <zyga> Pharaoh_Atem: golang package in fedora was busted but fantastic people from #fedora-devel figured it out
[15:48] <zyga> and now travis slots are not ready
[15:48] <zyga> popey: can you try edge?
[15:48] <zyga> popey: it should be fine on edge now
[15:48] <popey> now?
[15:48] <zyga> yes
[15:48] <popey> ok
[15:48] <zyga> though
[15:48] <zyga> well, let's see
[15:48] <popey> doing now
[15:49] <zyga> thank you
[15:49] <zyga> btw, sublime-text?
[15:49] <zyga> it's in edge now
[15:49] <popey> didnt we discuss this like 3 hours ago?
[15:49] <zyga> yeah but I want to understand what's the next step
[15:49] <zyga> sorry, I didn't mean to be nagging
[15:49] <popey> ask store people if they can rename st3 to st
[15:49] <zyga> just impatient
[15:49] <popey> np
[15:50] <popey> edge works with my nvidia application
[15:50] <zyga> great news
[15:50] <zyga> so that's coming in beta
[15:50] <zyga> soon (tm
[15:51] <zyga> well, I can take an hourly nap now, it looks like travis is starving us after the series of failed builds
[16:25]  * zyga EOWs
[16:25] <zyga> I'll do the release tomorrow
[16:25] <zyga> happy easter everyone :)
[16:28] <koza> zyga, happy easter and which version you release?
[16:36] <zyga> koza: 2.32.2
[16:36] <zyga> popey: sublime-text from edge doesn't start on 18.04
[16:36] <zyga> I'll check why and get bakc to you
[16:36] <popey> pretty sure it worked here when i tried it
[16:37] <zyga> the rpath
[16:37] <zyga>  0x000000000000000f (RPATH)              Library rpath: [/snap/core/current/lib/x86_64-linux-gnu:/snap/core/current/usr/lib/x86_64-linux-gnu]
[16:37] <zyga> it doesn't mention sublime-text the snap?
[16:38] <zyga> ah, there's nothing inside
[16:38] <zyga> no GTK anything
[16:39] <zyga> subl after removing the deb and using snap from edge on bionic https://www.irccloud.com/pastebin/oveW053Y/
[16:41] <zyga> popey: since neither the snap nore core ships them
[16:41] <zyga> popey: and since the snap has rpath set to just core
[16:41] <zyga> popey: those are expected to fail
[16:41] <zyga> but I don't know why it works for you
[16:41] <zyga> can you run it
[16:41] <zyga> and run my magic classic bug finding shell line?
[16:42] <popey> huh, not on this machine, maybe I tested on 16.04 laptop
[16:42] <zyga> I should make that a snap itself
[16:52]  * zyga does so
[17:03] <zyga> jdstrand: can system-observe grant r on /proc/pid/environ
[17:04] <zyga> jdstrand: in addition /proc/pid/maps would be nice
[17:04] <zyga> there's nothing that provides that onw
[17:04] <zyga> *now
[17:46] <jdstrand> zyga: I'm uneasy about environ, but added it to my todo to think about
[17:46] <jdstrand> well, I added both things
[17:46] <zyga> jdstrand: thanks
[17:46] <zyga> jdstrand: I'm seeing something odd on bionci
[17:47] <zyga> I'm editing the profile by hand
[17:47] <zyga> and I see a lot of this
[17:47] <zyga> ^[[AMar 30 19:44:17 t470 kernel: [25362.426443] audit: type=1400 audit(1522431857.647:15305): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="snap.classic-snap-analyzer.classic-snap-analyzer" pid=11067 comm="apparmor_parser"
[17:47] <zyga> after clearly making some changes
[17:47] <zyga> and I get a denial
[17:47] <zyga> I added "/proc/*/environ r,"
[17:47] <zyga> recompiled and got this message and a denial
[17:47] <zyga> I will try on artful next
[17:48] <jdstrand> zyga: please file a bug against apparmor with a reproducer and jj can take a look next week (he is off this week)
[17:49] <zyga> ack, thanks
[17:59] <zyga> it is obviously broken on the -13 kernel ;-)
[18:39]  * zyga found a bug with system key
[18:39] <zyga> :-(
[18:44] <zyga> build-id mismatch in system-key https://www.irccloud.com/pastebin/YbDY7GB9/
[18:45] <zyga> ah, this is "okay"
[18:45] <zyga> hmm hmm hmm
[19:25] <Caelum> zyga: getting an error trying to add our project as a repo: [snappy|http://download.opensuse.org/repositories/system:/snappy/openSUSE_Tumbleweed/] Valid metadata not found at specified URL
[19:25] <zyga> oh
[19:25]  * zyga looks
[19:26] <zyga> those are the docs we have: https://docs.snapcraft.io/core/install-opensuse
[19:26] <zyga> sudo zypper addrepo http://download.opensuse.org/repositories/system:/snappy/openSUSE_Tumbleweed/ snappy
[19:26] <zyga> though we should really update that to enable auto refresh
[19:27] <Caelum> that's what I was looking at and it gives me that error
[19:28] <Caelum> once I do a zypper in -y snapd
[19:28] <zyga> hmm hmm
[19:28] <zyga> no idea, I'm AFK from my suse box now
[19:29] <Caelum> I'll ask on #opensuse-factory
[20:15] <mup> PR snapcraft#2043 opened: cli: support exporting login to stdout <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2043>
[20:19] <zyga> Caelum: checking now
[20:20] <Caelum> the addrepo command works fine, it's once you try to install snapd
[20:20] <zyga> aha
[20:20] <zyga> booting now
[20:24] <zyga> 487 updates from tumbleweed :)
[20:24] <mup> Issue snapcraft#1675 closed: Base logic for metadata setters set- <Created by sergiusens> <Closed by kyrofa> <https://github.com/snapcore/snapcraft/issue/1675>
[20:24] <mup> PR snapcraft#2002 closed: many: add snapcraftctl command for scriptlets <Created by kyrofa> <Merged by kyrofa> <https://github.com/snapcore/snapcraft/pull/2002>
[20:28] <zyga> Caelum: I cannot reproduce your issue
[20:29] <zyga> I used "zypper update" to refresh the database
[20:29] <zyga> I trusted the key for the system:snappy repo
[20:29] <zyga> then I zypper installed snapd
[20:29] <zyga> installing Spotify
[20:30] <mup> Issue snapcraft#1671 closed: Rename prepare/build/install to pre-build/build/post-build, and deprecate prepare/build/install <Created by sergiusens> <Closed by kyrofa> <https://github.com/snapcore/snapcraft/issue/1671>
[20:40] <zyga> Caelum: and spotify starts and works correctly
[23:20] <mup> PR snapd#4960 closed:  interfaces/serial: change pattern not to exclude /dev/ttymxc (2.32) <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4960>
[23:20] <mup> PR snapd#4964 closed: tests: adjust canonical-livepatch test on GCE <Critical> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4964>