[00:07] kyrofa: ack, I will try tomorrow [00:08] jdstrand: ack === cwayne_ is now known as cwayne === rob-oi-ma_ is now known as rob-oi-ma === zyga_ is now known as zyga === davidcalle_ is now known as davidcalle === sdrobertw_ is now known as sdrobertw === stokachu_ is now known as stokachu === andyrock_ is now known as andyrock [00:13] pedronis: ack === caveat-_ is now known as caveat [05:06] Bug #1684014 changed: LibreOffice in snap won't print [07:05] Good morning [07:15] morning zyga! [07:15] Hey hey [07:15] It is Friday and everything looks terrible :-) [07:18] zyga: no kidding [07:20] pstolowski: fedora builds are broken [07:20] pstolowski: travis slots last night were all gone [07:20] pstolowski: and I didn't manage to release snapd [07:21] gosh [07:22] zyga: what's going on with fedora this time? [07:25] pstolowski: afaict go build itself (wich is mostly a wrapper process) dies with exit 2 just before doing the steps of building snapd main itself (after having built all other packages) [07:26] without printing any useful info [07:26] fun :/ [07:26] * pedronis -> afk [07:30] pstolowski: zyga: I tried to strace it, it was late so maybe I missed something, but afaict no smoking gun there, just after it writes sig.o to signal.a (the internal pack, internal meaning I discovered is code inside go build not an external exec) it just starts unlinking stuff from the work dir and then you get the exit 2 [07:30] so it's a sorf of unexplained semi orderly death [07:30] * pedronis really afk [07:31] were there any updates to fedora? [07:32] I don't know, golang claims to be from Mar 5 (so we should have been using it already? but I don't know how updates percolates there) [07:32] hmm [07:32] rpmbuild seems the same [07:33] but maybe some lib changed [07:33] hey pedronis, thank you for the analysis [07:34] zyga: for details, what I did sofar is set the few GO env vars and rerun the "go build" line in a -debug spread session, with -n (to see how far we go, as I said very far) and then with strace -f [07:36] it's also very consistent [07:36] dies the same way each time [07:37] pedronis: do you remember what you set? [07:37] I also double checked the content of the work dir , as I said signal.a looks right but ... /cmd/snapd/_obj is not yet created [07:37] zyga: it's in the log [07:38] just GOPATH and GOFLAGS [07:38] I think [07:38] ok [07:38] after of course cd-ing into /root/rpmbuild/.... [07:39] * zyga goes to reproduce this with fresh head [07:43] so, the list of updates is on https://bodhi.fedoraproject.org/updates/?releases=F27 [07:43] I'll see if there's something recently that could explain it [07:49] zyga: looking into this too [07:50] if we don't find anything quickly let's split focus and not all try to solve this [07:51] I went back 10 days and there's nothing that would indicate any relationship [07:51] the closest I saw was a clang update to ship a versioned compiler-version binary [08:05] I noticed that on 18.04 high-dpi regressed as compared to 17.10 (for snaps) [08:23] zyga: fwiw I got that error when building signal.a; then I executed /var/tmp/rpm-tmp.5TbNku manually and it succeeded [08:23] zyga: (fedora on linode) [08:23] interesting, I wonder what that tells us [08:24] exactly [08:34] zyga: i've rebuilt manually like that 3 times, worked [08:35] pstolowski: I'm doing a mock build to see how that works [08:35] zyga: do you know what creates and calls /var/tmp/rpm-tmp.5TbNku ? [08:36] I think it's part of some RPM macro but I don't know which [08:36] it could also be a recovery mechanism [08:36] when rpm fails, it writes "reproduce me" shell [08:37] zyga: yeah, ok, it's assembled from snapd.spec [08:38] Breakfast break [08:39] After that I will mock build [08:39] And try koji next [08:41] pstolowski: that is interesting because if I run the go build part alone again and again is still fails [08:46] pedronis: wow, interesting [08:47] pedronis, zyga and i'v just run the build via rpmbuild --with testkeys -bs and it failed [08:47] yes, I got that too [08:48] so repeating rpmbuild fails (different state each time), repeating go build fails (afaict) [08:48] but repeating the tmp script works? [08:48] fascinating [08:48] pedronis: i'm just going to try tmp script again [08:49] is it succedding in the sense of exit 0 [08:49] or in the sense it build a snapd binary? [08:51] I tried to remove the spread + linode|google magic out of the equation [08:51] I built an source tarball and srpm [08:51] now building it in mock [08:51] this is essentially how all of fedora is normally built [08:51] iff this fails [08:52] we can give this srpm to people and ask for help [08:52] pedronis: yes, exit 0, no error around signal.a, the execution ends with https://pastebin.ubuntu.com/p/gtHJGPbPmV/ [08:52] + mock is nicer than rpmbuild directly [08:52] pedronis: so, tmp script just succeeded again [09:02] so [09:02] ah, I didn't use test keys so maybe that's why [09:02] the source rpm + mock fail on pack but in a different spot [09:02] BUILDSTDERR: /usr/lib/golang/pkg/tool/linux_amd64/compile -o $WORK/golang.org/x/net/context/ctxhttp.a -trimpath $WORK -shared -goversion go1.9.4 -p golang.org/x/net/context/ctxhttp -complete -installsuffix shared -buildid ab9e3e1669ce2ef40183ac44ed64b157e02355b9 -D _/usr/share/gocode/src/golang.org/x/net/context/ctxhttp -I $WORK -I /usr/share/gocode/pkg/linux_amd64_shared -pack ./ctxhttp.go [09:03] pstolowski: how are you running it again, I get failed to create symbolic link because the link alreday exists [09:05] Pharaoh_Atem: hey [09:05] Pharaoh_Atem: so I have a srpm I can reproducibly fail to build in mock [09:06] pstolowski: are you rm -rf something before running it again? [09:06] pedronis: I get that symink error too but that doesn't interrupt the build [09:07] heh [09:07] it definitely does here [09:07] pstolowski: ah, are you running without -e ? [09:07] that's why it finishes [09:07] it ignores errors [09:07] nothing magic [09:07] ah [09:08] dammit [09:08] indeed [09:08] too bad [09:09] zyga: I have no clue what dimension changed (our code, fedora bit) though ... it stopped working quite randomly yesterday afaict [09:09] yes, I'm totally puzzled [09:09] I will try with -testkeys now [09:09] as my failure looks the same but in different spot [09:10] it seems a go build bug [09:10] but unclear why it appears now [09:10] what it triggers it [09:10] s/what it/what7 [09:10] * pedronis -> off [09:12] zyga: one thing to try would also be to copy over the BUILD directory to bionic and see if that go build command fails there too with 1.9, fwiw it failed for me also when I didn't pass some of the ldflags stuff [09:13] anyway as said, from the strace it seems to fail in the boring bits of driving the build [09:29] on my host it consistently fails elsewhere [09:30] maybe it depends number of CPUs? [09:31] oh, I'll try koi first [09:31] koji [09:48] fwiw it fails also with go1.9.5 and 1.10.1 on fedora [10:01] zyga: my btrfs died, I submitted that request from the console on a half-broken system, just reinstalled now and picking up the pieces, on ext4 this time [10:02] even just this fails: cd /root/rpmbuild/BUILD/snapd-1337.2.31.1 ; cd cmd/snapd; go build -v -x . with or without -a [10:02] it just exit 2 before building main [10:02] Caelum: ouch [10:02] sorry to hear that [10:02] Caelum: I made some changes since your patches [10:02] excellent [10:03] pedronis: I submitted a koji build https://koji.fedoraproject.org/koji/taskinfo?taskID=26057356 [10:03] so .32 is out [10:03] pstolowski: would be interesting to copy over the BUILD to an ubuntu system and see if it fails too, as I show there it seems to fail even without fancy options or anything [10:04] pedronis: yes, i'm going to try that next [10:04] it just refuses to build cmd/snapd(/maing.go) on fedora [10:04] the log is a little bit more useful [10:04] https://koji.fedoraproject.org/koji/getfile?taskID=26057357&volume=DEFAULT&name=build.log&offset=-4000 [10:04] (tail of the build log) [10:04] tail feature == good thing to copy [10:05] does this cp $WORK/b150/_pkg_.a /builddir/.cache/go-build/92/92ebdb22b8c38b43994a36a864adc877e4e85517bc1f1a7c1b22ee610a1af86d-d # internal [10:05] error: Bad exit status from /var/tmp/rpm-tmp.MIqip3 (%build) [10:05] say it is actually cp that failed? [10:05] that's a completely different kind of failure [10:05] though [10:05] we don't know [10:05] usually the last thing printed might have work or not [10:06] as I said in the other failure [10:06] the pack works [10:06] is the next step that is not taken [10:06] this is in koji, the fedora build service [10:06] I understand [10:06] just saying that looks like a failure [10:06] if this is kernel related they are (perhaps) on a different version of that [10:06] but a different failure [10:07] it's also an exit 1 vs exit 2 [10:07] note that the same SRPM was failing on pack on my host [10:07] I note that on linode the failure mode is very deterministic [10:08] as I said it seems to refuse to go to the last bit [10:08] and build cmd/snapd itself [10:08] (the pack itself works) [10:08] indeed it fails like that, exit 2 refusing to build main even in builds that don't have a pack bit [10:09] late [10:10] zyga: as I said the failure in that log, looks very different, it seems to be failing building one the prereq packages [10:10] on snapd itself [10:10] s/on snapd/not snapd/ [10:11] what golang is that btw? [10:11] 1.10? [10:11] I switched from F28 to F27 and now it's the same error I saw on my host [10:11] let me check [10:11] https://koji.fedoraproject.org/koji/getfile?taskID=26057448&volume=DEFAULT&name=build.log&offset=-4000 (the F27 build tail) [10:12] DEBUG util.py:439: golang x86_64 1.9.4-2.fc27 build 622 k [10:12] 1.9.4 [10:12] on F27 ? [10:12] on F28 ? [10:13] this is on F27 now [10:13] the prior build was on F28 and this is why it was different, it was also on using more recent golang [10:14] it just feels like they brok all go building [10:14] it's very strange [10:14] also the errors are all different [10:14] I'll ask around [10:18] I asked on fedora-devel [10:19] not sure what to do now [10:19] I feel we should disable fedora for the time being [10:19] this would unblock the release [10:22] zyga: interesting, not sure if related but I see, /usr/share/gocode/src/%{go_import_path} like that in the filesystem [10:22] like some package is really broken [10:22] oh [10:22] that looks like RPM macro that didn't expand [10:22] that's very interesting, let me look [10:23] pedronis: I'll disable fedora now [10:25] let's see if that passes [10:26] PR snapd#4962 opened: spread.yaml: switch Fedora 27 tests to manual [10:45] I’m debugging this in fedora-devel [10:52] zyga: pstolowski: so if I copy over to ubuntu the gocode dir and the rpmbuild dir and set GOPATH to those, even old go 1.6 fails to build cmd/snapd with exit 2 [10:53] cool find [10:53] but does that tree include effectively golang 1.9 source? [10:53] no [10:54] gocode has only our deps [10:54] goroot is the 1.6 one [10:54] pedronis: and that gocode dir contains that suspicious unexpanded macro? [10:54] yes [10:54] but yes, it would be better to try this with ubuntu go 1.9 [10:54] less moving parts [10:55] but is interesting nevertheless [10:55] it really seems is some issue with src code (ours or the one brought in by dep packages) [10:56] and to be clear, this is just a plain: go build -a -v -x . [10:56] nothing fancy [10:56] from cmd/snapd [10:56] let's try building older tree [10:56] anyway the %{} thing for sure cannot bring any good [10:57] # cd /home/zyga/go/.cache/govendor/gopkg.in/yaml.v2; git reset --hard 86f5ed62f8a0ee96bd888d2efdfd6d4fb100a4eb [10:57] fatal: Could not parse object '86f5ed62f8a0ee96bd888d2efdfd6d4fb100a4eb'. [10:57] mmmm [10:57] ./get-deps.sh failed [10:57] worked after I removed ~/go/.cache [10:58] pedronis: "go build ./cmd/snapd" works [10:58] ? [10:58] where, in which sense? [10:58] you mean with our usual vendor dir? [10:58] yes [10:58] in a checkout on F27 [10:58] not in the whole build machinery [10:59] so it seems some packages are broken [10:59] and render this %{} nonsense [10:59] I ran "go build -a -v -x" next and it printed a wall of text and exited with 130 [10:59] good morning [10:59] and that breaks things [10:59] (strangely in a very non visible way) [10:59] hey thresh [10:59] pedronis: go build -a -v -x in cmd/snapd also works [10:59] from our checkout [10:59] ls [11:02] pedronis: https://github.com/snapcore/snapd/pull/4962 [11:02] PR #4962: spread.yaml: switch Fedora 27 tests to manual [11:02] let's land this [11:03] +1 [11:11] zyga: pstolowski: yes definitely switching the share/gocode stuff with out usual deps makes thing work again [11:13] even using them not through vendor to be clear, but just in the GOPATH [11:14] zyga: pstolowski: so yesterday one of the deps packages was updated? or something changed in the rpm go stuff that affects one happens when they are installed? [11:14] I mean in fedora [11:15] pedronis: I looked at the list of changes published in bodhi and I dind't notice anything golang at all [11:16] pedronis: let's merge my PR and unblock master [11:18] and i've just removed that bogus /usr/share/gocode/%{go_import* path from fs but that didn't make a difference, must be something else in gocode dif [11:18] *dir [11:20] lunch break [11:23] pstolowski: yea, I tried the same, I don't get a missing package error, but even without the %{} nonsese, it still exit 2 [11:23] even go 1.6 [11:28] pedronis: I think it may be yaml related [11:28] https://zero.crans.org/?3e5975e22418a02e#Xv5Juf3UYkZ0FgzYq9Uu2h9AFG4FKFMWqbITXDLVZno= [11:28] I was thinking the same [11:28] a bit unclear exactly what [11:28] this is something fedora developer was helping me with [11:29] I'll revert the yaml version bump and build [11:32] but the run that bumped it was green ?? [11:33] reverted, no change [11:33] so back to where we started :/ [11:34] zyga: I fixed trailing " and there is no more backend_test.go failure in https://github.com/snapcore/snapd/pull/4944 [11:34] pedronis: shall we merge 4962 or do you want to resolve this before master can move [11:34] PR #4944: interfaces: add /var/lib/snapd/snap to @{INSTALL_DIR} [11:34] zyga: I should be off, I'm not taking decisions [11:34] ok [11:35] zyga: anyway it's not a bug in our code afaict [11:35] yes, that is clear, I thin [11:35] *think [11:53] PR snapd#4962 closed: spread.yaml: switch Fedora 27 tests to manual [12:11] woah, my forum post is flagged as spam [12:11] verynice [12:11] > Multiple community members flagged this post before it was hidden [12:11] what [12:12] seriously? [12:12] * zyga -> walk [12:15] how is https://forum.snapcraft.io/t/snapcraft-stage-packages-inconsistent-behaviour/4746 a spam ?! [12:15] and/or advertisement [12:15] is there a saner platform to collaborate with devs? [12:23] niemeyer: hi! quick 'snap switch' question. I have canonical-livepatch on a system. it was tracking stable, r26. yesterady I did 'sudo snap refresh --edge canonical-livepatch'. this refreshed to r38 and marked the snap as starting to track edge (based on snap info). this is all fine [12:24] niemeyer: then I did 'sudo snap switch --channel=stable canonical-livepatch'. this did not refresh the snap (good) and marked the snap as tracking stable (as seen with snap info. also good) [12:25] niemeyer: so at this point, I thought that everything was all set for the snap to only refresh the next time stable refreshed, however, at some point later in the day, the snap refreshed automatically in the background and downgraded to stable. is this expected behavior? [12:25] s/stable refreshed/stable's revision changed/ [12:27] thresh_: someone doesn't like you :) [12:27] thresh: ^ [12:28] niemeyer: here is a little more detail: [12:28] $ sudo snap changes canonical-livepatch [12:28] ID Status Spawn Ready Summary [12:28] 551 Done 2018-03-29T13:17:24Z 2018-03-29T13:17:31Z Refresh "canonical-livepatch" snap from "edge" channel [12:28] 552 Done 2018-03-29T13:18:04Z 2018-03-29T13:18:04Z Switch "canonical-livepatch" snap to stable [12:28] 553 Done 2018-03-29T20:49:59Z 2018-03-29T20:50:02Z Auto-refresh snap "canonical-livepatch" [12:28] 554 Done 2018-03-30T12:20:37Z 2018-03-30T12:20:40Z Refresh "canonical-livepatch" snap from "candidate" channel [12:28] niemeyer: it is '553' that did the downgrade back to stable [12:29] niemeyer: if it is operating as designed, that's fine (though, I wonder about the utility of 'switch' then), I just want to make sure it is working correctly [12:31] jdstrand: no, that's the correct behavior, it will refresh to whatever is on stable [12:31] s/correct/expected/ [12:31] remember there's no order in revisions [12:34] jdstrand: switch is more useful if say candidate and revision have the same revision, but I also think we changed refresh to work more like switch in that case [12:34] so it might not be as useful as it was [12:35] vidal72[m], but I'm amazing and nice [12:36] :) [12:39] Which post is that [12:45] I linked it half and hour ago here [12:45] afraid to post it twice now lol [12:46] niemeyer: nm, pedronis answered [12:47] pedronis: sure, I realize there is no order with revisions, but if I was on r26/stable then refreshed to r38/edge, then snap switched to stable, I didn't expect to go to r26/stable. I expected to go to !26/stable [12:48] pedronis: I'm not sure of the utility of snap switch then tbh. it ends up being a delayed refresh [12:49] pedronis: where I want a delayed refresh to a new thing in the switched to channel [12:51] * zyga is back from his walk [12:53] popey: what blocks sublime-text from release? [12:54] thresh: I didn't see the link, can you please re-post? [13:00] PR snapd#4955 closed: cmd/snap-confine: make /run/media an alias of /media [13:00] zyga: actually realised that we should store rename s-t3 to s-t, otherwise the hundreds of people with st3 won't get updates [13:01] can we even do this? [13:01] I don't think we can [13:01] I believe the store people can rename a snap yes [13:01] it keeps the id [13:01] ah, in that case, all for it [13:01] can I help somehow [13:01] I'd love to just use s-t from snaps already [13:01] are store people working today? [13:02] yeah, sorry, been on vacation the last few days so it's been back burner [13:03] zyga: standup, or skip? [13:08] oh [13:09] sorry [13:09] thresh: dunno why the forum is marking your posts as spam. I have unflagged them as such. Maybe niemeyer knows [13:09] pstolowski: let's skip maybe [13:09] pstolowski: google signed me out [13:09] pstolowski: is it just you and me? [13:10] ok, found my token [13:14] thank you popey [13:18] wtf. i just installed a snap and as soon as the apparmor connections were made the session exploded [13:18] i didnt even launch the snap, it ended immediately after install finished [13:18] popey: eom? [13:18] eom? [13:18] popey: probably udev [13:19] or oom [13:19] (out of memory) [13:19] unlikely [13:19] it has 64GB [13:19] but I saw people reporting bugs about touchpad issues when installing a snap [13:19] woah, nice machine! [13:19] just logged back in and I get a "xorg crashed" bug reporter [13:19] and it feels like udevadm trigger is not good [13:19] popey: can you ssh in [13:19] and run journalctl -f [13:19] and reproduce? [13:19] i logged back in, on a desktop [13:20] I'm on core beta [13:20] popey: is that a wayland session [13:20] when wayland crashes it takes the session with it [13:20] hah, not on your life [13:21] well, review your log [13:21] there's gotta be a backtrace somewhere :) [13:21] juat waiting for whoopsie to do its thing [13:28] https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers/+bug/1760104 whoospie says it's an nvidia driver bug, so who knows [13:29] popey: *bam* [13:30] maybe it's our bug [13:30] I got 404, it's marked as security [13:30] I wlll try and reproduce it [13:30] popey: look at your journal first [13:30] given I am building a new rev of the snap [13:30] there is likely a backtrace there already [13:31] will that be already attached to the bug? [13:31] I cannot see the bug [13:31] * zyga is super happy today [13:31] https://www.irccloud.com/pastebin/Ic5TwB7Y/ [13:31] that's all i see in journal [13:31]  no no, that's -f [13:32] that's "follow" [13:32] run journalctl [13:32] and page it [13:32] until you see something that's clearly a backtrace [13:32] it should be from this boot only [13:32] so not _too_ long perhaps [13:32] * zyga is super happy because his T470 has a working modem now :) [13:32] it's internal, antennas work and I have 100GB of data plan to use [13:33] wow [13:33] so I can now do what I love doing [13:33] work outside :) [13:33] work anywhere [13:33] the modem I got before was not on the whitelist (curse you lenovo) [13:33] but this one is flawless :) [13:34] I'm slowly getting used to the T series, X was way smaller [13:34] but this is not huge either, it's the okay size for work [13:34] and it's not heavy which is what I was worried about [13:39] https://paste.ubuntu.com/p/3FbykZXbx2/ [13:39] not a lot interesting in there? [13:41] many thanks popey [13:41] * thresh wanders off to friday evening endeavours [13:42] \o/ [13:42] Mar 30 14:15:58 hal compiz[23168]: XIO: fatal IO error 11 (Resource temporarily unavailable) on X server ":1" [13:42] but this is after X died [13:42] Mar 30 14:15:34 hal audit[1042]: AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.dosbox-x.dosbox-x" pid=1042 comm="apparmor_parser" [13:42] Mar 30 14:15:35 hal systemd[22791]: Starting Notification regarding a crash report... [13:42] Mar 30 14:15:36 hal update-notifier-crash[1108]: Xorg [13:42] we have the same mouse :) [13:42] hah [13:43] popey: haha 'eom'. I meant oom, yes :) [13:43] "End of memory" :D [13:43] hmmm [13:43] End Of Memory. hehe [13:43] Mar 30 14:15:33 hal systemd-udevd[689]: Process '/usr/lib/snapd/snap-device-helper change snap_vidcutter_vidcutter /devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card0 226:0' failed with exit code 2. [13:43] Mar 30 14:15:33 hal systemd-udevd[689]: Process '/usr/lib/snapd/snap-device-helper change snap_vlc_vlc /devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card0 226:0' failed with exit code 2. [13:43] Mar 30 14:15:33 hal systemd-udevd[689]: Process '/usr/lib/snapd/snap-device-helper change snap_writefull_writefull /devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card0 226:0' failed with exit code 2. [13:43] Mar 30 14:15:33 hal systemd-udevd[689]: Process '/usr/lib/snapd/snap-device-helper change snap_zzt_zzt /devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card0 226:0' failed with exit code 2. [13:43] jdstrand: ^ does this ring a bell [13:43] remember the nvidia udev bug? [13:43] is this back in some way? [13:44]