[04:00] PR snapcraft#1943 closed: Release/2.39.2+18.04 (DO NOT MERGE) [04:18] PR snapcraft#2078 opened: Release/2.41+18.04 [04:34] zyga: thanks a lot for your patience. so the most common way for a download tool like https://github.com/pbek/lmdownload is to tell the user that they only can download files to their home folder if they are installing the snap package? [05:10] morning === King_InuYasha is now known as Son_Goku [05:44] good morning [05:44] zyga: hello [05:45] pbek: I'm not sure I understand, is the download tool itsel a snap? [05:45] hey :) [05:45] * zyga is a bit hurt today :) [05:45] good morning, zyga [05:46] yes, the tool downloads pdfs to the current folder and is snapped [05:46] https://github.com/pbek/lmdownload/blob/develop/snap/snapcraft.yaml [05:47] but of course downloading just works in the /home folder when the tool is started as snap [05:47] pbek: you can check if you are in /var/lib/snapd/void, if so, tell the user to do something else [05:48] pbek: that directory is used if the original working directory cannot be represented in a snap [05:49] mvo: some good news, the leak is gone [05:49] mvo: but the case is not fully cloesd, we need to talk to the kernel team [05:51] zyga: strange, the tool never told me that it was in that directory (I print out the directory where the tool wants to strore the pdf file). the directory was always the one I really was in bash, but I got a write permissions error when I was storing the file. the tool is written in go [05:51] pbek: it only does so if it is in a directory that cannot be represented [05:51] pbek: it may be in a directory that is represented but cannot be written to [05:54] zyga: ok, I can output a more propper error message then, but I cannot get around the fact that the tool can only write to the home-folder while in snap confinement, right? [05:54] pbek: that's correct [05:55] pbek: snaps should wite to $SNAP_USER_DATA or $SNAP_USER_COMMON [05:55] thank you again for your patience, zyga! [05:55] pbek: eventually you may use xdg portals to write to more places [05:55] yes, I found that out already (and am using using it for storing settings) [05:55] zyga: hey, good morning. yeah, I saw the mail [06:37] PR snapd#5050 opened: many: add "stop-mode: sig{term,hup,usr[12]}{,-all}" instead of conflating that with refresh-mode #5043 [06:40] * zyga tries to get through his inbox today [06:54] PR snapd#5051 opened: snap,wrappers: address review feedback from #5043 [07:01] PR snapd#5044 closed: 'system' nickname for 'core' in snap get/set (2.32) [07:02] PR snapd#5045 closed: overlord/snapstate: poll for up to 10s if a snap is unexpectedly not mounted in doMountSnap (2.32) [07:16] morning [07:20] hey pawel [07:20] * zyga spent all of saturday and sunday outdoors, walking and biking a lot [07:20] and now endures the pain in all kinds of places :) [07:25] if anyone feels like doing some reading https://forum.snapcraft.io/t/snap-refresh-over-metered-connections/5001 [07:27] mborzecki: yeah [07:28] btw. i think there's a bug in nm, some signal not emitted at the right time/place [07:28] at least when switching connection to metered from the cli, you don't see the Metered property changed on the main NM object [07:28] had to restart NM and then it got updated [07:32] hi, is there a fix for the pip error when building a python package with snacraft: subprocess.CalledProcessError: Command '['/bin/sh', '/tmp/tmp7036_63f', '/home/ack/canonical/src/gitlptools/parts/gitlptools/install/usr/bin/python3', '-m', 'pip']' returned non-zero exit status 1 [07:39] mborzecki: replied [07:39] zyga: good points [07:40] good morning [07:41] Pharaoh_Atem: snap-confine package contains all kinds of things in fedora [07:42] Pharaoh_Atem: we should really disband that package and move those files over to snapd [07:43] disband ;) reminds me of civ [07:50] mborzecki: ha, yeah [07:50] disband militia in 2050 :) === mborzeck1 is now known as mborzecki [08:14] moin moin [08:15] wow, my network is flaky today [08:16] mvo: I will update the GitHub release page [08:16] although I plan to skip the .1 and .2 tarballs [08:16] just go to .4 [08:16] I will use it to make the openSUSE release now === mborzeck1 is now known as mborzecki [08:19] mvo: hi, are you going to create .5 today? [08:29] pedronis: yes, I will do that hopefully ready around lunch time. do you have anything pending? [08:30] no [08:31] I saw you merged the doMountSanp/readInfo stuff [08:32] pedronis: can you update https://forum.snapcraft.io/t/issues-to-address-ahead-of-ubuntu-18-04-release/4954 to include the latest status of that issue please [08:33] pedronis: yeah, I looked over it and it looked like a good idea to have it [08:36] I'll update it in a bit [08:36] #3 is wontfix, #4 has had some error-catching fixes, #5 is almost fixed right? [08:50] any prs needing reviews? [08:52] Can you look at my trespassing pr [08:53] zyga: #4889 ? already did [08:53] PR #4889: cmd/snap-update-ns: don't trespass on host filesystem [08:53] Ah, then no idea :-) [08:53] zyga: btw. it needs deconflicting [08:53] Oh thanks. I’ll do that shortly [09:05] #4983 needs a second review [09:05] PR #4983: osutil/sys, client: add sys.RunAsUidGid, use it for auth.json [09:12] can't linux mint run classic snaps? [09:14] is there a way to tell what type of confinement is supported by the system, other than running `snap debug confinement` ? [09:17] zyga: updated [09:17] thanks! [09:18] mborzecki: what do you mean? [09:20] mborzecki: http snapd:///v2/system-info | jq .result.confinement [09:20] mborzecki: ? [09:20] Chipaca: when you run `snap debug confinement` you actually see the confinement mode supported on the host (strict|partial), but I don't see a way to get this information using cli and not 'debug' command (which is hidden btw) [09:22] mborzecki: AFAIK there isn't [09:22] mborzecki: i mean, not via 'snap' [09:22] mborzecki: maybe we want it in 'snap version' or sth? [09:23] mborzecki: (what for?) [09:23] Chipaca: when the user runs hello-world.evil they will on unconfined system they will see the message 'If you see this line the confinement is not working correctly, please file a bug', but they may not have confinement in the first place [09:24] mborzecki: 'please file a bug with your distribution' :-p [09:24] i was thinking that we could update the message with eg. 'Try running snap version to check if your system supports confinement blah blah' [09:25] mborzecki: why not make .evil run 'snap debug confinement' or equivalent itself? [09:26] Chipaca: hm doable, but snap debug is hidden (or pretends to be at least), snap version would be nicer, https://paste.ubuntu.com/p/JRn57pP99Q/ [09:26] mborzecki: is this on arch btw? [09:27] Chipaca: yes [09:33] mborzecki: is there any info we could put next to 'arch' in the output of 'snap version'? [09:34] i know it's rolling, but maybe, the time of the last update? [09:34] dunno [09:34] mborzecki: we could add confinement to distro info instead of as a new line if the new line is frowned on [09:34] Chipaca: - ? [09:36] Chipaca: https://paste.ubuntu.com/p/CGJZ9XPnSv/ [09:37] mborzecki: aw :-) ok [09:38] IMO putting this on the distribution version is silly [09:38] let's just do it properly [09:38] confinement: seccomp(...), cgroup(device, freezer), namespace(mount), apparmor(...) [09:41] zyga: note that that's not exposed via system info yet [09:41] +1 to it or something like it if it were [09:42] PR snapd#5050 closed: many: add "stop-mode: sig{term,hup,usr[12]}{,-all}" instead of conflating that with refresh-mode #5043 [09:42] zyga: seccomp(...) as in actions? [09:42] yeah, some version details there [09:49] hmm maybe we should have a separate `snap confinement` command or some --verbose switch to snap version, this is getting a bit verbose [09:50] then we could print apparmor features, seccomp actions, available cgroups (those that are mounted), and namespaces (those listed under /proc/$$/ns of snapd daemon) [09:57] mvo: updated https://github.com/snapcore/snapd/releases/tag/2.32.4 [09:57] mborzecki: I think we have too many commands but I agree we should have some way to s how this [09:58] mborzecki: I doubt most users will undertstand more than I said initially on a single-line confinement value [09:58] ok, so maybe `confinement: (strict|partial)` in snap version is all that we need [09:59] zyga: thanks! .5 is rolling forward [10:01] mvo: so .5 doesn't include 5051 ? [10:02] PR snapd#5052 opened: cmd/snap: handle distros with no version ID [10:02] Chipaca: ^^ [10:02] pedronis: no, I think its fine, its cosmetic but if you feel strongly about it I can include it too [10:04] mvo: that's odd, we had that patched before [10:04] mborzecki: +1 [10:04] ah [10:04] I see what the patch does now [10:04] mvo: ping me when .5 tarball is up [10:05] diddledan: you havint trouble with irccloud today? [10:05] fails to launch here on core stable, and I just get 3 dots in the screen constantly [10:06] mvo: mostly fearing a bit of confusion about the log, and if we need to do a .6 [10:06] * zyga does /o\ [10:06] why would we need a 6? [10:06] exactly, we don't know [10:06] that's how that stuff happens [10:09] zyga: what is odd? [10:09] mvo: nothing, I misunderstood the output [10:09] mvo: - vs "" [10:09] pedronis: ok, I will pull it into release/2.32 just in case we need a .6 [10:09] pedronis: it does make sense [10:09] mvo: https://github.com/snapcore/snapd/pull/5053 [10:09] PR #5053: release-tools: handle the snapd-x.y.z version [10:10] zyga: ta, I have a look. .5 is available in the ppa, one sec, I look for a direct link for you [10:10] mvo: FYI I have tested jj's fix and it's stable over extended periods of time [10:10] mvo: oh, neat [10:10] just push the tag, I'll do the rest :) [10:10] zyga: tag is pushed as well [10:10] PR snapd#5051 closed: snap,wrappers: address review feedback from #5043 [10:10] PR snapd#5053 opened: release-tools: handle the snapd-x.y.z version [10:10] zyga: https://launchpad.net/~snappy-dev/+archive/ubuntu/image/+sourcepub/8990918/+listing-archive-extra has the build [10:10] Son_Goku: good morning [10:12] diddledan: must have been corrupted config, removed and reinstalled the snap and it's fine \o/ [10:12] PR snapd#5054 opened: snap,wrappers: address review feedback from #5043 (2.32) [10:15] PR snapd#5055 opened: release: 2.32.5 [10:20] mvo: https://github.com/snapcore/snapd/releases/tag/2.32.5 [10:21] can you please check if I got the refresh/stop mode bits right [10:23] mvo: .5 failed to build for ppc [10:23] zyga: looking, I suspect a stray failure [10:24] pedronis: https://bugs.launchpad.net/snappy/+bug/1764069 [10:25] Bug #1764069: Can't install snaps (X-Ubuntu-Series header is required - connection refused) [10:32] * zyga -> walk [10:32] popey: notepad++ looks bad on high-dpi system [10:32] not sure if there's anything we can do to tell wine" hide" [10:32] "hidpi" [10:32] Indeed [10:33] We don't pass through pixel doubling in some apps [10:33] not just WINE [10:37] popey: do you know what we need to pass through? [10:37] i dont think there is anything we can do for WINE, but for some other apps, Wimpress mentioned looked bad on his hidpi machine, i think he knows [10:38] * Chipaca tries to make a joke about sulphites, and fails [10:46] PR snapd#5052 closed: cmd/snap: handle distros with no version ID [10:46] 5052 for 2.32 in case we do a .6 release? ;) [10:48] mborzecki: too much regression potential [10:49] :-p [10:49] zyga: afaict that's a DNS problem [10:53] Ok [10:59] mvo: this changelog line looks strange: daemon: support 'system' as nickname of the core snapchange it is [10:59] + possible to do: [11:00] change it is ... seems spurious [11:03] pedronis: my fault, i tend to include how the command output looks like in the commit messages [11:04] I'm not sure, the whitespace is wrong either way [11:08] pedronis: yeah, the whitespace there is incorrect. good catch [11:08] pedronis: look like the script that generates them has a bug there, I look into it [11:16] PR snapd#5056 opened: cmd/snap: make confinement mode part of `snap version` output [11:17] * Chipaca trundles off in search of something to lunch [11:23] re [11:23] * zyga works on .5 release for opensuse === pstolowski is now known as pstolowski|lunch [11:25] zyga: .6 is where it's at [11:25] * Chipaca runs [11:25] Chipaca: what?!? [11:25] are we doing a .6 now [11:26] is this what "eventually released" software is like? [11:32] .5.4.3.2.1 [11:33] approximating towards 2.33 [11:34] Chipaca: should we use LazyUnmount= in our .mount units? [11:34] effectively :) [11:34] (that was about approx 2.33) [11:35] mvo, should I start testing .5 or I should wait for a .6? [11:35] cachio: I don't think we plan a .6 atm [11:36] pedronis, ah, ok [11:36] thanks, I'll run beta validation for .5 in that case [11:37] mborzecki, hey, amazon linux image is ready [11:37] but you just can use it [11:37] if you use this spread [11:37] 2.32.4 is building for all opensuse systems now [11:37] mborzecki, https://github.com/snapcore/spread/pull/57 [11:37] PR spread#57: Adding preserve option for systems to keep the default image size [11:37] mborzecki, beasuse the filesystem cannot be resized [11:38] cachio: what's the system name in spread.yaml? [11:38] mborzecki, basically it is not possible to shrink xfs filesystems [11:39] - amazon-linux-2-64: [11:39] workers: 1 [11:39] preserve: true [11:39] mborzecki, you can add that [11:39] with that branch [11:41] ok [11:41] mborzecki, I'll going the the kinesiology now, I'll explain better during the standup [11:41] cachio: i'll try to run it, thanks [11:42] mvo, there is not changelog for 2.32.4 -> 2.32.5 [11:45] cachio: I made "some" changelog [11:45] cachio: https://github.com/snapcore/snapd/releases/2.32.5 [11:51] Caelum: I updated snapd twice to 2.32.4 and now to 2.32.5 [11:51] Caelum: let me know if you run into any issues [11:51] Caelum: although this release should be very stable now [11:58] mvo: maybe I could look at updating the debian package [11:59] Pharaoh_Atem: I will have a small selinux patch for you to review soon [12:01] does anyone know what networkd-dispatcher is for? [12:02] mvo: how is our upload to bionic like? [12:02] mvo: can we upload .5 now? [12:02] and similar question about xenial SRU [12:03] off to get the kids [12:08] mborzecki: when you return - did you have any luck with nvidia on 18.04? I got from zyga that it's somewhat of a lost cause? [12:09] popey: with classic snaps [12:09] popey: otherwise it should work [12:09] popey: with classic snaps it can work but the snap needs to understand the particular layout of the host [12:09] This needs documenting somewhere. [12:13] popey: yes, no disagreement there [12:15] mvo: when we have more bases and more core18 things [12:16] mvo: I was thinking that we should drop the "series 16" line from snap version [12:16] we could show it on core devices [12:16] and actually show the name of the bootable base [12:16] I think right now it's not a very useful/informative thing [12:18] Bug #1764069 changed: Can't install snaps (X-Ubuntu-Series header is required - connection refused) [12:18] mvo: hey, is issue #5 solved with 2.32.5 now? [12:20] PR snapd#4833 closed: tests: add a detector for kernel bug https://pad.lv/1755563 [12:23] mvo: given that #5043 is for 2.32.x, do you still plan to make additional release (that is, .6)? [12:23] PR #5043: many: add "stop-mode: sig{term,hup,usr[12]}{,-all}" instead of conflating that with refresh-mode [12:28] zyga: reading backlog now [12:30] cachio: http://people.canonical.com/~mvo/core-changes/html/beta/2.32.4r4443_2.32.5r4486.html <- that looks ok (modulo missing space) [12:31] Son_Goku: hey! I've proposed go-udev https://bugzilla.redhat.com/show_bug.cgi?id=1567819 but just got a comment there that go packaging has been revamped? haven't yet looked at the guidelines [12:32] zyga: 5043 has lnaded in .5, no? [12:32] zyga: or what am I missing? [12:32] * kalikiana lunch [12:33] mvo: has it? the branch is open [12:33] ah [12:33] sorry [12:33] I mean 5054 [12:34] mvo: I think my question was about the status of the forum post, if issue #5 is fixed let's mark it as solved [12:34] zyga: checking [12:34] pstolowski|lunch, I think he's talking about this new form: https://src.fedoraproject.org/rpms/golang-gopkg-yaml/blob/master/f/golang-gopkg-yaml.spec [12:35] mup #5054 [12:35] PR #5054: snap,wrappers: address review feedback from #5043 (2.32) [12:35] mvo: offtopic, remember those 100s of threads? my bionic shows 11 threads in .5 [12:35] zyga: 5054 is just cosmetic, but given the amount of questions about it I should have included it. its now in release/2.32 so *if* there is anohter one it will be in [12:35] zyga: I do remember that [12:35] mvo: understood now, thanks [12:35] Son_Goku: okay, i'll dig into the new guideline after lunch [12:36] zyga: thank you! so the 100s threads I don't see on my two bionic machines [12:36] zyga: so not sure what is going on there [12:36] huh, gnome software has a 'restart and install' button [12:36] zyga: #5 *should* be fixed in beta but I hope to get confirmation from a real world test case later today [12:36] Chipaca: for installing updates? [12:37] Chipaca: yes, for some years now :) [12:37] cachio: thanks for the sru validation! [12:37] mvo: I'm guessing what it describes as firmware is a bios update [12:37] Chipaca: it's a very old systemd/fedora/pkgkit feature [12:37] cachio: also yeah beta validation for .5 sounds great! [12:37] it's not related to firmware [12:38] it's related to a mode where all updates are downloaded but not installed, the machine reboots and goes into a special boot mode where systemd installs all updates and reboots back to normal mode [12:39] pstolowski|lunch, guideline in question is documented here: https://fedoraproject.org/wiki/More_Go_packaging [12:39] zyga: I don't open gnome-software often (i've only recently had enough memory to keep it installed at all) [12:41] zyga: what is the link to the 18.04 post again? then I can update #5 [12:41] mvo: https://forum.snapcraft.io/t/issues-to-address-ahead-of-ubuntu-18-04-release/4954 [12:42] zyga: ta! [12:43] zyga: funny pr 4954 is also a fix for 2.32 as is the above forum topic (ok, maybe not that funny). i update it now [12:43] PR #4954: store: Sections and WriteCatalogs need to strictly send device auth only if the device has a custom store (2.32) [12:44] zyga: updated, thanks you [12:46] mvo: we have new pam and netplan [12:48] zyga: yeah, changelog looks mostly harmless [12:48] yes === pstolowski|lunch is now known as pstolowski [13:32] re [13:32] zyga: to be clear, the snap which I am having trouble with is _not_ a classic snap. [13:36] arch updated to 2.32.5 [13:36] I'll speak to snapcraft chaps, as it could be that. [13:37] popey: still mame thing? [13:37] yeah [13:37] i posted the logs from friday in the LP bug report [13:37] you mentioned rpath, but I dont know what *I* am doing wrong here [13:37] yeah, I saw. [14:04] niemeyer: will you have some time to talk about hotplug after your current meeting? [14:29] mvo, zyga: hello! would you all be alright with the apparmor memory leak bug (bug #1750594) being fixed in the first SRU kernel of 18.04 or do you feel it is urgent enough that the 18.04 kernel be respun again before the release? [14:29] Bug #1750594: Eventual OOM with profile reloads [14:30] tyhicks: I think it's not that critical, it only affects systems that run our test suite as far as we are aware [14:30] PR snapcraft#2074 closed: Release changelog for 2.41 [14:30] it's annoying but not user-critical from my POV [14:30] mvo: ^ [14:30] tyhicks: will live kernel patches fix it? [14:30] that's my understanding of the bug, as well [14:30] zyga: no, we don't plan to do a livepatch for it [14:31] we'd really only want to respin if it affected installs or made the out of box experience bad but, AFAIK, this bug doesn't show up in normal use cases [14:32] tyhicks: hi, in a meeting right now [14:32] tyhicks: let me reply as quick as I can [14:32] thanks [14:35] pstolowski: I'm just off and have a few minutes before lunch.. can you talk now? [14:36] niemeyer: yes [14:36] pstolowski: Ok, standup then === deanman_ is now known as deanman [14:47] PR snapd#5057 opened: tests: bring back one missing test in snap-service-stop-mode [14:53] niemeyer: 10MB unaccounted for after creating a bolt db. Will continue digging. [14:53] RSS of a simple thing goes from 4MB to 14MB [14:54] Chipaca: That sounds very promising [14:55] Chipaca: One data point worth gathering is what happens if we open that same file in RO mode [14:57] niemeyer: 8MB rss [14:57] pedronis: hey, will you find some time this week to make another pass on iterface hooks PR? i've addressed your last comments [14:58] Chipaca: Unaccounted or total? That is, 4 to 12, or 4 to 8? [14:58] niemeyer: actually 10MB if the code to create it is still there, just not used [14:58] niemeyer: total [14:58] * Chipaca needs to step away for a bit, will bbl [15:02] niemeyer: time.Sleep -> 4MB, ro bolt: 8MB, ro bolt and rw code unused: 10MB; rw bolt: 14MB [15:02] * Chipaca afk [15:04] mvo, https://paste.ubuntu.com/p/Z59BDC4VWq/ [15:04] running beta validation for 64 bits [15:05] mvo, it is like the system was rebooted after line 53 [15:05] but it shouldn't [15:06] cachio: yeah, this is stange, anything in syslog? [15:06] Chipaca: hacked a small tool that prints package sizes (*.a that end up in the binary, unless go does some lto at the end) https://gist.github.com/bboozzoo/a363e591075cb4d4faae39b4066ac411 [15:06] mvo, this is from syslog [15:07] mvo, I already saw that doing refresh from stable too [15:07] similar problem [15:08] cachio: uh uh [15:08] cachio: is this on a core device? or 16.04? [15:09] mvo, from journalctl https://paste.ubuntu.com/p/DsqmKfDGNv/ [15:10] it is core-16-64 [15:10] a vm [15:10] this is very uncommon [15:10] Apr 16 14:31:53 localhost.localdomain snapd[993]: ERROR cannot open snap: open /tmp/snapd-sideload-pkg-105860667: no such file or directory [15:10] PR #16: Feature/git buildpackage [15:10] mvo, yes, first time I see this [15:12] cachio: confusing [15:13] mvo, this has more info https://paste.ubuntu.com/p/2ty37JM39W/ [15:15] cachio: yeah, more info but still very confusing. I wonder what change from .4 -> .5 might cause this :( [15:15] Chipaca: Hmm.. are we keeping the file open? If we close the database and discard the structure, does the extra RSS memory go away? [15:15] the latest build of the chromium snap on launchpad failed automated review in the store ("checksums do not match"), is that a known issue with the store/snapcraft, or could it be just that specific snap? [15:15] cachio: is it reproducible? [15:15] Chipaca: If it doesn't, that must be a bug [15:15] Chipaca: If it does, then we should probably just do that for the release [15:15] mvo, I don't know [15:15] I'll try [15:15] cachio: thank you! [15:16] Chipaca: Or at least keep it RO, if it impacts runtime of advice too much [15:16] cachio: it looks very fishy, is this in a VM ? or on real HW? [15:16] * niemeyer lunch [15:16] mvo, it is a vm [15:16] cachio: hm, hm, then that is double confusing [15:17] cachio: I was wondering if there might be some HW issue that triggered the reboot but not in a VM [15:17] (usually) [15:17] in parallel I was running i386 [15:17] and it worked fine [15:17] cachio: it *might* be the oom issue but given that we have not seen that beforre on non !i386 its unlikely [15:17] cachio: ok, cool. at least that is good to know [15:36] I just rebuilt the chromium snap again on launchpad, and got the same error (checksums do not match) : https://code.launchpad.net/~chromium-team/+snap/chromium-snap-stable/+build/193033 [15:36] known issue? [15:37] ah, apparently it is bug #1763641 [15:37] Bug #1763641: Checksum mismatch error when uploading a snap to the store [15:42] however the bug says that this shouldn't happen with snapcraft 2.40, but the chromium snap is built on LP and the log says it's using snapcraft 2.40 (deb) [15:42] PR snapcraft#2079 opened: repo: catch error updating the package cache [15:43] zyga: nice, will look in a bit, and also post to factory list [15:51] Thanks! [15:54] mvo: that's the new checking code in doMountSnap [15:55] PR snapd#5055 closed: release: 2.32.5 [15:57] pedronis: oh, so it already caught a case where things did not quite work correctly? [15:57] PR snapd#5058 opened: packaging: fix incorrectly auto-generated changelog entry for 2.32.5 [15:57] mvo: maybe, or we are misunderstanding something [15:58] cachio: -^ [15:58] mvo, not yet [15:59] pedronis: cool, I need to leave in a wee bit to play hockey but I will read backlog, would be great to learn if we introduced a regression [15:59] working on that [15:59] cachio: thank you [15:59] zyga, jdstrand any insight on https://paste.ubuntu.com/p/ZP6R7WM4Kz/ ? [15:59] cachio: just wanted to let you know that pedronis has an idea where the error comes from but its not clear yet if it is just making an existing error visible or if its something new [16:00] mvo, ok [16:00] mvo, perhaps it is not new but it is cannot be reproduced easily [16:01] mvo, I'll make many executions today to see if I can reproduce it [16:01] mvo, pedronis I'll keep you updated in case I find something [16:02] cachio: thank you! [16:03] cachio: thanks to pedronis we have much improved error checking around this area of code now, so maybe it surfaces something that existed before. I will read backlog, need to play hockey now [16:03] mvo, nice, enjoy it [16:04] cachio: thank you! yeah, I am excited [16:10] cachio: it's strange because it seems we are rebooting in the middle of installing a local snap [16:11] Chipaca: what happens if we reboot when installing a local snap? it seems the snap is copied into /tmp, but tmp will be cleared on reboot [16:13] cachio: am I misreading that there's a reboot going on at the start of that test? [16:14] cachio: do we have a bug about stopping refreshes during some of the core tests? [16:17] * kalikiana eod === JanC_ is now known as JanC [16:19] pedronis, I don't think so, but let me re-check [16:28] pedronis: AFAIK it fails if it happens before the (snap.Container).Install, otherwise it succeeds [16:28] pedronis: (that's the bit that copies the local snap into /var/lib/snapd/snaps iirc) [16:28] it seems we are hitting the scenario in a test [16:28] hehe [16:29] but I'm not sure why there is rebooting going on though [16:30] Chipaca: do you agree that there is some system restarting going on here: https://paste.ubuntu.com/p/DsqmKfDGNv/ ? [16:31] there's too much starting, early, stuff and systemd talking about early boot stuff [16:31] niemeyer: we are not keeping the file open, and we do a commit and close on the db [16:31] Chipaca: another theory is that snapd and the test starts before the cleaning of tmp happens [16:31] niemeyer: the only reason snapd uses boltdb is to write that databse, having it ro is like not having it [16:32] That's pretty awkward. Why would the memory stay resident [16:32] pedronis: looking [16:32] bu that makes sense only if tmp is not a tmpfs [16:32] niemeyer: MAGIC [16:32] Chipaca: I guess Go's GC [16:33] Chipaca: There is (was?) a method in the runtime package to "encourage" the GC to give memory back to the system [16:34] pedronis: no that doesn't look like early boot stuff [16:34] pedronis: it looks like snapd starts, and systemd says 'ok i'm done' [16:34] niemeyer: runtime.GC()? [16:36] Chipaca: No, that just kicks the GC.. let me check [16:36] Chipaca: but we are after a reboot, right? [16:37] Chipaca: otherewise there would not be that Reached target stuff [16:37] niemeyer: runtime/debug's FreeOSMemory? [16:37] Yes! [16:38] pedronis, now error on i386 https://paste.ubuntu.com/p/hkxVgZDZgk/ [16:39] I don't see an "error" in that log [16:40] niemeyer: it helps a little (14 -> 12 for rw, 10->9 for ro w/unused rw) [16:40] pedronis, this is for 64 bits https://paste.ubuntu.com/p/5MHzmfjJP8/ [16:41] pedronis, no errors [16:41] but suddenly rebooted [16:41] ah [16:41] so we are getting reboots [16:41] pedronis, yes [16:42] I don't see anything about "core" in those logs [16:42] not sure what would provoke reboots [16:42] that is new [16:42] I understand why reboot would make other things fail [16:42] but not what make us reboot [16:42] * pedronis -> dinner [16:49] blame spectre [16:50] jdstrand, have you seen my comment on bug #1763641 ? If I read the bug correctly I shouldn't be seeing this error with snapcraft 2.40, but I am [16:50] Bug #1763641: Checksum mismatch error when uploading a snap to the store [16:50] I ran reviews-tool from edge on the snap and got no such error [16:56] sergiusens: did corebird work before? [16:56] PAM got updated recently but I don’t think we’d break abi [16:58] I might have told the build to not prime libkrb5 [17:13] zyga: we are getting random reboots on .5 [17:13] cachio: we didn't see these reboots with .4 ? [17:14] Hmmmm [17:14] What do we know? [17:14] not for beta scenario at least [17:14] .5 has new PAM and nplan [17:14] danke popey [17:15] I didnt know we had source-checksum! [17:15] TIL [17:15] inorite [17:15] pedronis, I just saw some problems but for factory release to beta [17:15] Changelog from for .5 is not extensive, mainly new API and a bugfix [17:16] Is there any chance this is caused by new stop restart mode [17:16] * Chipaca afk [17:16] pedronis, zyga perhaps something else in the beta image it is causing this [17:16] new kernel [17:16] Oh [17:16] Which one? [17:18] I will be home soon [17:21] pedronis: who reported this and how? [17:21] cachio [17:21] running the validation [17:21] zyga, at least for dragonboard there was a new one [17:22] pedronis, there are 2 scenarios, first is to create a beta image and run the tests for that image [17:22] last execution I didn't see any error [17:22] pedronis, this time I saw this reboot [17:23] pedronis, but it is not failing 100% of the cases [17:23] so, perhaps the same error was present in the previous validation and I did not reproduce it [17:24] but today first exec I saw the errror [17:24] for core-16-64 [17:24] and core-16-32 worked properly [17:24] 100% pass [17:25] cachio: on what hardware? [17:25] then I rexecuted both and both failed [17:25] I am running those on vms [17:25] probably worth trying a couple runs with .4 I suppose [17:25] So amd64/kvm [17:25] to see if it's really .5 [17:25] Yes [17:25] Let’s test .4 in the same host [17:26] ok [17:26] I'll do that [17:26] Thank you [17:27] of course if it's new kernel we need to separate that somehow [17:29] It could also be the oom bug [17:30] does it provokes reboots though? [17:30] It is disabled on 32bit systems but also happens on 64 big systems the same, just takes longer [17:30] or only errors? [17:30] It eats all ram over time till kernel crashes [17:30] With swap it’s a slow painful death [17:30] Without swap it is quick and clear [17:32] As an option we can do .6 with one of my workarounds [17:47] for the coders among us, Brenden Eich on Javascript's evolution: https://www.youtube.com/watch?v=3-9fnjzmXWA [17:54] there are coders among us ? [17:57] * popey steps back [17:58] Yes, but I heard the word "javascript" and ran [17:58] :-p [17:59] * zyga looks at the reboot issue [17:59] pedronis, cachio: we have a forum thread for the problem? [18:09] zyga, no yet [18:10] zyga, I'll create it [18:10] thank you [18:15] the pastebin I read looks very weird, I doubt that's OOM [18:16] it looks like we really trigger a graceful shutdown shomehow [18:16] cachio: does it happen in one spot or in some random place during testing [18:17] the pattern that I saw in the syslog is [18:18] https://paste.ubuntu.com/p/DVJNqD5yHP/ [18:18] in all the cases I saw something like this before the reboot [18:19] how can i change the owner of a snap in the store [18:20] zyga, and then this [18:20] https://paste.ubuntu.com/p/vTM3bYx2Ph/ [18:21] I am running now 16-2.32.4 to see if I can reproduce it [18:23] zyga, I thinks 2/3 cases happened installing snapd [18:23] snaps [18:23] * zyga dinner, I will check soon [18:26] i specifically need to adjust the ownership of git-ubuntu, git-ubuntu-stable and git-ubuntu-beta away from my user [18:41] nacc: to canonical, snapcrafters or another individual user? [18:45] nacc: post on the forum in the store category [19:06] Hey niemeyer, I can't get the LXD backend of spread to ever work for 17.10 or 18.04, they always end up failing with "2018-04-16 19:01:30 Discarding lxd:ubuntu-17.10 (spread-8-ubuntu-17-10), cannot connect: cannot connect to lxd:ubuntu-17.10 (spread-8-ubuntu-17-10): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain" [19:06] Any ideas? [19:09] popey: up ? [19:09] om26er: yo! [19:09] oh boy you are always online, cool. [19:10] popey: sublime is broken and I hope we can land its fix ASAP. [19:10] we do have it published in edge channel but it won't start [19:10] ok, let me take a look now. [19:10] oh, it's classic. Are you on nvidia? [19:11] popey: no, Intel [19:11] ok, lemme see [19:14] ok, merged your PR, lets see what that builds [19:14] great stuff. [19:14] thanks popey [19:14] np [19:17] cachio: any news [19:18] I ran 2.32.4 and I didn't see any error yet [19:18] zyga, I am re-executing now [19:19] can you upload the manifest [19:19] I created images using candidate [19:19] channel [19:19] and ran the tests [19:19] zyga, but no errors yet [19:19] zyga, the same it is failing 66% of the times on beta [19:19] cachio: so mvo suggested some ideas [19:19] can you upload the manifests somewhere [19:20] zyga, sure [19:20] mvo said it would be in /usr/share/dpkg/snap* something [19:20] also how big is the image [19:20] can you upload the failing image as you got it now [19:21] zyga, sure [19:21] thanks! [19:21] it is also very easy to generate it [19:21] and faster [19:21] with my wifi it could take long to upload [19:21] zyga, clone git@github.com:sergiocazzolato/validator.git [19:21] and then run [19:22] ./create.sh beta pc-amd64 [19:22] ./create.sh beta pc-i386 [19:22] trying [19:22] that will be faster [19:23] which ubuntu-image are you using? [19:23] from deb/snap [19:23] and which version [19:23] deb [19:24] ubuntu-image 1.3+16.04ubuntu2 [19:25] I'm on 1.3+18.04ubuntu2 [19:25] but I'll make an image and try this way first [19:25] if my image works can you upload your image [19:25] we can compare binary stuff inside [19:25] zyga, sure [19:26] maybe overnight [19:26] so we can analyze tomorrow [19:31] kyrofa: I think there is a patch from Saviq that fixes that, from long ago.. sorry for not having merged it yet.. will try to have a look today still [19:31] niemeyer, ah, excellent, okay thank you [19:31] cachio: my dpkg.list from /usr/share/snappy/dpkg.list http://paste.ubuntu.com/p/JWPqJFWH7D/ [19:31] cachio: can you check if you have the same one [19:32] cachio: I'm running kernel 4.4.0-121-generic [19:32] Yep, that looks like it [19:32] ok [19:39] cachio: are you running the same thing on your test machine? [19:40] zyga, checking [19:42] zyga, the kernel is the same [19:43] zyga, same [19:43] the list is the same [19:43] ok, thanks [19:43] zyga, I am running again now [20:01] cachio: I just chatted with zyga and he told me about the reboot issues (and you did so as well earlier). I just checked, might be worth running with the kernel from stable, looks like the kernel in beta got released just today [20:02] mvo, you mean the kernel snap? [20:03] yes [20:03] cachio: yes [20:03] let me cehck that [20:03] cachio: I was just looking at the things that changed recently and noticed that this might be something [20:04] cachio: I will also run the testsuite against a freshly build image to see if I can find any clues [20:04] mvo, I just created a new image 20 minutes ago [20:04] and I am running again against this new one [20:05] so far no errors [20:05] mvo, but just 31/199 yes [20:05] yet [20:06] mvo, I am in i386 and I have pc-kernel 4.4.0-120.144 111 beta canonical kernel [20:07] there is a new one in beta [20:07] 4.4.0-121.145 [20:07] I'll regenerate also i386 image to see if the problem persist [20:10] cachio: thanks. I just build an amd64 image, is that correct? [20:10] cachio: aha, nice if you have an i386 image with an older kernel its interessting to see if the new kernel changes anything [20:11] mvo, :( [20:11] I just delete it [20:11] overwrite it [20:12] cachio: no worries [20:26] popey: https://github.com/snapcrafters/sublime-text/pull/10 [20:26] PR snapcrafters/sublime-text#10: Remove architectures stanza as it breaks auto builds [20:29] ...and this https://github.com/snapcrafters/android-studio/pull/20 (though not urgent) [20:29] PR snapcrafters/android-studio#20: remove architectures stanza [20:30] om26er: merged both [20:30] thank you! [20:31] super! [20:33] we probably need to define some kind of "trust" chain (equivalent of MOTU/Core dev ?), so that we don't always have to bug you to get things landed. [20:33] there are others who have access :) [20:36] apart from Martin, ev and you ? talking about snapcrafters repos only [20:36] yeah, currently the three of us [20:36] its generally difficult to get a hold of flexiondotorg [20:37] he's Wimpress now :) [20:37] ev, I am not sure if he hangs around here. [20:37] he doesn't [20:37] aah [20:37] he hates irc [20:37] like, passionately :) [20:37] It is I [20:38] And popey will burn in the embers of the heat death of the universe for being a lying #&$% [20:38] alright, its the two of you then ;-) [20:38] cachio: tests with stable kernel still going strong, I had one issue with python3 /nsap/network-bind-consumer/x1/bin/consumer eating all cpu and producing log messages I think we need to ensure in the relevant test that we kill it [20:39] I see the reference to hating IRC was not attributed to me. [20:39] #carryon [20:39] popey: Wimpress still when snaps take over the world, MOTU/Core dev like access may be necessary. [20:39] mvo, which is the test? [20:39] om26er: well, ideally these would all be gone from snapcrafters and owned upstream :) [20:39] not today, but hopefully soon -__- [20:39] :) [20:40] mvo, I am rnnning refresh scenario too [20:40] mvo, let's see how it goes [20:41] cachio: I'm not sure what test triggers this, just noticed [20:41] mvo, ok, I'll check it [20:42] popey: sublime text is good now [20:42] \o/ [20:42] zyga: ^ its in edge [20:42] thanks! [20:42] mvo: I have a helper snap for redirecting systemctl [20:43] om26er: alias works too? [20:43] popey: yep, subl [20:43] wow, that launches fast [20:43] om26er: released. thanks so much for all your work on this! [20:43] Sorry it took so long. [20:44] https://snapcraft.io/sublime-text [20:44] cachio: and I noticed sometihng interessting too, when I log into the VM while the test is running I see a "system is going gown" message, so there is a planned reboot triggered by a test, for me the planned reboot message starts at 22:38 and the following tests run then http://paste.ubuntu.com/p/d3xsB4KQMP/ so one of these tests triggers a change that includes core/kernel which triggers the reboot [20:44] popey: no problem, lets see if the upstream wants it now [20:45] mvo, weird [20:45] maybe wait till we promote it and get more installs :) [20:45] I didn't see any reboot scheduled [20:45] well, there is a reboot schedulet but for a really long time [20:46] which should not be affecting the testst [20:46] sure [20:46] cachio: if you ssh into the machine, do you get Broadcasting messages as well? [20:46] mvo, no [20:46] I just saw in the logs [20:46] in the vm I didn't see any broadcastig message [20:54] mvo, cachio: sudo snap install systemctl-redirector --beta [20:54] zyga, what's this? [20:54] then touch /var/log/systemctl.fake.log [20:55] then sudo /snap/systemctl-redirector/current/bin/install [20:55] then run all tests [20:55] in the end collect the log file [20:55] this logs all invocations of systemctl [20:55] PR snapd#5059 opened: tests: add pending shutdown detection [20:55] (reboot and all kinds of systemctl commands) [20:55] mvo: ideas for improvement welcome [20:57] zyga: thanks, I also pushed the above pR to break when we detect something that triggered a shutdown [20:57] yeah [20:57] looks good [20:57] zyga: lets attack it from multiple angles [20:58] cachio: iif you cherry pick the change from 5059 into your tree hopefully this also gives you an error when the shutdown is triggered [20:58] cachio: when running -debug maybe it gives us clues [20:58] cachio: if you run tests with this snap installed and with the install script invoked we should collect more info about what is going on [20:58] mvo, zyga, ok [20:59] I'll run it now [20:59] ta [20:59] thanks [20:59] I will soon need to crahs :/ [20:59] but I start the tests now to see if I also notice anything [20:59] me too [20:59] and also login the VM to see any wall messages [20:59] with the last kernel seems to be working better [20:59] I didn't see any reboot yet} [21:02] cachio: I refreshed the systemctl-redirector snap to improve the log messages [21:04] ok, I'll start a run in 10 minutes [21:04] zyga, could you reproduce the error? [21:04] with the new kernel snap? [21:05] I didn't run full test run yet [21:05] zyga, I regenerated the image with the new kernel snap and I couldn't reproduce it anymore [21:06] zyga, wait [21:06] I did :) [21:07] did you install and configure my snap on your VM [21:08] zyga, https://paste.ubuntu.com/p/KMR33JvpyG/ [21:08] not for this run [21:08] I'll do it for next one [21:09] Apr 16 20:45:41 localhost kernel: [ 3071.152512] modprobe: page allocation failure: order:7, mode:0x2080024 [21:09] PR #16: Feature/git buildpackage [21:09] zyga, because if I install it during execution it is gonna be deleted on reset [21:09] zyga, yes [21:09] on reset? [21:09] ah [21:09] snaps are removed :/ [21:09] looks like we ran out of ram [21:09] zyga, I am configuring vms with -m 1500 [21:10] zyga so far it was enough [21:11] cachio, mvo: the allocation that failed was for 524288 bytes [21:11] not a lot [21:11] it looks like we ran out of ram, plain and simple [21:11] zyga: I also run with a 1500m VM, interessting [21:11] cachio: which kernel is that? [21:11] I keep it running [21:13] 4.4 [21:13] zyga, pc-kernel 4.4.0-121.145 113 beta [21:14] right, I see [21:14] guys, I need to go to sleep [21:14] cachio: before you EOD write what you know on the forum [21:14] zyga, sure [21:14] zyga, go to bed :) [21:16] Thank you. Good night! [21:17] zyga, cachio I need to crash as well, I will keep the tests running and check in my morning, I run with beta kernel and the shutdown detection patch from the PR above [21:17] * mvo waves [21:37] nacc, any idea why building git-ubuntu on arm64 would fail unable to find ffi.h? libffi-dev seems to be in stage-packages [21:38] Does it need to be a build-package as well? [21:38] Not sure how this is setup [21:43] kyrofa: log? [21:45] nacc, https://pastebin.ubuntu.com/p/mGjrtbBdfJ/ [21:50] nacc, note that this worked fine on amd64 [21:51] kyrofa: otp, looking now [21:51] No problem [21:54] kyrofa: i have no idea :/ [21:54] kyrofa: i've never tried to build on arm64 [21:54] nacc, ah, that's what I was going to ask [21:54] Okay, that's not unusual === sergiusens_ is now known as sergiusens