=== JoshStrobl is now known as JoshStrobl|zzz [05:35] PR snapd#3782 closed: client: fix go vet 1.7 errors [07:33] jamesh: hey, I added some small comments for 3581, very excited that this can land [07:33] mvo: okay. I was just writing a reply to niemeyer's last comment. [07:34] jamesh: great! [07:36] pedronis: how are things looking from your side for 2.28 - did we need to make the tweaks you mentioned yesterday? [08:02] Buon giorno, principesse e principi! [08:04] good morning [08:04] hey Chipaca, good to see you [08:04] how were your holidays? [08:04] tiring :-) [08:04] how were things? [08:04] I think mostly okay [08:05] mvo: hi, not we didn't yet [08:05] tests are greener than earlier [08:05] anything igneous that shouldn't be? [08:05] Chipaca: haha, yes, mystery of my VM with high error rate [08:05] Chipaca: I have a VM that when I run tests on, things massively fail, randomly [08:05] Chipaca: (in unit tests) [08:05] mvo: I need to prepare a PR with a change this morning [08:06] pedronis: ok, please tag with 2.28 then [08:06] zyga-suse: is it accidentally a Win10 vm? [08:06] Chipaca: haha, no, it's my Arch VM [08:06] pedronis: I went over the open ones to see what we want for 2.28 and milestoned a bunch [08:06] Chipaca: tests failures are pretty odd, like "cannot parse this constant yaml string" [08:07] Chipaca: the host has good memory (tested) [08:07] zyga-suse: how big is the vm image? [08:07] Chipaca: let me look [08:07] 14GB [08:07] Chipaca: note that it fails randomly but mostly so when I unpack the release tarball again [08:07] so feels like something gets corrupted randomly [08:08] Chipaca: the releast tarball md5sum checks out (weak but useful) [08:08] Chipaca: buon giorno [08:09] zyga-suse: and it's a regular VM thing like kvm? [08:09] yes [08:09] zyga-suse: you're not emulating ryzen or something [08:10] Chipaca: no, on a 3rd gen core i5 [08:10] and this machine has loads of other VMs with nothing failing anywhere [08:10] Chipaca: the kernel there is fresh, I forgot how fresh [08:10] Chipaca: the golang bits are 1.8.3 [08:11] Chipaca: if you want I can show you [08:11] zyga-suse: can you build a fresh arch vm of the same version and compare checksums? [08:11] Chipaca: checksums of all the files on disk? [08:11] yes [08:11] Chipaca: arch is hand made (early parts) so I bet things will turn out different [08:11] well, not _all_, but say /lib [08:11] Chipaca: I could reinstall all packages [08:11] Chipaca: one thing I did [08:11] Chipaca: that is interesting [08:11] pstolowski: 3569 is ready, right? test look a bit unhappy right now, but I can baby sit, I think we wan tthis for 2.28 :) [08:11] Chipaca: is to save yaml to /tmp/FAILING.yaml whenever I cannot parse it [08:12] Chipaca: and I got the string ` - : `or something like that [08:12] o_O [08:12] I should do incremental files to see what other strings may fail [08:12] and note that it is not random at all [08:12] er [08:12] it **is** random [08:12] zyga-suse: do you always get the same yaml? hexdump -C on the file would be great [08:12] mvo: so far yes but as I said, I only save one file [08:12] mvo: so I could get the same one because it's the first/last [08:12] zyga-suse: but that file is always the same? [08:12] yes [08:13] each time I saw it (~10 [08:13] zyga-suse: thats a good start [08:13] mvo: I've pushed some changes based on your review comments. Assuming CI is happy with it, can you merge it? [08:13] and note that since I built the package so I managed to run everything at least once without any failures [08:13] zyga-suse: ` - :` probably is bad yaml tho [08:13] :-p [08:13] speaking of which, I wanted to upload that package [08:13] Chipaca: I figure :) [08:13] pstolowski: does 3526 needs a review from gustavo? or just a second review? [08:13] let me get more facts [08:13] I find it annoying and curious at the same time [08:14] jamesh: \o/ thank you [08:14] jamesh: I have a look and if CI is green I will merge it, really nice to see this sorted, this is a huge win for usability [08:14] mvo, hey, yes 3569 is ready but has been failing tests for random reasons unrelated to the PR itself, i restarted it a couple of times in the last few days [08:14] zyga-suse: you sure its ` - :` and not `( - :` ;) ? [08:14] mvo: hahahaha [08:15] mvo: I've got more I'd like to do w.r.t. polkit support: this was just a minimal change to get the foot in the door. [08:15] pstolowski: yeah, I pushed master, maybe today is a good day [08:15] mvo: yaml-smiley-virus-2017 [08:15] mvo, and yes, 3526 needs a review from Gustavo [08:15] mvo: replaces random in-memory yaml with a non-parsing smile [08:15] :D [08:15] jamesh: anything that is ready or almost ready that we want for 2.28 (which we plan to tag in the next few days)? [08:15] mvo, :) yes let's hope today is THE DAY ;) [08:16] pstolowski: I added gustavo for 3526, lets poke him in the standup :) [08:16] mvo, thanks@ [08:16] mvo: It's probably best not to wait. I'd like to add polkit auth for a few more APIs, but I understand that is somewhat controversial [08:17] mvo: less controversial is having the snap client support the textmode polkit agent and indicate whether it is interactive [08:17] jamesh: ok, sounds good! what is needed for the textmode polkit agent? I assumed this is transparent for the API user, no? [08:18] morphis_: want me to have a look at the open comments for 3260? it seems its almost there [08:18] mvo: essentially just have it fork "pkttyagent --fallback", like systemctl does [08:19] re [08:19] Chipaca: have a look [08:19] mvo: this would let you run privileged operations without sudo, having pkttyagent prompt if needed [08:19] https://paste.gnome.org/pecbkpb4e [08:19] mvo: if you have spare cycles for that, that would awesome [08:19] this is from [08:19] https://paste.gnome.org/prw9lmfbx [08:19] jamesh: aha, nice! [08:19] mvo: but more in general I still waiting for niemeyer to approve this from an architectural point [08:20] Chipaca: ideas appreciated, I can patch and rebuild easily [08:20] jamesh: if this PR is there before 2.28 is taged I will milestone it so that it gets at least priority reviews :) [08:21] mvo: okay [08:21] morphis_: I can raise it in the standup, I was uner the impression he is happy with it now but I have not talked with him in detail [08:21] mvo: that was my guess too, but he still has his denial on the PR [08:22] mvo: otherwise let me try to get through zyga-arch's comments this afternoon [08:22] morphis_: I tagged it for 2.28, lets see if that helps speeding it along. I also check the open question and will push changes [08:22] :-) [08:22] zyga-arch: and which test is it that fails with this? [08:22] morphis_: lets make a deal, I look at things now and what is left you can have in the afternoon :) [08:22] sounds good :-) [08:22] Chipaca: *any* tests that touches yaml, I saw it fail all over the source code [08:23] zyga-arch: an example? [08:26] essentially any that calls snap.InfoFromSnapYaml for insatance FirstBootTestSuite.TestPopulateFromSeedHappyMultiAssertsFiles [08:26] (note that this is a first that I saw this particular test failing) [08:28] mvo: question, did you upgrade your laptop to >> xenial? [08:28] (hence the spellchcek and govet changes) [08:33] Chipaca: I just patched it to collect each failure, let's see what we get [08:45] https://paste.gnome.org/pjiw7rerp [08:45] more interesting failure log [08:46] zyga-arch: what filesystem is this on? [08:47] * Chipaca thinks "lardfs" [08:48] Chipaca: ext4 [08:48] as boring as it gets, right? [08:48] Linux archlinux 4.12.8-2-ARCH #1 SMP PREEMPT Fri Aug 18 14:08:02 UTC 2017 x86_64 GNU/Linux [08:48] the kernel is fresh though [08:50] zyga-suse: can you try with a kernel that isn't newer than the host's? (not that it should matter) [08:52] Chipaca: not sure, I don't know how to get old arch kernels [08:53] zyga-arch: snap install linux-2.0.43 [08:53] no? [08:53] aw :-( [08:54] heh, not yet [08:54] I improved logging to save the failing yaml and the error message [08:54] maybe some of those are genuine errors from the test suite [08:54] “snap set core locales en_GB.UTF-8”, WTDT? [08:54] WDYT* [08:55] Chipaca, wont work [08:55] ogra_: wy? [08:55] because there is no locale-gen === tinwood_ is now known as tinwood [08:56] ogra_: not _yet_ :-) [08:56] Chipaca, i wont add 20MB (or what was the locales stuff) [08:56] * ogra_ forgot the exact size but it wasnt small iirc [08:57] ogra_: oooh, we can discuss this like civilized people, or we can fight! [08:57] * Chipaca gets his gloves [08:57] *ding* FIGHT [08:57] and you konw [08:57] * Chipaca runs the fastest [08:57] the finishing move will be called LOCALITY! [08:58] * Chipaca makes chicken noises and waves his elbows while running [08:58] * ogra_ throws a de_DE.UTF-8 ... in a perl variant ! [08:58] * Chipaca is in a perl hackers list somewhere on the internet still [08:59] * Chipaca mutates the glob behind the variant and throws it back [08:59] Chipaca: on the interenet nobody knows you are a perl interpreter [08:59] Chipaca, i'll trade though ... find something that compensates enough for the added stuff and we can talk ;) [08:59] ogra_: but, serioulsy, if it's 20MB i'd agree, but i think what we want is the locale-gen, not the locales themselves [09:00] there is definitely still other stuff we can drop [09:00] ogra_: here's $0.01 for the flash cost ;) [09:00] Chipaca: here's another interesting failure [09:00] ---------------------------------------------------------------------- [09:00] PANIC: firstboot_test.go:445: FirstBootTestSuite.TestPopulateFromSeedMissingBootloader [09:00] ... Panic: cmd: "mksquashfs . /tmp/check-717607699634245797/61/snapsrc/core_1.0_all.snap -noappend -comp xz -no-xattrs" failed: exit status 1 ("Parallel mksquashfs: Using 4 processors\nCreating 4.0 filesystem on /tmp/check-717607699634245797/61/snapsrc/core_1.0_all.snap, block size 131072.\n\r[===================================================================|] 1/1 100%writer: Lseek on destination failed because Bad file descri [09:00] ptor, offset=0x60\nFATAL ERROR:Probably out of space on output filesystem\n\n\nExportable Squashfs 4.0 filesystem, xz compressed, data block size 131072\n\tcompressed data, compressed metadata, compressed fragments, no xattrs\n\tduplicates are removed\nFilesystem size 0.35 Kbytes (0.00 Mbytes)\n\t101.72% of uncompressed filesystem size (0.34 Kbytes)\nInode table size 98 bytes (0.10 Kbytes)\n\t100.00% of uncompressed inode table [09:00] size (98 bytes)\nDirectory table size 55 bytes (0.05 Kbytes)\n\t100.00% of uncompressed directory table size (55 bytes)\nNumber of duplicate files found 0\nNumber of inodes 3\nNumber of files 1\nNumber of fragments 1\nNumber of symbolic links 0\nNumber of device nodes 0\nNumber of fifo nodes 0\nNumber of socket nodes 0\nNumber of directories 2\nNumber of ids (unique uids + gids) 1\nNumber of uids 1\n\tzyga (1000)\nNumber of gid [09:00] s 1\n") (PC=0x458E88) [09:01] /dev/sda2 20G 9,5G 9,1G 52% / [09:01] hmmmm [09:01] zyga-arch: how [09:01] 's space doing? [09:01] tmpfs 1,5G 2,1M 1,5G 1% /tmp [09:01] ah [09:01] not bad [09:01] zyga-arch: what does dmesg say? [09:01] really weird, right? [09:02] did you add locales back ? [09:02] :P [09:02] nothing at all [09:02] https://paste.gnome.org/psa6vgmnj [09:02] the last timestamp is old there [09:02] zyga-arch: yeah, I run on zesty [09:02] mvo: I'm tempted to update my xenial x250 [09:02] PR snapd#3569 closed: snapd,snapctl: decode json using Number [09:02] any issues? [09:02] zyga-arch: its doing fine on my x250 [09:03] zyga-arch: i'll move this weirdness to the backburner for a bit [09:03] let see if something percolates [09:04] Chipaca: yeah, me too; thank you for the time! [09:04] is it my machiine, or are the travis logs so slow they're almost unusable? [09:07] my network is shitty enough not to handle 2 IRC connections (and music streaming :P) [09:09] yay! thanks mvo :) [09:09] pstolowski: thank you :) [09:58] janisozaur: hey, around? [09:59] o/ [09:59] hey :) [09:59] do you have a moment to help me out? [09:59] with? [10:00] zyga-arch: I am seeing https://paste.ubuntu.com/25375579/ right now on my production system [10:00] janisozaur: I'd like you to build a package for me a few times [10:00] I'm seeing random bugs that feel like kernel memory corruption or something similar [10:00] and I need 2nd opinions [10:00] on Arch? [10:00] yes [10:01] morphis: oh that's a new thing, feels I dind't patch the apparmor profile [10:01] morphis: where are you seeing this? [10:01] zyga-arch: Ubuntu 16.04 [10:01] I don't have access to my system right now, maybe in the evening? [10:01] I think with proposed enabled [10:01] janisozaur: that's ok, thank you! [10:01] morphis: I think we need to fix this, mvo ^^^ [10:01] mvo: we may need to patch 2.27.x [10:01] let me check though [10:01] 2.27.3 is the version of the snapd deb [10:02] zyga-arch: that's on fully updated system, right? [10:02] yes [10:03] morphis: can you look at your apparmor profile [10:03] morphis: look for mount options=(rw bind) @SNAP_MOUNT_DIR@/{,ubuntu-}core/*/etc/nsswitch.conf -> /tmp/snap.rootfs_*/etc/nsswitch.conf, [10:03] morphis: this should be in /etc/apparmor.d/ [10:03] yeah, it's there [10:03] morphis: can you check that it is present for the core snap version as well [10:04] zyga-arch: hm, an apparmor_parser -r /etc/apparmor.d/usr.lib.snapd.snap-confine.real fixed it [10:04] morphis: !!! [10:04] zyga-arch: the core snap is at 16-2.26.14 [10:04] morphis: can you please open a forum topic [10:04] I think we've seen it before [10:04] but now it's a smoking-gun-bug [10:04] morphis: and please save your journal [10:04] sure, but can't share that in the public [10:05] as there are log entries for each time the profiles are loaded [10:07] jdstrand: ^ [10:07] mvo: ^ (possible bug in our packaging or in dh_apparmor in general) [10:09] zyga-arch: mvo: https://forum.snapcraft.io/t/apparmor-profile-for-snapd-confine-not-reloaded-on-deb-update/1820 [10:10] zyga-arch: if I remember right I've rebooted in between [10:10] zyga-suse: ok, we need to figure out if its a regression, fortunately its not too late yet to do another 2.27 [10:11] mvo: I don't think so, we've seen this randomly a few times [10:11] morphis: what is the timeline, i.e. did you apt upgrade and after that the snapd services did stop working? [10:11] mvo: but it is serious, it means we are running on a broken configuration [10:11] zyga-arch: or did a reboot happen in between? [10:11] mvo: I don't know, it's morphis that is affected [10:11] zyga-arch: yeah, definitely serious [10:12] mvo: if I remember right, I did a manual apt upgrade [10:12] did a reboot and then after some time wondered why specific services coming from a snap were not running [10:12] * zyga-arch wonders where to upload the binary release for Arch [10:12] :/ [10:13] morphis: can you please add the info that there was a reboot in between? that might be useful to know [10:13] done [10:13] morphis, zyga-*: I think we want the input from jdstrand on https://forum.snapcraft.io/t/apparmor-profile-for-snapd-confine-not-reloaded-on-deb-update/1820 [10:13] zyga-arch: github releases? [10:14] janisozaur: oh, good idea [10:14] mvo: can I add a file to our 2.27.3 github release page? [10:15] zyga-arch: I bet its apparmor cache related [10:15] mvo: I checked that the debhelper scripts uses an option to skip the cache [10:15] morphis: silly question, but another reboot will not bring the problem back, right? I assume apparmor_parser -r will update all caches [10:15] zyga-arch: oh? woah [10:15] mvo: didn't tried yet [10:15] mvo: not sure, caches are writen to explicitly [10:15] zyga-arch: so the file on disk is correct but the wrong profile is loaded even after reboot? [10:16] I think the early startup script that loads profiles [10:16] mvo: not sure how it uses or not uses caches [10:16] zyga-arch: re file for release> what file exactly? [10:16] mvo: binary build for arch [10:16] and a source package for arch [10:16] zyga-arch: sure [10:16] thank you [10:16] zyga-arch, morphis: so just to double check - the apparmor profile on disk is correct, the reboot has happend and yet the kernel has an incorrect profile loaded? [10:17] mvo: yes, that is what happened [10:17] can it be a problem that the core snap was still at 2.26? [10:17] morphis: next time, don't reload the profile [10:17] morphis: please collect the loaded profile from sysfs [10:17] morphis: would you mind also putting this as a TL;DR in the forum post :) ? [10:17] morphis: no, each profile is distinct [10:18] it applies to a specific binary via path [10:18] morphis: this makes it also sound as alarming as it should sound :) [10:18] morphis: profiles can be extracted from memory [10:18] morphis: via /sys/kernel/apparmor/profiles/ [10:18] morphis: but you need to copy the raw policy [10:18] zyga-arch: 3260 is ready for a re-review [10:18] morphis: you cannot tarball the directory [10:18] (or you may need to give tar a smart option to ignore size) [10:18] mvo: noted, I'll open it now [10:19] PR snapd#3581 closed: daemon: add polkit support to /v2/login [10:19] * mvo hugs jamesh [10:20] mvo: thanks! [10:20] mvo, zyga-arch: btw. shouldn't I better file a bug report on launchpad for this? [10:21] morphis: I prefer the forum really [10:21] morphis: people look there x10 more often [10:21] zyga-arch: so is that were you now track bugs? [10:21] morphis: track TODOs really, bugs and other [10:22] morphis: a bug is fine as well but my point is that exposure there is low [10:22] * zyga-arch spends $ETERNITY to upload the release [10:22] just wondering what the actual policy is [10:22] morphis: no idea [10:22] :-D [10:22] morphis: forum >> bug for sure but bugs are used too [10:26] I'll focus on the review from mvo/morphis now, ping me for attention === jospoortvliet is now known as jospoortvliet_ === jospoortvliet_ is now known as jospoortvliet === ShalokShalom_ is now known as ShalokShalom [10:31] PR snapd#3784 opened: cmd/snap: in "snap info', don't print a newline between tracks [10:37] morphis: quick question - there is code in cmd_userd.go that looks like you created unittests, there is no cmd_userd_test.go on your hdd that is not git added ;) [10:38] mvo: the bigger question is, how can you see morphis' hdd [11:14] PR snapd#3785 opened: asserts,overlord/devicestate: accept generic serials only if the model has generic-serials: true [11:15] mvo: this is the PR ^ , marked for 2.28 and needs Gustavo to look at it [11:21] mvo: need to check but I guess I never ported them over from the former implementation when it was snapd --user [11:22] mvo, zyga-arch: there seems to be another problem, I am now seeing seccomp enabled for my devmode snaps with 2.27.3 [11:24] morphis: can you please collect as much stuff as you can [11:24] sure [11:24] morphis: can you look at the bpf profile [11:24] zyga-arch: how do I see if a process is running with seccomp attached? [11:24] morphis: and at the legacy profiles that we don't remove [11:24] morphis: let me check, I don't know [11:25] mvo: devmode snaps get an ALLOW bpf, right? [11:25] mvo: or nothing at all? [11:25] hm, the profile has @complain [11:25] morphis: which specific file has this? [11:26] morphis: note that we have new profile location now so you may be looking at the old palce [11:26] zyga-arch: the profile for my snap app [11:26] place [11:26] morphis: right, _which_ file? :-) [11:26] /var/lib/snapd/seccomp/profiles/snap.anbox.container-manager [11:26] that's the old file [11:26] it's not used anymore [11:26] and not removed either [11:26] morphis: look at seccomp/bpf [11:26] there are two files per profile [11:26] binary and source [11:27] those are compiled with snap-seccomp [11:27] hm, /var/lib/snapd/seccomp/bpf/snap.anbox.container-manager.src has @complain too [11:27] okay so _so far_ so good [11:27] mvo: is the person to talk as he wrote that part [11:27] but I see a lot denials in dmesg: https://paste.ubuntu.com/25375999/ [11:27] I'll return to your code review [11:28] mq_timedreceive ? [11:28] or is this an i386 binary? [11:28] it's x86_64 [11:28] so this is mq_timedreceive, weird [11:28] I'll leave that to mvo for now [11:29] but feels like another forum topic to open [11:30] zyga-arch: can I reliable switch back to the older snapd or will that break things even more? [11:30] you should be [11:30] (able to) [11:31] ok, but lets wait for mvo before I do that [11:36] seems there are still issues with the fakestore sometimes [11:39] we probably want to print the logs there [11:39] morphis: I commented on the 3260 PR about one aspect of the spread test [11:39] morphis: can you look please [11:39] I'm looking at the rest [11:47] zyga-suse: when I get to it today, I will for sure :-) [11:47] zyga-suse: thanks [12:02] zyga-suse, morphis: seccomp is allow all mode [12:03] mvo: aha, then something wonky is going on [12:04] morphis: strange, @complain should be fine and should not kill anything, the generated bpf file ".bpf" would be nice, could you mail that to me? [12:05] zyga-suse, morphis: I can look at the test issue in 3260, I'm keen to land that, will also look at the cmd_userd.go bits and see if I can come up with a test [12:06] mvo: Chipaca: as I told Gustavo yesterday I have errands today that overlap with the standup so I won't be there [12:07] pedronis: ok, thank you [12:11] mvo: sure [12:12] mvo: sent [12:12] mvo: so these messages in dmesg are no denials? [12:14] morphis: those are hard denials [12:14] (SIGKILL) [12:14] there's no complain mode for seccomp being used yet [12:15] good, that is what I thought [12:18] morphis: bpf disassembled looks ok, what architecutre is this? amd64? [12:19] yes, amd64 [12:19] mvo: installed with snap install --devmode --edge anbox [12:20] morphis: ok, let me try that [12:22] morphis: I just did a snap install --devmode --edge anbox and systemctl status snap.anbox.container-manager is happy [12:22] morphis: what should I see? [12:34] ppisati, ".config:9871:warning: override: reassigning to symbol ARCH_MEDIATEK" what does that mean ? [12:34] (building a patched arm64 mediatek kernel here, is the above fatal in any way ? i have never seen it before) [12:41] morphis: fyi, syscall 243 is mq_timedreceive and that isn't included in any interfaces. the denial indicates you are not actually in complain mode [12:42] mvo, zyga-arch: ^ [12:42] also, I commented in the forum topic [12:42] morphis: ^ questions for you there [12:42] jdstrand: looking [12:47] mvo: based on morphis' denial, it seems clear that seccomp.NewFilter(seccomp.ActAllow) isn't being run [12:48] mvo: that should be a very small bpf based on lines 616-624 (as you know) [12:49] morphis: can you paste the entirety of /var/lib/snapd/seccomp/bpf/snap.anbox.container-manager.src? [12:53] jdstrand: morphis mailed me the bpf and it looks like this http://paste.ubuntu.com/25376337/ [12:53] I get a change after running get-deps.sh [12:53] jdstrand: so it looks like the normal allow bpf [12:53] golang.org/x/sys/unix changes checksum [12:57] PR snapd#3786 opened: vendor: update vendor.json after (presumed) manual edits [12:57] mvo: ^ [12:57] mvo: that's interesting. seems that the old non-bpf profile is being preferred, which suggests a problem with the reexec code [12:59] jdstrand: oh, that is interessting [12:59] morphis: what do you get with `snap version` on the system that has the anbox problem? [13:00] jdstrand: yes, I was thinking the same thing [13:01] jdstrand: the old profile gets used because the old snap-confine runs [13:11] zyga-suse: based on the forum topic, it looks like morphis has the stable core installed (16-2.26.14) and upgraded to 2.27.3. I tried that exact upgrade path and couldn't reproduce the apparmor bits [13:11] (upgraded t 2.27.3 deb) [13:11] jdstrand: I haven't been able to reproduce that [13:11] jdstrand: but I've seen it happen once in front of my eyes [13:12] mvo, jdstrand: back now, let me read through the messages [13:13] mvo: snap version output is on the forum thread [13:14] zyga-suse: based on the forum topic alone, it looks like the system (not the reexec'd) snap-confine is used [13:14] yes, I agree [13:15] zyga-suse: I wonder if it is a race between the snapd restart and the deb packaging [13:15] mmmm [13:16] re-execing snap (as in snap run) will always use the new snap-confine [13:16]