[06:35] <mborzecki> morning
[07:36] <mborzecki> re
[07:36] <mborzecki> hate the morning traffic
[08:04] <pstolowski> morning
[08:06] <mborzecki> pstolowski: hey
[08:08] <mborzecki> pstolowski: can you take a look at https://github.com/snapcore/snapd/pull/7702 ? maybe we could land it
[08:08] <mup> PR #7702: tests: adding fedora 31 <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7702>
[08:08] <pstolowski> mborzecki: sure, will do
[08:08] <mborzecki> pstolowski: thanks!
[10:11] <mborzecki> quick errand, brb
[10:39] <pstolowski> hmm something is not right with tumbleweed
[10:54] <mborzecki> re
[10:55] <mborzecki> pstolowski: got logs?
[10:59] <pstolowski> mborzecki: https://paste.ubuntu.com/p/PkdyMvHYjT/
[10:59] <pstolowski> home/gopath/src/github.com/snapcore/snapd/tests/lib/quiet.sh: line 30:   993 Segmentation fault      (core dumped) "$@" &> "$tf"
[11:00] <mborzecki> duh, zypper segfaulted?
[11:01] <pstolowski> mborzecki: happened on 2 independent PRs and now when run manually
[11:02] <pstolowski> although i have no proof it was the same error; but they were definaltely tumbleweed prepare failures
[11:47] <om26er> popey ping! re: building close-source project's snaps with launchpad, did you get a chance to make a video for that ?
[11:47] <om26er> or popey_ ^
[11:59] <mup> PR snapd#7731 opened: usersession/userd: add "apt" to the white list of URL schemes handled by xdg-open (LP: #1776873) <Created by oSoMoN> <https://github.com/snapcore/snapd/pull/7731>
[12:51] <zyga> o/
[12:51] <zyga> good morning
[13:00] <pstolowski> hey zyga!
[13:00] <zyga> hey guys, how's the day going?
[13:02] <zyga> mborzecki: can we chat here
[13:03] <mborzecki> zyga: call would be quicker ;) unless you're in pyjamas or whatnot
[13:03] <zyga> mborzecki: yes but yes :)
[13:03] <zyga> ok, let's use meet
[13:04] <zyga> https://meet.google.com/mym-ngpj-ynj
[13:46]  * zyga goes for breakfast
[13:50] <ijohnson> morning folks
[14:02] <zyga> hey ijohnson
[14:02] <ijohnson> morning zyga
[14:02] <roadmr> zyga: no breakfast yet 😢
[14:02] <roadmr> (unless you go out?)
[14:03] <zyga> roadmr: yeah, I had a call with my family; I think brekfast starts later
[14:03] <zyga> oh
[14:03] <zyga> shiny
[14:03] <zyga> https://www.apple.com/macbook-pro-16/
[14:03] <roadmr> zyga: yep need to wait 27 min :)
[14:04] <roadmr> zyga: haha magic keyboard ?? I anticipate the "yes, magic - press a key, who knows what'll come out on screen 🤣  " memes
[14:05]  * zyga sells his 15" 
[14:05] <roadmr> 🤑
[14:06] <zyga> 64GB ram maximum, wow
[14:08] <zyga> pstolowski: ^
[14:09] <pstolowski> zyga: uhm, your predictions were right
[14:09] <zyga> pstolowski: I will call it the vim macbook
[14:09] <zyga> pstolowski: it has physical escape key :)
[14:10] <zyga> pstolowski: but geez, if the keyboard is now "regular" this is a super shiny machine
[14:10] <zyga> pstolowski: 8TB SSD max, 64GB RAM max
[14:10] <pstolowski> zyga: wooot, esc? no way :D
[14:10] <zyga> pstolowski: (obviously $$$$ but still)
[14:10] <zyga> pstolowski: yep, physical escape key + rest of touchbar as before
[14:10] <zyga> pstolowski: but the base model is not unlike other laptops from dell / lenovo
[14:11] <zyga> pstolowski: 11-14K PLN for a really nice setup
[14:13] <pstolowski> zyga: i see it replaced 15' in the store, only 13 & 16 now
[14:15] <zyga> Yes, it seems so
[14:15] <zyga> I wonder what the GPU is
[14:20] <pstolowski> zyga: huh, going with 32GB and ssd >1TB makes the price skyrocket
[14:21] <mborzecki> cachio: is there a more recent tumbleweed image that we could use by any chance?
[14:22] <cachio> mborzecki, so, we are creating tumbleweed based on the leap 15.0
[14:22] <pstolowski> cachio: re install slowness, please grab 'snap debug timings <change id>'
[14:22] <cachio> mborzecki, and then we update it weekly
[14:23] <cachio> pstolowski, sure
[14:23] <cachio> mborzecki, which is the problem with the current one?
[14:23] <cachio> perhaps we could create a new one
[14:24] <mborzecki> cachio: this failed for instance https://api.travis-ci.org/v3/job/611337121/log.txt
[14:24] <mborzecki> cachio: this is the relevant part https://paste.ubuntu.com/p/VGYvRc2wG4/
[14:25] <mborzecki> cachio: looks like there's a segfault in zypper (?)
[14:25] <mborzecki> cachio: although when i run a shell manually, i don't seem to be able to reproduce it with the same zypper command on a clean image
[14:25] <cachio> mborzecki, mborzecki I could create a new one based on leap 15.1 latest image
[14:26] <cachio> mborzecki, not sure if it is gonna help
[14:26] <mborzecki> cachio: do you create a new image from scratch each time, or do you rather keep updating the same image?
[14:27]  * ijohnson drools over the new macbook
[14:27] <cachio> I initially created a tumbleweed image from leap 15.0
[14:27] <cachio> this is called base image
[14:27] <cachio> then every week I update that and install all the test dependencies
[14:27] <cachio> in a new image
[14:27] <cachio> the final image
[14:28] <cachio> this is the image we use for snapd
[14:28] <cachio> mborzecki, sometimes I create new base images or update the base image
[14:30] <cachio> mborzecki, what I see is that opensuse-cloud added an image for leap 15.1 that we could use
[14:31] <zyga> pstolowski: the important part is what the baseline has
[14:31] <zyga> No more 256GB 8GB model
[14:31] <zyga> It is a great upgrade!
[14:32] <mborzecki> cachio: can you maybe try and update the current image to the latest TW snapshot?
[14:32] <cachio> mborzecki, sure
[14:32] <mborzecki> cachio: hmm looks like we're using 20191109 already
[14:33]  * ijohnson stops drooling after looking at prices and gets back to work
[14:33] <cachio> mborzecki, yes
[14:34] <cachio> mborzecki, I updated 2 days ago
[14:34] <pstolowski> zyga: for the gpu it apparently has AMD Radeon Pro 5300M and 5600M with 4GB DDR6
[14:36] <mborzecki> zyga: looks like tehre's some updates, i think it's worth trying the latest snapshot anyway
[14:37] <mborzecki> pstolowski: still i9 cpu booo
[14:38] <mborzecki> pstolowski: zyga: wonder is i9 is vulnerable https://mdsattacks.com/#ridl-ng
[14:39] <zyga> pstolowski: not sure how that GPU compares to consumer models
[14:39] <mborzecki> zyga: i'm sure it's not as hot
[14:39] <zyga> pstolowski: i9 -> no zen 2nd gen mobile yet, also very unlikely apple would move to amd on a laptop still
[14:39] <zyga> mborzecki: well, apple store is 200 m away
[14:40] <zyga> I'll check it out :)
[14:40] <mborzecki> hahah
[14:40] <pstolowski> zyga: :)
[14:41] <pstolowski> i can sense zyga returning from the sprint with a new laptop ;)
[14:41] <zyga> pstolowski: never, I want to get it back home as proper purchase with vat and stuff
[14:41] <zyga> pstolowski: here it'd be just more complex
[14:41] <mborzecki> zyga: for sure it's gonna hit the new split payment regulations :)
[14:41] <zyga> mborzecki: mhm
[14:42] <zyga> mborzecki: I really really wonder how that works now
[14:42] <pstolowski> yeah it's confusing
[14:44] <zyga> I cannot deny that it is tempting though
[14:44] <zyga> it looks like great piece of hardware
[14:44] <pstolowski> degville: hey, i think https://snapcraft.io/docs/hotplug-support needs a minor update, the feature is still behind experimental flag and not made widely available with 2.39 as indicated there
[14:45] <cachio> pstolowski, https://paste.ubuntu.com/p/Y6TY4WMYXG/
[14:45] <pstolowski> zyga: my santa is not reach enough ;). maybe next year..
[14:45] <degville> pstolowski: thanks for letting me know -I'll update it now.
[14:45] <pstolowski> *rich
[14:45] <zyga> pstolowski: my santa is the same, it would depend on selling the 15" first
[14:48] <cachio> pstolowski, id: 25
[14:48] <cachio> and 24
[14:49] <pstolowski> cachio: thanks, looking
[14:54] <pstolowski> cachio: it takes 1.5s on my local VM. i'm not sure there is anything wrong there. great chunk of that is seccomp profiles compiler. note this snap has 9 apps in it
[14:55] <pstolowski> cachio: not sure why copy-snap-data took 1.7s though. was there any old data?
[14:55] <cachio> pstolowski, yes, but it takes more than 2 seconds
[14:55] <cachio> seems to be so much compared with the other times
[14:56] <pstolowski> cachio: what do you mean by other times?
[14:56] <cachio> pstolowski, compared with the other steps and with apparmor
[14:57] <pstolowski> cachio: ah, sure. yes, seccomp compiler got very slow a few months ago with a new version
[14:57] <cachio> apparmor= 725ms seccomp=2201ms
[14:57] <pstolowski> cachio: a few weeks ago i presented some tests on the standup, seccomp got a few times slower afair
[14:59] <pstolowski> cachio: so yes, setup-profiles is one of the most expensive tasks, a few seconds is normal, unfortunately
[15:00] <cachio> pstolowski, https://paste.ubuntu.com/p/yYnDCPpm7m/
[15:00] <pstolowski> degville: thanks for updating hotplug doc!
[15:00] <cachio> in this case I installed over
[15:00] <degville> pstolowski: np!
[15:00] <cachio> and didnt setup seccomp
[15:01] <cachio> but did the setup for apparmor
[15:01] <cachio> pstolowski, do you know why?
[15:07] <pstolowski> cachio: seccomp backend computes new profiles and compares them with what's already on the disk, if there is no change then they are not reloaded. same logic applies to apparmor, i suppose they were reloaded because they have snap paths (specific to snap revision) in them
[15:13] <cachio> pstolowski, still weird that if it computes the new profiles it makes that super fast
[15:13] <cachio> and also it does not appear in the timings
[15:14] <pstolowski> cachio: we don't save timings if something is faster than 5ms
[15:14] <cachio> pstolowski, we have some stuff with 4ms
[15:14] <pstolowski> cachio: computing profiles is fast, loading them with seccomp profiles is slow
[15:15] <cachio> pstolowski, ahh
[15:16] <pstolowski> cachio: sorry, i wasn't precise: we will still record times of individual tasks (like 4ms with post refresh hook), but we don't store detailed timing breakdown (nested timings) underneath
[15:16] <cachio> pstolowski, ah, ok
[15:16] <cachio> pstolowski, thanks for the explanation
[15:17] <pstolowski> cachio: if we compute a profile and find it's the same, we don't invoke seccomp or apparmor parser at all
[15:19] <cachio> pstolowski, perhaps we could see why we are calling apparmot
[15:19] <cachio> it I am installing the same snap twice
[15:20] <pstolowski> cachio: but it ends up with different revision, no?
[15:21] <cachio> pstolowski, ye
[15:21] <cachio> s
[15:21] <pstolowski> cachio: so, paths are differnt in the aa profiles
[15:21] <cachio> pstolowski, ah, ok
[15:22] <cachio> makes sense
[15:27] <cachio> pstolowski, so seccomp takes more time depending on the number of apps the snap has?
[15:28] <cachio> pstolowski, for example the snap test-snapd-framebuffer with 2 apps and 1 interface takes 493ms on seccomp
[15:28] <cachio> and 0ms to copy snap data
[15:31] <cachio> pstolowski, and first time I saw: 1791ms            -  Copy snap "test-snapd-tools" data
[15:31] <cachio> perhaps we could update the snap to make it faster to be installed
[15:32] <cachio> because it is being used on many tests
[15:34] <pstolowski> cachio: what is the data of this snap? it seems we're copying data over to the new revision in the test, maybe it makes no sense
[15:35] <pstolowski> cachio: i mean, maybe we should simply rm -rf .. on snap data to avoid copying
[15:39] <cachio> pstolowski, I think in the test there is something weird
[15:39] <cachio> I ran the samein the pi3 and got 2212ms            -    setup security backend "seccomp" for snap "test-snapd-tools-core18"
[15:39] <cachio> seccom takes same time on pi3 than in the vm
[15:39] <cachio> hhehe
[15:41] <cachio> pstolowski, take a look to this please https://paste.ubuntu.com/p/MKD5MCTtxn/
[15:42] <cachio> pstolowski, do we use different seccomps for core than classic?
[15:46] <pstolowski> cachio: is it a fast vm?
[15:50] <cachio> pstolowski, it is same we use for tests on google
[15:51] <cachio> pstolowski, but it should be faster then the pi3
[15:52] <cachio> in fact in the pi3 apparmor takes more than 2 seoncds
[15:52] <cachio> compared with 700ms on the vm
[15:52] <cachio> but seccomp is the same time in both
[15:55] <pstolowski> cachio: i see. i don't know why. this the time of seccomp compiler itself. maybe mborzecki or zyga know what is it doing that explains no difference on pi3 vs vm
[15:55] <mborzecki> pstolowski:  hm? why it's so slow?
[15:56] <cachio> mborzecki, well, in pi3 is slow as in a vm
[15:56] <pstolowski> mborzecki: why it takes same time on pi3 vs vm
[15:56] <cachio> mborzecki, https://paste.ubuntu.com/p/MKD5MCTtxn/
[15:56] <pstolowski> mborzecki: where it should in theory be faster on vm
[15:57] <cachio> pstolowski, but apparmor is much faster in the vm
[15:57] <cachio> mborzecki, ~
[15:57] <mborzecki> pstolowski: iirc it's cpu bound and single threaded
[15:57] <pstolowski> mborzecki: aah, here you go
[15:57] <cachio> mborzecki, ahh, that explains it
[15:58] <cachio> mborzecki, pstolowski thanks for the explanations
[15:58] <mborzecki> cachio: pstolowski: we could run the compiler in parallel, but you'd only benefit when there's more than 1 app
[15:59] <mborzecki> setting up security backends in parallel would give more gains i believe
[15:59] <cachio> mborzecki, for testing that should be really usefull
[15:59] <pstolowski> yep.. test-snapd-tools has 9 apps ;)
[15:59] <cachio> is it possible to so it by configuration?
[15:59] <pstolowski> cachio: no, needs coding
[15:59] <cachio> pstolowski, :(
[15:59] <zyga> cachio, pstolowski: snap-seccomp needs a cache layer
[15:59] <zyga> virtually all snaps use the same seccomp profile in the end
[15:59] <mborzecki> pstolowski: cachio: i had the patches somewhere, iirc i pointed pawel to the commits at some point
[16:00] <zyga> there is far less variability compared to apparmor
[16:00] <mborzecki> cachio: if you feel like it i can try to revive the branch with paralell compilation of seccomp profiles and we can try to benchmark that
[16:00] <mborzecki> brb
[16:00] <cachio> mborzecki, that should be really nice
[16:01] <cachio> on tests we have many snaps with more than 1 app and reviewing times I see most of the time it is installing snaps
[16:01] <cachio> and most of that time it is creating seccomp profiles
[16:02] <cachio> mborzecki, if you have a branch I could try it
[16:02] <cachio> and see the time improvement
[21:06] <mborzecki> ijohnson: hi, still around?
[21:06] <ijohnson> hey mborzecki
[21:06] <mup> PR snapd#7732 opened: [PoC] many: extracted snaps mode <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7732>
[21:06] <mborzecki> ijohnson: opened a branch for you ^^
[21:06] <ijohnson> thanks mborzecki, will give it a spin
[21:07] <mborzecki> ijohnson: i think it should mostly work, but you may see some issues with layouts, iirc some logic depends on statfs()ing some paths and observing squashfs magic there
[21:08] <mborzecki> ijohnson: and you'll need to set SNAPD_EXTRACT_SNAPS in your environment
[21:08] <ijohnson> right I see that from the patch
[21:08] <ijohnson> that's interesting about layouts needing to introspect the squashfs
[21:08] <ijohnson> I'll see what I can find in any case
[21:10] <mborzecki> ijohnson: if you see bugs or issues feel free to patch it ofc :P
[21:10] <ijohnson> sounds good
[21:12] <mborzecki> ijohnson: thanks! really looking forward to the numbers, i suppose next thing we could try afterwards is repacking the squashfs locally
[21:13] <ijohnson> yeah I don't want to get too far into repacking without Samuele's blessing however :-)
[21:15] <mborzecki> ijohnson: yup, agreed, this should give us some insight though
[21:15] <ijohnson> indeed
[21:16] <mborzecki> all right, wrapping it up, till tomorrow
[22:11] <mup> PR snapd#7733 opened: tests: disable nova from install-snaps test <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7733>
[22:28] <mup> PR snapcraft#2786 closed: cli: add support for 'http-proxy' and 'https-proxy' parameters <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2786>
[22:46] <mup> PR snapcraft#2769 closed: extensions: skip icon cache creation for theme and runtime snaps  <Created by galgalesh> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2769>