[00:41] PR snapcraft#1929 closed: sources: proper errors for invalid handlers [00:44] PR snapcraft#1942 opened: Release changelog for 2.39.2 [01:05] snappy-m-o autopkgtest 1926 xenial:armhf xenial:amd64 xenial:arm64 [01:05] elopio: I've just triggered your test. [01:52] snappy-m-o autopkgtest 1942 xenial:armhf xenial:amd64 xenial:arm64 [01:52] elopio: I've just triggered your test. [03:39] yo guys, my snaps seem to not have any internet connection... Spotify says no internet connection and so does Writefull [03:40] i made sure the network plug was enabled [04:41] snappy-m-o autopkgtest 1943 bionic:amd64 bionic:arm64 bionic:armhf [04:41] sergiusens: I've just triggered your test. [04:42] PR snapcraft#1943 opened: Release/2.39.2+18.04 (DO NOT MERGE) [06:07] morning [06:24] mvo: morning [06:24] hey mborzecki ! good morning [06:25] mvo: i noticed this in one of the travis jobs: https://paste.ubuntu.com/p/qYKjfqYdxF/ is this the restart on refresh mode thing? [06:27] mborzecki: yes - looks like a race, do you have the full log? there should be logging to syslog around this to help tracking this down [06:28] mvo: https://api.travis-ci.org/v3/job/343470218/log.txt [06:35] mborzecki: hm, hm, the log looks ok, the right restart reason and all that. puzzling [06:38] mvo: maybe the log comes from before the actual test action happened? [06:39] mborzecki: yeah, I wonder why [06:41] mvo: backend calls systemd.Restart(), that's implemented as stop & start [06:43] mborzecki: oh, thats an interessting clue, a restart with no restart reason. when is backend calling this? [06:44] mvo: i believe the path is ifacestate.setupSnapSecurity() -> backend.Setup() -> systemd.Restart() [06:46] ta [07:45] mvo: do you need any help with that issue? [07:46] mborzecki: help would be great, I'm still fighting with the 2.31.1 stable release and the oom error and the autopkgtests [07:47] mvo: oom error? you got me interested now [07:47] mborzecki: on bionic/i386 the interfaces-many tests triggers an oom error [07:47] mborzecki: its not a new bug, I can reproduce it with 2.29.x too [07:48] mborzecki: but its very unclear right now what is causing it [07:48] mborzecki: its reliable to reproduce on i386 [07:49] mvo: is snapd getting killed? [07:50] mborzecki: everything is getting killed :) its also strange as htop does not show any user space grwoing like mad. also the swtich from ok(ish) to oom comes very fast [07:50] mborzecki: http://paste.ubuntu.com/p/gVgDtTqPBg/ this is the script [07:51] mborzecki: I can prepare a tar that includes the two snaps to run this [07:53] mvo: no need, if they are in tests/lib/snaps i can build them myself [07:54] mborzecki: I will do it anyway I think because I want to run it on an amd64 and compare the logs. it seems like th eoom condition is not triggered there [07:54] mborzecki: might be kernel, might be udev [07:55] mborzecki: each connect seems to take in the range of 12mb [07:55] mborzecki: it could be apparmor too, we load profiles on each connect [07:55] mvo: wow, that's a lot [07:55] mborzecki: well, thats what "free" tells me, its not clear (to me yet) where this memory actually goes [07:56] mvo: i'll try to play with pprof, see if anything comes up [07:57] mvo: there's a bunch of maps flying around in the code, also we may be collecing some output from processses that we run [08:00] mborzecki: my gut feeling is that its external to snapd, we need more data, but afaict this is fine on xenial/i386 [08:01] mvo: interesting, is ti possible to say, disable apparmor and not call apparmor_parser? [08:01] mborzecki: yeah, that might be worth a shot [08:02] mborzecki: its puzzling, free telle me I have "479404" used and "784648" free - thats confusing (but memory reporting apparently is) [08:03] mborzecki: I also see a lot of kmalloc-8192 in slab-info [08:03] mborzecki: the objects there grow from 500 to 5000 [08:03] (which is still not that much in total memory size about 40mb) [08:06] mornings [08:09] mborzecki: my current theory is that for some reason "normal" memory runs out on i386. there is plenty of "highmem" free but the normal memory falls below the threshold of 20mb [08:10] pstolowski: morning [08:10] hey pstolowski ! good morning [08:12] mvo, heya, is this re to the 'killed...' issue on connect from yesterday? [08:15] pstolowski: yes [08:16] interesting [08:17] PR snapd#4676 closed: timeutil: introduce helpers for checking it time falls inside the schedule [08:20] mvo: hm the test only connects the interfaces, so maybe temporarily linking apparmor_parser to /bin/true would allow to skip this path if that's what you suspect [08:23] pstolowski: if you're up for reviews https://github.com/snapcore/snapd/pull/4695 needs looking at :) [08:23] PR #4695: wrappers: generator for systemd OnCalendar schedules [08:26] mvo: xenial i386 vm is updating, meanwhile i'll look at refresh mode thing [08:26] mborzecki, sure! [08:26] pstolowski: great, thanks :) [09:02] mborzecki: I will write a forum post in a wee bit that summarizes my current findinds [09:05] snappy-m-o autopkgtest 1942 xenial:armhf xenial:amd64 xenial:arm64 [09:05] sergiusens: I've just triggered your test. [09:06] sergiusens: morning! (?) [09:06] Chipaca: intermittent, just woke up to trigger these darn tests :-) [09:12] snappy-m-o autopkgtest 1943 bionic:amd64 bionic:arm64 bionic:armhf [09:12] sergiusens: I've just triggered your test. [09:31] PR snapd#4708 opened: overlord/snapstate: use spread in the default refresh schedule [09:36] mvo: so no apparmor no leak? [09:37] mborzecki: thats what it looks like to me at least [09:39] that's interesting observation [09:39] , udev/seccomp/cgroups is all there right? [09:39] mborzecki: yes, I just disabled this one [09:42] PR snapd#4709 opened: tests: fixes for autopkgtest in bionic [09:43] mvo: can you try with something more recent, like 4.15.4? [09:43] mvo: i mean the kernel ofc [09:47] mborzecki: yeah, I am preparing a more automatic thing now [09:48] mborzecki: I mean a tarfile so that one can just run ./test.sh [09:48] mborzecki: and then I can test on more systems (artful etc) to see when it started [09:49] mvo: i did not observe any issues on xenial, but that's a really old kernel [09:51] mborzecki: interessting [09:53] mvo: https://paste.ubuntu.com/p/bWVBw4Yp6v/ [09:53] LowTotal: 858144 kB [09:53] LowFree: 723696 kB [09:53] mborzecki: this is xenial? [09:54] mvo: yes [09:55] mborzecki: it seems to have a different lowmem split [09:55] mborzecki: i.e. more available (my system only had ~400mb) [09:55] mborzecki: is it also shrinking fast? [10:02] mborzecki: same here, no craziness on 16.04-32 [10:03] mvo, hey [10:03] mvo, can you help us? andyrock needs some input on changes he would like to suggest for snapd that he needs for his livepatch work [10:04] mvo: https://imgur.com/a/xDeVR [10:04] seb128: sure, what kind of changes does he have in mind? [10:04] mvo: the test script stars around 20s mark [10:05] mvo: so we need to login in snapd from the installer. The problem is that we can't talk with the to-be-installed snapd in ubiquity [10:06] mvo: the chroot hack does not work (in a nutshell the chroot is not ready when we need it) [10:06] mborzecki: what is the y axis? [10:06] mvo: kB, LowFree [10:06] mborzecki: ta [10:06] mvo: would be possible to have a way to seed the auth state? something like this: https://paste.ubuntu.com/p/sSbKstQ2Sy/ [10:08] andyrock: interesting! pedronis, what do you think about the ability to put auth.json into the seed directory and import it on firstboot? this is for the desktop installer use-case [10:10] mvo: I'm a bit missign the context [10:11] pedronis: I've been asked to add snapd login in the installer [10:11] pedronis: they allow people to login into the store/snapd in the livecd session. but the snapd that is running in there is not the snapd that is copied to the hardisk. the harddisk snapd will have an empty state.json [10:11] pedronis: so the user would have to login again after the system was installed [10:13] what is that auth.json? the one from the homedir? [10:13] or something else [10:13] something else [10:13] the one in the homedir doesn't have all the data [10:13] something we need to take from the livecd state.json [10:14] we can take all under auth: {...} [10:14] no you don't want everything [10:14] because there's device info there [10:14] kk [10:14] and I think we are trying not to make livece session count the same wa [10:14] as full installs [10:18] pedronis: so last-id, users, and macaroon-key? [10:18] or last-id and users are enough? [10:27] andyrock: you need the macaroon-key as well, if you plan to port ~/.snap/auth.json over or the gnome-software equivalent [10:27] kk [10:28] pedronis: so are you willing to accept something like that? [10:28] maybe, is not just my decision [10:29] it needs also to be a bit more paranoid about checking the perms of that file [10:29] kk who else should we ask? [10:31] you need niemeyer +1 as well [10:33] I'm mildly worried that it allows to clone the same creds on a lot of images, it's not your use case but it can be used that way [10:36] pedronis: should we move the discussion the forum? [10:36] *in the [10:37] yes [10:37] you should make a proposal there I suppose === ikey|afk is now known as ikey [10:49] pedronis, andyrock another option might be to join the standup as a guest [10:50] o/ [10:50] I'm sick and I won't be around today [10:50] I just got off the bed to let you guys know [10:50] sorry about that :/ [10:50] zyga_: get well [10:50] * mvo hugs zyga_ [10:50] thank you, I be back as soon as I can [10:51] PR snapd#4704 closed: tests: store journal in autopkgtest artiffacts and add 18.04-32 to qemu [11:09] mborzecki: fwiw, artful seems to show the same behavior [11:10] random question, is there a pattern for dealing with migrating configs and stuff on a snap package upgrade? [11:11] or do you just bake something into the start script? [11:15] mvo: I'm told there's a PR in flight somewhere which will enable a snap (like gnome-calculator) to auto-download & install a platform snap (like the gnome one). Do you know what stage that's at? [11:15] popey: that is #4103 [11:15] PR #4103: snapstate: auto install default-providers for content snaps [11:16] popey: we want to get it in for 2.32, its a bit hard because there is some work to do still but its super important to us [11:16] mvo: excellent, thanks. [11:16] Yes, that would be awesome. [11:32] * pstolowski lunch [11:35] mvo: that problem with test-snapd-service endure, the only idea i have atm is that we do journalctl --rotate, but no journalctl --sync, if a test using test-snapd-service was run before that one, we could end up with a log from the previous run [11:36] mborzecki: interessting, that sounds like a good theory [11:37] funnily enough, looking at the source code of journalctl, you cannot use `journalctl --sync --rotate` in a single command, both flags are set to the same variable [11:45] PR snapd#4710 opened: tests/lib/prepare-restore: sync journal before rotating and vacuuming [11:46] mvo: ^^ a simple fix for now, let's see if this is enough [11:46] PR snapd#4709 closed: tests: fixes for autopkgtest in bionic [11:47] mborzecki: \o/ [12:00] mborzecki: mvo: #4708 gtg [12:00] PR #4708: overlord/snapstate: use spread in the default refresh schedule [12:02] PR snapd#4708 closed: overlord/snapstate: use spread in the default refresh schedule [12:07] Chipaca, mborzecki thanks, cherry-picked into 2.31.1 [12:09] mvo, hey [12:10] hey cachio [12:10] test-snapd-hello-classic in the store for s390x, stil waiting a review [12:10] mvo, updating the kernel in the qemu machine [12:10] cachio: aha, cool, I can do that after lunch [12:30] mvo, same issue with an old kernel [12:30] cachio: oom? [12:31] mborzecki, yes [12:32] cachio: how much ram are those vms getting? [12:32] mborzecki, well, I could not connect anymore [12:32] mborzecki, 3500 MB [12:34] cachio: tried this today on 16.04 32bit, with -mem 4096 and haven't observed any issues, lowmem did drop a bit though, see ~20s mark https://imgur.com/a/xDeVR [12:36] mborzecki, I ran some scripts yesterday and it is like in 18.04 i386 the garbage collections is not done [12:37] so you connect disconnect and do stuff and the memory is not release [12:37] d [12:40] cachio: can you try to collect this `while true; do echo "$(date +%s) $(awk '/LowFree/ {print $2}' < /proc/meminfo)"; sleep 1; done` while the test is run and paste is somewhere? [12:40] mborzecki, sure [13:05] mborzecki, https://paste.ubuntu.com/p/JTBkvh5PqN/ [13:14] hmm it's really low to begin with [13:21] jdstrand: ping [13:22] hey Chipaca [13:22] jdstrand: pm [13:23] jdstrand: hey, https://forum.snapcraft.io/t/oom-for-interfaces-many-on-bionic-i386/4101 might be interessting for you guys, I suspect it is apparmor related (but maybe not) [13:25] stgraber: hi there, was wondering if there is a probe command in lxd to figure out that we have been `init'ed` [13:54] i'm experimenting with `snap set core refrsh.timer=<...>` and it's a bit weird, you see the current setting appear right away in `snap refresh --time` but it becomes effective only after restart or when the current next expires [13:57] mborzecki: shouldn't be like that looking at the code, but it might take 5 minutes to take effect if we are not careful [13:59] pedronis: how often is autorefresh.Ensure() called? [14:02] mvo, mborzecki I ran with this kernel and I for the oom issue too https://paste.ubuntu.com/p/4X9s7jtybd/ [14:02] mborzecki: if nothing is going on, each 5 minutes [14:02] that's where my 5 minutes comes from [14:03] PR snapd#4710 closed: tests/lib/prepare-restore: sync journal before rotating and vacuuming [14:04] cachio: oh, interessting. [14:04] mborzecki: it's ensureInterval in overlord/overlord.go, probably worth getting familiar with that bit [14:04] pedronis: yup, looking at it now [14:05] mborzecki: we have state.EnsureBefore to trigger it earlier, various places use it [14:05] some in taskrunner.go [14:06] right, i've seen this one before :) === pbek_ is now known as pbek [14:07] mvo, is it any way to disable apparmor and make the tests work? [14:07] mvo, to discard that? [14:09] mvo, I'll try just removing it [14:09] cachio: I did: "mv /sbin/apparmor_parser /sbin/apparmor_parser.saved" [14:09] pedronis: there's a funny effect, that once a refresh runs as is successful, next becomes time.Time{}, so when you do `snap refresh --time`, next will be 'n/a' [14:10] cachio: cp /bin/true /sbin/apparmor_parser [14:10] mvo: interesting. As you said, this sounds like a kernel issue since you couldn't reproduce on 4.10 or lower [14:11] mvo, you already tried that and it didn't work? [14:11] mvo: you said 4.10 was not affected. 4.13 (artful) is? why are we only seeing this now? is it just now is when you decided to get to the bottom of artful and bionic issues? [14:11] jdstrand: take it with a grain of salt, cachio said he sees it with 4.10 too, I will double check [14:11] cachio: that worked for me [14:11] cachio: when I moved the apparmor_parser away I did not see the mem leak anymore [14:11] mvo: how much memory is in the vm? [14:12] jdstrand: I think its some factors: the interfaces-many test got added relatively recently, we don't run much on i386 and autopkgtest for 2.30 did not run because we had problems getting this snap accepted into the archive at all [14:12] jdstrand: the vm has 1500mb [14:12] jdstrand: but it seems like the LowMem is what triggers the oom on i386 which is just 400mb [14:12] mvo: this is i386 only/ [14:13] ? [14:13] jdstrand: I suspect its amd64 too, it just takes a lot longer as it has the full 1500mb of memory [14:13] jdstrand: for some reason its the lowmem on i386 that runs out [14:13] jdstrand: the oom-test.tar.gz should have all that is needed to reproduce [14:14] jdstrand: let me run it (in a loop) on amd64 to see if it eventually also dies [14:14] mvo: yes. I'd like to remove snapd from the equation though so I commented in the topic [14:15] jdstrand: I see ~15mb per "snap connect" decrease in free memory with 4.13. maybe with 4.10 (and below) the leak is just smaller that is why cachio sees it (because he was looking harder) and I don't (because I was just looking for the oom to trigger) [14:15] mvo: the interfaces-many test is a bit of a hog, it has so many changes that towards the end it takes seconds to do each one [14:16] Chipaca: indeed [14:16] jdstrand: I run it now on amd64 in a loop just to see what will happen (bionic with 4.13) [14:16] mvo: what if you try to reload apparmor profiles in a loop? [14:16] mvo: note that an apparmor profile does take kernel memory. it should not take that much and it shouldn't lose 15mb with a replace operation [14:17] jdstrand: *nod* [14:18] off to pick up the kids [14:26] jdstrand: fwiw, it looks like amd64 is also affect, just takes longer because of the different memory available (1480 vs 390) [14:28] mvo, https://paste.ubuntu.com/p/YsCpXvfgNg/ [14:28] 56MB free [14:28] preparing the suite [14:36] re [14:49] snappy-m-o autopkgtest 1942 xenial:armhf [14:49] sergiusens: I've just triggered your test. === rharper` is now known as rharper [15:18] cachio: yeah, same here, eventually amd64 also runs out of memory [15:22] mvo, I am gonna try with an older bionic image [15:23] cachio: thank you [15:24] mvo: https://forum.snapcraft.io/t/oom-for-interfaces-many-on-bionic-i386/4101/5 [15:25] oh heh, you already liked it [15:25] jdstrand: very much indeed! thanks for the reproducer [15:25] cachio: -^ [15:25] cachio: looks like jdstrand has an even better reproducer that does not involve snapd, probably easy(ish) to bisect using this [15:26] Hi guys! I installed bluez in Ubuntu Core Snappy and I'm having this problem: [15:26] bluez.sdptool browse local [15:26] Failed to connect to SDP server on FF:FF:FF:00:00:00: No such file or directory [15:26] mvo: I think for now the test should be disabled on i386 [15:26] Does anyone knows how can I solve this issue? [15:26] mvo: the test really only needs to be run on one arch imho [15:27] yes+ [15:27] well [15:27] the snapd part of the test is only useful on one arch [15:28] * mvo nods [15:28] the fact that it showed an issue on i386 is certainly helpful :) [15:38] o/ can anyone help me with this: https://forum.snapcraft.io/t/refreshing-disabled-snap-kubectl-not-supported/4060/11 [15:39] Hi guys! I installed bluez in Ubuntu Core Snappy and I'm having this problem: $bluez.sdptool browse local Failed to connect to SDP server on FF:FF:FF:00:00:00: No such file or directory [15:39] Does anyone knows how can I solve this issue? [15:40] brunosfer, it is probably best to ask this on the forum ... koza can probably help but i think he is not around currently (so he can more easily pick up the question on the forum later) [15:40] also make sure to describe what HW you run and if all your interfaces are connected properly etc [15:41] how much later? because I posted that issue since December 17th and so far I'm stuck and nobody replied... [15:42] brunosfer, where did you post it ? [15:42] Chipaca: o/ [15:42] Everytime I make this question here people redirect me to the forum and it's a dead end... [15:42] (got a link to the forum post ?) [15:42] https://forum.snapcraft.io/t/failed-to-connect-to-sdp-server-permission-problem/3085 [15:47] hm, hm, today https://forum.snapcraft.io/t/all-revisions-of-snaps-are-mounted-when-they-dont-need-to-be/2294 came up, I wonder if anyone ever looked what is taking >500ms [15:48] mvo: fyi, jjohansen is aware of a profile replace memory leak and will continue to look into it [15:55] marcoceppi: o/! [15:55] marcoceppi: how's things? [15:55] marcoceppi: so, tell me more about this messed-up snapd you have [15:55] great except for my broken snaps :( [15:55] I mean, it's my laptop [15:55] marcoceppi: ok. Can you enabled debug, if you haven't already? [15:56] enable* [15:56] 16.04 full patched, few days ago a new kubectl snap was published and since then it's not been well [15:56] marcoceppi: that's add SNAPD_DEBUG=1 to /etc/environment and 'systemctl restart snapd' [15:56] yeah, restarting snapd hangs [15:57] niiice [15:57] marcoceppi: hangs how? [15:57] marcoceppi: what exactly is broken? does snapd itself misbehave? or one (or muliple) snaps? [15:57] mvo: https://forum.snapcraft.io/t/refreshing-disabled-snap-kubectl-not-supported/4060/11 [15:57] mvo: https://forum.snapcraft.io/t/refreshing-disabled-snap-kubectl-not-supported/4060/7 [15:57] Chipaca: restart command blocks indefinitely [15:57] haha [15:57] youch [15:57] marcoceppi: what does 'journalctl -u snapd' show? [15:57] I can straight up kill it [15:58] Chipaca: https://paste.ubuntu.com/p/fDhqFCPGkC/ [15:59] marcoceppi: is that the _end_ of the journal? [15:59] yeah, I mean, I just did an -f and it's printing a few more things [15:59] like sysctl just SIGKILL'd the proc [15:59] with the SNAPD_DEBUG, and it hanging, i'd expect lots of stuff [15:59] nnngh [15:59] oh, no, still waiting for it to restart [15:59] so it can get SNAPD_DEBUG [16:00] ok, so just kill it [16:00] something is bad [16:00] * marcoceppi nods [16:00] and the oom will kill one thread and leave go wondering [16:00] so when that happens you need to manually kill the whole thing [16:00] kubectl is classic fwiw [16:00] ack [16:01] yes, but lxd isnt pedronis and it has the same problem [16:01] pedronis: i mean, if it were just the snap misbehaving i'd kick it over to sergio :-) [16:01] pedronis: but this is snapd doing something weeeird [16:01] but you said it started when you got the new kubectl [16:01] that's when I noticed it exhibiting issues, yes [16:01] marcoceppi: how's the kill-everything-and-start-over going? [16:01] it's started [16:02] Chipaca: https://paste.ubuntu.com/p/vR9Nky4bhx/ [16:03] marcoceppi: can you 'curl -s -N -0 --unix-socket /run/snapd.socket http://localhost/v2/changes?select=all | python -m json.tool | pastebinit'? [16:03] marcoceppi: assuming you have pastebinit, otherwise you know what i mean [16:03] that's a mouthful [16:04] http://paste.ubuntu.com/p/nRTM6jyyvT/ [16:06] marcoceppi: can you do a 'sudo du -sh /root/snap/* /home/user/snap/* /var/snap/*' ? [16:06] (depending on your permissions you might need to tweak that for the glob to expand everywhere) [16:07] Chipaca: https://paste.ubuntu.com/p/HNq9hGGm66/ [16:08] marcoceppi: if you say 'snap watch 481' does it still say it's copying stuff? [16:08] yes [16:08] marcoceppi: that's kubefed, yes? [16:08] correct [16:08] marcoceppi: so my question for you is, how does it take this long to copy 28k of data [16:09] well, that's my question for you - isn't it :D [16:09] SSD with over 60GB of free space on EXT4 [16:09] marcoceppi: can you try copying that directory somewhere, by hand? ie 'cp -a /var/snap/kubefed /var/tmp' ? [16:10] Chipaca: [16:10] time cp -a /var/snap/kubefed /var/tmp/ [16:10] real 0m0.003s [16:10] user 0m0.000s [16:10] sys 0m0.000s [16:10] worked without issue [16:11] mvo, using an 2 months old bionic image I see the same oom error [16:11] cachio: ta [16:11] marcoceppi: the other one you had in doing was copying the lxd data [16:11] marcoceppi: that one could take a bit longer, it being 5 gigs [16:12] cachio: jdstrand told me the memory leak is something that jjohansen is investigating. maybe he knows (roughly) when this started ? if not we might be able to help bisecting [16:12] yeah, but it's been over 24 hours [16:12] and LXD was a problem after 24 hours of the kubectl issue [16:12] I feel like snapd is unable to perform this action [16:13] marcoceppi: is there anything new in the logs? [16:13] Feb 20 11:03:59 T430 snapd[28114]: 2018/02/20 11:03:59.146915 daemon.go:233: DEBUG: pid=28499;uid=1000;@ GET /v2/changes?select=all 1.369681ms 200 [16:13] that is the only new line [16:13] that's you [16:14] yes, I realize that, but wanted to be complete in my reporting [16:14] yes yes, i was just confirming [16:14] I don't understand what it thinks it's doing [16:14] can I - like delete this change attempt and just start over? [16:15] I really /really/ DO not want to lose my lxd containers [16:15] but kubectl has no userdata [16:15] marcoceppi: i mean, you could, but it'd be hairy [16:15] marcoceppi: can you stop snapd, and start it straced? [16:16] you got what I should invoke for snapd to be strace'd? [16:17] marcoceppi: so, first, systemctl stop snapd.\* [16:18] mvo, so, are you preparing 2.31.1 today, or we need to wait until this is fixed? [16:18] marcoceppi: and then: sudo strace -E SNAPD_DEBUG=1 -f -s 9999 -o /tmp/snapd.trace /usr/lib/snapd/snapd [16:18] mvo, or any other fix? [16:20] Chipaca: 5G in versioned data, that's a problem in itself, we though common is for that [16:20] pedronis: is it versioned? [16:20] well if it's being copied [16:20] we don't copy common data, no? [16:20] we don't [16:20]