zygagood morning05:56
mborzeckizyga: morning06:06
zygahey :)06:06
mborzeckimvo: hey06:33
mvohey mborzecki06:34
zygahey mvo06:45
zygaquick comment, do you remember the remodel kernel work? there's a test that reliably fails in osc build, there's a trello card for it06:45
zygaI wanted to look now but if you know more I'd love to get help06:45
mvozyga: can you point me to the log?06:46
mvozyga: the remodel kernel test takes a while iirc, we had to increase the settle timeout on arm06:46
mvozyga: is it maybe that?06:46
zygathat's the same tet06:46
zygabut it failed on my xeon ;D06:46
mvoalso with a timeout?06:48
zygait's hard to say, there's very little useful stuff when settle fails06:48
zygaI complained that it should not be time-based at all but this is not a simple problem to solve06:48
zygamvo: https://trello.com/c/foU3iOrs/321-investigate-testremodelswitchtodifferentkernel-failure06:49
zygamvo: the whole output is06:52
mvozyga: this happens on your opensuse as well?06:52
zygano, it only happens in 'osc build'06:52
zygathough my master is "after" 2.4206:52
mvozyga: is it easy to submit test builds? I could add something that prints more debug06:52
zygamvo: it's not remote06:52
zygait's local06:52
zygamvo: it's like dpkg-buildpackage06:53
zygamvo: anything extra can be added, just make a patch against 2.4206:53
zygamvo: if you push a branch I can extract the patch and run it easily06:53
zygamvo: I would like to make some general changes to the tests later on06:53
zygathough I doubt this will be fasst06:53
zyga1) log all handlers that executed during failed test06:54
zyga2) change settle in tests to just do more stuff as long as handlers want to run, remove the timeout entirely06:54
zyga3) patch tests that measure failure to introduce callback to break further loops06:54
zygacurrent time based code is simply not reliable and not engineered correctly as tests IMO06:55
=== pstolowski|afk is now known as pstolowski
zygapstolowski: hey :)07:22
zygaman, I'm so sleepy today07:22
zygathe rain, the low pressure07:23
zygacoffee, sorry07:23
mvohey pstolowski - good morning!07:24
zygamvo: sorry if I came across cranky, I just think we need to acknowledge shortcomings and not just paper over them with "this is how it's been done"07:24
mvozyga: not cranky at all07:25
mvozyga: let me just finish this one thing here and then I can do something, probably something like for _, t := range chg.Tasks() { fmt.Printl(t.Kind(), t.Status()) }07:26
zygaflashing the card again, last night I had issues and somehow I could not get the card to copy from macos07:29
zyga(new sandbox for all apps)07:29
zygamvo: macos catalina switched to ... read only image as root filesystem07:29
zygamvo: yeah :)07:29
zygamvo: ubuntu personal07:29
zygamvo: /Users (/home for macos) and /Applications are writable07:30
zygarest is not07:30
zygaand they live on a different apfs dataset (like zfs thing)07:30
mborzeckizyga: updated #757107:38
mupPR #7571: sandbox/cgroup: refactor process cgroup helper to support v2 and named hierarchies <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7571>07:38
mborzeckizyga: is there a place like /var/log on osx where system logs land? wonder how it's combined with the read only filesystem image07:39
zygamborzecki: actually macos has /home /var /usr and all that07:39
zygathey are just hidden07:39
zygathis is the current mount table07:39
zygamborzecki: and yeah, there is /var/log07:40
zygamborzecki: this is also interesting07:41
zyganote that /home is /System/Volumes/Data/home07:42
zygawhere Data is the new writable part of apfs07:42
zygamborzecki: the new PR looks nice07:44
zygamborzecki: approved07:46
mborzeckizyga: thx07:49
pedronismvo: hi, I made this comment:  https://github.com/snapcore/snapd/pull/7538#issuecomment-539882778 let me know if you prefer me to go over line by line07:50
mupPR #7538: tests: use `snap model` instead of `snap known model` in tests <Created by mvo5> <https://github.com/snapcore/snapd/pull/7538>07:50
mupPR snapd#7575 closed: cmd/model: add authority-id to verbose fields <Needs Samuele review> <Simple 😃> <Created by anonymouse64> <Closed by pedronis> <https://github.com/snapcore/snapd/pull/7575>07:55
mvopedronis: thanks, looking now08:00
mvopedronis: no need to go over it line by line, thanks, I will make the changes08:02
mvozyga: I created github.com/mvo5/snappy:debug-remodel-kernel-zyga with some extra debug (quite small right now). please run when you get a chance and let me know what it outputs08:08
zygamvo: thank you08:08
zygamvo: I've rebased on 2.42, trying now08:10
zygamvo: new log https://www.irccloud.com/pastebin/EUXdHkdl/08:13
mvozyga: aha, looks like something in the mock restart is not working08:15
zygamvo: I'm logged into the image from rogpeppe08:21
zygarefreshing snaps now08:21
zygawe'll know in a second if it crashes on reboot08:21
rogpeppezyga: sometimes it doesn't :)08:22
zygarogpeppe: fingers crossed08:22
zygaI have serial output08:22
zygaso we'll know what is attempted at least08:22
rogpeppezyga: the other thing is that it surely shouldn't be rebooting very often08:22
zygaand I should send the raspi-tool to snapd into some contrib/ directory08:22
rogpeppezyga: how often would you expect? is it usual for it to try to reboot once or twice a day?08:23
mvozyga: will look into this a little bit08:23
zygarogpeppe: not sure, one sec08:23
zygaChipaca: from that device08:23
mvorogpeppe: the only valid reason to reboot that often is if it tracks edge for core/kernel everything else is most likely a bug08:24
zygaChipaca: system-shutdown messages https://www.irccloud.com/pastebin/sSWsFr69/08:24
mvorogpeppe: thank you so much for helping us tracking this down!08:24
mvozyga: thank you, super curious about this08:24
rogpeppemvo: i hope you do! :)08:24
zygarogpeppe: it updated to 2.42 correctly08:24
Chipacazyga: those messages are fine08:24
zygait reboots instantly?08:24
zygaI snap refreshed08:25
zygait rebooted08:25
zygathen it reboots again08:25
zygastraight away08:25
zygamvo: suggestion for improvement08:26
zygamvo: uboot script should log basics08:26
zygalike booting that kernel and this base08:26
zygaand in this mode08:26
zygawould be great to see08:26
zygamvo: snapd change after update08:27
zygait seems that 2nd reboot was the only one08:27
zygasnap change on successful 2nd reboot https://www.irccloud.com/pastebin/4X2z8O6R/08:27
zygaI'll reboot manually to see if it boots ok08:27
zygarogpeppe: sadly, it works :/08:29
pedronismborzecki: question in #744308:32
mupPR #7443: timeutil: fix schedules with ambiguous nth weekday spans <Bug> <Needs Samuele review> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7443>08:32
zygarogpeppe: I'll leave it running08:32
rogpeppezyga: one thing you could do is just leave it running08:32
rogpeppezyga: lol08:32
zygarogpeppe: to see how it ages for a few days08:32
rogpeppezyga: yup. i guess it might be that we've got two pi's both with buggy h/w08:33
zygarogpeppe: could it be the SD card?08:33
Chipacacould it be that the additional hardware somehow destabilises the pies?08:33
rogpeppezyga: i used a different SD card too08:34
zygahow is the hardware connecteD?08:34
rogpeppezyga: and it's a decent quality SD card, not much used.08:34
rogpeppezyga: connected to the network?08:34
zygano, any custom stuff added to your pi?08:34
zyganot network08:34
rogpeppezyga: nothing custom added08:35
Chipacathere goes that hypothesis then :)08:35
rogpeppezyga: there was an RTC and a pi-glow on the old pi, but not on this one08:36
zygaI can send you my pi if you want to swap :)08:36
rogpeppezyga: that's an interesting thought. the other thing i might try (sorry) is to use a different OS.08:37
zygathat would be an interesting data point as well08:37
pstolowskiChipaca: hey, health-check task is not executed on install if flags&skipConfigure != 0 (because we return early and don't schedule config hook - see the bottom of doInstall); intended or a bug?08:52
mborzeckipedronis: iirc it's not strictly needed, though i kind of highlights the thing happens every day08:54
pedronismborzecki: highlights or confuses people like me :)08:55
mborzeckipedronis: hah ok, i can drop it08:55
pedronismborzecki: one possible reading is that it triggers every day of the month08:55
pedronisanyway with /108:55
pedronisthe "spec" is not great08:55
mborzeckiduh, it is not :/08:56
pedronismborzecki: if it's not strictly needed I would drop it, less mental burden to read it. The spec says " Values may be suffixed with "/" and a repetition value, which indicates that the value itself and the value plus all multiples of the repetition value are matched. Two values separated by ".." may be used to indicate a range of values; ranges may also be followed with "/" and a repetition value." of which I'm not sure how to09:01
pedronisinterpret in practice09:01
mborzeckii'm looking at the spec, and we may have a bug there, at laest on older systemd versions09:02
mborzeckithere's no .. range on xenial :/09:02
mborzeckibut out snap with timers does not fail, so, the bug is only for numbered ranges09:03
mborzeckianother conclusion is that since we had no bug reports or complains about snap install failing, then people do not use it at least in snap.yaml09:04
pedronismborzecki: we'll need to update this, right? https://forum.snapcraft.io/t/timer-string-format/656209:06
pedronisalso  before you could do mon1-fri2,  right?   now you need something like  mon1-sun,,mon2-fri ?09:07
mborzeckiyes, mon1-fri or mon-fri2 (or mon2-fri)09:11
mborzeckiso at most the range can describe 8 days, eg mon1-mon09:12
pedronisok, not sure I'm worried,  also the other one does quite do the same thing09:12
pedronisjust making sure I understand09:12
pedronis*does not quite do09:12
mvozyga: I updated the kenrel-remodel debug PR, will probably produce a lot of output now, could you please run it?09:14
zygaone sec09:14
zygamvo: try to rebase on 2.42 next time09:15
pedroniszyga: mvo: are you looking into TestRemodelSwitchToDifferentKernel ?09:16
zygamvo: building now09:17
mvopedronis: yeah, it seems to fail for zyga in obs09:17
zygapedronis: mvo is really, I'm just the test runner09:17
zygamvo: in osc, not obs (I didn't push yet)09:17
mvopedronis: its a bit strange I cannot reproduce and I'm a bit lost09:17
pedronisthere's a card, I will move it and put you on it09:17
zygaosc - dpkg-buildpackage, obs -> that thing in the cloud09:17
zygamvo: new log coming up09:17
mvozyga: thanks, is there an ocd as well?09:17
zygaocd? :D09:18
mvozyga: just kidding09:18
zygaI was suspecting some :D09:18
mvozyga: we also have obi here09:18
mvozyga: SCNR09:18
zygasame here09:18
zygavery useful09:18
mvozyga: sorry-could-not-resist09:18
mvozyga: these short acronyms are just too hard for me to remember :)09:19
mvozyga: anyway, hope it gives me clues but I have *no* idea right now09:19
pedronisI can look as well at some point if it remains a mystery, not quite right now09:20
zygamvo: the log is large and got truncated from my screen, give me a sec to grab it from disk09:21
mvozyga: only the bits before the test failure are of intersst09:21
mvozyga: you could also modify the buld to just run the remodel test during the build09:22
zygathe build takes a second, it's just the timeout that we are waiting for09:22
pedronispstolowski: in practice we use SkipConfigure only in firstboot code, doesn't really answer either way09:22
zygaok, collecting now09:22
mvomborzecki: 7572 has an arch specific failure in the spread logs that I have not seen before, could you please have a look before I restart the build?09:23
mvomborzecki: (not urgent :)09:23
Chipacapstolowski: bug09:24
pedronisChipaca: pstolowski: notice that in practice we don't want that behavior, is right now immaterial, we don't want to run health-checks before at least the essential snaps are installed, so what we do for configure there, would apply also for health-checks09:25
pedronisso in practice is a feature, just obscurely expressed09:26
pstolowskipedronis: true09:26
ogramvo, zyga, ondra showed me a bootchart yesterday and we noticed that there are a gazillion unsquashfs calls during seeding ... why is that (i.e. if we extract metadata or some such, why dont we mount/umount the squashfs and cp the file which is a ton times faster and less CPU hungry)09:26
zygaogra: we unsquash to look at info AFAIR09:26
ograright, thats was my guess09:27
zygaogra: but yeah, I agree09:27
pedronispstolowski: basically renaming the flag SkipConfigureAndHealthCheck would also work09:27
Chipacaogra: at one point we determined it to be cheaper to unsquash it to extract just one file, than to mount it09:29
mvowe also do some checks before we consider a snap safe to mount but I don't remember the specifics there for firstboot09:30
zygawe could save on one op entirely by mounting to /tmp/$RANDOM and then moving the mount over to final location09:30
zygamvo: IMO our checks are not in the "safe to mount" category09:30
zygamvo: it's valid vs invalid but not safe09:30
pedroniseither way, we will do nothing without measuring09:31
pedronisalso it's a delicate area under churn09:31
pedronisso also nothing very soon09:31
pedronisogra: I recommend to create a forum topic with some of those logs, and we'll see09:32
ograChipaca, well, on first boot of a low end system it significantly eats CPU09:32
zygamvo: it's still running, it's odd - I had a look at the log file it's running under now and it keeps printing open /dev/null and open /bin/false09:32
Chipacaogra: yes. I sent an email about this three years ago.09:32
Chipacaogra: I profiled it09:32
zygamvo: the log is crashing pastebin :D09:34
zygaone sec09:34
mvozyga: heh, I just need the last few lines09:34
mvozyga: well, I need to see why its not converging, i.e. what it tries to run over and over again09:35
zyga[  618s] trying to run: runMgrForever Doing 1... []09:35
zyga[  618s] trying to run: runMgrForever Doing 1... []09:35
zyga[  618s] trying to run: runMgr1 Do 1... []09:35
zyga[  618s] trying to run: unknown Do ... []09:35
pedronisrunMgrForever is in overlord_test.go ?09:36
pedronisno managers_test.go09:36
zygathere's a long sequence of those09:36
zygabut earlier there's also this09:36
pedronisdo we have a test that doesn't clean up?09:36
pedronisanyway it seems unrelated09:36
zygathen there's a long wall of09:37
mvozyga: how big is the log? maybe you can put it into google drive?09:37
zygasure, one sec09:37
mvozyga: something along the lines of "] Done link-snap Make snap "brand-kernel" (2) available to the system [2019-10-09T08:11:58Z INFO Requested system restart.]" is what I'm hoping for09:38
zygagedit hangs for a while while pasting09:38
Chipacaogra: mail subject 'booting the pi', 26 Jan 201709:38
Chipacaogra: fwiw09:38
zygamvo: 32M09:39
zygamvo: uploading now09:39
mvozyga: woah09:40
ograChipaca, hmm, that talks about sha3 and keccac09:41
Chipacaogra: yes09:41
Chipacaogra: having profiled, those were the slow things09:41
Chipacaogra: that's what we mean by 'we will do nothing without measuring': measure first, determine what is slow, see how to improve it09:42
Chipacaogra: optimising away the calls to mskquashfs will do nothing if those calls are not the thing slowing down first boot09:42
Chipacaof course, it's entirely possibe that those calls are now slow09:43
Chipacabut after having received exactly 0 responses to my original mail I'm not going to waste time on this again09:43
ograChipaca, well, they eat CPU via userspace while mount/umount via the kernel dont09:43
ograthe outcome of your mail was in the end that ijohnson developed assembly to improve sha3 (even though that has never been merged or is in use anywhere still)09:44
mupPR snapd#7538 closed: tests: use `snap model` instead of `snap known model` in tests <Created by mvo5> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/7538>09:45
mupPR snapd#7566 closed: snap-repair: add missing check in TestRepairBasicRun <Simple 😃> <Created by mvo5> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/7566>09:48
* mvo hugs pedronis 09:49
pedronismvo: #7568 needs a tweak (or I am confused)09:52
mupPR #7568: snap: when running `snap repair` without arguments, show hint <Simple 😃> <Created by mvo5> <https://github.com/snapcore/snapd/pull/7568>09:52
pedronisthere is some mark down in the error message which I don't think we do09:52
pedronisheh, Markdown09:53
mvopedronis: sure, let me fix this09:56
mborzeckigrr old systemd09:57
ondraChipaca I'm collecting some bootcharts I can share with you10:07
ondraChipaca those are the one ogra mentioned10:07
* zyga is somewhat hungry10:08
zygamaking progress on cgroup branches, I'll have something to share soon10:08
zygatrying to eliminate the extra binary now10:08
zygawell, it works, just needs some cleanup still10:09
zygamborzecki: ^ FYI10:09
Chipacaondra: I'm not sure what you expect to do with them though10:12
zygaheh, I just noticed that we have cmd/snapd-generator that claims to be snapd-workaround-generator10:13
Chipacaondra: ogra: if I'm reading the bootcharts right, the unsquashfs calls are not a big contributor to the overall boot? most don't even register as taking any time at all, only three (of 23) take more than a second10:15
ograChipaca, they eat CPU .. all of them have pretty bold blue bars .... onn a multicore system that wont be significant, on single cores it will bite10:17
Chipacaogra: sure. So if we avoid them we save... .01% of the overall boot time? or how much?10:17
ogranot on multicore where your IO is rather serialized10:18
Chipacathat boot time is, still, over 5 minutes. The total sum of all time spent on unsquashfs is less than 10 seconds.10:18
ograpoint is it wastese ressources that we could avoid10:18
ograsure, there is something else going on with the system ondra took the bootchart10:19
ograi'm not saying this needs urgent fixing or anything but it is an obvious place for improvement10:19
ondraChipaca my main focus is what is happening between ~40 and ~140 seconds10:19
ograright, it *feels* like there is a Sleep(100) somewhere in snapd's code :D10:20
ondraChipaca it takes almost 100s to start snapd from snapd snap, and I don't seem to get timing for this anywhere, or logs10:20
Chipacaondra: have you enabled debug logs?10:20
ograbut thats unrelated to the unsquashfs ... i'm just pointing out that unsquashfs is wasteful and it would be nic to improve it10:20
ograi'm not asking for an urgent fix but for considering to improve it if there is time10:21
pedronisogra: please write something on the forum with data, these IRC discussions will go nowhere10:22
ograwill do10:22
Chipacaogra: what pedronis said. Thanks!10:22
ondraChipaca I'd need to use custom snapd snap I guess10:22
ondraChipaca or how to do it at first boot10:22
Chipacaondra: no, you just need to drop a file in etc/systemd/system/snapd.service.d10:23
ondraChipaca also it seems like it's before we start snapd, so logs in snapd will not help10:23
Chipacaondra: you can do that if you pause the bootloader for ex10:23
ograit would be cool if we could enable debug options for snpd on the kernel cmdline BT10:23
ondraChipaca can you give me example of that service, I will drop it to the image10:23
ograadjusting the cmdline on firstboot is very easy .. while injecting a file to the rootfs is hard10:24
Chipacaondra: $  cat /etc/systemd/system/snapd.service.d/debug.conf10:24
ChipacaEnvironment=SNAPD_DEBUG=1 SNAPD_DEBUG_HTTP=710:24
Chipacaondra: ^ I also sent an email asking anybody developing things on snapd to have this, ages ago (and it's all over the forum)10:24
Chipacaondra: or, it might be easier, to just drop those two env vars in /etc/environment10:25
* pstolowski walk, bbiab10:25
Chipacaondra: in core18, given it's a funkier service, you might need to adjust the service name10:26
ondraChipaca Ok thanks, I will try that10:27
Chipacaondra: i.e. /etc/systemd/system/core18.start-snapd.service.d/debug.conf10:27
Chipacamvo: I notice this service doesn't source /etc/environment, that's probably not good10:27
Chipacaondra: or maybe it's snapd.seeding.service?10:28
Chipacaondra: at this point I don't know which service actually starts snapd :-|10:28
Chipacain core1810:28
mvoChipaca: oh, good point10:29
pedronisChipaca: different ones at different points10:29
Chipacamvo: do you know how that dance happens?10:29
Chipacai can't find a snapd.seeding.service but it's in the bootchart10:29
pedronisthere's a bootstrapping one then it does a bit of seeding then snapd restarts10:29
mvoChipaca: yes, snapd itself writes it10:30
pedronisas the real service I think10:30
Chipacamvo: from where?10:30
mvoChipaca: i.e. when snapd starts and there is nothing on the system yet it calls wrappers/core18.go10:30
Chipacapedronis: but there isn't a seeding there either10:30
pedronisare you sure it's seeding?10:31
pedronisthere is no such thing afaik10:31
pedronisthere's a seeded service10:31
pedronisthat one doesn't snapd10:31
Chipacapedronis: https://private-fileshare.canonical.com/~okubik/bootchart-20191008-1710.svg10:31
Chipacapedronis: there's a snapd-seeding.service10:32
ondraChipaca I think this one core18.start-snapd.service10:32
pedronisChipaca: it's not a service, it's an ephemeral unit10:34
pedronisChipaca: from here (as ogra said): https://github.com/snapcore/core18/blob/master/static/usr/lib/core18/run-snapd-from-snap10:35
* pedronis lunch10:35
Chipacaondra: wrt not sure if it's snapd or not, snapd is running from t=39s at least10:36
Chipacaondra: so debug logs should shed some light on what's going on there10:36
ograChipaca, pedronis https://forum.snapcraft.io/t/unsquashfs-vs-mount-during-firstboot-seeding/13614/1 ... (really just a suggestion for times where you guys dont have other things to do :) )10:37
ondraChipaca https://private-fileshare.canonical.com/~okubik/boot-plot.svg10:38
ondraChipaca if you look here, we only mount snap-snapd-5038.mount  around 130 second10:39
ondraChipaca and snapd is started at 140 second10:39
ondraChipaca and my interest is before 14010:40
ondraChipaca also from boot-chart you can see that during that time system is relatively idle10:41
ondraonce it starts seeding, it pegs on core well, there we would need to consider parallel jobs to speed it up. but that first 100s seems to be as easier to address now10:41
Chipacaondra: what is it waiting for?10:43
ondraChipaca you tell me :)10:44
ondraChipaca why do you thing I'm digging into it, I want to know as well :)10:44
Chipacaondra: can you stop it inside the initramfs just before it pivots in?10:46
Chipacai.e. before systemd kicks in10:46
ograChipaca, pedronis, ondra https://forum.snapcraft.io/t/allowing-snapd-debug-options-to-be-set-on-kernel-cmdline/1361510:46
Chipacabut after everything is mounted10:46
Chipacaogra: tks10:46
ograwe need whishlist tags in the forum :D10:47
ondraChipaca possibly, what do you want to do there?10:47
Chipacaondra: add 'set -x' to run-snapd-from-snap :-)10:48
ograondra, break=bottom ... then you can modify /writable10:48
ogra(indeed only stuff thats not inside snaps :P )10:48
ondraChipaca but that one is core18 snap10:49
Chipacaondra: copy it somewhere, edit it, bind mount it back10:49
ondraChipaca you are bringing big guns right away :)10:50
mupPR snapd#7576 opened: spread.yaml,.gitignore: sync ignored files <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/7576>10:50
ondraChipaca I guess I can also create own core18 snap can't I?10:52
Chipacaondra: probably, yes :)10:52
pedronispstolowski: btw, are you using core18 for your spike ?10:53
ondraogra is that right to have in core18 snap things like dev/null dev/random dev/urandom dev/zero ?10:54
ograit doesnt do harm10:55
ondraogra seems to me useless10:55
ogra(but in general these things should be handled by devtmpfs which we mount already in the initrd)10:55
ograyeah, they likely are ... unless you have a kernel without devtmpfs10:55
ogra(these are rare but i guess they still exist)10:56
ondraChipaca anything else you want me to modify in core18 snap? set -eux -> set -x?10:56
ondraChipaca I will sprinkle that file with few echos once on it10:56
ograset -x should be enough ... you dont need eu (said boris johnson ...)10:58
Chipacaondra: get some timestamps in there as well10:59
Chipacaondra: dunno where the log will end up :)10:59
Chipacahopefully the journal?10:59
Chipacaif so no timestamps needed10:59
ondraChipaca I will see10:59
ondraChipaca let me run test boot11:00
mborzeckipstolowski: pedronis: pushed a fix to #7443, we had an actual bug on systems with systemd older than 23311:18
mupPR #7443: timeutil: fix schedules with ambiguous nth weekday spans <Bug> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7443>11:18
pedronismborzecki: thanks, I'm a bit confused about the places where the comments mention 28 and the ranges have 2911:21
mborzeckipedronis: heh, right so the 28th day is part of the range genrated by actual calculation, while 29-31 is the bit that varies from month to month (or year), but technically a month can have anywhere from 28 to 31 days11:24
mborzeckiperhaps i should leave a note that 28th is included in the range added further down the code11:25
* Chipaca takes a break11:27
pstolowskipedronis: no, not yet, i'm using cloud image (core + lxd)11:32
pedronispstolowski: ok, that will have it's complications11:33
* zyga -> tea11:36
mupPR snapd#7577 opened: overlord: set fake sertial in TestRemodelSwitchToDifferentKernel <Created by mvo5> <https://github.com/snapcore/snapd/pull/7577>11:38
mborzeckipedronis: force pushed a comment about 29-31 day range11:50
ondraChipaca https://paste.ubuntu.com/p/fQQXtCBSnQ/11:56
ondraChipaca with different formatting https://paste.ubuntu.com/p/q5MFkKy5cw/11:57
ondraChipaca attempts to overlay that file failed though11:57
Chipacaondra: and is the time you need to know about the time between 11:36:25 (131.827907) and 11:38:20 (247.375225), there?11:59
Chipacaondra: because if so, that _is_ snapd running, and you should be able to set env vars to get it to print more debug info12:00
Chipacaondra: add --setenv="SNAPD_DEBUG=1 SNAPD_DEBUG_HTTP=7" to the systemd-run call, there12:01
ondraChipaca but 'Started Start the snapd services from the snapd snap.' is only printed at the end12:02
ondraChipaca and don't you connect eligible plugs and slots before you start snapd?12:02
Chipacaondra: yes, but "started" does not mean "just did the exec"12:02
Chipacaondra: snapd is the thing doing the connecting12:02
ondraChipaca true12:02
Chipacathose log lines about connecting come from snapd12:02
Chipacaoooh! kernel parameter!12:03
Chipaca systemd.setenv=SNAPD_DEBUG=112:03
Chipaca^ that should work!12:03
Chipacasays the manpage12:03
ondraChipaca OK let me test12:03
Chipacaondra: you can pass it more than once (if you also need _HTTP) but just SNAPD_DEBUG=1 should be enough12:04
ondraChipaca OK12:04
ondraChipaca I might got bind mount working12:05
ondraChipaca so i will get logs from this boot first12:06
ondraChipaca do you want logs from snapd.service then?12:06
Chipacaondra: just the ones from that first run12:06
ondraChipaca but from which service?12:07
Chipacaondra: snapd-seeding.service12:07
ondraChipaca do you mean snapd.seeded.service ?12:13
Chipacaondra: no, I mean snapd-seeding.service12:13
Chipacaondra: as in what systemd-run starts in that script12:13
Chipacathat's the mystery snapd12:14
Chipacathat's the one you want to look at12:14
ondraChipaca I cannot see such a service12:14
Chipacaondra: where are you looking?12:14
ondrasystemctl status snapd-seeding.service12:14
ondraUnit snapd-seeding.service could not be found.12:14
Chipacaondra: it doesn't have a unit file12:15
Chipacaondra: journalctl will know of it though12:15
Chipacaondra: journalctl -u snapd-seeding.service should have it12:15
ondraChipaca OK I need to re-run boot as logs rotated now12:15
Chipacaondra: wait a mo'12:16
ondraChipaca how can I increase journal buffer?12:16
Chipacaondra: there was a way to make it non-ephemeral, maybe ogra knows offhand12:16
Chipacaondra: systemd.journald.forward_to_syslo12:16
Chipacaondra: in the commandline12:17
ondraChipaca but on core18 we have no syslog12:20
ograi didnt know there is a cmdline option ... i only know you can create a dir (was it /var/log/journal ???) that will then automagically be used for journald logs12:20
Chipacaondra: I didn't know that :)12:20
ograyeah, no more syslog12:20
ondraChipaca will it create it?12:20
ondraChipaca then no syslog12:20
Chipacabah, i dunno12:20
ograi guess it will just try to write to logger12:20
Chipacaondra: but12:21
ogra(and not find it ...)12:21
ondraChipaca so I do actually have a log from seeding service12:21
Chipacaondra: edit that script, and redirect the output from systemd-run to somewhere12:21
Chipacathat should work12:21
ograwell, creating that dir should also work for persistent logs12:21
Chipacaondra: actually it might _not_ be in the journal, it might all be spewed out from the script12:21
ondraChipaca but it's useless https://paste.ubuntu.com/p/sw5sNBp6WK/12:21
ograthey are binary files though12:21
ChipacaI could check but I'm not sure it's the same in 16 and 1812:21
Chipacaondra: that's not debug output :-|12:22
ondraChipaca booting now with debug env from cmd line12:22
Chipacaand I can confirm the output from the systemd-run does reach the journal (but I think your recent pastebin does, too)12:24
ondraChipaca actually reading logs over ssh seems to messed it up, check this https://paste.ubuntu.com/p/ZFB78ycbMz/12:26
Chipacaondra: if the rotation is still getting in the way, we can either tweak journald.conf, or have it forward to kmsg and bump the kmsg buffer (but that might be expensive on a low lowmem board)12:26
ondraChipaca it's fine if I know what logs to get there is time to do it once I can access seeded device12:27
Chipacaondra: is that with -u snapd-seeding.service ?12:27
Chipacaondra: it looks like the logs of the start-snapd-wotsit script12:27
Chipacawhich would not be in snapd-seeding.service log12:27
Chipaca(and yes there is noise in that that could be avoided... :-| )12:28
ondraChipaca sudo journalctl --all -u snapd-seeding.service is always empty12:30
Chipacaondra: what was https://paste.ubuntu.com/p/sw5sNBp6WK/ ?12:30
ondraChipaca core18.start-snapd.service12:31
ograwelcome to our world :P12:31
Chipacaondra: and if you foraward_to_kmsg ?12:32
ogra(debugging the indebuggable)12:32
Chipacaondra: I've got another option12:33
ondraChipaca OK now with extra verbosity I do not get start of the log :(12:34
Chipacaondra: forwarding to kmsg?12:34
Chipacaondra: or without?12:34
ChipacaI don't know which of all the options you've gone for :)12:34
ondraChipaca without12:34
ondraChipaca BTW it's clear from journalcrl of whole system, there is only kernel chatty from 40 to 140 second12:35
ondraChipaca no changes to where journal goes for now12:35
Chipacaondra: ok, I'm downloading a core18 for amd64, and will play around with it until I can get logs for that first boot snapd12:37
Chipacaondra: I'll let you know12:37
ondraChipaca looks like we have no journald.conf at all, so it should be easy to add one to etc12:37
Chipacaondra: feel free to do so :)12:37
Chipacaseems silly that it doesn't have a kernel option for setting storage as it has for everything else12:38
Chipacaondra: so, on amd64, all I need to do is set systemd.setenv=SNAPD_DEBUG=1 in the kernel commandline, and when I log in after creating my account and etc I have logs for snapd-seeding.service12:44
Chipacaondra: https://paste.ubuntu.com/p/kSmxKc6zKw/12:45
ograogra@pi4:~$ ls /etc/systemd/journald.conf12:46
ogramy core18 image has journald.conf here12:46
ondraogra not mine12:46
ondraogra I can add it, but how do I increase max size12:47
ogragiven we are most likely using the exact same core18 snap12:47
ondrait's set to 10M12:47
Chipacaondra, ogra: where did you get yours from?12:47
ondraogra also it is not in core snap12:47
ogramine is my pi4 custom image ... but that uses core18 from stable12:48
ograogra@pi4:~$ snap info core18|grep refresh12:48
ograrefresh-date: 6 days ago, at 22:09 UTC12:48
Chipacajournald.conf is there, and writable, in the amd64 core12:49
ograsame here12:49
ograogra@pi4:~$ sudo touch /etc/systemd/journald.conf12:49
ondraogra ah RuntimeMaxUse=12:49
Chipacaondra: what core18 are you running, that it doesn't have this file?12:50
ondraChipaca edge12:51
ondraChipaca rev 120212:51
Chipacaondra: what arch?12:52
ograsame here ... but stable12:53
Chipacaondra: core18 1202 has etc/systemd/journald.conf12:54
ondraChipaca armhs?12:55
Chipaca1202 is armhf12:55
ograwould be surprising if it vamished all of a sudden12:55
Chipacaondra: UBUNTU_STORE_ARCH=armhf snap download core18 --edge12:55
Chipacaondra: unsquashfs core18_1202.snap etc/systemd/journald.conf12:55
Chipacaondra: less squashfs-root/etc/systemd/journald.conf12:56
Chipacaogra: yeah it'd be a bug12:56
ondraChipaca OK my bad, I must have had type when searching12:56
ondrasorry I have it too :)12:56
ackksergiusens, hi, any chance you could take a look at https://github.com/sergiusens/snapcraft-preload/pull/34 and https://github.com/sergiusens/snapcraft-preload/pull/36 ?12:57
mupPR sergiusens/snapcraft-preload#34: preserve LD_PRELOAD from the calling environment (fixes #33) <Created by albertodonato> <https://github.com/sergiusens/snapcraft-preload/pull/34>12:57
mupPR sergiusens/snapcraft-preload#36: use SNAP_INSTANCE_NAME instead of SNAP_NAME for prefixing /dev/shm fi… <Created by albertodonato> <https://github.com/sergiusens/snapcraft-preload/pull/36>12:57
Chipacaondra: so now I'm suspicious of your "it's empty" :-|12:57
Chipacaanyway, i've got an up to stand12:57
ondraChipaca non it is not :)12:58
ondraChipaca ogra anyway it only exists in core snap, and not in overlay /etc12:59
ondrawhich is probably good, as I can add mine12:59
zygasorry auth12:59
ondraI'm increasing buffer 4times so I should have now more time12:59
Chipacacore18 has an overlay /etc ??12:59
ondraChipaca overlay of /etc/systemd/13:00
ondraChipaca how would we add new units :)13:00
ograogra@pi4:~$ grep etc/systemd /etc/system-image/writable-paths13:02
ogra/etc/systemd                            auto                    persistent  transition  none13:02
ograogra@pi4:~$ cat /etc/issue13:02
ograUbuntu Core 18 on \4 (\l)13:02
ograif you dont have the journald.conf file then something with your initrd is broken13:03
ogra(transition means it copies the content from core to /writale on first mount)13:04
Chipacaondra: sorry, overlay is a bad word (overlayfs ...)13:04
ograwell, the point is that it *should* be in the overlay /etc13:05
ograif it isnt, there is a bigger prob13:05
ograogra@pi4:~$ ls /writable/system-data/etc/systemd/journald.conf13:05
ondraogra did we find more bugs ? :)13:05
ograsmells like ... if you are sure the file inst there at least13:05
ondraogra if I have file there, will ill override it?13:06
ograif you have the file there it is fully writable13:06
ograso you ccan just edit13:06
ondraogra no if I have already file there, it will not nuke it, right?13:07
ondraogra ignore me it should not13:07
ograyeah, it wont13:08
ogratransition only copies once on first mount ... in subsequent mounts it just uses whats there13:09
ogra(so you could even create a new file there)13:09
ondraChipaca https://paste.ubuntu.com/p/bq6b39PBJ4/13:10
=== ricab is now known as ricab|lunch
Chipacaondra: so there you go13:11
Chipacaondra: and, snap debug timings 1, for more info13:11
ondraChipaca ?13:13
ondraChipaca how to get more logs?13:13
ogra"error trying to compare the snap system key: system-key missing on disk"13:13
ograis that relevant ?13:13
Chipacaogra: nah, that's expected on first boot13:13
ograor does the system-key only come in with initialize-device ?13:13
ograso the gap is before "Load: validated crc32" ... i guess the checksumming is slow then ?13:14
ogra(once again)13:14
roadmrcrc32 wow what a throwback13:15
ondraogra no that is check of lk boot env13:15
ondraogra that takes 1ms13:16
ograhmm, well, there is a long gap before it runs13:17
Chipacaondra: did you run 'snap debug timings 1'?13:17
ondraogra exactly, and I want to know why13:17
Chipacaondra: or even just 'snap changes'13:18
* ogra humms david bowie13:18
ondraChipaca https://paste.ubuntu.com/p/FsdG3V4Fjg/13:18
mupPR snapd#7578 opened: sandbox/cgroup, overlord/snapstate: move helper for listing pids in group to the cgroup package <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7578>13:19
mborzeckizyga: ^^13:19
ograondra, that doesnt seem to list the bootloader stuff ... perhaps check change 2 instead ?13:19
Chipacaondra: also, --verbose13:20
ondraogra that only shows last 2 changes :)13:20
ondraChipaca no it always start at number 313:21
Chipacathose are task numbers, not change numbers13:21
Chipacaondra: snap debug timings --verbose 113:21
Chipacaondra: and, as ogra said, also look at change 213:22
Chipacachange 2 will have the keys generation13:23
zygamborzecki: look again13:24
ondraChipaca it does not give me change 1 or 2 https://paste.ubuntu.com/p/Yj5Q2HYpp9/13:26
Chipacaondra: the "1" in "snap debug timings --verbose 1" is \change 113:27
Chipacaondra: please also run the same command but for change 2, i.e. "snap debug timings --verbose 2"13:27
ondraChipaca and 'snap debug timings --verbose 2' will only print timing around change 20613:27
Chipacaondra: the numbers under the ID column in that output are about tasks13:27
ondraChipaca ah sorry13:28
ograsoooo complicated !13:28
Chipacaondra: what else do you have in "snap changes"?13:28
ondraChipaca so change 2 is two tasks, and both take 1 combined13:28
ondraChipaca just starting services and initialize device13:29
Chipacaondra: can I see the output of 'snap changes' please13:29
ograwhat do you pay ?13:30
ondraChipaca https://pastebin.canonical.com/p/pzJTRfPtdq/13:30
Chipacaondra: ok, the changes you want to look at are 1 and 613:30
ograyeah, 613:30
ondraChipaca https://paste.ubuntu.com/p/dtgwn5C9SK/13:31
ondrais that waiting 56s on key generation?13:31
Chipacaondra: yes13:32
Chipacaondra: so13:32
Chipacaondra: your random is probably empty13:32
mborzeckizyga: replied, i had a really trivial check in mind there13:32
ogratwo times actually, no ?13:32
ondrabut is that blocking boot?13:32
Chipacaondra: get more randomses13:32
Chipacaogra: no, one is the sub of the other13:32
ograah, k13:32
Chipacaondra: yes, you can't boot with an empty random pool13:32
ograright, i see the indendation in the summary when looking closer :)13:33
ograondra, did you already add the cmdline changes for rng_core ?13:33
Chipacaondra: your device probably has a hardware rng you can plug in … somehow?13:33
ondraChipaca still even if blocking, and that's a bit odd to block at that stage, it only explain 56s there13:33
ondraogra yep rng_core is there13:34
ondraogra rng_core.default_quality=70013:34
ograyeah, that should have improved a little bit ... (i'd expect a few seconds less ... so 80-90 vs the 100 before)13:35
ondraogra  I do not think it improved13:36
ograwell, perhaps a missing kernel config ? rng_core is tied to the kernel SW rng13:36
ogra(and will bee quietly ignored if the kernel doesnt have it)13:36
ondraogra what config would be that?13:37
ondraChipaca when do we need that random number first time?13:37
ondraogra CONFIG_HW_RANDOM ?13:38
ondraogra as that is enabled13:39
ograCONFIG_HW_RANDOM and sub-options of that (see menuconfig)13:39
=== chesty_ is now known as chesty
ondraogra is there actually device node which I should see?13:41
ograondra, tyhicks was the one that went over the whole rng stuff back then ... he might know what other options are needed (he also suggested the cmdline option ... which back then helped a lot on a beaglebone where we saw the slowness first913:41
ogranot with in-kernel rng13:42
ograif you want to use a device node there is likely some other kernel config option thats specific to that HW13:42
ograand gives you HW rng ... (if the SoC actually has it at all)13:42
ograyou probably want to take a look at the original android config13:43
ondraogra like CONFIG_HW_RANDOM_MTK :)13:43
ondraogra and enabled13:43
ografooor example :)13:43
ogratat might need userspace bits ... not sure13:43
Chipacaondra: you probably have things in dmesg about not having enough entropy?13:45
Chipacawhen we need it → to generate keys13:45
ograwe should just generate them at build time ... makes everything easier ... and everyone has the same key ... :)13:46
ograondra, btw, do you also see the slowness on second boot ?13:47
ogra(i assume not)13:47
ondraogra nope, sub 30 seconds13:47
ograperhaps we should simply tell the customer to boot the device once in-factory13:47
ondraogra from kernel code, there is driver using TZ to generate rn, so this should be fast13:48
ogramake it part of the flashing procedure13:48
ogra"power on ... leave it sit for 5min"13:49
ogratimer wont be enough to get entropy13:55
ograyou need interrupts13:55
ograattach a mouse and wiggle it :)13:55
ondraChipaca stupid question but task ID is assigned when task is started, is that correct?13:55
ondraChipaca and tasks from change id 1 will run before tasks from change 213:55
ondraChipaca because random number generation is run as task 200+13:55
ondraChipaca this is way after snaps were seeded13:55
=== ^arcade_droid is now known as zarcade_droid
=== cjp256_ is now known as cjp256
=== arnatious_ is now known as arnatious
ondrawow lot's of dead ppl here13:56
* Chipaca got better13:56
ondraChipaca did you see my question about task IDs?13:56
ondraChipaca is that ID assigned when task was started?13:57
ondraChipaca so I can assume they go in sequence?13:57
Chipacaondra: I did not see your question13:57
ondraChipaca OK last two question I did ask now13:57
Chipacaondra: they go in sequence at the time of task creation, which might not line up exactly with task execution13:57
ondraChipaca OK, so then this random key generation might take some time, but it's run way way later13:58
ondraChipaca so it's not that 100s we are looking into13:58
ondraChipaca it's task id 200+ and change 613:58
ondraChipaca we are looking into 100s before or at the beginning of change 113:59
ondraChipaca like task id 1 and 213:59
Chipacaondra: I … what? why?13:59
Chipacaondra: if it's before snapd starts, then it's somebody else you should be talking to13:59
ondraChipaca that 100 is before we start seeding, rsa key is generated after seeding14:00
ondraChipaca nope, it's this log https://paste.ubuntu.com/p/5jx9kcfZXV/14:01
ondraChipaca or in that seeding log, there is that pause14:02
ondraChipaca rsa-key comes after seeding14:02
Chipacaondra: ok14:03
ondraChipaca so can I get more logs there?14:05
Chipacaondra: did you ever get me a 'snap debug timings --verbose 1'? i might have closed it already14:07
ondraChipaca yes I did, https://paste.ubuntu.com/p/Yj5Q2HYpp9/14:07
ondraChipaca task 4 there is too late, this is after that quiet period14:08
Chipacaondra: 101ms is too late?14:09
Chipacaondra: can you actual timestamps on the journal output, or is it too late for that?14:10
Chipacaondra: the way I read the one you pasted is probably wrong14:10
ondraChipaca probably too late now14:10
ondraChipaca I have timing from start of the boot14:11
Chipacaondra: ok, maybe you can help me then14:11
Chipacaondra: https://paste.ubuntu.com/p/bq6b39PBJ4/14:11
ondraChipaca which is then 014:11
Chipacain that one14:11
Chipacaondra: the numbers in the left column are seconds in offset since the 'logs start', yes?14:11
ondraChipaca time in that log is relative from 0 which is device power on14:11
Chipacaondra: ah14:11
ondraChipaca since device power on14:11
Chipacaondra: and how do I compare with other things?14:12
ondraChipaca and yes, seconds from device power on14:12
ondraChipaca all the logs and boot chart is in seconds from device power on14:12
ondraChipaca snap changes are not though14:12
Chipacanor is '-- Logs begin at Wed 2019-10-09 13:00:46 UTC, end at Wed 2019-10-09 13:10:02 UTC. --'14:13
ondraChipaca so depends what do you want to compare it to?14:13
ondraChipaca OK there you go, you have also log start14:13
Chipacaondra: that doesn't help, does it?14:14
Chipacaondra: what wall clock time does the gap happen at?14:14
ograbootchart looks at the kernel counter14:15
ograwhich starts at 0 when the kernel fires up14:15
ondraChipaca but what are you interested in? Absolute time when service started, or actual time processing14:15
ondraogra and I did same for journal logs14:15
ogra(same thing you get with dmesg if you dont use -T )14:15
ondraogra so you can link them to bootchart14:16
ondraogra and same you get with journalctl -o short-monotonic14:16
ograah, i didnt knwo that one (i rarely want the kernel timer anyway)14:16
ondraChipaca but if I tell you wall clock time wha will you compare it to?14:17
Chipacaondra: what's the output of 'snap debug timings --verbose --ensure=seed' ?14:17
ondraChipaca https://pastebin.canonical.com/p/3NRRGZYJ9S/14:18
ondraChipaca '75739ms            -  state-from-seed             populate state from seed'?14:18
Chipacaondra: how many more seconds do you need to find?14:18
ondraChipaca what is that?14:18
ondraChipaca what do you mean 'how many more seconds'?14:19
Chipacaondra: last time I found you 50s and your first reaction was "i need more"14:19
Chipacaso here i am finding you more14:19
Chipacathat's a chunk of time14:19
ondraChipaca no, that 50 seconds you found is when services run14:19
ondraChipaca device communicated14:20
ondraChipaca so they are not helping me at all14:20
=== ricab|lunch is now known as ricab
Chipacayes, that was your second reaction14:20
ondraChipaca that's 50 seconds to connect to the store to check for update, but at that point device will be already communicating with user, so that 50s is for me irrelevant14:21
Chipacaondra: I regret trying to help you, now14:21
ondraChipaca well then just tell me "yep, during first boot we stop for 100s before we start seeding, this is feature"14:22
Chipacathe first thing I told you before trying to help you was that years ago I pointed out that first boot on armhf takes 5+ minutes because of things14:23
Chipacaand now, several hours of trying to walk you through it all I point you at the logs that show you that same bit of information14:23
ChipacaI have no idea what you're so upset about at this point14:23
ondraChipaca this is smart speaker, it takes 5 minutes to boot, before it can connect to the phone. that 5 minutes does not include or is affected by that 50 seconds key generation, as you do not have internet anyway.14:23
Chipacabut you have the data, right there14:23
ondraChipaca but you can connect over the BT to provision WiFi, while device is generating rsa-key14:24
ChipacaI am not talking about those 50 seconds for key generation, I am talking about _these_ 300 seconds of seeding14:24
ondraChipaca so it's not having any effect on user experience14:24
ondraChipaca I accept seeding takes time, related to number of snaps14:25
ondraChipaca but snapd snap seeds for 150 seconds, that seems odd14:25
ondraChipaca and nobody complains about that, I'm just wondering why we have 100s of nothing, Now you explain me this is feature. And you should just take it14:27
ondraChipaca thanks for help14:27
Chipacaondra: at what point did I say this was a feature and you should just take it14:31
ondraChipaca at point that you said I was not happy with 50s discovery, which are post seeding14:32
ondraChipaca I told you those make no difference14:32
Chipacaondra: I still don't see how I told you that. I see you saying I did, though.14:34
Chipacaor maybe saying I _should_14:35
Chipacabut I won't, because it isn't14:35
Chipacaondra: could you please snap install http, and pastebin 'http snapd:///v2/changes select==all'14:39
ondraChipaca https://pastebin.canonical.com/p/k8Q6ryzwhn/14:45
Chipacaondra: where does 2019-10-09T13:06:29.005541125Z fall WRT the missing time?14:47
Chipacais that before or after the time we're looking for?14:47
ondraChipaca that's after14:48
ondraChipaca I think we are looking for things before 2019-10-09 13:03:0014:50
ondraChipaca more precise between 2019-10-09 13:01:26 and 2019-10-09 13:03:0014:51
Chipacaondra: where do those timestamps fall in https://paste.ubuntu.com/p/bq6b39PBJ4/ ?14:51
ondraChipaca between 40 and 14014:55
ondraChipaca from what I understand, logs star is when kernel started14:56
* zyga calls it a day15:03
Chipacaondra: just in case, is there anything in any of the other --ensure= options?15:10
Chipacaanything in the times we're looking at15:10
ondraChipaca I did get whole journal if you want15:13
ondraChipaca I could not see anything there though15:13
ondraChipaca let me paste it for you15:14
Chipacaondra: there's a little more detail in the timings output15:14
Chipacaondra: can you patch the snapd for first boot, or is that too much?15:15
ondraChipaca not that is actually fairly easy15:15
ondraChipaca unlike core as I learnt15:15
ondraChipaca https://pastebin.canonical.com/p/Bzv44GKwmD/15:16
Chipacaondra: out of curiosity, what are these about?15:17
Chipaca[   42.310781] localhost kernel: CPU3 killed.15:17
ondraChipaca just some MTK kernel verbosity :)15:19
ograChipaca, at least it is a friendly killer ... i have seen other boards where daemons kill their children and such ... way more blood there15:33
Chipacaondra: so if I give you a patch for snapd you can apply it to your snapd?15:33
ondraChipaca yep15:34
ondraChipaca that wont' be problem15:34
Chipacaondra: http://paste.ubuntu.com/p/DWYXndYs5C/15:35
Chipacaondra: won't be too informative but hopefully it'll give us somewhere to start looking15:35
Chipacaondra: it'll need the debug trick still15:36
ondraChipaca sure I will let you know once I have it15:38
=== pstolowski is now known as pstolowski|afk
* cachio lunch15:47
ondraChipaca it will take some time, as my LP git sync is now hanging to run sync 'as soon as possible' for past 15 minutes15:52
ondraChipaca time is relative thing15:52
ograblame einstein16:03
Chipacaonly important if you're going fast, so it doesn't apply to this board16:04
* Chipaca hides16:04
cjwatsonIf you're routinely waiting for an LP git import, it's probably a sign that you should be hosting the repository directly on LP instead.  Imports are best-effort16:07
ograbut thats less funny because you cant complain then ...16:12
ondracjwatson it's a bit pain to push to two locations as snapd is in github, but I guess best option16:20
ondracjwatson as now I'm waiting over half an hour and still nada16:20
roadmrfolks, so: mir-kiosk has a plugs: - x11, so it offers x11 for other snaps to connect to it, right?16:23
cjwatsonondra: Hm, one of our importds doesn't seem to be doing much though, let me see16:23
ondracjwatson thanks16:27
ondracjwatson I did create copy git repo to kick build now, so it should be OK16:27
Chipacaondra: I'm off but I'll check back later tonight16:32
ondraChipaca no worries I have some other tasks anyway, let's check tomorrow16:32
cjwatsonondra: Repaired the stuck worker, so it should be doing a better job of keeping up with imports now anyway17:16
ondracjwatson thank you :)17:18
ondraChipaca https://paste.ubuntu.com/p/WMKkCPSBjg/17:32
* cachio afk17:47
ograroadmr, i think that plug is only for running mir-kiosk on desktop systems .... to actually run an x11 app on top of mir-kiosk (which only offers wayland to apps) your app needs to ship Xwayland and set that up through a wrapper .... the app also needs a loopback slot/plug x11 setup18:00
ograroadmr, i.e. line 50,56 and 81 in https://github.com/ogra1/opencv-demo-snap/blob/master/snap/snapcraft.yaml18:01
roadmrthanks ogra, that'll be useful as a reference19:24
mupPR snapd#7579 opened: interfaces/network-setup-observe: add Info D-Bus method accesses <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7579>20:25

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!