| zyga | good morning | 05:56 |
|---|---|---|
| mborzecki | zyga: morning | 06:06 |
| zyga | hey :) | 06:06 |
| mborzecki | mvo: hey | 06:33 |
| mvo | hey mborzecki | 06:34 |
| zyga | hey mvo | 06:45 |
| zyga | quick comment, do you remember the remodel kernel work? there's a test that reliably fails in osc build, there's a trello card for it | 06:45 |
| zyga | I wanted to look now but if you know more I'd love to get help | 06:45 |
| mvo | zyga: can you point me to the log? | 06:46 |
| mvo | zyga: the remodel kernel test takes a while iirc, we had to increase the settle timeout on arm | 06:46 |
| mvo | zyga: is it maybe that? | 06:46 |
| zyga | that's the same tet | 06:46 |
| zyga | but it failed on my xeon ;D | 06:46 |
| mvo | also with a timeout? | 06:48 |
| zyga | it's hard to say, there's very little useful stuff when settle fails | 06:48 |
| zyga | I complained that it should not be time-based at all but this is not a simple problem to solve | 06:48 |
| zyga | mvo: https://trello.com/c/foU3iOrs/321-investigate-testremodelswitchtodifferentkernel-failure | 06:49 |
| zyga | mvo: the whole output is | 06:52 |
| zyga | https://www.irccloud.com/pastebin/8NhFWypY/ | 06:52 |
| mvo | zyga: this happens on your opensuse as well? | 06:52 |
| zyga | no, it only happens in 'osc build' | 06:52 |
| zyga | though my master is "after" 2.42 | 06:52 |
| mvo | zyga: is it easy to submit test builds? I could add something that prints more debug | 06:52 |
| zyga | mvo: it's not remote | 06:52 |
| zyga | it's local | 06:52 |
| zyga | mvo: it's like dpkg-buildpackage | 06:53 |
| zyga | mvo: anything extra can be added, just make a patch against 2.42 | 06:53 |
| zyga | mvo: if you push a branch I can extract the patch and run it easily | 06:53 |
| zyga | mvo: I would like to make some general changes to the tests later on | 06:53 |
| zyga | though I doubt this will be fasst | 06:53 |
| zyga | 1) log all handlers that executed during failed test | 06:54 |
| zyga | 2) change settle in tests to just do more stuff as long as handlers want to run, remove the timeout entirely | 06:54 |
| zyga | 3) patch tests that measure failure to introduce callback to break further loops | 06:54 |
| zyga | current time based code is simply not reliable and not engineered correctly as tests IMO | 06:55 |
| === pstolowski|afk is now known as pstolowski | ||
| pstolowski | mornings | 07:03 |
| zyga | pstolowski: hey :) | 07:22 |
| zyga | man, I'm so sleepy today | 07:22 |
| zyga | the rain, the low pressure | 07:23 |
| zyga | coffee, sorry | 07:23 |
| mvo | hey pstolowski - good morning! | 07:24 |
| zyga | mvo: sorry if I came across cranky, I just think we need to acknowledge shortcomings and not just paper over them with "this is how it's been done" | 07:24 |
| mvo | zyga: not cranky at all | 07:25 |
| mvo | zyga: let me just finish this one thing here and then I can do something, probably something like for _, t := range chg.Tasks() { fmt.Printl(t.Kind(), t.Status()) } | 07:26 |
| zyga | yeah | 07:26 |
| zyga | flashing the card again, last night I had issues and somehow I could not get the card to copy from macos | 07:29 |
| zyga | (new sandbox for all apps) | 07:29 |
| zyga | mvo: macos catalina switched to ... read only image as root filesystem | 07:29 |
| zyga | mvo: yeah :) | 07:29 |
| zyga | mvo: ubuntu personal | 07:29 |
| zyga | mvo: /Users (/home for macos) and /Applications are writable | 07:30 |
| zyga | rest is not | 07:30 |
| zyga | and they live on a different apfs dataset (like zfs thing) | 07:30 |
| mborzecki | zyga: updated #7571 | 07:38 |
| mup | PR #7571: sandbox/cgroup: refactor process cgroup helper to support v2 and named hierarchies <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7571> | 07:38 |
| zyga | checking | 07:38 |
| mborzecki | zyga: is there a place like /var/log on osx where system logs land? wonder how it's combined with the read only filesystem image | 07:39 |
| zyga | mborzecki: actually macos has /home /var /usr and all that | 07:39 |
| zyga | they are just hidden | 07:39 |
| zyga | this is the current mount table | 07:39 |
| zyga | https://www.irccloud.com/pastebin/r4LiHBk7/ | 07:39 |
| zyga | mborzecki: and yeah, there is /var/log | 07:40 |
| zyga | mborzecki: this is also interesting | 07:41 |
| zyga | https://www.irccloud.com/pastebin/fRUAdcOV/ls%20-la%20%2F%20 | 07:42 |
| zyga | note that /home is /System/Volumes/Data/home | 07:42 |
| zyga | where Data is the new writable part of apfs | 07:42 |
| zyga | mborzecki: the new PR looks nice | 07:44 |
| zyga | mborzecki: approved | 07:46 |
| mborzecki | zyga: thx | 07:49 |
| pedronis | mvo: hi, I made this comment: https://github.com/snapcore/snapd/pull/7538#issuecomment-539882778 let me know if you prefer me to go over line by line | 07:50 |
| mup | PR #7538: tests: use `snap model` instead of `snap known model` in tests <Created by mvo5> <https://github.com/snapcore/snapd/pull/7538> | 07:50 |
| mup | PR snapd#7575 closed: cmd/model: add authority-id to verbose fields <Needs Samuele review> <Simple 😃> <Created by anonymouse64> <Closed by pedronis> <https://github.com/snapcore/snapd/pull/7575> | 07:55 |
| mvo | pedronis: thanks, looking now | 08:00 |
| mvo | pedronis: no need to go over it line by line, thanks, I will make the changes | 08:02 |
| mvo | zyga: I created github.com/mvo5/snappy:debug-remodel-kernel-zyga with some extra debug (quite small right now). please run when you get a chance and let me know what it outputs | 08:08 |
| zyga | mvo: thank you | 08:08 |
| zyga | mvo: I've rebased on 2.42, trying now | 08:10 |
| zyga | mvo: new log https://www.irccloud.com/pastebin/EUXdHkdl/ | 08:13 |
| mvo | zyga: aha, looks like something in the mock restart is not working | 08:15 |
| zyga | mvo: I'm logged into the image from rogpeppe | 08:21 |
| zyga | refreshing snaps now | 08:21 |
| zyga | we'll know in a second if it crashes on reboot | 08:21 |
| rogpeppe | zyga: sometimes it doesn't :) | 08:22 |
| zyga | rogpeppe: fingers crossed | 08:22 |
| zyga | I have serial output | 08:22 |
| zyga | so we'll know what is attempted at least | 08:22 |
| rogpeppe | zyga: the other thing is that it surely shouldn't be rebooting very often | 08:22 |
| zyga | and I should send the raspi-tool to snapd into some contrib/ directory | 08:22 |
| rogpeppe | zyga: how often would you expect? is it usual for it to try to reboot once or twice a day? | 08:23 |
| mvo | zyga: will look into this a little bit | 08:23 |
| zyga | rogpeppe: not sure, one sec | 08:23 |
| zyga | Chipaca: from that device | 08:23 |
| mvo | rogpeppe: the only valid reason to reboot that often is if it tracks edge for core/kernel everything else is most likely a bug | 08:24 |
| zyga | Chipaca: system-shutdown messages https://www.irccloud.com/pastebin/sSWsFr69/ | 08:24 |
| mvo | rogpeppe: thank you so much for helping us tracking this down! | 08:24 |
| mvo | zyga: thank you, super curious about this | 08:24 |
| rogpeppe | mvo: i hope you do! :) | 08:24 |
| zyga | rogpeppe: it updated to 2.42 correctly | 08:24 |
| Chipaca | zyga: those messages are fine | 08:24 |
| zyga | ok | 08:24 |
| zyga | oh | 08:24 |
| zyga | it reboots instantly? | 08:24 |
| zyga | what? | 08:24 |
| zyga | I snap refreshed | 08:25 |
| zyga | it rebooted | 08:25 |
| zyga | then it reboots again | 08:25 |
| zyga | straight away | 08:25 |
| zyga | mvo: suggestion for improvement | 08:26 |
| zyga | mvo: uboot script should log basics | 08:26 |
| zyga | like booting that kernel and this base | 08:26 |
| zyga | and in this mode | 08:26 |
| zyga | would be great to see | 08:26 |
| zyga | mvo: snapd change after update | 08:27 |
| zyga | it seems that 2nd reboot was the only one | 08:27 |
| zyga | snap change on successful 2nd reboot https://www.irccloud.com/pastebin/4X2z8O6R/ | 08:27 |
| zyga | I'll reboot manually to see if it boots ok | 08:27 |
| zyga | rogpeppe: sadly, it works :/ | 08:29 |
| pedronis | mborzecki: question in #7443 | 08:32 |
| mup | PR #7443: timeutil: fix schedules with ambiguous nth weekday spans <Bug> <Needs Samuele review> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7443> | 08:32 |
| zyga | rogpeppe: I'll leave it running | 08:32 |
| rogpeppe | zyga: one thing you could do is just leave it running | 08:32 |
| rogpeppe | zyga: lol | 08:32 |
| zyga | rogpeppe: to see how it ages for a few days | 08:32 |
| zyga | :D | 08:32 |
| rogpeppe | zyga: yup. i guess it might be that we've got two pi's both with buggy h/w | 08:33 |
| zyga | rogpeppe: could it be the SD card? | 08:33 |
| Chipaca | could it be that the additional hardware somehow destabilises the pies? | 08:33 |
| rogpeppe | zyga: i used a different SD card too | 08:34 |
| zyga | interesting | 08:34 |
| zyga | how is the hardware connecteD? | 08:34 |
| rogpeppe | zyga: and it's a decent quality SD card, not much used. | 08:34 |
| rogpeppe | zyga: connected to the network? | 08:34 |
| zyga | no, any custom stuff added to your pi? | 08:34 |
| zyga | not network | 08:34 |
| rogpeppe | zyga: nothing custom added | 08:35 |
| Chipaca | there goes that hypothesis then :) | 08:35 |
| rogpeppe | zyga: there was an RTC and a pi-glow on the old pi, but not on this one | 08:36 |
| zyga | I can send you my pi if you want to swap :) | 08:36 |
| rogpeppe | zyga: that's an interesting thought. the other thing i might try (sorry) is to use a different OS. | 08:37 |
| zyga | that would be an interesting data point as well | 08:37 |
| pstolowski | Chipaca: hey, health-check task is not executed on install if flags&skipConfigure != 0 (because we return early and don't schedule config hook - see the bottom of doInstall); intended or a bug? | 08:52 |
| mborzecki | pedronis: iirc it's not strictly needed, though i kind of highlights the thing happens every day | 08:54 |
| pedronis | mborzecki: highlights or confuses people like me :) | 08:55 |
| mborzecki | pedronis: hah ok, i can drop it | 08:55 |
| pedronis | mborzecki: one possible reading is that it triggers every day of the month | 08:55 |
| pedronis | anyway with /1 | 08:55 |
| pedronis | the "spec" is not great | 08:55 |
| mborzecki | duh, it is not :/ | 08:56 |
| pedronis | mborzecki: if it's not strictly needed I would drop it, less mental burden to read it. The spec says " Values may be suffixed with "/" and a repetition value, which indicates that the value itself and the value plus all multiples of the repetition value are matched. Two values separated by ".." may be used to indicate a range of values; ranges may also be followed with "/" and a repetition value." of which I'm not sure how to | 09:01 |
| pedronis | interpret in practice | 09:01 |
| mborzecki | i'm looking at the spec, and we may have a bug there, at laest on older systemd versions | 09:02 |
| mborzecki | there's no .. range on xenial :/ | 09:02 |
| mborzecki | but out snap with timers does not fail, so, the bug is only for numbered ranges | 09:03 |
| mborzecki | another conclusion is that since we had no bug reports or complains about snap install failing, then people do not use it at least in snap.yaml | 09:04 |
| pedronis | mborzecki: we'll need to update this, right? https://forum.snapcraft.io/t/timer-string-format/6562 | 09:06 |
| mborzecki | yes | 09:06 |
| pedronis | also before you could do mon1-fri2, right? now you need something like mon1-sun,,mon2-fri ? | 09:07 |
| mborzecki | yes, mon1-fri or mon-fri2 (or mon2-fri) | 09:11 |
| mborzecki | so at most the range can describe 8 days, eg mon1-mon | 09:12 |
| pedronis | ok, not sure I'm worried, also the other one does quite do the same thing | 09:12 |
| pedronis | just making sure I understand | 09:12 |
| pedronis | *does not quite do | 09:12 |
| mvo | zyga: I updated the kenrel-remodel debug PR, will probably produce a lot of output now, could you please run it? | 09:14 |
| zyga | sure | 09:14 |
| zyga | one sec | 09:14 |
| zyga | mvo: try to rebase on 2.42 next time | 09:15 |
| pedronis | zyga: mvo: are you looking into TestRemodelSwitchToDifferentKernel ? | 09:16 |
| zyga | mvo: building now | 09:17 |
| mvo | pedronis: yeah, it seems to fail for zyga in obs | 09:17 |
| zyga | pedronis: mvo is really, I'm just the test runner | 09:17 |
| zyga | mvo: in osc, not obs (I didn't push yet) | 09:17 |
| mvo | pedronis: its a bit strange I cannot reproduce and I'm a bit lost | 09:17 |
| pedronis | there's a card, I will move it and put you on it | 09:17 |
| zyga | osc - dpkg-buildpackage, obs -> that thing in the cloud | 09:17 |
| zyga | mvo: new log coming up | 09:17 |
| mvo | zyga: thanks, is there an ocd as well? | 09:17 |
| zyga | ocd? :D | 09:18 |
| mvo | zyga: just kidding | 09:18 |
| zyga | hahaha | 09:18 |
| zyga | I was suspecting some :D | 09:18 |
| mvo | zyga: we also have obi here | 09:18 |
| mvo | zyga: SCNR | 09:18 |
| zyga | same here | 09:18 |
| zyga | very useful | 09:18 |
| zyga | SCNR? | 09:18 |
| mvo | zyga: sorry-could-not-resist | 09:18 |
| mvo | zyga: these short acronyms are just too hard for me to remember :) | 09:19 |
| mvo | zyga: anyway, hope it gives me clues but I have *no* idea right now | 09:19 |
| pedronis | I can look as well at some point if it remains a mystery, not quite right now | 09:20 |
| pedronis | though | 09:21 |
| zyga | mvo: the log is large and got truncated from my screen, give me a sec to grab it from disk | 09:21 |
| mvo | zyga: only the bits before the test failure are of intersst | 09:21 |
| mvo | zyga: you could also modify the buld to just run the remodel test during the build | 09:22 |
| zyga | the build takes a second, it's just the timeout that we are waiting for | 09:22 |
| pedronis | pstolowski: in practice we use SkipConfigure only in firstboot code, doesn't really answer either way | 09:22 |
| zyga | ok, collecting now | 09:22 |
| mvo | mborzecki: 7572 has an arch specific failure in the spread logs that I have not seen before, could you please have a look before I restart the build? | 09:23 |
| mvo | mborzecki: (not urgent :) | 09:23 |
| Chipaca | pstolowski: bug | 09:24 |
| pedronis | Chipaca: pstolowski: notice that in practice we don't want that behavior, is right now immaterial, we don't want to run health-checks before at least the essential snaps are installed, so what we do for configure there, would apply also for health-checks | 09:25 |
| pedronis | so in practice is a feature, just obscurely expressed | 09:26 |
| pstolowski | pedronis: true | 09:26 |
| ogra | mvo, zyga, ondra showed me a bootchart yesterday and we noticed that there are a gazillion unsquashfs calls during seeding ... why is that (i.e. if we extract metadata or some such, why dont we mount/umount the squashfs and cp the file which is a ton times faster and less CPU hungry) | 09:26 |
| zyga | ogra: we unsquash to look at info AFAIR | 09:26 |
| ogra | right, thats was my guess | 09:27 |
| zyga | ogra: but yeah, I agree | 09:27 |
| pedronis | pstolowski: basically renaming the flag SkipConfigureAndHealthCheck would also work | 09:27 |
| ogra | *that | 09:27 |
| Chipaca | ogra: at one point we determined it to be cheaper to unsquash it to extract just one file, than to mount it | 09:29 |
| mvo | we also do some checks before we consider a snap safe to mount but I don't remember the specifics there for firstboot | 09:30 |
| zyga | we could save on one op entirely by mounting to /tmp/$RANDOM and then moving the mount over to final location | 09:30 |
| zyga | mvo: IMO our checks are not in the "safe to mount" category | 09:30 |
| zyga | mvo: it's valid vs invalid but not safe | 09:30 |
| pedronis | either way, we will do nothing without measuring | 09:31 |
| pedronis | also it's a delicate area under churn | 09:31 |
| pedronis | so also nothing very soon | 09:31 |
| pedronis | ogra: I recommend to create a forum topic with some of those logs, and we'll see | 09:32 |
| ogra | Chipaca, well, on first boot of a low end system it significantly eats CPU | 09:32 |
| zyga | mvo: it's still running, it's odd - I had a look at the log file it's running under now and it keeps printing open /dev/null and open /bin/false | 09:32 |
| Chipaca | ogra: yes. I sent an email about this three years ago. | 09:32 |
| Chipaca | ogra: I profiled it | 09:32 |
| zyga | mvo: the log is crashing pastebin :D | 09:34 |
| zyga | one sec | 09:34 |
| mvo | zyga: heh, I just need the last few lines | 09:34 |
| mvo | zyga: well, I need to see why its not converging, i.e. what it tries to run over and over again | 09:35 |
| zyga | [ 618s] trying to run: runMgrForever Doing 1... [] | 09:35 |
| zyga | [ 618s] trying to run: runMgrForever Doing 1... [] | 09:35 |
| zyga | [ 618s] trying to run: runMgr1 Do 1... [] | 09:35 |
| zyga | [ 618s] trying to run: unknown Do ... [] | 09:35 |
| pedronis | runMgrForever is in overlord_test.go ? | 09:36 |
| pedronis | no managers_test.go | 09:36 |
| zyga | there's a long sequence of those | 09:36 |
| zyga | but earlier there's also this | 09:36 |
| pedronis | do we have a test that doesn't clean up? | 09:36 |
| zyga | https://www.irccloud.com/pastebin/8PTTypwd/ | 09:36 |
| pedronis | anyway it seems unrelated | 09:36 |
| zyga | then there's a long wall of | 09:37 |
| mvo | zyga: how big is the log? maybe you can put it into google drive? | 09:37 |
| zyga | sure, one sec | 09:37 |
| mvo | zyga: something along the lines of "] Done link-snap Make snap "brand-kernel" (2) available to the system [2019-10-09T08:11:58Z INFO Requested system restart.]" is what I'm hoping for | 09:38 |
| zyga | gedit hangs for a while while pasting | 09:38 |
| zyga | 2019 | 09:38 |
| Chipaca | ogra: mail subject 'booting the pi', 26 Jan 2017 | 09:38 |
| Chipaca | ogra: fwiw | 09:38 |
| zyga | mvo: 32M | 09:39 |
| zyga | mvo: uploading now | 09:39 |
| mvo | zyga: woah | 09:40 |
| ogra | Chipaca, hmm, that talks about sha3 and keccac | 09:41 |
| Chipaca | ogra: yes | 09:41 |
| Chipaca | ogra: having profiled, those were the slow things | 09:41 |
| Chipaca | ogra: that's what we mean by 'we will do nothing without measuring': measure first, determine what is slow, see how to improve it | 09:42 |
| Chipaca | ogra: optimising away the calls to mskquashfs will do nothing if those calls are not the thing slowing down first boot | 09:42 |
| Chipaca | unsquashfs* | 09:42 |
| Chipaca | of course, it's entirely possibe that those calls are now slow | 09:43 |
| Chipaca | but after having received exactly 0 responses to my original mail I'm not going to waste time on this again | 09:43 |
| ogra | Chipaca, well, they eat CPU via userspace while mount/umount via the kernel dont | 09:43 |
| ogra | the outcome of your mail was in the end that ijohnson developed assembly to improve sha3 (even though that has never been merged or is in use anywhere still) | 09:44 |
| mup | PR snapd#7538 closed: tests: use `snap model` instead of `snap known model` in tests <Created by mvo5> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/7538> | 09:45 |
| mup | PR snapd#7566 closed: snap-repair: add missing check in TestRepairBasicRun <Simple 😃> <Created by mvo5> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/7566> | 09:48 |
| * mvo hugs pedronis | 09:49 | |
| pedronis | mvo: #7568 needs a tweak (or I am confused) | 09:52 |
| mup | PR #7568: snap: when running `snap repair` without arguments, show hint <Simple 😃> <Created by mvo5> <https://github.com/snapcore/snapd/pull/7568> | 09:52 |
| pedronis | there is some mark down in the error message which I don't think we do | 09:52 |
| pedronis | heh, Markdown | 09:53 |
| mvo | pedronis: sure, let me fix this | 09:56 |
| mborzecki | grr old systemd | 09:57 |
| ondra | Chipaca I'm collecting some bootcharts I can share with you | 10:07 |
| ondra | Chipaca those are the one ogra mentioned | 10:07 |
| * zyga is somewhat hungry | 10:08 | |
| zyga | making progress on cgroup branches, I'll have something to share soon | 10:08 |
| zyga | trying to eliminate the extra binary now | 10:08 |
| zyga | well, it works, just needs some cleanup still | 10:09 |
| zyga | mborzecki: ^ FYI | 10:09 |
| Chipaca | ondra: I'm not sure what you expect to do with them though | 10:12 |
| zyga | heh, I just noticed that we have cmd/snapd-generator that claims to be snapd-workaround-generator | 10:13 |
| Chipaca | ondra: ogra: if I'm reading the bootcharts right, the unsquashfs calls are not a big contributor to the overall boot? most don't even register as taking any time at all, only three (of 23) take more than a second | 10:15 |
| ogra | Chipaca, they eat CPU .. all of them have pretty bold blue bars .... onn a multicore system that wont be significant, on single cores it will bite | 10:17 |
| Chipaca | ogra: sure. So if we avoid them we save... .01% of the overall boot time? or how much? | 10:17 |
| ogra | not on multicore where your IO is rather serialized | 10:18 |
| ogra | err | 10:18 |
| ogra | s/multi/single/ | 10:18 |
| Chipaca | that boot time is, still, over 5 minutes. The total sum of all time spent on unsquashfs is less than 10 seconds. | 10:18 |
| ogra | point is it wastese ressources that we could avoid | 10:18 |
| ogra | sure, there is something else going on with the system ondra took the bootchart | 10:19 |
| ogra | i'm not saying this needs urgent fixing or anything but it is an obvious place for improvement | 10:19 |
| ondra | Chipaca my main focus is what is happening between ~40 and ~140 seconds | 10:19 |
| ogra | right, it *feels* like there is a Sleep(100) somewhere in snapd's code :D | 10:20 |
| ondra | Chipaca it takes almost 100s to start snapd from snapd snap, and I don't seem to get timing for this anywhere, or logs | 10:20 |
| Chipaca | ondra: have you enabled debug logs? | 10:20 |
| ogra | but thats unrelated to the unsquashfs ... i'm just pointing out that unsquashfs is wasteful and it would be nic to improve it | 10:20 |
| ogra | *nice | 10:21 |
| ogra | i'm not asking for an urgent fix but for considering to improve it if there is time | 10:21 |
| pedronis | ogra: please write something on the forum with data, these IRC discussions will go nowhere | 10:22 |
| ogra | will do | 10:22 |
| Chipaca | ogra: what pedronis said. Thanks! | 10:22 |
| ondra | Chipaca I'd need to use custom snapd snap I guess | 10:22 |
| ondra | Chipaca or how to do it at first boot | 10:22 |
| Chipaca | ondra: no, you just need to drop a file in etc/systemd/system/snapd.service.d | 10:23 |
| ondra | Chipaca also it seems like it's before we start snapd, so logs in snapd will not help | 10:23 |
| Chipaca | ondra: you can do that if you pause the bootloader for ex | 10:23 |
| ogra | it would be cool if we could enable debug options for snpd on the kernel cmdline BT | 10:23 |
| ogra | W | 10:23 |
| ondra | Chipaca can you give me example of that service, I will drop it to the image | 10:23 |
| ogra | adjusting the cmdline on firstboot is very easy .. while injecting a file to the rootfs is hard | 10:24 |
| Chipaca | ondra: $ cat /etc/systemd/system/snapd.service.d/debug.conf | 10:24 |
| Chipaca | [Service] | 10:24 |
| Chipaca | Environment=SNAPD_DEBUG=1 SNAPD_DEBUG_HTTP=7 | 10:24 |
| Chipaca | ondra: ^ I also sent an email asking anybody developing things on snapd to have this, ages ago (and it's all over the forum) | 10:24 |
| Chipaca | ondra: or, it might be easier, to just drop those two env vars in /etc/environment | 10:25 |
| * pstolowski walk, bbiab | 10:25 | |
| Chipaca | ondra: in core18, given it's a funkier service, you might need to adjust the service name | 10:26 |
| ondra | Chipaca Ok thanks, I will try that | 10:27 |
| Chipaca | ondra: i.e. /etc/systemd/system/core18.start-snapd.service.d/debug.conf | 10:27 |
| Chipaca | mvo: I notice this service doesn't source /etc/environment, that's probably not good | 10:27 |
| Chipaca | ondra: or maybe it's snapd.seeding.service? | 10:28 |
| Chipaca | ondra: at this point I don't know which service actually starts snapd :-| | 10:28 |
| Chipaca | in core18 | 10:28 |
| mvo | Chipaca: oh, good point | 10:29 |
| pedronis | Chipaca: different ones at different points | 10:29 |
| Chipaca | mvo: do you know how that dance happens? | 10:29 |
| Chipaca | i can't find a snapd.seeding.service but it's in the bootchart | 10:29 |
| pedronis | there's a bootstrapping one then it does a bit of seeding then snapd restarts | 10:29 |
| mvo | Chipaca: yes, snapd itself writes it | 10:30 |
| pedronis | as the real service I think | 10:30 |
| Chipaca | mvo: from where? | 10:30 |
| pedronis | wrappers/core18.go | 10:30 |
| mvo | Chipaca: i.e. when snapd starts and there is nothing on the system yet it calls wrappers/core18.go | 10:30 |
| Chipaca | pedronis: but there isn't a seeding there either | 10:30 |
| pedronis | are you sure it's seeding? | 10:31 |
| pedronis | there is no such thing afaik | 10:31 |
| pedronis | there's a seeded service | 10:31 |
| pedronis | that one doesn't snapd | 10:31 |
| Chipaca | pedronis: https://private-fileshare.canonical.com/~okubik/bootchart-20191008-1710.svg | 10:31 |
| Chipaca | pedronis: there's a snapd-seeding.service | 10:32 |
| ondra | Chipaca I think this one core18.start-snapd.service | 10:32 |
| pedronis | Chipaca: it's not a service, it's an ephemeral unit | 10:34 |
| pedronis | Chipaca: from here (as ogra said): https://github.com/snapcore/core18/blob/master/static/usr/lib/core18/run-snapd-from-snap | 10:35 |
| * pedronis lunch | 10:35 | |
| Chipaca | ondra: wrt not sure if it's snapd or not, snapd is running from t=39s at least | 10:36 |
| Chipaca | ondra: so debug logs should shed some light on what's going on there | 10:36 |
| ogra | Chipaca, pedronis https://forum.snapcraft.io/t/unsquashfs-vs-mount-during-firstboot-seeding/13614/1 ... (really just a suggestion for times where you guys dont have other things to do :) ) | 10:37 |
| ondra | Chipaca https://private-fileshare.canonical.com/~okubik/boot-plot.svg | 10:38 |
| ondra | Chipaca if you look here, we only mount snap-snapd-5038.mount around 130 second | 10:39 |
| ondra | Chipaca and snapd is started at 140 second | 10:39 |
| Chipaca | yep | 10:39 |
| ondra | Chipaca and my interest is before 140 | 10:40 |
| ondra | Chipaca also from boot-chart you can see that during that time system is relatively idle | 10:41 |
| ondra | once it starts seeding, it pegs on core well, there we would need to consider parallel jobs to speed it up. but that first 100s seems to be as easier to address now | 10:41 |
| Chipaca | ondra: what is it waiting for? | 10:43 |
| ondra | Chipaca you tell me :) | 10:44 |
| ondra | Chipaca why do you thing I'm digging into it, I want to know as well :) | 10:44 |
| Chipaca | ondra: can you stop it inside the initramfs just before it pivots in? | 10:46 |
| Chipaca | i.e. before systemd kicks in | 10:46 |
| ogra | Chipaca, pedronis, ondra https://forum.snapcraft.io/t/allowing-snapd-debug-options-to-be-set-on-kernel-cmdline/13615 | 10:46 |
| Chipaca | but after everything is mounted | 10:46 |
| ogra | :) | 10:46 |
| Chipaca | ogra: tks | 10:46 |
| ogra | we need whishlist tags in the forum :D | 10:47 |
| ondra | Chipaca possibly, what do you want to do there? | 10:47 |
| Chipaca | ondra: add 'set -x' to run-snapd-from-snap :-) | 10:48 |
| ogra | ondra, break=bottom ... then you can modify /writable | 10:48 |
| ogra | (indeed only stuff thats not inside snaps :P ) | 10:48 |
| ondra | Chipaca but that one is core18 snap | 10:49 |
| Chipaca | ondra: copy it somewhere, edit it, bind mount it back | 10:49 |
| ogra | heh | 10:49 |
| ondra | Chipaca you are bringing big guns right away :) | 10:50 |
| mup | PR snapd#7576 opened: spread.yaml,.gitignore: sync ignored files <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/7576> | 10:50 |
| ondra | Chipaca I guess I can also create own core18 snap can't I? | 10:52 |
| Chipaca | ondra: probably, yes :) | 10:52 |
| pedronis | pstolowski: btw, are you using core18 for your spike ? | 10:53 |
| ondra | ogra is that right to have in core18 snap things like dev/null dev/random dev/urandom dev/zero ? | 10:54 |
| ogra | it doesnt do harm | 10:55 |
| ondra | ogra seems to me useless | 10:55 |
| ogra | (but in general these things should be handled by devtmpfs which we mount already in the initrd) | 10:55 |
| ogra | yeah, they likely are ... unless you have a kernel without devtmpfs | 10:55 |
| ogra | (these are rare but i guess they still exist) | 10:56 |
| ondra | Chipaca anything else you want me to modify in core18 snap? set -eux -> set -x? | 10:56 |
| ondra | Chipaca I will sprinkle that file with few echos once on it | 10:56 |
| ogra | set -x should be enough ... you dont need eu (said boris johnson ...) | 10:58 |
| Chipaca | ondra: get some timestamps in there as well | 10:59 |
| Chipaca | ondra: dunno where the log will end up :) | 10:59 |
| Chipaca | hopefully the journal? | 10:59 |
| Chipaca | if so no timestamps needed | 10:59 |
| Chipaca | dunno | 10:59 |
| ondra | Chipaca I will see | 10:59 |
| ondra | Chipaca let me run test boot | 11:00 |
| mborzecki | pstolowski: pedronis: pushed a fix to #7443, we had an actual bug on systems with systemd older than 233 | 11:18 |
| mup | PR #7443: timeutil: fix schedules with ambiguous nth weekday spans <Bug> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7443> | 11:18 |
| pedronis | mborzecki: thanks, I'm a bit confused about the places where the comments mention 28 and the ranges have 29 | 11:21 |
| mborzecki | pedronis: heh, right so the 28th day is part of the range genrated by actual calculation, while 29-31 is the bit that varies from month to month (or year), but technically a month can have anywhere from 28 to 31 days | 11:24 |
| mborzecki | perhaps i should leave a note that 28th is included in the range added further down the code | 11:25 |
| * Chipaca takes a break | 11:27 | |
| pstolowski | re | 11:31 |
| pstolowski | pedronis: no, not yet, i'm using cloud image (core + lxd) | 11:32 |
| pedronis | pstolowski: ok, that will have it's complications | 11:33 |
| pedronis | maybe | 11:33 |
| * zyga -> tea | 11:36 | |
| mup | PR snapd#7577 opened: overlord: set fake sertial in TestRemodelSwitchToDifferentKernel <Created by mvo5> <https://github.com/snapcore/snapd/pull/7577> | 11:38 |
| mborzecki | pedronis: force pushed a comment about 29-31 day range | 11:50 |
| ondra | Chipaca https://paste.ubuntu.com/p/fQQXtCBSnQ/ | 11:56 |
| ondra | Chipaca with different formatting https://paste.ubuntu.com/p/q5MFkKy5cw/ | 11:57 |
| ondra | Chipaca attempts to overlay that file failed though | 11:57 |
| Chipaca | ondra: and is the time you need to know about the time between 11:36:25 (131.827907) and 11:38:20 (247.375225), there? | 11:59 |
| Chipaca | ondra: because if so, that _is_ snapd running, and you should be able to set env vars to get it to print more debug info | 12:00 |
| Chipaca | ondra: add --setenv="SNAPD_DEBUG=1 SNAPD_DEBUG_HTTP=7" to the systemd-run call, there | 12:01 |
| ondra | Chipaca but 'Started Start the snapd services from the snapd snap.' is only printed at the end | 12:02 |
| ondra | Chipaca and don't you connect eligible plugs and slots before you start snapd? | 12:02 |
| Chipaca | ondra: yes, but "started" does not mean "just did the exec" | 12:02 |
| Chipaca | ondra: snapd is the thing doing the connecting | 12:02 |
| ondra | Chipaca true | 12:02 |
| Chipaca | those log lines about connecting come from snapd | 12:02 |
| Chipaca | oooh! kernel parameter! | 12:03 |
| Chipaca | systemd.setenv=SNAPD_DEBUG=1 | 12:03 |
| Chipaca | ^ that should work! | 12:03 |
| Chipaca | says the manpage | 12:03 |
| ondra | Chipaca OK let me test | 12:03 |
| Chipaca | ondra: you can pass it more than once (if you also need _HTTP) but just SNAPD_DEBUG=1 should be enough | 12:04 |
| ondra | Chipaca OK | 12:04 |
| ondra | Chipaca I might got bind mount working | 12:05 |
| ondra | Chipaca so i will get logs from this boot first | 12:06 |
| ondra | Chipaca do you want logs from snapd.service then? | 12:06 |
| Chipaca | ondra: just the ones from that first run | 12:06 |
| ondra | Chipaca but from which service? | 12:07 |
| Chipaca | ondra: snapd-seeding.service | 12:07 |
| ondra | Chipaca do you mean snapd.seeded.service ? | 12:13 |
| Chipaca | ondra: no, I mean snapd-seeding.service | 12:13 |
| Chipaca | ondra: as in what systemd-run starts in that script | 12:13 |
| Chipaca | that's the mystery snapd | 12:14 |
| Chipaca | that's the one you want to look at | 12:14 |
| ondra | Chipaca I cannot see such a service | 12:14 |
| Chipaca | ondra: where are you looking? | 12:14 |
| ondra | systemctl status snapd-seeding.service | 12:14 |
| ondra | Unit snapd-seeding.service could not be found. | 12:14 |
| Chipaca | ondra: it doesn't have a unit file | 12:15 |
| Chipaca | ondra: journalctl will know of it though | 12:15 |
| Chipaca | ondra: journalctl -u snapd-seeding.service should have it | 12:15 |
| ondra | Chipaca OK I need to re-run boot as logs rotated now | 12:15 |
| Chipaca | ondra: wait a mo' | 12:16 |
| ondra | Chipaca how can I increase journal buffer? | 12:16 |
| Chipaca | ondra: there was a way to make it non-ephemeral, maybe ogra knows offhand | 12:16 |
| Chipaca | ondra: systemd.journald.forward_to_syslo | 12:16 |
| Chipaca | syslog* | 12:16 |
| Chipaca | ondra: in the commandline | 12:17 |
| ondra | Chipaca but on core18 we have no syslog | 12:20 |
| ogra | i didnt know there is a cmdline option ... i only know you can create a dir (was it /var/log/journal ???) that will then automagically be used for journald logs | 12:20 |
| Chipaca | ondra: I didn't know that :) | 12:20 |
| ogra | yeah, no more syslog | 12:20 |
| ondra | Chipaca will it create it? | 12:20 |
| Chipaca | no | 12:20 |
| ondra | Chipaca then no syslog | 12:20 |
| Chipaca | bah, i dunno | 12:20 |
| ogra | i guess it will just try to write to logger | 12:20 |
| Chipaca | but | 12:21 |
| Chipaca | ondra: but | 12:21 |
| ogra | (and not find it ...) | 12:21 |
| ondra | Chipaca so I do actually have a log from seeding service | 12:21 |
| Chipaca | ondra: edit that script, and redirect the output from systemd-run to somewhere | 12:21 |
| Chipaca | that should work | 12:21 |
| ogra | well, creating that dir should also work for persistent logs | 12:21 |
| Chipaca | ondra: actually it might _not_ be in the journal, it might all be spewed out from the script | 12:21 |
| ondra | Chipaca but it's useless https://paste.ubuntu.com/p/sw5sNBp6WK/ | 12:21 |
| ogra | they are binary files though | 12:21 |
| Chipaca | I could check but I'm not sure it's the same in 16 and 18 | 12:21 |
| Chipaca | ondra: that's not debug output :-| | 12:22 |
| ondra | Chipaca booting now with debug env from cmd line | 12:22 |
| Chipaca | and I can confirm the output from the systemd-run does reach the journal (but I think your recent pastebin does, too) | 12:24 |
| ondra | Chipaca actually reading logs over ssh seems to messed it up, check this https://paste.ubuntu.com/p/ZFB78ycbMz/ | 12:26 |
| Chipaca | ondra: if the rotation is still getting in the way, we can either tweak journald.conf, or have it forward to kmsg and bump the kmsg buffer (but that might be expensive on a low lowmem board) | 12:26 |
| ondra | Chipaca it's fine if I know what logs to get there is time to do it once I can access seeded device | 12:27 |
| Chipaca | ondra: is that with -u snapd-seeding.service ? | 12:27 |
| Chipaca | ondra: it looks like the logs of the start-snapd-wotsit script | 12:27 |
| Chipaca | which would not be in snapd-seeding.service log | 12:27 |
| Chipaca | (and yes there is noise in that that could be avoided... :-| ) | 12:28 |
| ondra | Chipaca sudo journalctl --all -u snapd-seeding.service is always empty | 12:30 |
| Chipaca | ondra: what was https://paste.ubuntu.com/p/sw5sNBp6WK/ ? | 12:30 |
| ondra | Chipaca core18.start-snapd.service | 12:31 |
| Chipaca | gr | 12:31 |
| ogra | welcome to our world :P | 12:31 |
| Chipaca | ondra: and if you foraward_to_kmsg ? | 12:32 |
| ogra | (debugging the indebuggable) | 12:32 |
| Chipaca | ondra: I've got another option | 12:33 |
| ondra | Chipaca OK now with extra verbosity I do not get start of the log :( | 12:34 |
| Chipaca | ondra: forwarding to kmsg? | 12:34 |
| Chipaca | ondra: or without? | 12:34 |
| Chipaca | I don't know which of all the options you've gone for :) | 12:34 |
| ondra | Chipaca without | 12:34 |
| ondra | Chipaca BTW it's clear from journalcrl of whole system, there is only kernel chatty from 40 to 140 second | 12:35 |
| ondra | Chipaca no changes to where journal goes for now | 12:35 |
| Chipaca | ondra: ok, I'm downloading a core18 for amd64, and will play around with it until I can get logs for that first boot snapd | 12:37 |
| Chipaca | ondra: I'll let you know | 12:37 |
| ondra | Chipaca looks like we have no journald.conf at all, so it should be easy to add one to etc | 12:37 |
| Chipaca | ondra: feel free to do so :) | 12:37 |
| Chipaca | seems silly that it doesn't have a kernel option for setting storage as it has for everything else | 12:38 |
| Chipaca | hmm | 12:44 |
| Chipaca | ondra: so, on amd64, all I need to do is set systemd.setenv=SNAPD_DEBUG=1 in the kernel commandline, and when I log in after creating my account and etc I have logs for snapd-seeding.service | 12:44 |
| Chipaca | ondra: https://paste.ubuntu.com/p/kSmxKc6zKw/ | 12:45 |
| ogra | ogra@pi4:~$ ls /etc/systemd/journald.conf | 12:46 |
| ogra | /etc/systemd/journald.conf | 12:46 |
| ogra | my core18 image has journald.conf here | 12:46 |
| ondra | ogra not mine | 12:46 |
| ogra | weird | 12:47 |
| ondra | ogra I can add it, but how do I increase max size | 12:47 |
| ogra | given we are most likely using the exact same core18 snap | 12:47 |
| ondra | it's set to 10M | 12:47 |
| Chipaca | ondra, ogra: where did you get yours from? | 12:47 |
| ondra | ogra also it is not in core snap | 12:47 |
| ogra | mine is my pi4 custom image ... but that uses core18 from stable | 12:48 |
| ogra | ogra@pi4:~$ snap info core18|grep refresh | 12:48 |
| ogra | refresh-date: 6 days ago, at 22:09 UTC | 12:48 |
| Chipaca | journald.conf is there, and writable, in the amd64 core | 12:49 |
| Chipaca | core18* | 12:49 |
| ogra | same here | 12:49 |
| ogra | ogra@pi4:~$ sudo touch /etc/systemd/journald.conf | 12:49 |
| ogra | ogra@pi4:~$ | 12:49 |
| ondra | ogra ah RuntimeMaxUse= | 12:49 |
| Chipaca | ondra: what core18 are you running, that it doesn't have this file? | 12:50 |
| ondra | Chipaca edge | 12:51 |
| ondra | Chipaca rev 1202 | 12:51 |
| Chipaca | ondra: what arch? | 12:52 |
| ondra | armhf | 12:52 |
| ogra | same here ... but stable | 12:53 |
| Chipaca | ondra: core18 1202 has etc/systemd/journald.conf | 12:54 |
| ondra | Chipaca armhs? | 12:55 |
| Chipaca | 1202 is armhf | 12:55 |
| ogra | 1202 | 12:55 |
| ogra | :) | 12:55 |
| ogra | would be surprising if it vamished all of a sudden | 12:55 |
| Chipaca | ondra: UBUNTU_STORE_ARCH=armhf snap download core18 --edge | 12:55 |
| Chipaca | ondra: unsquashfs core18_1202.snap etc/systemd/journald.conf | 12:55 |
| Chipaca | ondra: less squashfs-root/etc/systemd/journald.conf | 12:56 |
| Chipaca | ¯\_(ツ)_/¯ | 12:56 |
| Chipaca | ogra: yeah it'd be a bug | 12:56 |
| ondra | Chipaca OK my bad, I must have had type when searching | 12:56 |
| ondra | sorry I have it too :) | 12:56 |
| ackk | sergiusens, hi, any chance you could take a look at https://github.com/sergiusens/snapcraft-preload/pull/34 and https://github.com/sergiusens/snapcraft-preload/pull/36 ? | 12:57 |
| mup | PR sergiusens/snapcraft-preload#34: preserve LD_PRELOAD from the calling environment (fixes #33) <Created by albertodonato> <https://github.com/sergiusens/snapcraft-preload/pull/34> | 12:57 |
| mup | PR sergiusens/snapcraft-preload#36: use SNAP_INSTANCE_NAME instead of SNAP_NAME for prefixing /dev/shm fi… <Created by albertodonato> <https://github.com/sergiusens/snapcraft-preload/pull/36> | 12:57 |
| Chipaca | ondra: so now I'm suspicious of your "it's empty" :-| | 12:57 |
| Chipaca | anyway, i've got an up to stand | 12:57 |
| ondra | Chipaca non it is not :) | 12:58 |
| ondra | Chipaca ogra anyway it only exists in core snap, and not in overlay /etc | 12:59 |
| ondra | which is probably good, as I can add mine | 12:59 |
| zyga | sorry auth | 12:59 |
| ondra | I'm increasing buffer 4times so I should have now more time | 12:59 |
| Chipaca | core18 has an overlay /etc ?? | 12:59 |
| ondra | Chipaca overlay of /etc/systemd/ | 13:00 |
| ondra | Chipaca how would we add new units :) | 13:00 |
| ogra | ogra@pi4:~$ grep etc/systemd /etc/system-image/writable-paths | 13:02 |
| ogra | /etc/systemd auto persistent transition none | 13:02 |
| ogra | ogra@pi4:~$ cat /etc/issue | 13:02 |
| ogra | Ubuntu Core 18 on \4 (\l) | 13:02 |
| ogra | if you dont have the journald.conf file then something with your initrd is broken | 13:03 |
| ogra | (transition means it copies the content from core to /writale on first mount) | 13:04 |
| Chipaca | ondra: sorry, overlay is a bad word (overlayfs ...) | 13:04 |
| ogra | well, the point is that it *should* be in the overlay /etc | 13:05 |
| ogra | if it isnt, there is a bigger prob | 13:05 |
| ogra | ogra@pi4:~$ ls /writable/system-data/etc/systemd/journald.conf | 13:05 |
| ogra | /writable/system-data/etc/systemd/journald.conf | 13:05 |
| ondra | ogra did we find more bugs ? :) | 13:05 |
| ogra | smells like ... if you are sure the file inst there at least | 13:05 |
| ogra | *isn't | 13:06 |
| ondra | ogra if I have file there, will ill override it? | 13:06 |
| ogra | if you have the file there it is fully writable | 13:06 |
| ogra | so you ccan just edit | 13:06 |
| ondra | ogra no if I have already file there, it will not nuke it, right? | 13:07 |
| ondra | ogra ignore me it should not | 13:07 |
| ogra | yeah, it wont | 13:08 |
| ogra | transition only copies once on first mount ... in subsequent mounts it just uses whats there | 13:09 |
| ogra | (so you could even create a new file there) | 13:09 |
| ondra | Chipaca https://paste.ubuntu.com/p/bq6b39PBJ4/ | 13:10 |
| === ricab is now known as ricab|lunch | ||
| Chipaca | ondra: so there you go | 13:11 |
| Chipaca | ondra: and, snap debug timings 1, for more info | 13:11 |
| ondra | Chipaca ? | 13:13 |
| ondra | Chipaca how to get more logs? | 13:13 |
| ogra | "error trying to compare the snap system key: system-key missing on disk" | 13:13 |
| ogra | is that relevant ? | 13:13 |
| Chipaca | ogra: nah, that's expected on first boot | 13:13 |
| ogra | or does the system-key only come in with initialize-device ? | 13:13 |
| ogra | right | 13:13 |
| ogra | so the gap is before "Load: validated crc32" ... i guess the checksumming is slow then ? | 13:14 |
| ogra | (once again) | 13:14 |
| roadmr | crc32 wow what a throwback | 13:15 |
| ondra | ogra no that is check of lk boot env | 13:15 |
| ondra | ogra that takes 1ms | 13:16 |
| ogra | hmm, well, there is a long gap before it runs | 13:17 |
| Chipaca | ondra: did you run 'snap debug timings 1'? | 13:17 |
| ondra | ogra exactly, and I want to know why | 13:17 |
| Chipaca | ondra: or even just 'snap changes' | 13:18 |
| ogra | ch-ch-ch-changesss | 13:18 |
| * ogra humms david bowie | 13:18 | |
| ondra | Chipaca https://paste.ubuntu.com/p/FsdG3V4Fjg/ | 13:18 |
| mup | PR snapd#7578 opened: sandbox/cgroup, overlord/snapstate: move helper for listing pids in group to the cgroup package <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7578> | 13:19 |
| mborzecki | zyga: ^^ | 13:19 |
| zyga | thanks | 13:19 |
| ogra | ondra, that doesnt seem to list the bootloader stuff ... perhaps check change 2 instead ? | 13:19 |
| Chipaca | ondra: also, --verbose | 13:20 |
| ondra | ogra that only shows last 2 changes :) | 13:20 |
| ondra | Chipaca no it always start at number 3 | 13:21 |
| Chipaca | those are task numbers, not change numbers | 13:21 |
| Chipaca | but | 13:21 |
| Chipaca | ondra: snap debug timings --verbose 1 | 13:21 |
| Chipaca | ondra: and, as ogra said, also look at change 2 | 13:22 |
| Chipaca | change 2 will have the keys generation | 13:23 |
| zyga | mborzecki: look again | 13:24 |
| ondra | Chipaca it does not give me change 1 or 2 https://paste.ubuntu.com/p/Yj5Q2HYpp9/ | 13:26 |
| Chipaca | ondra: the "1" in "snap debug timings --verbose 1" is \change 1 | 13:27 |
| Chipaca | ondra: please also run the same command but for change 2, i.e. "snap debug timings --verbose 2" | 13:27 |
| ondra | Chipaca and 'snap debug timings --verbose 2' will only print timing around change 206 | 13:27 |
| Chipaca | ondra: the numbers under the ID column in that output are about tasks | 13:27 |
| ondra | Chipaca ah sorry | 13:28 |
| ogra | soooo complicated ! | 13:28 |
| Chipaca | ondra: what else do you have in "snap changes"? | 13:28 |
| ondra | Chipaca so change 2 is two tasks, and both take 1 combined | 13:28 |
| ondra | Chipaca just starting services and initialize device | 13:29 |
| Chipaca | ondra: can I see the output of 'snap changes' please | 13:29 |
| ogra | what do you pay ? | 13:30 |
| ondra | Chipaca https://pastebin.canonical.com/p/pzJTRfPtdq/ | 13:30 |
| ogra | :P | 13:30 |
| Chipaca | ondra: ok, the changes you want to look at are 1 and 6 | 13:30 |
| ogra | yeah, 6 | 13:30 |
| ondra | Chipaca https://paste.ubuntu.com/p/dtgwn5C9SK/ | 13:31 |
| ondra | is that waiting 56s on key generation? | 13:31 |
| Chipaca | ondra: yes | 13:32 |
| Chipaca | ondra: so | 13:32 |
| Chipaca | ondra: your random is probably empty | 13:32 |
| mborzecki | zyga: replied, i had a really trivial check in mind there | 13:32 |
| ogra | two times actually, no ? | 13:32 |
| ondra | but is that blocking boot? | 13:32 |
| Chipaca | ondra: get more randomses | 13:32 |
| Chipaca | ogra: no, one is the sub of the other | 13:32 |
| ogra | ah, k | 13:32 |
| Chipaca | ondra: yes, you can't boot with an empty random pool | 13:32 |
| ogra | right, i see the indendation in the summary when looking closer :) | 13:33 |
| ogra | ondra, did you already add the cmdline changes for rng_core ? | 13:33 |
| Chipaca | ondra: your device probably has a hardware rng you can plug in … somehow? | 13:33 |
| ondra | Chipaca still even if blocking, and that's a bit odd to block at that stage, it only explain 56s there | 13:33 |
| ondra | ogra yep rng_core is there | 13:34 |
| ondra | ogra rng_core.default_quality=700 | 13:34 |
| ogra | yeah, that should have improved a little bit ... (i'd expect a few seconds less ... so 80-90 vs the 100 before) | 13:35 |
| ondra | ogra I do not think it improved | 13:36 |
| ogra | well, perhaps a missing kernel config ? rng_core is tied to the kernel SW rng | 13:36 |
| ogra | (and will bee quietly ignored if the kernel doesnt have it) | 13:36 |
| ondra | ogra what config would be that? | 13:37 |
| ondra | Chipaca when do we need that random number first time? | 13:37 |
| ondra | ogra CONFIG_HW_RANDOM ? | 13:38 |
| ondra | ogra as that is enabled | 13:39 |
| ogra | CONFIG_HW_RANDOM and sub-options of that (see menuconfig) | 13:39 |
| === chesty_ is now known as chesty | ||
| ondra | ogra is there actually device node which I should see? | 13:41 |
| ogra | ondra, tyhicks was the one that went over the whole rng stuff back then ... he might know what other options are needed (he also suggested the cmdline option ... which back then helped a lot on a beaglebone where we saw the slowness first9 | 13:41 |
| ogra | not with in-kernel rng | 13:42 |
| ogra | if you want to use a device node there is likely some other kernel config option thats specific to that HW | 13:42 |
| ogra | and gives you HW rng ... (if the SoC actually has it at all) | 13:42 |
| ogra | you probably want to take a look at the original android config | 13:43 |
| ondra | ogra like CONFIG_HW_RANDOM_MTK :) | 13:43 |
| ondra | ogra and enabled | 13:43 |
| ogra | fooor example :) | 13:43 |
| ogra | tat might need userspace bits ... not sure | 13:43 |
| Chipaca | ondra: you probably have things in dmesg about not having enough entropy? | 13:45 |
| Chipaca | when we need it → to generate keys | 13:45 |
| roadmr | 🔑 | 13:46 |
| ogra | we should just generate them at build time ... makes everything easier ... and everyone has the same key ... :) | 13:46 |
| ogra | ondra, btw, do you also see the slowness on second boot ? | 13:47 |
| ogra | (i assume not) | 13:47 |
| ondra | ogra nope, sub 30 seconds | 13:47 |
| ogra | perhaps we should simply tell the customer to boot the device once in-factory | 13:47 |
| ondra | ogra from kernel code, there is driver using TZ to generate rn, so this should be fast | 13:48 |
| ogra | make it part of the flashing procedure | 13:48 |
| ogra | "power on ... leave it sit for 5min" | 13:49 |
| ogra | (unconnected) | 13:49 |
| ogra | timer wont be enough to get entropy | 13:55 |
| ogra | you need interrupts | 13:55 |
| ogra | attach a mouse and wiggle it :) | 13:55 |
| ondra | Chipaca stupid question but task ID is assigned when task is started, is that correct? | 13:55 |
| ondra | Chipaca and tasks from change id 1 will run before tasks from change 2 | 13:55 |
| ondra | Chipaca because random number generation is run as task 200+ | 13:55 |
| ondra | Chipaca this is way after snaps were seeded | 13:55 |
| === ^arcade_droid is now known as zarcade_droid | ||
| === cjp256_ is now known as cjp256 | ||
| === arnatious_ is now known as arnatious | ||
| ondra | wow lot's of dead ppl here | 13:56 |
| * Chipaca got better | 13:56 | |
| ondra | Chipaca did you see my question about task IDs? | 13:56 |
| ondra | Chipaca is that ID assigned when task was started? | 13:57 |
| ondra | Chipaca so I can assume they go in sequence? | 13:57 |
| Chipaca | ondra: I did not see your question | 13:57 |
| ondra | Chipaca OK last two question I did ask now | 13:57 |
| Chipaca | ondra: they go in sequence at the time of task creation, which might not line up exactly with task execution | 13:57 |
| ondra | Chipaca OK, so then this random key generation might take some time, but it's run way way later | 13:58 |
| ondra | Chipaca so it's not that 100s we are looking into | 13:58 |
| ondra | Chipaca it's task id 200+ and change 6 | 13:58 |
| ondra | Chipaca we are looking into 100s before or at the beginning of change 1 | 13:59 |
| ondra | Chipaca like task id 1 and 2 | 13:59 |
| Chipaca | ondra: I … what? why? | 13:59 |
| Chipaca | ondra: if it's before snapd starts, then it's somebody else you should be talking to | 13:59 |
| ondra | Chipaca that 100 is before we start seeding, rsa key is generated after seeding | 14:00 |
| ondra | Chipaca nope, it's this log https://paste.ubuntu.com/p/5jx9kcfZXV/ | 14:01 |
| ondra | Chipaca or in that seeding log, there is that pause | 14:02 |
| ondra | Chipaca rsa-key comes after seeding | 14:02 |
| Chipaca | ondra: ok | 14:03 |
| ondra | Chipaca so can I get more logs there? | 14:05 |
| Chipaca | ondra: did you ever get me a 'snap debug timings --verbose 1'? i might have closed it already | 14:07 |
| ondra | Chipaca yes I did, https://paste.ubuntu.com/p/Yj5Q2HYpp9/ | 14:07 |
| ondra | Chipaca task 4 there is too late, this is after that quiet period | 14:08 |
| Chipaca | ondra: 101ms is too late? | 14:09 |
| Chipaca | ondra: can you actual timestamps on the journal output, or is it too late for that? | 14:10 |
| Chipaca | ondra: the way I read the one you pasted is probably wrong | 14:10 |
| ondra | Chipaca probably too late now | 14:10 |
| ondra | Chipaca I have timing from start of the boot | 14:11 |
| Chipaca | ondra: ok, maybe you can help me then | 14:11 |
| Chipaca | ondra: https://paste.ubuntu.com/p/bq6b39PBJ4/ | 14:11 |
| ondra | Chipaca which is then 0 | 14:11 |
| Chipaca | in that one | 14:11 |
| Chipaca | ondra: the numbers in the left column are seconds in offset since the 'logs start', yes? | 14:11 |
| ondra | Chipaca time in that log is relative from 0 which is device power on | 14:11 |
| Chipaca | ondra: ah | 14:11 |
| ondra | Chipaca since device power on | 14:11 |
| Chipaca | ondra: and how do I compare with other things? | 14:12 |
| ondra | Chipaca and yes, seconds from device power on | 14:12 |
| ondra | Chipaca all the logs and boot chart is in seconds from device power on | 14:12 |
| ondra | Chipaca snap changes are not though | 14:12 |
| Chipaca | nor is '-- Logs begin at Wed 2019-10-09 13:00:46 UTC, end at Wed 2019-10-09 13:10:02 UTC. --' | 14:13 |
| ondra | Chipaca so depends what do you want to compare it to? | 14:13 |
| ondra | Chipaca OK there you go, you have also log start | 14:13 |
| Chipaca | ondra: that doesn't help, does it? | 14:14 |
| Chipaca | ondra: what wall clock time does the gap happen at? | 14:14 |
| ogra | bootchart looks at the kernel counter | 14:15 |
| ogra | which starts at 0 when the kernel fires up | 14:15 |
| ondra | Chipaca but what are you interested in? Absolute time when service started, or actual time processing | 14:15 |
| ondra | ogra and I did same for journal logs | 14:15 |
| ogra | (same thing you get with dmesg if you dont use -T ) | 14:15 |
| ondra | ogra so you can link them to bootchart | 14:16 |
| ondra | ogra and same you get with journalctl -o short-monotonic | 14:16 |
| ogra | ah, i didnt knwo that one (i rarely want the kernel timer anyway) | 14:16 |
| ondra | Chipaca but if I tell you wall clock time wha will you compare it to? | 14:17 |
| Chipaca | ondra: what's the output of 'snap debug timings --verbose --ensure=seed' ? | 14:17 |
| ondra | Chipaca https://pastebin.canonical.com/p/3NRRGZYJ9S/ | 14:18 |
| ondra | Chipaca '75739ms - state-from-seed populate state from seed'? | 14:18 |
| Chipaca | ondra: how many more seconds do you need to find? | 14:18 |
| ondra | Chipaca what is that? | 14:18 |
| ondra | Chipaca what do you mean 'how many more seconds'? | 14:19 |
| Chipaca | ondra: last time I found you 50s and your first reaction was "i need more" | 14:19 |
| Chipaca | so here i am finding you more | 14:19 |
| Chipaca | that's a chunk of time | 14:19 |
| ondra | Chipaca no, that 50 seconds you found is when services run | 14:19 |
| ondra | Chipaca device communicated | 14:20 |
| ondra | Chipaca so they are not helping me at all | 14:20 |
| === ricab|lunch is now known as ricab | ||
| Chipaca | yes, that was your second reaction | 14:20 |
| ondra | Chipaca that's 50 seconds to connect to the store to check for update, but at that point device will be already communicating with user, so that 50s is for me irrelevant | 14:21 |
| Chipaca | ondra: I regret trying to help you, now | 14:21 |
| ondra | Chipaca well then just tell me "yep, during first boot we stop for 100s before we start seeding, this is feature" | 14:22 |
| Chipaca | the first thing I told you before trying to help you was that years ago I pointed out that first boot on armhf takes 5+ minutes because of things | 14:23 |
| Chipaca | and now, several hours of trying to walk you through it all I point you at the logs that show you that same bit of information | 14:23 |
| Chipaca | I have no idea what you're so upset about at this point | 14:23 |
| ondra | Chipaca this is smart speaker, it takes 5 minutes to boot, before it can connect to the phone. that 5 minutes does not include or is affected by that 50 seconds key generation, as you do not have internet anyway. | 14:23 |
| Chipaca | but you have the data, right there | 14:23 |
| ondra | Chipaca but you can connect over the BT to provision WiFi, while device is generating rsa-key | 14:24 |
| Chipaca | I am not talking about those 50 seconds for key generation, I am talking about _these_ 300 seconds of seeding | 14:24 |
| ondra | Chipaca so it's not having any effect on user experience | 14:24 |
| ondra | Chipaca I accept seeding takes time, related to number of snaps | 14:25 |
| ondra | Chipaca but snapd snap seeds for 150 seconds, that seems odd | 14:25 |
| ondra | Chipaca and nobody complains about that, I'm just wondering why we have 100s of nothing, Now you explain me this is feature. And you should just take it | 14:27 |
| ondra | Chipaca thanks for help | 14:27 |
| Chipaca | ondra: at what point did I say this was a feature and you should just take it | 14:31 |
| ondra | Chipaca at point that you said I was not happy with 50s discovery, which are post seeding | 14:32 |
| Chipaca | ...? | 14:32 |
| ondra | Chipaca I told you those make no difference | 14:32 |
| Chipaca | ondra: I still don't see how I told you that. I see you saying I did, though. | 14:34 |
| Chipaca | or maybe saying I _should_ | 14:35 |
| Chipaca | but I won't, because it isn't | 14:35 |
| Chipaca | ondra: could you please snap install http, and pastebin 'http snapd:///v2/changes select==all' | 14:39 |
| ondra | Chipaca https://pastebin.canonical.com/p/k8Q6ryzwhn/ | 14:45 |
| Chipaca | ondra: where does 2019-10-09T13:06:29.005541125Z fall WRT the missing time? | 14:47 |
| Chipaca | is that before or after the time we're looking for? | 14:47 |
| ondra | Chipaca that's after | 14:48 |
| ondra | Chipaca I think we are looking for things before 2019-10-09 13:03:00 | 14:50 |
| ondra | Chipaca more precise between 2019-10-09 13:01:26 and 2019-10-09 13:03:00 | 14:51 |
| Chipaca | ondra: where do those timestamps fall in https://paste.ubuntu.com/p/bq6b39PBJ4/ ? | 14:51 |
| ondra | Chipaca between 40 and 140 | 14:55 |
| ondra | Chipaca from what I understand, logs star is when kernel started | 14:56 |
| * zyga calls it a day | 15:03 | |
| Chipaca | ondra: just in case, is there anything in any of the other --ensure= options? | 15:10 |
| Chipaca | anything in the times we're looking at | 15:10 |
| ondra | Chipaca I did get whole journal if you want | 15:13 |
| ondra | Chipaca I could not see anything there though | 15:13 |
| ondra | Chipaca let me paste it for you | 15:14 |
| Chipaca | ondra: there's a little more detail in the timings output | 15:14 |
| Chipaca | ondra: can you patch the snapd for first boot, or is that too much? | 15:15 |
| ondra | Chipaca not that is actually fairly easy | 15:15 |
| ondra | Chipaca unlike core as I learnt | 15:15 |
| ondra | Chipaca https://pastebin.canonical.com/p/Bzv44GKwmD/ | 15:16 |
| Chipaca | ondra: out of curiosity, what are these about? | 15:17 |
| Chipaca | [ 42.310781] localhost kernel: CPU3 killed. | 15:17 |
| ondra | Chipaca just some MTK kernel verbosity :) | 15:19 |
| Chipaca | hmm | 15:19 |
| ogra | Chipaca, at least it is a friendly killer ... i have seen other boards where daemons kill their children and such ... way more blood there | 15:33 |
| Chipaca | ondra: so if I give you a patch for snapd you can apply it to your snapd? | 15:33 |
| ondra | Chipaca yep | 15:34 |
| ondra | Chipaca that wont' be problem | 15:34 |
| Chipaca | ondra: http://paste.ubuntu.com/p/DWYXndYs5C/ | 15:35 |
| Chipaca | ondra: won't be too informative but hopefully it'll give us somewhere to start looking | 15:35 |
| Chipaca | hmm | 15:36 |
| Chipaca | ondra: it'll need the debug trick still | 15:36 |
| ondra | Chipaca sure I will let you know once I have it | 15:38 |
| === pstolowski is now known as pstolowski|afk | ||
| * cachio lunch | 15:47 | |
| ondra | Chipaca it will take some time, as my LP git sync is now hanging to run sync 'as soon as possible' for past 15 minutes | 15:52 |
| ondra | Chipaca time is relative thing | 15:52 |
| ogra | blame einstein | 16:03 |
| Chipaca | only important if you're going fast, so it doesn't apply to this board | 16:04 |
| * Chipaca hides | 16:04 | |
| cjwatson | If you're routinely waiting for an LP git import, it's probably a sign that you should be hosting the repository directly on LP instead. Imports are best-effort | 16:07 |
| ogra | but thats less funny because you cant complain then ... | 16:12 |
| ondra | cjwatson it's a bit pain to push to two locations as snapd is in github, but I guess best option | 16:20 |
| ondra | cjwatson as now I'm waiting over half an hour and still nada | 16:20 |
| roadmr | folks, so: mir-kiosk has a plugs: - x11, so it offers x11 for other snaps to connect to it, right? | 16:23 |
| cjwatson | ondra: Hm, one of our importds doesn't seem to be doing much though, let me see | 16:23 |
| ondra | cjwatson thanks | 16:27 |
| ondra | cjwatson I did create copy git repo to kick build now, so it should be OK | 16:27 |
| Chipaca | ondra: I'm off but I'll check back later tonight | 16:32 |
| ondra | Chipaca no worries I have some other tasks anyway, let's check tomorrow | 16:32 |
| cjwatson | ondra: Repaired the stuck worker, so it should be doing a better job of keeping up with imports now anyway | 17:16 |
| ondra | cjwatson thank you :) | 17:18 |
| ondra | Chipaca https://paste.ubuntu.com/p/WMKkCPSBjg/ | 17:32 |
| * cachio afk | 17:47 | |
| ogra | roadmr, i think that plug is only for running mir-kiosk on desktop systems .... to actually run an x11 app on top of mir-kiosk (which only offers wayland to apps) your app needs to ship Xwayland and set that up through a wrapper .... the app also needs a loopback slot/plug x11 setup | 18:00 |
| ogra | roadmr, i.e. line 50,56 and 81 in https://github.com/ogra1/opencv-demo-snap/blob/master/snap/snapcraft.yaml | 18:01 |
| roadmr | thanks ogra, that'll be useful as a reference | 19:24 |
| mup | PR snapd#7579 opened: interfaces/network-setup-observe: add Info D-Bus method accesses <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7579> | 20:25 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
