[07:22] <ackk> hi, I'm getting a weird error when testing the maas snap. In logs for various processes I see "mkdir: cannot create directory ‘/var/snap/maas’: Read-only file system"
[07:22] <ackk> I tried both restarting the snap and the whole system
[07:34] <pedronis> mvo: thanks for reviewing again verify-boot, I merged it now
[07:51] <ackk> anyone might have an idea on how to debug this issue?
[07:51] <ackk> I'm using the non-root-user enabled snapd 2.40
[07:57] <mvo> pedronis: cool, thank you!
[08:30] <pedronis> Chipaca: hi
[08:31] <Chipaca> pedronis: hiya
[08:31] <Chipaca> pedronis: not ready for you yet i'm afraid
[08:31] <Chipaca> :-|
[08:32] <pedronis> Chipaca: late today, or are we talking monday?
[08:32] <Chipaca> pedronis: hmm. I might make it for late today, but monday is safer -- what'd you rather?
[08:33] <pedronis> Chipaca: monday is easier
[08:33] <pedronis> Chipaca: can you put an actual meeting in the calendar for monday when it works for you
[08:34] <Chipaca> pedronis: monday it is then
[08:34] <Chipaca> i'll probably sneak in some time on this over the weekend to make up for this week which has been less than productive
[08:34] <Chipaca> hopefully school being out and the boys wanting to spud out for a while will be conducive
[08:36] <Chipaca> pedronis: calendaered
[08:37] <Chipaca> pedronis: you've got Edit, so you can move it if you need/want to
[08:38] <pedronis> Chipaca: you picked the wrong monday I think
[08:38] <pedronis> 29 vs 22
[08:38] <Chipaca> pedronis: augh
[08:38] <Chipaca> moved
[08:38] <Chipaca> oh wait now it conflicts with my 1:1
[08:38] <Chipaca> moving again
[08:39] <Chipaca> done
[08:41] <pedronis> Chipaca: thanks
[08:43] <pedronis> btw my other PRs are ready for reviews,  and 6923 from pawel is unblocked and needs a 2nd review
[09:36] <Chipaca> pedronis: +1 to both of yours, and +1 to pawel's even though I hate «for _, cstrs1 := range cstrs»
[09:36] <Chipaca> casters? coasters? castraters?
[09:38] <Chipaca> and with that, i'm out for a few hours
[09:39] <Chipaca> should be back in ~4h modulo everything
[09:54] <pedronis> mvo: https://forum.snapcraft.io/t/broken-dependency-of-content-snaps-during-seeding/11566/18 you meant (classic) image builds instead of deb packages ?
[09:57] <mvo> pedronis: eh, yes, let me fix htis
[09:57] <mvo> pedronis: thanks
[09:57] <pedronis> np
[09:57] <ogra> gra@pi4:~$ grep PRETTY /etc/os-release
[09:57] <ogra> PRETTY_NAME="Ubuntu Core 18"
[09:57] <ogra> ogra@pi4:~$ ps ax|grep unattended
[09:57] <ogra>  2209 ?        Ssl    0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
[09:57] <ogra> mvo, ^^^^ ??????
[09:58] <mvo> ogra: check with rbalint
[09:58] <mvo> ogra: but it seems thats the only reliable way these days to hook into the shutdown
[09:58] <ogra> mvo, oh, so the name is misleading ?
[10:00] <mvo> ogra: eh, uh - this is on core18
[10:00] <ogra> yes
[10:00] <mvo> ogra: sorry, missed that - that is - unexpected
[10:00] <mvo> ogra: and unwanted :/
[10:01] <mvo> ogra: I wonder when this sneaked in :(
[10:01] <ogra> right ... that package shoundt be there at all
[10:01] <mvo> (and why)
[10:01] <mvo> ogra: totally
[10:01] <mvo> ogra: there should also be a big dependency issue, like apt is removed during the build
[10:02] <ogra> note that i have a few lxd containers running ... i wonder if that makes it sneak in
[10:03] <mvo> ogra: that could be, let me look at the build logs
[10:03] <ogra> ogra@pi4:~$ lxc stop xenial
[10:03] <ogra> ogra@pi4:~$ ps ax|grep unattended
[10:03] <ogra> 16700 pts/0    S+     0:00 grep --color=auto unattended
[10:03] <ogra> there we go !
[10:03] <mvo> ogra: aha, ok
[10:03] <mvo> ogra: thats fine then
[10:03] <ogra> mvo, sorry, false alarm then
[10:03] <ogra> yeah
[10:05] <zyga> pre-last day of driving home, see you all on Monday
[10:08] <mvo> ogra: no worries
[10:56] <ogra> jdstrand, hmm, i'm trying to snap a script that reads /proc/diskstats and blinks one of the RPi LED's based n changes there but i cant seem to find any interface that allows me to write to /sys/class/leds/led1/trigger and /sys/class/leds/led1/brightness
[10:57] <ogra> (seems only display-control allows access to the kbd LED's)
[10:57] <ogra> ... but nothing for generic system ones
[10:57] <abeato> zyga, hey, do you know which is the current status for snaps in yocto?
[10:58] <ogra> abeato, https://forum.snapcraft.io/t/yocto-rocko-core-snap-panic/3261 perhaps ?
[10:59] <ogra> hmm, there are newer threads too ...
[11:00] <ogra> https://forum.snapcraft.io/t/question-about-meta-snappy-layer-for-yocto/10315
[11:00] <abeato> ogra, thanks, I see there are links to intructions in snapd repo
[11:00] <ogra> apparently you want https://github.com/morphis/meta-snappy
[11:02] <abeato> ha, nice
[11:23] <mup> Bug #1837209 opened: Splash screen fails to display on recent pi core18 images <Snappy:New> <https://launchpad.net/bugs/1837209>
[11:27] <mup> PR snapcraft#2632 closed: pluginhandler, repo: find stage-packages from DT_NEEDED on host <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2632>
[12:00] <jdstrand> ogra: hey, bool-file can do leds. you need a gadget like with gpio
[12:01] <ogra> jdstrand, ah, thanks ! will look into that (we should have something by default in the pi gadget then)
[12:02] <jdstrand> ogra: that said, it is quite old and doesn't operate exactly like gpio. it looks like the slot is expected to do the export/unexport
[12:09] <ogra> /usr/share/subiquity/console-conf-wrapper: line 13: snap: command not found
[12:09] <ogra> Press enter to configure.
[12:09] <ogra> HRM !
[12:13] <diddledan> HRM?
[12:13] <ogra> yeah
[12:13] <diddledan> I know not of this acronym
[12:13] <ogra> no snap command on current core18 pi images ... or my SD is screwed up
[12:14] <diddledan> oh dear :-(
[12:15] <diddledan> is it because snapd was originally in core?
[12:15] <ogra> might be .. let me re-try
[12:15]  * ogra flashes afresh
[12:16] <diddledan> need netboot on these pis
[12:16] <diddledan> I don't suppose uboot can do network booting so you can save the root image on a server someplace and just have uboot on the sdcard?
[12:17] <diddledan> might speed up development if you can do that
[12:21] <ogra> ok ... it was the SD ... (i was debugging boot stuff and had pulled it out in former boots so i guess something got corrupted ... fresh flash works fine)
[12:21] <diddledan> oops :-p
[12:21] <diddledan> the sdcard is the main issue I have with pis
[12:22] <ogra> well ... i run my pi4 off an USB3 SSD ...
[12:22] <ogra> a lot more fun !
[12:22] <diddledan> ooh
[12:22] <diddledan> I got one of those that I pulled from my old mac
[12:22] <ogra> still using the SD for the boot partition indeed
[12:22] <diddledan> it's a USB3-SATA enclosure thingy
[12:22] <ogra> yeah, that should work well
[12:23] <ogra> nopw if u-boot would only allow using the full 4GB
[12:23] <diddledan> is that a hardcoded limit in it's pi support that we can reconfigure?
[12:23] <ogra> thats the only missing bit to have an awesom build machine
[12:23] <diddledan> or is it more systemic?
[12:24] <ogra> its missing code in the u-boot port for pi4
[12:24] <ogra> it isnt fully done yet
[12:24] <ogra> the pi4 port is based on pi3 code so it uses whats currently there to init the RAM
[12:58] <diddledan> ogra: if you're working on u-boot https://github.com/raspberrypi/firmware/issues/1191
[12:59] <mup> PR snapd#7125 opened: snapstate: make progress reporting less granular <Created by mvo5> <https://github.com/snapcore/snapd/pull/7125>
[13:00] <diddledan> that might be the same person as wrote the port you're using, https://github.com/agherzan/u-boot
[13:02] <diddledan> (I don't know whether that's the same port you're working with or not)
[13:27] <mup> PR snapd#7126 opened: tests: part3 making tests work on ubuntu-core-18 <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7126>
[13:32] <ogra> diddledan, yes, thats the one
[13:40] <diddledan> ogra: it looks like this is calling into videocore rather than relying on the dtb: https://github.com/agherzan/u-boot/blob/ag/rpi4/board/raspberrypi/rpi/rpi.c#L296
[13:40] <diddledan> prolly needs a separate branch for pi4
[13:41] <diddledan> something #ifdef'd
[13:42] <ogra> diddledan, well, u-boot never actually loads the dtb but reads the dtb values from memor ... the proprietary bootloader needs to load the dtb else the overlays and confi.txt stuff for configuring the HW doesnt work
[13:43] <ogra> diddledan, https://github.com/agherzan/u-boot/blob/ag/rpi4/arch/arm/dts/bcm2838-rpi-4-b.dts#L9 is the issue i think
[13:44] <diddledan> aah good find
[13:50] <diddledan> so that's a devicetree file, right? shouldn't the pi firmware be populating the devicetree memory, not uboot?
[13:50] <ogra> uboot needs a small devicetree to know about the HW
[13:50] <ogra> it gets merged into the binary
[13:50] <diddledan> aah ok
[13:51]  * diddledan learning
[13:51] <ogra> hmm, but blindly patching the dts file doesnt help
[13:52] <ogra> still only 1GB
[13:55] <ogra> well, agherzan seems to work on it already ... we'll just need to wait
[14:05] <mup> PR snapd#7127 opened: tests: removing support for ubuntu cosmic on spread test suite <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7127>
[14:30]  * cachio afk
[14:42] <ackk> jdstrand, hi, around?
[14:47] <ackk> mvo, a while ago you merged this https://github.com/snapcore/snapd/pull/6943 to allow adjtimex, but I still see it failing here with the time-control interface
[14:47] <mup> PR #6943: interfaces: add missing adjtimex to time-control <Simple 😃> <Created by mvo5> <Merged by jdstrand> <https://github.com/snapcore/snapd/pull/6943>
[14:48] <ackk> Jul 19 14:46:37 maas audit[13116]: SECCOMP auid=4294967295 uid=0 gid=0 ses=4294967295 pid=13116 comm="chronyd" exe="/snap/maas/x1/usr/sbin/chronyd" sig=0 arch=c000003e syscall=159 compat=0 ip=0x7f4eb1d700c7 code=0x50000 is the error I see
[14:56] <popey_> https://forum.snapcraft.io/t/refresh-failing-after-some-days-of-downtime/12384
[14:56] <popey_> :(
[14:59] <ChipAway> popey_: you have a snap named 's'?
[15:00] <popey_> ChipAway: i can't "snap info s" to confirm or deny that
[15:00] <popey_> but it's possible, yes
[15:00] <Chipaca> popey_: can you 'snap list s'?
[15:00] <popey_> no
[15:01] <popey_> no snap commands work
[15:01] <Chipaca> popey_: Jul 19 15:51:43 KinkPad-K450 systemd[1]: snapd.service: Found left-over process 978 (apparmor_parser) in control group while starting unit. Ignoring.
[15:01] <Chipaca> popey_: looks suspicious to me
[15:01] <popey_> i see apparmor_parser eating cpu repeatedly
[15:01] <Chipaca> popey_: can you systemctl stop snapd.*
[15:02] <popey_> and snapd eating 100%
[15:02] <popey_> done
[15:02] <Chipaca> popey_: and now sudo SNAPD_DEBUG=1 /usr/lib/snapd/snapd
[15:04] <popey_> Chipaca: added to thread on forum
[15:04] <Chipaca> popey_: … and then?
[15:05] <popey_> thats it
[15:05] <Chipaca> D:
[15:05] <pedronis> mvo: I have the fix for the download in remodel OTOH I discover we still send ancient headers with downloads
[15:05] <pedronis> but that's a different issue
[15:05] <pedronis> for another time
[15:05] <pedronis> I put some FIXME in though
[15:05] <popey_> ooh, now it moved
[15:06] <mvo> ackk: oh no - I wonder what is going on. is this happening with the core from edge?
[15:06] <mvo> pedronis: thank you \o/
[15:06] <popey_> it seems to be moving now :(
[15:06] <ackk> mvo, no. but ifit's just snapd, I'm using the 2.40-based build jdstrand gave us for the snap_daemon user
[15:06] <mvo> ackk: let me double check 2.40
[15:07] <Chipaca> popey_: can you paste a bit more of the output?
[15:07] <Chipaca> ah
[15:07]  * Chipaca reads
[15:07] <Chipaca> right
[15:07] <Chipaca> so that's the problem
[15:07] <Chipaca> 2+ minutes between starting and done
[15:08] <Chipaca> meaning, probably, systemd started freaking out
[15:08] <mvo> ackk: it should be in the 2.40 branch, do you know what branch jdstrand build your daemon from? maybe he branched before this got merged (his PR was wip for a while)
[15:08] <Chipaca> we need to do something about that
[15:08] <ackk> mvo, version reports 2.40+git227.g5ce5ff1f0
[15:08] <mvo> ackk: you could workaround by just adding "adjtimex" to the seccomp profile and recompile it but I'm not sure that is what you want
[15:08] <popey_> and now yes, i can tell you i do have a snap called 's' installed
[15:08] <Chipaca> popey_: :)
[15:09] <jdstrand> ackk: time-control is connected?
[15:09] <ackk> jdstrand, yes
[15:09] <jdstrand> ackk: note that you will have to stop/start the process after connected with seccomp
[15:10] <jdstrand> ackk: apparmor can reload the policy for a running process, seccomp cannot
[15:11] <jdstrand> ackk: so, if you verify it is connected (grep adjtimex /var/lib/snapd/seccomp/bpf/snap.maas.yourcommand.src) and then stop chrony, then start it, do you see the denial?
[15:11] <Chipaca> popey_: if you stop snapd and start it again in the terminal does it still wait a long time like that?
[15:12] <popey_> will try when this refresh finishes
[15:13] <Chipaca> thank you
[15:13] <mup> PR snapd#7128 opened: overlord: DeviceCtx must find the remodel context for a remodel change <Created by pedronis> <https://github.com/snapcore/snapd/pull/7128>
[15:14] <Chipaca> pedronis: mvo: I suspect we're doing too much initialisation before starting the watchdog, and it's making systemd kill us
[15:15] <pedronis> Chipaca: I suspect is loading the state
[15:15] <pedronis> anyway we can find out now
[15:15] <Chipaca> pedronis: 2+ minutes of it?
[15:16] <pedronis> snap debug timings --startup=load-state
[15:16] <pedronis> snap debug timings --starup=ifacemgr
[15:16] <pedronis> should give some info about that
[15:16] <mvo> Chipaca: are we maybe re-generating security profiles?
[15:16] <Chipaca> mvo: probably
[15:16] <pedronis> that would go under ifacemgr
[15:16] <Chipaca> popey_: can you do those debug commands? ^^
[15:16] <pedronis> there I think
[15:18] <pedronis> Chipaca: mvo: I proposed 7128 (part of remodeling stuff)
[15:18] <pedronis> it will conflict with the kernel branch (less things to do there though)
[15:21] <popey_> Chipaca: seems a little faster to start this time
[15:22] <Chipaca> popey_: can you stop it, and remove the system key, and start it again?
[15:22] <popey_> the second debug command seems malformed
[15:22] <pedronis> yea, is startup, sorry
[15:22] <popey_> i dont know what system key is
[15:22] <popey_> error: cannot find startup: ifacemgr
[15:22] <Chipaca> popey_: the key for the system, DUH
[15:22] <popey_> no, that didnt work either
[15:22] <Chipaca> :-P
[15:22] <pedronis> Chipaca: you can pass --all
[15:22] <pedronis> to those commands
[15:22] <pedronis> it will find all the timings (still kept) instead of the last one
[15:22] <Chipaca> pedronis: /var/lib/snapd/system-key
[15:23] <pedronis> only
[15:23] <Chipaca> er
[15:23] <Chipaca> popey_: /var/lib/snapd/system-key
[15:23] <pedronis> Chipaca: :)
[15:23] <popey_> sorry, to be clear, snap debug timings --startup=ifacemgr is not valid
[15:24] <pedronis> mmh,
[15:24] <Chipaca> popey_: 'cannot find'?
[15:24] <pedronis> maybe it's only in 2.41
[15:24] <ackk> jdstrand, ok that's weird. I grepp'd, it was there. stopped and started the snap, now it works
[15:24] <Chipaca> popey_: "cannot find" means it doesn't have any
[15:24] <popey_> i am on 2.40
[15:24] <Chipaca> popey_: if it's wrong it says "ALlowed values are: ..."
[15:24] <popey_> your error messages are weird
[15:24] <pedronis> it might be that systemd is killing us too fast
[15:24]  * Chipaca covers his error messages' ears
[15:25] <pedronis> we would need to increase the timeout in the service file
[15:25] <pedronis> and try again
[15:25] <popey_> so am i safe to remove /var/lib/snapd/system-key ?
[15:25] <Chipaca> pedronis: or we can tell systemd what's going on so we're not playing whack-a-mole with it
[15:25] <pedronis> ?
[15:25] <Chipaca> popey_: if you could stop snapd, remove that file, and start it again, that'd be good
[15:25] <pedronis> Chipaca: you mean the real fix?
[15:25] <Chipaca> pedronis: yeah
[15:26] <pedronis> we still don't know where we are spending time
[15:26] <Chipaca> pedronis: i mean there's a protocol to tell systemd to wait a bit more afaik
[15:26] <pedronis> yes
[15:26] <pedronis> there is
[15:26] <pedronis> doesn't make it a fix
[15:26] <pedronis> I mean I suspect this is not trivial to fix
[15:26] <pedronis> without shuffling things around
[15:26] <pedronis> somewhat
[15:27] <popey_> ok
[15:29] <popey_> yes, 2 mins delay
[15:29] <popey_> see forum
[15:30] <Chipaca> popey_: nice
[15:30] <Chipaca> popey_: and now do you have ahnything in the ifacestate timings?
[15:30] <Chipaca> ifacemgr*
[15:30] <ackk> jdstrand, could it be that for some reason that profile didn't get applied right way?
[15:32] <pedronis> Chipaca: anyway I think the issue is that we should reorg things so that we don't do slow things inside a New function, it's a bit misleading
[15:32] <Chipaca> pedronis: mhmm
[15:32] <Chipaca> pedronis: but even if we did it in Init or Start it'd still be before the watchdog started
[15:32] <pedronis> Chipaca: ?
[15:33] <pedronis> we decide when to do what
[15:33] <Chipaca> pedronis: snapd does d := daemon.New(); d.Init(); d.Start(); runWatchdog()
[15:33] <pedronis> anyway my main point is that this is not a 5 line fix
[15:33] <Chipaca> ok
[15:35] <Chipaca> popey_: so you should be ok to let systemd start snapd again, at least until you run some development version again
[15:35] <Chipaca> popey_: as long as it's not re-doing the system key (which happens on upgrade)
[15:35] <jdstrand> ackk: yes, that was what I was trying to say. with a snap declaration, the interface is not auto-connected. therefore on install, chronyd starts with the interface disconnected and has the denial. you connect the interface. it will continue to get a denial until you restart it
[15:35] <Chipaca> popey_: as a workaround until we sort this you can bump the start timeout in a config file snippet
[15:35] <ackk> jdstrand, well chrony is not started on install
[15:36] <ackk> jdstrand, I install the snap, connect  the interfaces, then run maas init which configures stuff and starts everythin
[15:36] <ackk> jdstrand, at least it shuoldn't be, lemme confirm :)
[15:36] <jdstrand> ackk: you could use connection hook to deal with that
[15:37] <jdstrand> ackk: (not your last two comments, the fact that something is starting before a connection)
[15:37] <popey_> Chipaca: added timings to forum
[15:37] <popey_> Chipaca: thanks for the help
[15:37] <ackk> jdstrand, yeah it's a bit tricky at the moment as we only have one service, which is supervisord, which manages everything else
[15:37] <Chipaca> popey_: i'll try to reproduce your issue here so we don't have to guinea pig you
[15:38] <jdstrand> ackk: it can take a little while for snapd to compile the new security policies (work is planned this cycle to make that faster), so it is possible you connected, then ran maas init before it was done connecting
[15:38]  * jdstrand shrugs
[15:38] <ackk> jdstrand, so I guess the reason is that supervisord is started before connections are made, so children inherit the profile without connections?
[15:39] <jdstrand> ackk: I can say that snapd will not restart services when connecting interfaces, but on install won't start services until everything is connected
[15:39] <jdstrand> ackk: yes
[15:39] <ackk> jdstrand, so if the service starts before connecting and spawns other processes, they won't get the connection right?
[15:40] <jdstrand> mvo: fyi ^ tldr; chronyd started before the interface was connected
[15:40] <jdstrand> ackk: that is correct
[15:40] <ackk> jdstrand, would autoconnect fix this?
[15:40] <jdstrand> ackk: yes
[15:40] <jdstrand> ac
[15:40] <jdstrand> ackk: so would be smart wrt a connection hook
[15:40] <jdstrand> being*
[15:42] <jdstrand> supervisord (or something else) could detect that the interface isn't connected. I believe there is a mechanism to restart a service on interface connection
[15:42] <ackk> jdstrand, yeah but if you connect 5/6 interfaces one after another then you get a restart storm
[15:43] <jdstrand> if maas is known to break if all interfaces aren't connected, you probably want some smarts to make sure everything is connected and fail to start if not (with some appropriate log message that tells the user what to do)
[15:43] <ackk> jdstrand, is there a way to get interfaces status in snapctl?
[15:44] <jdstrand> ackk: I thought so, I'm not the best to answer that. perhaps ask in #snapcraft? these are the sorts of things snap publishers deal with all the time...
[15:44] <jdstrand> ackk: they may be able to advise on your storm comment, etc
[15:50] <pedronis> no, it's a request but we haven't worked on it yet
[16:33] <mup> PR core-build#47 opened: initramfs: restore wait-for-root calls <Created by cmatsuoka> <https://github.com/snapcore/core-build/pull/47>
[16:57] <cachio> mvo, any idea about how to create a fat fs on core 18?
[16:57] <cachio> I need it for the assertrions disk
[16:58] <cachio> do you know if that could work using a differet fs?
[16:59] <mvo> cachio: we could create a snap with mtools
[16:59] <cachio> mvo, seems to work using ext3
[17:00] <mvo> cachio: yes it does
[17:00] <cachio> mvo, is any special reason why we use vfat  on the tests?
[17:00] <cachio> mkfs.vfat
[17:04] <mvo> cachio: not really, just to test if it works with that, its the most common format on usb sticks
[17:05] <cachio> mvo, ah ok, I'll leave some tests using vfat and others using ext3
[17:05] <cachio> so we can test also core18
[17:06] <cachio> mvo, should be ok?
[17:06] <mvo> cachio: should be ok, yes
[17:07] <cachio> mvo, tx
[17:10] <mup> PR snapcraft#2635 opened: Legacy errors <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2635>
[17:49]  * mvo declares victory over swtpm
[18:10] <mup> PR snapd#7129 opened: Allow setting default-url-scheme-handler <Created by jwheare> <https://github.com/snapcore/snapd/pull/7129>
[20:04] <mup> PR snapcraft#2633 closed: remote-build: increase number of launchpad start_build request attempts <Created by cjp256> <Closed by cjp256> <https://github.com/snapcore/snapcraft/pull/2633>