/srv/irclogs.ubuntu.com/2019/07/19/#snappy.txt

=== harrisj_ is now known as harrisj
=== Girtablulu is now known as girtablulu|work
ackkhi, I'm getting a weird error when testing the maas snap. In logs for various processes I see "mkdir: cannot create directory ‘/var/snap/maas’: Read-only file system"07:22
ackkI tried both restarting the snap and the whole system07:22
pedronismvo: thanks for reviewing again verify-boot, I merged it now07:34
ackkanyone might have an idea on how to debug this issue?07:51
ackkI'm using the non-root-user enabled snapd 2.4007:51
mvopedronis: cool, thank you!07:57
pedronisChipaca: hi08:30
Chipacapedronis: hiya08:31
Chipacapedronis: not ready for you yet i'm afraid08:31
Chipaca:-|08:31
pedronisChipaca: late today, or are we talking monday?08:32
Chipacapedronis: hmm. I might make it for late today, but monday is safer -- what'd you rather?08:32
pedronisChipaca: monday is easier08:33
pedronisChipaca: can you put an actual meeting in the calendar for monday when it works for you08:33
Chipacapedronis: monday it is then08:34
Chipacai'll probably sneak in some time on this over the weekend to make up for this week which has been less than productive08:34
Chipacahopefully school being out and the boys wanting to spud out for a while will be conducive08:34
Chipacapedronis: calendaered08:36
Chipacapedronis: you've got Edit, so you can move it if you need/want to08:37
pedronisChipaca: you picked the wrong monday I think08:38
pedronis29 vs 2208:38
Chipacapedronis: augh08:38
Chipacamoved08:38
Chipacaoh wait now it conflicts with my 1:108:38
Chipacamoving again08:38
Chipacadone08:39
pedronisChipaca: thanks08:41
pedronisbtw my other PRs are ready for reviews,  and 6923 from pawel is unblocked and needs a 2nd review08:43
Chipacapedronis: +1 to both of yours, and +1 to pawel's even though I hate «for _, cstrs1 := range cstrs»09:36
Chipacacasters? coasters? castraters?09:36
Chipacaand with that, i'm out for a few hours09:38
Chipacashould be back in ~4h modulo everything09:39
=== Chipaca is now known as ChipAway
pedronismvo: https://forum.snapcraft.io/t/broken-dependency-of-content-snaps-during-seeding/11566/18 you meant (classic) image builds instead of deb packages ?09:54
mvopedronis: eh, yes, let me fix htis09:57
mvopedronis: thanks09:57
pedronisnp09:57
ogragra@pi4:~$ grep PRETTY /etc/os-release09:57
ograPRETTY_NAME="Ubuntu Core 18"09:57
ograogra@pi4:~$ ps ax|grep unattended09:57
ogra 2209 ?        Ssl    0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal09:57
ogramvo, ^^^^ ??????09:57
mvoogra: check with rbalint09:58
mvoogra: but it seems thats the only reliable way these days to hook into the shutdown09:58
ogramvo, oh, so the name is misleading ?09:58
mvoogra: eh, uh - this is on core1810:00
ograyes10:00
mvoogra: sorry, missed that - that is - unexpected10:00
mvoogra: and unwanted :/10:00
mvoogra: I wonder when this sneaked in :(10:01
ograright ... that package shoundt be there at all10:01
mvo(and why)10:01
mvoogra: totally10:01
mvoogra: there should also be a big dependency issue, like apt is removed during the build10:01
ogranote that i have a few lxd containers running ... i wonder if that makes it sneak in10:02
mvoogra: that could be, let me look at the build logs10:03
ograogra@pi4:~$ lxc stop xenial10:03
ograogra@pi4:~$ ps ax|grep unattended10:03
ogra16700 pts/0    S+     0:00 grep --color=auto unattended10:03
ograthere we go !10:03
mvoogra: aha, ok10:03
mvoogra: thats fine then10:03
ogramvo, sorry, false alarm then10:03
ograyeah10:03
zygapre-last day of driving home, see you all on Monday10:05
mvoogra: no worries10:08
ograjdstrand, hmm, i'm trying to snap a script that reads /proc/diskstats and blinks one of the RPi LED's based n changes there but i cant seem to find any interface that allows me to write to /sys/class/leds/led1/trigger and /sys/class/leds/led1/brightness10:56
ogra(seems only display-control allows access to the kbd LED's)10:57
ogra... but nothing for generic system ones10:57
abeatozyga, hey, do you know which is the current status for snaps in yocto?10:57
ograabeato, https://forum.snapcraft.io/t/yocto-rocko-core-snap-panic/3261 perhaps ?10:58
ograhmm, there are newer threads too ...10:59
ograhttps://forum.snapcraft.io/t/question-about-meta-snappy-layer-for-yocto/1031511:00
abeatoogra, thanks, I see there are links to intructions in snapd repo11:00
ograapparently you want https://github.com/morphis/meta-snappy11:00
abeatoha, nice11:02
mupBug #1837209 opened: Splash screen fails to display on recent pi core18 images <Snappy:New> <https://launchpad.net/bugs/1837209>11:23
mupPR snapcraft#2632 closed: pluginhandler, repo: find stage-packages from DT_NEEDED on host <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2632>11:27
jdstrandogra: hey, bool-file can do leds. you need a gadget like with gpio12:00
ograjdstrand, ah, thanks ! will look into that (we should have something by default in the pi gadget then)12:01
jdstrandogra: that said, it is quite old and doesn't operate exactly like gpio. it looks like the slot is expected to do the export/unexport12:02
ogra/usr/share/subiquity/console-conf-wrapper: line 13: snap: command not found12:09
ograPress enter to configure.12:09
ograHRM !12:09
diddledanHRM?12:13
ograyeah12:13
diddledanI know not of this acronym12:13
ograno snap command on current core18 pi images ... or my SD is screwed up12:13
diddledanoh dear :-(12:14
diddledanis it because snapd was originally in core?12:15
ogramight be .. let me re-try12:15
* ogra flashes afresh12:15
diddledanneed netboot on these pis12:16
diddledanI don't suppose uboot can do network booting so you can save the root image on a server someplace and just have uboot on the sdcard?12:16
diddledanmight speed up development if you can do that12:17
ograok ... it was the SD ... (i was debugging boot stuff and had pulled it out in former boots so i guess something got corrupted ... fresh flash works fine)12:21
diddledanoops :-p12:21
diddledanthe sdcard is the main issue I have with pis12:21
ograwell ... i run my pi4 off an USB3 SSD ...12:22
ograa lot more fun !12:22
diddledanooh12:22
diddledanI got one of those that I pulled from my old mac12:22
ograstill using the SD for the boot partition indeed12:22
diddledanit's a USB3-SATA enclosure thingy12:22
ograyeah, that should work well12:22
ogranopw if u-boot would only allow using the full 4GB12:23
diddledanis that a hardcoded limit in it's pi support that we can reconfigure?12:23
ograthats the only missing bit to have an awesom build machine12:23
diddledanor is it more systemic?12:23
ograits missing code in the u-boot port for pi412:24
ograit isnt fully done yet12:24
ograthe pi4 port is based on pi3 code so it uses whats currently there to init the RAM12:24
=== ricab is now known as ricab|lunch
diddledanogra: if you're working on u-boot https://github.com/raspberrypi/firmware/issues/119112:58
mupPR snapd#7125 opened: snapstate: make progress reporting less granular <Created by mvo5> <https://github.com/snapcore/snapd/pull/7125>12:59
diddledanthat might be the same person as wrote the port you're using, https://github.com/agherzan/u-boot13:00
diddledan(I don't know whether that's the same port you're working with or not)13:02
mupPR snapd#7126 opened: tests: part3 making tests work on ubuntu-core-18 <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7126>13:27
ogradiddledan, yes, thats the one13:32
diddledanogra: it looks like this is calling into videocore rather than relying on the dtb: https://github.com/agherzan/u-boot/blob/ag/rpi4/board/raspberrypi/rpi/rpi.c#L29613:40
diddledanprolly needs a separate branch for pi413:40
diddledansomething #ifdef'd13:41
ogradiddledan, well, u-boot never actually loads the dtb but reads the dtb values from memor ... the proprietary bootloader needs to load the dtb else the overlays and confi.txt stuff for configuring the HW doesnt work13:42
ogradiddledan, https://github.com/agherzan/u-boot/blob/ag/rpi4/arch/arm/dts/bcm2838-rpi-4-b.dts#L9 is the issue i think13:43
diddledanaah good find13:44
diddledanso that's a devicetree file, right? shouldn't the pi firmware be populating the devicetree memory, not uboot?13:50
ograuboot needs a small devicetree to know about the HW13:50
ograit gets merged into the binary13:50
diddledanaah ok13:50
* diddledan learning13:51
ograhmm, but blindly patching the dts file doesnt help13:51
ograstill only 1GB13:52
ograwell, agherzan seems to work on it already ... we'll just need to wait13:55
=== ricab|lunch is now known as ricab
=== girtablulu|work is now known as Girtablulu
mupPR snapd#7127 opened: tests: removing support for ubuntu cosmic on spread test suite <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7127>14:05
* cachio afk14:30
ackkjdstrand, hi, around?14:42
ackkmvo, a while ago you merged this https://github.com/snapcore/snapd/pull/6943 to allow adjtimex, but I still see it failing here with the time-control interface14:47
mupPR #6943: interfaces: add missing adjtimex to time-control <Simple 😃> <Created by mvo5> <Merged by jdstrand> <https://github.com/snapcore/snapd/pull/6943>14:47
ackkJul 19 14:46:37 maas audit[13116]: SECCOMP auid=4294967295 uid=0 gid=0 ses=4294967295 pid=13116 comm="chronyd" exe="/snap/maas/x1/usr/sbin/chronyd" sig=0 arch=c000003e syscall=159 compat=0 ip=0x7f4eb1d700c7 code=0x50000 is the error I see14:48
popey_https://forum.snapcraft.io/t/refresh-failing-after-some-days-of-downtime/1238414:56
popey_:(14:56
ChipAwaypopey_: you have a snap named 's'?14:59
popey_ChipAway: i can't "snap info s" to confirm or deny that15:00
=== ChipAway is now known as Chipaca
popey_but it's possible, yes15:00
Chipacapopey_: can you 'snap list s'?15:00
popey_no15:00
popey_no snap commands work15:01
Chipacapopey_: Jul 19 15:51:43 KinkPad-K450 systemd[1]: snapd.service: Found left-over process 978 (apparmor_parser) in control group while starting unit. Ignoring.15:01
Chipacapopey_: looks suspicious to me15:01
popey_i see apparmor_parser eating cpu repeatedly15:01
Chipacapopey_: can you systemctl stop snapd.*15:01
popey_and snapd eating 100%15:02
popey_done15:02
Chipacapopey_: and now sudo SNAPD_DEBUG=1 /usr/lib/snapd/snapd15:02
popey_Chipaca: added to thread on forum15:04
Chipacapopey_: … and then?15:04
popey_thats it15:05
ChipacaD:15:05
pedronismvo: I have the fix for the download in remodel OTOH I discover we still send ancient headers with downloads15:05
pedronisbut that's a different issue15:05
pedronisfor another time15:05
pedronisI put some FIXME in though15:05
popey_ooh, now it moved15:05
mvoackk: oh no - I wonder what is going on. is this happening with the core from edge?15:06
mvopedronis: thank you \o/15:06
popey_it seems to be moving now :(15:06
ackkmvo, no. but ifit's just snapd, I'm using the 2.40-based build jdstrand gave us for the snap_daemon user15:06
mvoackk: let me double check 2.4015:06
Chipacapopey_: can you paste a bit more of the output?15:07
Chipacaah15:07
* Chipaca reads15:07
Chipacaright15:07
Chipacaso that's the problem15:07
Chipaca2+ minutes between starting and done15:07
Chipacameaning, probably, systemd started freaking out15:08
mvoackk: it should be in the 2.40 branch, do you know what branch jdstrand build your daemon from? maybe he branched before this got merged (his PR was wip for a while)15:08
Chipacawe need to do something about that15:08
ackkmvo, version reports 2.40+git227.g5ce5ff1f015:08
mvoackk: you could workaround by just adding "adjtimex" to the seccomp profile and recompile it but I'm not sure that is what you want15:08
popey_and now yes, i can tell you i do have a snap called 's' installed15:08
Chipacapopey_: :)15:08
jdstrandackk: time-control is connected?15:09
ackkjdstrand, yes15:09
jdstrandackk: note that you will have to stop/start the process after connected with seccomp15:09
jdstrandackk: apparmor can reload the policy for a running process, seccomp cannot15:10
jdstrandackk: so, if you verify it is connected (grep adjtimex /var/lib/snapd/seccomp/bpf/snap.maas.yourcommand.src) and then stop chrony, then start it, do you see the denial?15:11
Chipacapopey_: if you stop snapd and start it again in the terminal does it still wait a long time like that?15:11
popey_will try when this refresh finishes15:12
Chipacathank you15:13
mupPR snapd#7128 opened: overlord: DeviceCtx must find the remodel context for a remodel change <Created by pedronis> <https://github.com/snapcore/snapd/pull/7128>15:13
Chipacapedronis: mvo: I suspect we're doing too much initialisation before starting the watchdog, and it's making systemd kill us15:14
pedronisChipaca: I suspect is loading the state15:15
pedronisanyway we can find out now15:15
Chipacapedronis: 2+ minutes of it?15:15
pedronissnap debug timings --startup=load-state15:16
pedronissnap debug timings --starup=ifacemgr15:16
pedronisshould give some info about that15:16
mvoChipaca: are we maybe re-generating security profiles?15:16
Chipacamvo: probably15:16
pedronisthat would go under ifacemgr15:16
Chipacapopey_: can you do those debug commands? ^^15:16
pedronisthere I think15:16
pedronisChipaca: mvo: I proposed 7128 (part of remodeling stuff)15:18
pedronisit will conflict with the kernel branch (less things to do there though)15:18
popey_Chipaca: seems a little faster to start this time15:21
Chipacapopey_: can you stop it, and remove the system key, and start it again?15:22
popey_the second debug command seems malformed15:22
pedronisyea, is startup, sorry15:22
popey_i dont know what system key is15:22
popey_error: cannot find startup: ifacemgr15:22
Chipacapopey_: the key for the system, DUH15:22
popey_no, that didnt work either15:22
Chipaca:-P15:22
pedronisChipaca: you can pass --all15:22
pedronisto those commands15:22
pedronisit will find all the timings (still kept) instead of the last one15:22
Chipacapedronis: /var/lib/snapd/system-key15:22
pedronisonly15:23
Chipacaer15:23
Chipacapopey_: /var/lib/snapd/system-key15:23
pedronisChipaca: :)15:23
popey_sorry, to be clear, snap debug timings --startup=ifacemgr is not valid15:23
pedronismmh,15:24
Chipacapopey_: 'cannot find'?15:24
pedronismaybe it's only in 2.4115:24
ackkjdstrand, ok that's weird. I grepp'd, it was there. stopped and started the snap, now it works15:24
Chipacapopey_: "cannot find" means it doesn't have any15:24
popey_i am on 2.4015:24
Chipacapopey_: if it's wrong it says "ALlowed values are: ..."15:24
popey_your error messages are weird15:24
pedronisit might be that systemd is killing us too fast15:24
* Chipaca covers his error messages' ears15:24
pedroniswe would need to increase the timeout in the service file15:25
pedronisand try again15:25
popey_so am i safe to remove /var/lib/snapd/system-key ?15:25
Chipacapedronis: or we can tell systemd what's going on so we're not playing whack-a-mole with it15:25
pedronis?15:25
Chipacapopey_: if you could stop snapd, remove that file, and start it again, that'd be good15:25
pedronisChipaca: you mean the real fix?15:25
Chipacapedronis: yeah15:25
pedroniswe still don't know where we are spending time15:26
Chipacapedronis: i mean there's a protocol to tell systemd to wait a bit more afaik15:26
pedronisyes15:26
pedronisthere is15:26
pedronisdoesn't make it a fix15:26
pedronisI mean I suspect this is not trivial to fix15:26
pedroniswithout shuffling things around15:26
pedronissomewhat15:26
popey_ok15:27
popey_yes, 2 mins delay15:29
popey_see forum15:29
Chipacapopey_: nice15:30
Chipacapopey_: and now do you have ahnything in the ifacestate timings?15:30
Chipacaifacemgr*15:30
ackkjdstrand, could it be that for some reason that profile didn't get applied right way?15:30
pedronisChipaca: anyway I think the issue is that we should reorg things so that we don't do slow things inside a New function, it's a bit misleading15:32
Chipacapedronis: mhmm15:32
Chipacapedronis: but even if we did it in Init or Start it'd still be before the watchdog started15:32
pedronisChipaca: ?15:32
pedroniswe decide when to do what15:33
Chipacapedronis: snapd does d := daemon.New(); d.Init(); d.Start(); runWatchdog()15:33
pedronisanyway my main point is that this is not a 5 line fix15:33
Chipacaok15:33
Chipacapopey_: so you should be ok to let systemd start snapd again, at least until you run some development version again15:35
Chipacapopey_: as long as it's not re-doing the system key (which happens on upgrade)15:35
jdstrandackk: yes, that was what I was trying to say. with a snap declaration, the interface is not auto-connected. therefore on install, chronyd starts with the interface disconnected and has the denial. you connect the interface. it will continue to get a denial until you restart it15:35
Chipacapopey_: as a workaround until we sort this you can bump the start timeout in a config file snippet15:35
ackkjdstrand, well chrony is not started on install15:35
ackkjdstrand, I install the snap, connect  the interfaces, then run maas init which configures stuff and starts everythin15:36
ackkjdstrand, at least it shuoldn't be, lemme confirm :)15:36
jdstrandackk: you could use connection hook to deal with that15:36
jdstrandackk: (not your last two comments, the fact that something is starting before a connection)15:37
popey_Chipaca: added timings to forum15:37
popey_Chipaca: thanks for the help15:37
ackkjdstrand, yeah it's a bit tricky at the moment as we only have one service, which is supervisord, which manages everything else15:37
Chipacapopey_: i'll try to reproduce your issue here so we don't have to guinea pig you15:37
jdstrandackk: it can take a little while for snapd to compile the new security policies (work is planned this cycle to make that faster), so it is possible you connected, then ran maas init before it was done connecting15:38
* jdstrand shrugs15:38
ackkjdstrand, so I guess the reason is that supervisord is started before connections are made, so children inherit the profile without connections?15:38
jdstrandackk: I can say that snapd will not restart services when connecting interfaces, but on install won't start services until everything is connected15:39
jdstrandackk: yes15:39
ackkjdstrand, so if the service starts before connecting and spawns other processes, they won't get the connection right?15:39
jdstrandmvo: fyi ^ tldr; chronyd started before the interface was connected15:40
jdstrandackk: that is correct15:40
ackkjdstrand, would autoconnect fix this?15:40
jdstrandackk: yes15:40
jdstrandac15:40
jdstrandackk: so would be smart wrt a connection hook15:40
jdstrandbeing*15:40
jdstrandsupervisord (or something else) could detect that the interface isn't connected. I believe there is a mechanism to restart a service on interface connection15:42
ackkjdstrand, yeah but if you connect 5/6 interfaces one after another then you get a restart storm15:42
jdstrandif maas is known to break if all interfaces aren't connected, you probably want some smarts to make sure everything is connected and fail to start if not (with some appropriate log message that tells the user what to do)15:43
ackkjdstrand, is there a way to get interfaces status in snapctl?15:43
jdstrandackk: I thought so, I'm not the best to answer that. perhaps ask in #snapcraft? these are the sorts of things snap publishers deal with all the time...15:44
jdstrandackk: they may be able to advise on your storm comment, etc15:44
pedronisno, it's a request but we haven't worked on it yet15:50
mupPR core-build#47 opened: initramfs: restore wait-for-root calls <Created by cmatsuoka> <https://github.com/snapcore/core-build/pull/47>16:33
cachiomvo, any idea about how to create a fat fs on core 18?16:57
cachioI need it for the assertrions disk16:57
cachiodo you know if that could work using a differet fs?16:58
mvocachio: we could create a snap with mtools16:59
cachiomvo, seems to work using ext316:59
mvocachio: yes it does17:00
cachiomvo, is any special reason why we use vfat  on the tests?17:00
cachiomkfs.vfat17:00
mvocachio: not really, just to test if it works with that, its the most common format on usb sticks17:04
cachiomvo, ah ok, I'll leave some tests using vfat and others using ext317:05
cachioso we can test also core1817:05
cachiomvo, should be ok?17:06
mvocachio: should be ok, yes17:06
cachiomvo, tx17:07
mupPR snapcraft#2635 opened: Legacy errors <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2635>17:10
* mvo declares victory over swtpm17:49
mupPR snapd#7129 opened: Allow setting default-url-scheme-handler <Created by jwheare> <https://github.com/snapcore/snapd/pull/7129>18:10
mupPR snapcraft#2633 closed: remote-build: increase number of launchpad start_build request attempts <Created by cjp256> <Closed by cjp256> <https://github.com/snapcore/snapcraft/pull/2633>20:04
=== msalvatore_ is now known as msalvatore

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!