[06:16] morning [06:49] PR snapd#6450 closed: release: 2.37.1 [07:21] arch packages update [07:21] d [07:37] hi, is it expected that $SNAP_DATA and $SNAP_COMMON on parallel installs point at the same dirs on all installs? [07:41] Hello [07:42] ackk: yes, check out the data though [07:42] It is all good :-) [07:42] mborzecki: thank you [07:43] ackk: yes, inside the mount namespace those appear as the same paths, but outside each instance has its own dir [07:43] zyga: hey [07:44] oh, I see [07:44] ackk: however, this is not the case for $SNAP_USER_DATA, $SNAP_USER_COMMON and $XDG_RUNTIME_DIR [07:45] ackk: https://forum.snapcraft.io/t/parallel-installs/7679 has some info about the paths and lists some limitations of the current implementation [07:47] mborzecki, but why are SNAP_DATA and SNAP_COMMON paths not instance-based as for the USER ones? [07:47] ackk: just technical limitation [07:48] We should be able to handle that eventually [07:48] 2.39 [07:48] Perhaps [07:48] ackk: they are, each instance has a separate directory, but insdie a snap we bind mount the instance specific dir on top of the non instance one to make it easier to use [07:50] ackk: for instance, snap foo_123 has /var/snap/foo_123/common and /var/snap/foo_123/current, inside the filesystem view of the snap, /var/snap/foo_123/common is also avaialble as /var/snap/foo/comon, but that's stil the same directory [07:51] ackk: not sure i explained that well enough :) [07:52] mborzecki, yeah thanks I understand [07:56] mborzecki, ok, the issue I was trying to track down seems to be this: if you have a "foo" snap with a config hook and you snap install foo_2, the hook is not found [08:06] ackk: ok, let me look into this, maybe there's a bug after all [08:06] mborzecki, thanks === pstolowski|afk is now known as pstolowski [08:09] morning [08:09] pstolowski: hey [08:10] mborzecki: thanks for the arch update [08:10] zyga: fwiw, the failure in the last from last night was a fluke [08:10] zyga: I re-run the entire suite [08:10] zyga: and no errors, so yay [08:11] Good morning pawel, michael [08:11] Good news [08:11] zyga: and good morning [08:14] I will be around soon. Feeling a little under the weather today === phoenix_firebrd is now known as murthy [08:43] at my keyboard [08:47] PR snapd#6451 opened: tests: update smoke/sandbox test for armhf [08:48] ackk: i tried my demo snap and it seems the hooks are called as expected [08:48] ackk: in your case, do you need foo to be installed before foo_2 is installed? [08:49] mborzecki, that's the case that doesn't work, when you just snap install foo_2 and don't have foo installed [08:49] pstolowski: hey, good morning! if you have some cycles and a dragonboard or a pi3 running some of the beta validation would be great, then we can parallelize as much as possible [08:49] zyga: or do you happen do have a dragonboard? [08:49] mvo: I do :) [08:49] ackk: can you use this demo snap https://github.com/bboozzoo/parallel-installs-demo ? snap pack and install --dangerous .. --name parallel-installs-demo_foo [08:50] ackk: the configure hooks drops a log in $SNAP_COMMON, so there should be /var/snap/parallel-installs-demo_foo/common/run.log file once it's installed [08:51] mvo: sure, happy to help, i've pi3, shall i run specific tests on it? [08:51] mborzecki, yeah that works but IDK why my snap doesn't then [08:51] mborzecki, "snap install --edge h2static_2" [08:52] mborzecki, then set listen=:8080 [08:52] (or any port) [08:52] ackk: ok, trying [08:52] pstolowski, mvo: for the benefit of everyone let's discuss the testing process here [08:52] my memory of how it goes is rusty [08:53] zyga: yeah, i've never done this before, not familiar with the process at all [08:53] zyga: its all in the mail from sergio [08:53] https://github.com/snapcore/snapd/blob/master/tests/external-backend.md [08:53] zyga: his examples are really good [08:53] oh, let me look [08:54] zyga, pstolowski https://github.com/sergiocazzolato/snappy-qa-jobs [08:54] https://github.com/sergiocazzolato/snappy-qa-jobs [08:54] right :) [08:54] zyga: heh [08:54] mvo: I think we really want to help sergio merge the code to master [08:54] pstolowski: yeah, sorry for this. special circumstances because sergio is on vac and we need to deal with this regrssion :/ [08:54] or at least link to it from master [08:55] zyga: yeah, I think we need to look into brining this into our tree [08:57] mvo: don't mention it, happy to help, we really need to spread knowledge and workloads more [08:57] pstolowski: yeah, it also makes me appreciate sergios work more (I always did of course) but its nice to see how much this is automated [08:58] pstolowski, zyga I added you to the card and there is a list of things to do, if you take one, just add *name* in the checlist and once its done a pastebin link (those are generated automatically) [08:59] ackk: can you add https://github.com/bboozzoo/parallel-installs-demo/blob/master/meta/snap.yaml#L10-L11 to your snap.yaml? [08:59] pstolowski: do you know if configure hooks are automatically discovered? [09:00] mborzecki, oh wait, I thought I had that, lemme try [09:02] mborzecki: yes they are [09:04] mborzecki, ah, that works [09:04] mborzecki, thanks [09:13] ackk: did you figure out your 'hooks aren't running on parallel installs' issue? [09:14] sparkiegeek, yeah, see above. I was missing the hook: section in the snapcraft.yaml [09:14] ackk: ah, so PEBKAC [09:14] sparkiegeek, well, afaiu it's only mandatory if you need to set plugs. the hook is called on the "normal" install [09:15] pstolowski: mborzecki: hook should be outdiscovered, does autodiscovery gets confused if there's a instance name? [09:15] sparkiegeek: might be a bug too looking into it, for now declaring it explicitly works [09:15] mo'in [09:15] Chipaca: hello [09:15] Chipaca: moin moin [09:15] pedronis: trying to find that out [09:19] pedronis: question about lanes: why do tasks have a list of lanes? when would a task be on more than 1? [09:21] mvo: clarification needed - if i'm going to test fresh core18 install on pi3, after flashing the image i do the initial consoleconf setup manually right? the test scripts doesn't automate this? [09:21] pstolowski: correct [09:21] mvo: good, thanks [09:24] pstolowski: if you notice anything where the docs should be clearer feel free to make notes and pass to sergio [09:25] Chipaca: afaik we haven't used that feature so far (putting a task in more than one lane) [09:25] mborzecki: fwtw we rely on auto-discovered configure in many of our test snaps in spread [09:25] popey: Wimpress: where was that "how to make your snap more shiny" tutorial with stats and such? [09:26] pedronis: what would it do if we did? :-) [09:26] Chipaca: something complicate if you abort [09:26] we have logic for it (complicated) [09:27] Chipaca: tbh, at this point I would be ok for us to error if we try to do that (add a task to more than one lane) [09:28] ok, I'll not worry about it for now then [09:28] Chipaca: https://snapcraft.io/blog/make-your-snap-store-page-pop [09:28] pedronis: can we talk in malta about using a database for state? [09:28] popey: thanks! [09:29] pstolowski: pedronis: looks like there is a bug [09:29] Chipaca: we can talk about it [09:29] PR core18#116 opened: Support arm64 with efi bootloader [09:29] Chipaca: do we have an obvious candidate for it tough? [09:30] pstolowski: pedronis: https://github.com/snapcore/snapd/blob/master/snap/info.go#L1011-L1019 [09:31] mborzecki: I suppose the bug is inside those functions [09:31] ah we set instance key [09:31] late [09:31] funny us [09:31] yeah [09:31] mborzecki: anyway need a test [09:31] mhm [09:32] mborzecki: ? [09:32] pedronis: just confirming that's the bug, and we do need a test [09:36] grr github acting up on me, can't see that diff [09:37] mborzecki: do we have any tests for hooks and instances? === murthy is now known as phoenix_firebrd_ [09:43] pstolowski: no, i don't think there's a spread test specifically targeting hooks with parallel instances [09:44] so declaring the hooks explicit works, but autodiscovery does not [09:45] mborzecki, pedronis: patch for 2.37.2 :) ? [09:45] zyga: likely [09:45] are we doing .2 already? :P [09:45] the tests run so much faster with 1.10 [09:47] mborzecki: inevitably [09:49] Chipaca: especially the ones are cache and don't run [09:49] PR snapd#6452 opened: overlord/snapstate: format the refresh time for the log [09:50] mborzecki: travis's "short" unit run went from ~9m to ~5m [09:50] locally it seems to be even faster [09:50] Chipaca: oh, those aren't cached for sure [09:50] ^^^ a very very trivial PR (honest) === phoenix_firebrd_ is now known as murthy [09:54] ehh merge conflict in #6415 [09:54] PR #6415: snapstate, snap: allow update/switch requests with risk only channel to DTRT [09:55] mborzecki: yeah [10:03] pedronis, mvo, is there any coordination between the cloud-init people and the core team going on ... the cloud-init documentation regarding core seems to be quite a mess (see: https://forum.snapcraft.io/t/ubuntu-core-18-rpi-3-not-seeded/9728/9 ) [10:04] ogra: not really so far [10:05] yeahm thats what i thought [10:08] blackboxsw: ^ [10:11] pedronis: wrt re-refresh, it's to be done on all snaps that were requested in the original update, not on all snaps that were updated, right? [10:12] ie if the user said "snap refresh a b c d", re-refresh checks those 4 even if just c had an update? [10:12] Chipaca: should we write it down somehwere [10:12] pedronis: I'll add a comment :-) [10:12] we are reshaing the same thing again and again [10:12] pedronis: last time it was just-those-that-had-epoch-bump vs "all" [10:12] now i'm clarifying what "all" meant [10:12] Chipaca: we need to ask about snaps that have had a successful refresh, of these we should refresh the ones that have a new epoch change lined up [10:13] *ask the store [10:13] pedronis: yes, and for that I check the lanes [10:13] pedronis: but in the task description, I wanted to say what the re-refresh was _for_ [10:14] I guess just the ones that get updated is the better superset [10:15] Chipaca: yes, but that's also the set we put in the original summary, no? [10:16] I don't think the summary has a b c d [10:16] if only c got an update [10:16] (but maybe I'm misremembering) [10:17] pedronis: the change summary yes, we set that in daemon [10:17] the task summary no [10:17] (there's no other task that's whole-update like rerefresh is) [10:18] (afaik) [10:18] but we have the info [10:18] mvo: should we upload 2.37.1-1 to disco? [10:18] Chipaca: we are problably discussing a detail that doesn't merit discussing here, it should be clear enough in the code, np? [10:19] pedronis: i think so [10:21] mvo: my validation tests seem to be stuck on failover:emptysystemd [10:22] pstolowski: hm, ok might be worthwhile to investigate [10:23] pstolowski: I saw some fluke failures in different tests, especially in core18 I think we need to look at those too, looks like the suite is not quite as robust on core18 yet [10:26] nooo... /usr/bin/pastebinit [10:26] Uploding execution log to paste.ubuntu.com [10:26] Failed to contact the server: [Errno socket error] The write operation timed out [10:27] mvo: is it ok to re-run the tests over same image, or should i reflash? [10:28] pstolowski: worth a try to re-run but my experience is that it might be broken from some invalid restore etc [10:29] pstolowski: via the docs in tests/external-devices.md you can also re-run individual tests [10:30] mvo: ack, thanks [10:41] hmm [10:41] 2019/01/30 10:40:32.282838 taskrunner.go:420: DEBUG: Running task 293320 on Do: Make snap "core" (6259) available to the system [10:41] 2019/01/30 10:40:32.321291 link.go:75: cannot update fontconfig cache: cannot get fc-cache-v6 from core: open /snap/core/current/bin/fc-cache-v6: no such file or directory [10:41] suspect something there is not waiting for the mount [10:46] Chipaca: we had a fix for that I tought [10:47] ah, maybe we do (i'm switching channels a lot) [10:48] Chipaca: https://github.com/snapcore/snapd/pull/6317 [10:48] PR #6317: overlord/snapstate/backend: call fontconfig helpers from the new 'current' [10:57] pedronis: i've cleaned up https://github.com/snapcore/snapd/pull/5962 and unblocked it. diff is ~1100 lines, but 700 lines are new tests. [10:57] PR #5962: ifacestate/hotplug: hotplug handlers [10:58] mvo: pastebin step of the validation tests always fails for me :/ [11:00] pstolowski: even if you have pastebinit installed? strange it works pretty reliable for me (18.04 and 18.10) [11:01] the Pi test just takes forever :/ [11:01] pstolowski: ok, thanks, will try to get to it this week still [11:02] pstolowski: does it include the bits about avoiding to recreate slots already in the gadget, or is that separate? [11:03] pedronis: yes it does [11:05] mvo: yes, i can use pastebinit manually just fine [11:10] pstolowski: did you find out if we have problem with the implicit slots logic as it relates to hotplug? [11:10] or it will just work [11:11] pedronis: should just work; it remains a bit messy and brittle though [11:12] ok, thanks [11:19] * Chipaca takes a break [11:58] heh, hookstate tests are missing some cleanup, added new test there and the previous ones get stuck :/ === philroche_ is now known as philroche [12:51] PR snapd#6453 opened: snap: fix hook autodiscovery for parallel installed snaps [12:51] pedronis: pstolowski: fix for autodiscovery of hooks ^^ [13:00] ty [13:02] mvo: hey [13:02] sorry for the slow update [13:03] I'm having issues setting up the dragon board [13:03] it gets the ip address but console-conf is unhappy [13:07] mvo: hey, fyi, test-snapd-dbus-consumer and test-snapd-dbus-provider have a bunch of 'Store upload failed for...' messages that for some reason the security team is getting emails for, but I don't see where we own those. did the macaroon expire? [13:07] jdstrand: yes, macaroon expiry is the likely cause [13:32] jdstrand: oh, stange - sould you forward me one of those? [13:32] jdstrand: why the security team of all people :) [13:36] mvo: I think the dragonboard image doesn't work [13:36] zyga: uh [13:36] mvo: it boots, I can associate with ap's [13:36] mvo: but I cannot leave console conf [13:36] mvo: some kernel error flies on the screen briefly during setup [13:36] zyga: uc16 or uc18 or both? [13:36] mvo: and then the message from console-conf is "Network configuration timed out; please verify your settings" [13:37] uc16 [13:37] I've been trying to use different AP's move the device around etc [13:37] mvo: note that it does connect, it gets an IP address [13:37] but that's that [13:37] the error that flashes is "] wcn36xx: ERROR hal_remove_b6" [13:38] zyga: hm, hm, bad. I can't comment, I don't know enough about the DB, I think sergio mentioned we can run some tests via testflinger, I have a look at this now [13:38] mvo: I can make a uc18 image next [13:39] I may also have a usb-ethernet adapter at home somewhere [13:39] but I think that's not sufficient to say "this is good" [13:39] I doubt it's our fault (3.37 -> 3.37.1) [13:39] so I would not block on that [13:40] zyga: yeah, lets see if testflinger is more happy [13:43] plars: ^ fyi [13:46] booting uc18 now [13:46] random thing from logs: FAT: Misaligned buffer address (00000000bd0f48f0) [13:46] ubuntu-image issue? [13:47] mvo: uc18 feels less happy [13:48] ah [13:48] no, just veeeery slow :) [13:49] mvo: uc18 works [13:53] mvo: ok, I bounced them to you [13:53] zyga: probably slow because of entropy [13:53] jdstrand_: ta [13:53] zyga: running testflinger now, fingers crossed [13:54] mvo: is this with a fresh image created with the core in beta? [13:54] cwayne: I hope so, this is done via a script from sergio, I have not looked at the inner workings yet [13:55] cwayne: it says "2019-01-30 13:52:47,210 dragonboard_002 INFO: DEVICE AGENT: Flashing Test Image [13:55] " [13:55] ok, just checking as we've done the core beta snap tests and dragonboard was a-ok [13:55] cwayne: aha, so my test is redundant? [13:55] mvo: not if its a new image [13:55] we just snap refresh [13:55] cwayne: its the beta 2.37.1 [13:55] cwayne: aha, I see [13:55] cwayne: I will check, I am not sure which method is used [13:55] IIRC this test is creating an image with ubuntu-image, which i guess is technically different [13:55] cwayne: indeed, I double check [13:55] mvo: question about sergio's scripts, is the snapd branch?release/2.37? [13:56] zyga: he has examples [13:56] I'm jumping through the maze of includes [13:56] zyga: but its more like a TAG [13:56] zyga: yeah, the layout is a bit unconventional :) [13:56] k [13:56] thanks [13:56] zyga: I had very good results just following the examples [13:57] it's running now ... let's see === chrisccoulson_ is now known as chrisccoulson [14:04] core doesn't like motd: /etc/update-motd.d/50-motd-news: 59: /etc/update-motd.d/50-motd-news: cannot create /var/cache/motd-news: Read-only file system [14:08] zyga: hm, we killed motd on core I thought [14:08] not that hard apparently [14:08] where should the bugs go? [14:09] we still have /etc/update-motd.d/50-motd-news [14:09] we actually have more [14:09] -rwxr-xr-x 1 root root 1220 Apr 9 2018 00-header [14:09] -rwxr-xr-x 1 root root 4264 Aug 19 23:44 50-motd-news [14:09] -rwxr-xr-x 1 root root 348 Jan 30 05:01 60-unminimize [14:09] zyga: meh, I think we need to kill that [14:09] it's in /etc, I hope that's not another writable-dirs hell === ricab is now known as ricab|lunch [14:20] 2019-01-30 15:19:34 WARNING: external:ubuntu-core-16-arm-64 (external:ubuntu-core-16-arm-64:tests/main/failover:rclocalcrash) running late. Current output: ... <- feels this guy got stuck [14:20] zyga: uh, apparently that happend to pstolowski as well - sounds like we need to look into this why its fragile [14:24] mvo: another random error observation, pam complains about Jan 30 14:08:45 localhost login[2870]: pam_env(login:session): Unable to open env file: /etc/default/locale: No such file or directory [14:25] mvo: tests failed [14:25] http://paste.ubuntu.com/p/D699fhS5BK/ [14:25] it looks like [14:25] real issue /home/gopath/src/github.com/snapcore/snapd/tests/lib/boot.sh: line 33: fw_setenv: command not found [14:26] zyga: i saw that a lot [14:26] https://www.irccloud.com/pastebin/uSQ850Vs/ [14:27] zyga: woah [14:27] zyga: is thre /etc/default/locale? [14:27] no [14:27] mvo: where should I report those [14:27] irc doesn't work well [14:28] core18 github? === jdstrand_ is now known as jdstrand [14:40] pstolowski: btw what did we conclude about spread tests for hotplug? [14:44] pedronis: i've something based on groundwork set by Sergio (nested vm tests), a simple scenario with virtual serial port worked, but it will require some effort still [14:56] pstolowski: ok [14:58] mvo: doh, you won't believe what prevented ssh-agent from working for me [14:58] pstolowski: I'm curious now :) [14:59] zyga: yeah, core18 github [14:59] mvo: thank you, on it [15:03] mvo: i had two rsa keys, one very old and unused (i don't even remember what it was); it was also very weak; turns out just having it around caused an issue; got rid of it and ssh agent is now happy about the other key and doesn't prompt for password anymore [15:05] Issue core18#117 opened: update-motd is installed but cannot operate on read-only / [15:20] * zyga -> tea and lunch === ricab|lunch is now known as ricab === phoenix_firebrd is now known as murthy [15:43] #6252 is green now, yay [15:44] PR #6252: userd: handle help urls which requires prepending XDG_DATA_DIRS [16:08] mvo #6453 needs a 2nd review [16:08] PR #6453: snap: fix hook autodiscovery for parallel installed snaps [16:08] Chipaca: 6452 is green [16:08] pedronis: in a wee bit [16:09] mvo: np, pointing to it because it would go in 2.37.2 [16:09] pedronis: whee [16:09] PR snapd#6452 closed: overlord/snapstate: format the refresh time for the log [16:09] pedronis: sounds good [16:10] re [16:12] brb, walking the dog for a bit [16:13] mvo: it's not a regression but core18 has a real bug [16:14] zyga: which bug? the motd? [16:14] no, the fw_printenv, hold on [16:15] https://github.com/snapcore/core18/issues/118 [16:15] Issue core18#118 opened: fw_printenv is not present in core18 [16:16] zyga: aha, right [16:16] zyga: yeah, I wonder why we did nto notice - oh well. we need to fix it so that we can run the tests. we could even bring it in as a snap or provide an internal command (we have support to write uboot) [16:16] it feels like a core18 or a gadget binary [16:16] it's not great that we didn't notice before release [16:17] zyga: well, I think its only needed for tests [16:17] I'm not familiar with how we use that binary to say where it belongs [16:17] zyga: I'm 99% sure [16:17] mvo: if it is just for tests that's not bad then [16:17] zyga: yeah, we would be in trouble otherwise [16:17] but also means nobody noticed? [16:17] that is, we released ignoring or not testing that [16:17] zyga: yeah, thats the part that is concerning [16:17] zyga: correct [16:17] I will file more bugs [16:17] took a break, got some cold/flu meds [16:17] feeling better now [16:17] zyga: well, what test was this? [16:18] zyga: I am running tests here right now on a dragonboard [16:18] zyga: and they are passing so I wonder *why* [16:18] as core18 or core16? [16:18] zyga: maybe we exclude some tests somewhere for the wrong reasons (systems: [-...]) [16:18] zyga: aha, sorry - this one runs on core16 [16:18] right [16:18] zyga: I will restart on core18 once this is done [16:18] it's not broken on core18 [16:18] zyga: sorry ! [16:18] we should generate a diff between file list of core and core18 [16:19] mvo: it is likely pawel saw the same issue on pi [16:19] pstolowski: which raspberry pi model did you use and which snapcraft model did you use? [16:21] I amended the bug report with more details [16:21] zyga: pi3 model b v1.2, 2015 [16:21] mvo: there's no snapcore/dragonboard-kernel [16:21] where should those bug reports go? [16:21] zyga: what do you mean by snapcraft model? i just built beta images following the guide, with a script [16:22] pstolowski: snap known model [16:22] what does that say on your device? [16:22] ah, let me check [16:26] mvo: we only use fw_setenv for testing, that's okay then [16:27] zyga: yeah, we still need to make sure how to get that in the tests and figure out why we didn't notice this before [16:29] This sounds like a reason for us to do this full beta image testing in our lab as well [16:29] cwayne: indeed [16:30] cwayne: having nightly runs against edge for pi3/db for core and core18 would be awesome [16:30] hey jdstrand, https://forum.snapcraft.io/t/the-u2f-devices-interface/9722 mentions 2.37+, yet the interface is not in 2.37.1, is that an oversight? [16:31] zyga: my pi3 system is borked, snapd daemon doesn't respond to client anymore :( [16:31] that's ok [16:31] which image did you flash? [16:32] zyga: pi3-18-beta [16:32] pstolowski: thank yo [16:32] you* [16:32] so that's that, the same bug is present across core18 devices [16:32] mvo: this is double worrying now, it means core18 was not really tested? [16:32] it would not have passed anywhere [16:35]