=== KindTwo is now known as KindOne
=== KindTwo is now known as KindOne
=== KindTwo is now known as KindOne
=== mcphail7 is now known as mcphail
zygaGood morning06:06
zygaI will start around 906:06
mupPR snapd#8842 closed: tests: fix bug waiting for snap command to be ready <Simple 😃> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8842>06:49
mupPR snapd#8816 closed: tests: port 2 uc20 part1 <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8816>06:59
pedronismvo_: hi07:00
zygaHey everyone07:13
zygaI’m around, going to address PR feedback now07:13
zygaI will be off later for the mri07:13
mvopedronis: hi, not sure my earlier "hi" made it to you, for some reason I was put into "quiet" mode by freenode07:19
mvopstolowski: good morning07:19
mvozyga: good luck for later today07:19
pedronismvo: if your nick is not the registered one (like mvo_) you can't talk here07:19
alazredHi there! I've asked that question on #snapcraft but I think it fit best here. I have a snap that break after each reboot. it show me this error https://pastebin.com/X3VCfyua. If I reinstall the snap it work again. I'm running snapd on manjaro-arm on the pinebook pro.  Any pointers of what might be going on ? thanks in advance!07:36
zygaalazred: this means that nothing is loading apparmor profiles07:37
zygaalazred: what is the status of snapd.apparmor.service?07:38
zygaalazred: it is responsible for loading snapd apparmor profiles on system startup07:38
alazredzyga: https://pastebin.com/i1eMy3Md07:40
alazredzyga: sorry not the good one07:41
zygahmm, too bad there are no logs07:41
zygado you have persistent journal enabled?07:41
zygait would be good to make sure you do (mkdir /var/log/journal)07:41
zygaand reboot07:41
zygaand then check the status of that service07:41
alazredzyga: It's dead  https://pastebin.com/eX3ccSgz07:42
zygathe service is not enabled07:43
zygasystemctl enable --now snapd.apparmor.service07:43
alazredIve enabled it and it's now working !07:45
alazredzyga: thanks  !07:45
zygaalazred: maybe file a bug in manjaro07:45
zygaalazred: do you remember if it was disabled out of the box?07:46
alazredzyga: Yeah i will go there also !  The error was not super clear  and since the other apparmor.service was enalble i tought I was ok on that side.07:47
zygayeah, until apparmor 3 ships we need a custom service07:47
alazredzyga: I'm pretty sure it was07:47
zygaapparmor 3 won't need this07:47
alazredbut since it's arch base I don't know if it enable those things on install.07:48
alazredI'll try to know more on that. Thanks again for your time!07:48
zygaalazred: thanks for reaching out :)07:49
alazredzyga: They dosen't do mention of snapd.apparmor.service in the manjaro wiki. Only to enable snapd.socket.07:59
zygaalazred: I see, I think it needs fixing08:00
zygaI reached out to some manjaro friends and let them know08:00
alazredzyga: Ok thanks !08:00
mupPR snapd#8841 closed: gadget: drop dead code, hide exports that are not used externally <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8841>08:05
mupPR snapd#8844 opened: asserts/internal: add some iteration benchmarks, potential opt  <Created by pedronis> <https://github.com/snapcore/snapd/pull/8844>08:05
pedronispstolowski: I tried to answer your comment in #8823, also made some small changes based on mborzecki feedback08:24
mupPR #8823: asserts/internal: limit Grouping size switching to a bitset representation <Bulk assert refresh :scroll::scroll::scroll:> <Created by pedronis> <https://github.com/snapcore/snapd/pull/8823>08:24
zygamvo: I'm seeing this error, not sure if related to my PR or just random08:48
mvozyga: oh, fun. looks like the earlier incorrect loop masked some other problem :(08:49
mvozyga: how often/where do you see it?08:49
zygajust noticed a few systems failed this way on my RFC PR08:50
mupPR #8843: [RFC] many: export tools from core/snapd to mount namespaces <Needs Samuele review> <Created by zyga> <https://github.com/snapcore/snapd/pull/8843>08:50
zygalook at the colors on the left08:50
zygalots of green, a few reds08:50
mvozyga: quire a few reds :/08:50
zygaI wonder what was going on before08:50
mvozyga: ok, I have a look. odd that it did not appear in the original PR08:50
zygait doesn't seem like a new bug08:50
zygajust surfaced08:50
mvozyga: yeah, I'm sure the superlong timeout just masked the issue08:51
zygait *might* be something my PR does08:51
zygabut I'm not sure08:51
zygaone sec08:51
zygaWed, 10 Jun 2020 08:30:40 GMT08:51
zygaretry: next attempt in 1.0 second(s) (attempt 6 of 120)08:51
zygaWed, 10 Jun 2020 08:30:40 GMT08:51
zyga+ snap wait system seed.loaded08:51
zygaWed, 10 Jun 2020 08:30:40 GMT08:51
zygaerror: cannot communicate with server: timeout exceeded while waiting for response08:52
zygawith time stamps08:52
zygathis is not after a long wait08:52
zygabut this is surprising08:52
zygaso retry *succeeded*08:52
zygaafter 6 seconds08:52
mvozyga: yeah, I suspect snapd is restarting or something08:52
zygabut the "snap wait" immediately after that fails08:53
zygamaybe snap wait should be more robust08:53
zygaI think this may be what was randomly failing earlier as well08:53
zygabtw, those timestamps *rock*08:53
mvozyga: yeah, snap should actually retry :/08:54
zygaretry snap wait08:54
pedroniswe have the wait.go logic but is meant for things that give back a Change08:58
pedronisbut we need some bits of it08:58
pedronisfor wait08:58
pedronisthis kind of code: https://github.com/snapcore/snapd/blob/master/cmd/snap/wait.go#L9609:01
mvopedronis: thank you09:02
mupPR snapd#8845 opened: [RFC] many: add "system.service.snapd-autoimport.disable" setting <Created by mvo5> <https://github.com/snapcore/snapd/pull/8845>09:45
mvozyga: I can look at the wait issue now09:47
zygamvo: thank you09:47
mupPR snapd#8808 closed: riscv64: bump timeouts <Created by xnox> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8808>10:00
=== alazred_ is now known as alazred
mborzeckizyga: are you still using fish?10:24
mvozyga: it's all broken10:25
zygamvo: what?10:27
zygamborzecki: yes10:27
zygamvo: what's going on?10:27
zygasorry, not having most productive day10:27
zygaI feel the best in a position where typing is not possible10:27
mborzeckizyga: can you try `snap <TAB>`, does it show the commands and their help?10:28
mvozyga: the wait thing you pointed me to is broken in 3 different ways on 16/18/20 :(10:30
mvozyga: anyway, I'm looking10:30
zygamvo: woah10:30
zygamborzecki: trying10:30
zygait super nice way :)10:30
zygahow do we screenshot on linux?10:30
mupPR snapd#8846 opened: tests: move update-related tests to snapstate_update_test.go <Test Robustness> <Created by stolowski> <https://github.com/snapcore/snapd/pull/8846>10:30
zygamborzecki: check this out https://usercontent.irccloud-cdn.com/file/K78qpiep/fish-tab-completion.png10:31
mborzeckiyeah, it's in the upstream https://github.com/fish-shell/fish-shell/blob/master/share/completions/snap.fish10:32
zygasuper pretty10:32
zyganot using bash helpers but I guess for some reasons10:33
mborzeckizyga: there was some problems reported in the forum though https://forum.snapcraft.io/t/tab-completion-for-snap-connect-in-fish-spams-errors/1808310:33
mborzeckiit wfm though10:33
zygathat is broken10:33
mborzeckiyeah, it's the connect/disconnect thing10:34
mborzeckithat warning should probably go to stderr10:34
zygamborzecki: snap connect <tab> SNAFU https://usercontent.irccloud-cdn.com/file/KdWyk66Z/snap-connect-broken-completion.png10:34
zygamvo: can you tell some more10:35
zygamvo: it's really interesting10:35
mvozyga: the first (16) stops with a connection timeout exceeded10:36
mvozyga: the second (18) hangs forever10:36
mvozyga: and 20 fails with dpkg --purge dir not empty10:36
zygaand it was just this broken/10:36
zygahow did we miss it?10:36
mvozyga: I start with 20 now that looks most simple10:36
mvozyga: no idea, 20 could be just coincidence10:36
mborzeckizyga: heh it calls `snap interfaces $snap | string replace -r '[- ]*([^ ]*)[ ]+([^ ]+)' '$2$1' | string match -v "*Slot*"`10:36
mborzeckihm i'll to fix it upstream10:38
zygaone thing I love in fish10:39
zygais that $foo is not unsafe10:39
zygaas fish does not expand strings this way10:39
zygait's weird coming from bash10:39
zygabut nice10:39
zygamvo: sorry for dragging you into this10:41
mvozyga: my own fault for doing the original PR10:44
pstolowskimvo, mborzecki : is #8784 still a draft?10:44
mupPR #8784: snap: add new `snap run --experimental-gdbserver` option <Created by mvo5> <https://github.com/snapcore/snapd/pull/8784>10:44
mborzeckipstolowski: switched it to 'ready'10:45
mvopstolowski: I think we can switch it10:45
pstolowskiindeed, thanks10:45
mborzeckiit's probably missing some tests tho, we can add those once we get some reviews10:45
mborzeckizyga: this should fix it https://github.com/fish-shell/fish-shell/pull/710410:54
mupPR fish-shell/fish-shell#7104: completions/snap: workaround snap interfaces deprecation notice <Created by bboozzoo> <https://github.com/fish-shell/fish-shell/pull/7104>10:54
mborzeckihm zomething we could distro patch for instance in ubuntu?10:55
zygaI'm sure we can11:00
zygamvo:  cna you dput to ubuntu?11:00
mborzecki(once it lands upstream first)11:01
mvozyga: dput what?11:01
zygamvo: fish11:01
mvozyga: I can/could to groovy, focal will need a SRU11:02
zygagrovy would be good11:02
mborzeckii'll file a bug first11:04
pedronismborzecki: looks like #8830 can be merged?11:06
mupPR #8830: bootloader: introduce bootloarder assets, import grub.cfg with an edition marker <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8830>11:06
mborzeckiit's green yay! at last11:07
mvozyga: and none of the wait issues you saw in the PR is reproducible so far :(11:07
zygamaybe my PR is at fault11:08
zygaare you trying on master or on that branch?11:08
mvozyga: master11:08
mvozyga: let me try your PR11:08
pedronisis the export PR?11:08
zygaif it's just that PR it's not worth spending time on it11:08
zygaI thought you found some deeper issues on master11:08
mvozyga: ok, I park it for now (if the current run is also not failing)11:10
mupPR snapd#8830 closed: bootloader: introduce bootloarder assets, import grub.cfg with an edition marker <UC20> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/8830>11:11
mvopedronis, zyga a log of the 20.04 "device not ready" we talked about yesterday with the debug log https://github.com/snapcore/snapd/pull/8845/checks?check_run_id=757318867 - I'm now trying to make sense of it11:16
mupPR #8845: [RFC] many: add "system.service.snapd-autoimport.disable" setting <Needs Samuele review> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8845>11:16
zygaJun 10 10:09:33 jun100957-926530 snapd[35371]: snapmgr.go:298: cannot read snap info of snap "snapd" at revision 7777: cannot find installed snap "snapd" at revision 7777: missing file /snap/snapd/7777/meta/snap.yaml11:18
zygathat's super clear IMO11:18
zygamvo: this is the bug that we talked earlier about11:18
zygamvo: about someone showing a system where booting lost interfaces11:18
zygabecause core was not mounted on time11:18
zygahow can this be happening!?11:19
zygathis is what is so myserious about it11:19
zygaare we racing with mounting that we do ourselves?11:19
pedronishere there is no core snap at all11:19
pedronisto be clear11:19
pedronisthere is no snapd snap either11:19
zygathis is also weird: Jun 10 10:00:06 jun100957-926530 systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing.11:19
mvozyga: oh, the mount units could be too slow to start?11:19
zygaand this killing was after we did this11:20
zygaJun 10 10:00:06 jun100957-926530 systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing.11:20
zygaand this kicks off the snapd.failure unit Jun 10 10:00:06 jun100957-926530 systemd[1]: snapd.service: Triggering OnFailure= dependencies.11:20
zygamvo: is it possible that snapd failing unmounts snap units?11:21
zygalike undoing some dependency chain or something11:21
mvozyga: possibly seems unlikely but I guess we need to explore whatever we can find11:21
pedronismvo: but remember there should be no snapd at all at that point here11:22
mvozyga: hm, two interessting things - the image has snaps seeded already, I think that's the only image we have with that property11:22
pedroniswe just reinstalled the snapd deb11:23
mvozyga: there is also a error auto-refresh change11:23
pedronisit seems like there's state from something else around11:23
mvopedronis: it looks like the image already has snapd in the image (and lxd/core18)11:23
pedronisand removing snapd doesn't do the right thing11:23
zygaoh interesting!11:24
pedronisthis is prepare.sh11:24
pedronisit was written for systems which just the snapd deb11:24
pedronisbut possibly with 20.04 there's a ton of stuff there11:24
pedroniswe just install the new deb?11:27
mvopedronis: yes11:27
mvopedronis: it sounds like we need to purge snapd first11:27
pedronisthat probably doesn't work well11:27
mvomaybe that's enough11:27
pedronisif snapd is in the middle11:27
pedronisof seeding11:27
pedronisthere's a real bug here I suppose11:27
pedroniswhat should happen if you upgrade the deb while snapd is seeding11:27
pedronisnot saying that we need to fix that to fix prepare.sh11:29
pedronisbut we defintely need to check what happens, what we want to happen11:29
pedronisin that scenario11:29
pedronisit seems you can get a fairly weird state11:30
mvopedronis: looking at the looks it seems like the seeding is already done, it happend 43 days ago (probably when the image was booted first and preapared for our needs)11:34
mvopedronis: I suspect the issue happens when a auto-refresh is in progress while we update the deb11:34
mvopedronis: but it's a similar issue I guess(?)11:35
pedronisI don't think we have thought through this11:35
pedronismaybe it's only relevant if the deb has a newer snapd?11:36
mvoyeah - and it seems real given that andy had this bug on a ubuntu 20.04 system some days ago11:36
mvopedronis: oh, that's interessting11:36
pedronisnaively the main issue in prepare.sh is that we have a state.json with seeded11:39
pedronisbut it doesn't relate to the installing snapd11:39
pedronisbut I don't know if there's other interference possible11:39
pedronisespecially if the deb version is newer11:39
pedronisit's a bit confusing because when it works we end up with no snaps except core2011:41
mvopedronis: what is most interessting to me is that we see the mount units vanishing, this looks like the bug we had the other day. I would like to understand this :/ the fix in prepare.sh is easy (like you said) we can purge and things should work. anyway, I have lunch first but this looks interessting and like there is something real lurking here11:43
* mvo lunch11:43
pedronismmh, we have also our strange hack that removes the seed11:47
pedronisthough I suppose only if it's invalid11:47
* zyga preps for the MRI12:00
zygasee you laster12:00
ogragood luck12:00
zygamvo: purge unmounts12:01
zygamvo: so snapd may be doing stuff while we purge and purge and purge12:01
zyga(just a guess)12:01
* pstolowski going to the vet, bbl12:18
clmsyhi everyone12:51
clmsyi have a question about building a custom image with ubuntu core 2012:51
clmsyim specially interested in trying to build with grade:secure option12:51
clmsyi modified my model json file and trying to sign it12:51
clmsyi get this error: error: cannot assemble assertion model: cannot specify a grade for model without the extended snaps header12:51
clmsynow it kinda makes sense because on this page, every snap is put with its identifier: https://docs.ubuntu.com/core/en/releases/uc2012:52
clmsybut the sign function from snapcraft expects a json12:52
clmsydo i have to somehow insert this information to required-snaps list12:52
clmsyi guess maybe i have to remove "require-snaps" andd replace it in json with "snaps" as a map not just a list of strs12:54
mvoclmsy: hey, check https://github.com/snapcore/models/blob/master/ubuntu-core-20-amd64.json for an example12:55
mvoclmsy: there are some more in https://github.com/snapcore/models12:55
clmsyaaah its already in here thanks a lot!12:55
=== stoopkid_ is now known as stoopkid
=== alazred_ is now known as alazred
clmsydid anyone get pc-kernel snap not found while trying to build a core20 image ?13:35
ijohnsonclmsy: what branch are you using for your kernel snap you need to be using the 20/ channel13:35
clmsyi mean surely its there since snap info shows: 20/beta:          5.4.0-34.38.1  2020-06-08 (524) 271MB -13:37
ijohnsonclmsy: hmm then I don't know what the cause of that issue is then13:38
* mvo hands pstolowski the "biggest-PR-of-the-year" award13:56
pstolowski /o\13:56
mvopstolowski: heh - not your fault that snapstate_test grew to such monstrous size :) thanks for cleaning it up !13:58
* mvo hands pstolowski the "Augean Stables" award instead13:58
zygaBack home14:10
zygaNeed rest14:10
zygaDarn pain14:10
* mvo hugs zyga (gently)14:12
zygaI won’t be doing those 12 hour flights anytime soon14:17
mupBug #1882957 opened: Snapd `internal error: connection "[slot] [plug]" not found in state` <Snappy:New> <https://launchpad.net/bugs/1882957>14:19
cachiopedronis, this is what I updated in spread: https://github.com/snapcore/spread/pull/7214:19
mupPR spread#72: Skip tests <Blocked> <Reviewed> <Created by sergiocazzolato> <https://github.com/snapcore/spread/pull/72>14:19
cachiopedronis, today I'll push another implementation using SKIP command14:19
cachioI need to make that work better14:20
cachiobefore pushing14:20
clmsyim almost able to build a custom image w core20, one last road bump im having14:23
clmsyerror: cannot add snap "network-manager" without also adding its base "core" explicitly14:23
clmsynow i have type:base core2014:23
clmsydo i need to add say core16 without type:base or somehow add core16 base information to snap header of network-manager14:23
ograeither add "core" to your required-snaps in the model assertion or find a different track for network-manager14:24
ogra(i'm sure at least a core18 track exists for it)14:24
clmsyok thank you, ill try figure that out14:24
pedronisclmsy: you need to use type: core for core14:24
clmsyok i guess as long as type:base is core20 i am allowed to add more type:core14:25
ograwell, the NM he installs has seemingly "base: core"14:25
ograso "core" needs to be in the image as well ... or an NM with core18 or core20 base14:25
pedronisyes, he can do that by having an entry in snaps with type: core  name: core id: of core14:26
clmsythanks for the fast feedback!14:26
ograah, in the model assertion ? (thats new !)14:26
pedronisogra: in core20 models requires-snaps is called snaps and has much more syntax14:26
pedronisit's a list of maps14:26
ogra(i have admittedly not looked a lot at 20 yet)14:27
pedronisogra: https://github.com/snapcore/models/blob/master/ubuntu-core-20-amd64.model for an example14:27
ogra(some docs, howtos and tutorials would be really helpful)14:28
mupBug #1882957 changed: Snapd `internal error: connection "[slot] [plug]" not found in state` <Snappy:New> <https://launchpad.net/bugs/1882957>14:28
ograoh, this is really cool !14:28
ogracan you omit the id filed for using local snaps though ?14:28
clmsyyeah i like the syntax and the reason i jumped into trying to utilize is that, one of our main images that we use, i needed to implement full disk encryption and secureboot14:28
clmsythan i saw that with core20 i can utilise grade:secured for that feature14:28
ogra(local snaps are pretty essential during development)14:29
pedronisogra: yes, but only in grade: dangerous14:29
ograthats good enough14:29
pedronisyes, it's supported exactly for development14:29
ograyeah 🙂14:29
mupBug #1882957 opened: Snapd `internal error: connection "[slot] [plug]" not found in state` <Snappy:New> <https://launchpad.net/bugs/1882957>14:31
mvocachio: silly question - I have one spread run for ubuntu-core-20-64 (https://github.com/snapcore/snapd/pull/8845/checks?check_run_id=757318867) that uses an image that has snaps installed. when I run the same test here with my local spread against google I get a ubuntu 20.04 base image without snaps. any idea how this can happen? why would we get different images for 20.04?14:33
mupPR #8845: [RFC] many: add "system.service.snapd-autoimport.disable" setting <Needs Samuele review> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8845>14:33
cachiomvo, let me check the pr14:34
cachioone sec14:34
zygamvo: interesting failure in repartition magic14:36
mupPR #8829: sandbox/cgroup: add tracking helpers <Created by zyga> <https://github.com/snapcore/snapd/pull/8829>14:36
mupPR snapd#8847 opened: tests: fail in setup_reflash_magic() if there are snaps already <Test Robustness> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8847>14:36
zygaI would add retry test -e /dev/loop1p2 before that settle call14:37
zygaI suspect we just call settle in a racy way14:37
zygasince the partition may not exist yet14:37
zyga(on line 547 in the log)14:37
cachiomvo, agree with zyga, it has to be a race14:37
zygaI can send patches later but I really need to rest in a neutral position now14:38
zygaso away from screen and kbd14:38
cachiomvo, I'll try to reproduce it here to see how to fix it14:39
clmsywell guys i think you might wanna add multiple core support on the model because the image tool is not allowing this14:40
clmsyerror: core snap has unexpected type: base14:40
clmsyhere ill show you how the model assertion looks like14:40
clmsyi tried removing type information from core18 and core as well14:41
clmsystill complains14:41
ijohnsonclmsy: core18 should not have `type: core`14:42
ijohnsonclmsy: core18 should have `type: base`14:42
ijohnsonI think14:42
clmsyi can try that sure14:42
clmsygut feeling was that i thought having 3 snaps set as type: base might cause an issue14:43
ijohnsonwhich snap is used as the actual base for the image is decided by the base header in the root of the model, not by the type: base in the snaps header iirc14:43
pedronisijohnson: correct14:50
pedronisthat's the reason it exists, we don't support many gadgets or kernels, but there can be many bases, but only one is the boot/root base14:51
clmsyyes thank you :)14:54
=== pedronis_ is now known as pedronis
mupPR snapcraft#3166 closed: plugin handler: load legacy plugins prefixed with 'x-' <bug> <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3166>15:01
mvocachio: is there any way to figure out what base image https://github.com/snapcore/snapd/pull/8845/checks?check_run_id=757318867 was using? i.e. is there a way for me to login into that image? I tried spread -debug google:ubuntu-core-20-64 but the image I get from that is different, it has no snaps but the log there shows that there are snaps in the image that failed in the test15:16
mupPR #8845: [RFC] many: add "system.service.snapd-autoimport.disable" setting <Needs Samuele review> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8845>15:16
cachiomvo, yes15:17
cachiobut in case of core is difficult because the base image is focal and we update it during the prepare15:17
cachiomvo, so, to login to core-20 image then only way is to add an exit 1 in some place and use -debug15:18
cachioif you want to log into the focal image used as base you can do the following in spread-images project15:19
cachiospread -shell google:ubuntu-20.04-64:tasks/google/start-instance15:20
cachioyou need to comment the service account line in the spread.yaml to make that work15:20
mupPR snapcraft#3167 opened: unit tests: move to pytest <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/3167>15:22
mvocachio: right, I got to this point but even with exit in setup_reflash_magic I get an image that looks different, i.e. something without snaps at this point15:22
mvocachio: but in the log of 8845 I see there is core18/lxd installed in the image15:22
mvocachio: and I wonder how this is possible :/15:22
cachiomvo, lxd?15:24
cachiomvo, you see that in snap changes output right?15:25
mvocachio: yeah, if you look at https://github.com/snapcore/snapd/pull/8845/checks?check_run_id=757318867 and look at line 36515:25
mupPR #8845: [RFC] many: add "system.service.snapd-autoimport.disable" setting <Needs Samuele review> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8845>15:25
mvocachio: aha, I can even link to it https://github.com/snapcore/snapd/pull/8845/checks?check_run_id=757318867#step:5:36515:25
cachiomvo, yes, let me check the base image15:26
mvocachio: thanks, please check if we have any base image with that, if so I would love to login into one to debug this further :)15:26
cachiomvo, for ubuntu core 20 we are using the image ubuntu-2004-64-virt-enabled as base15:28
cachionot sure why we are not using the regular one15:28
cachiocan't remember15:28
mvocachio: can you check on this base image if it has snaps?15:29
cachiomvo, starting the image15:29
cachioalmost started15:29
mvocachio: thank you15:29
cachiobut takes few minutes15:29
cachiomvo, core18 and lxd installed15:30
cachiomvo, let me check why15:31
mvocachio: oh, nice. let's keep it this way for now, I want to do some debugging15:31
mvocachio: so here is something crazy - if I setup an "exit 1" in setup_reflash_magic and run spread google:ubuntu-core-20-64:tests/main/smoke I get a shell that does not contain snaps. am I just confused?15:33
cachiomvo, hehe, the snaps should not be there for sure, and are not installed during images are updated15:36
zygaijohnson, mborzecki: can you please look at https://github.com/snapcore/snapd/pull/8848 and think if we have a similar problem anywhere in non-test code?15:36
mupPR #8848: tests: wait after creating partitions with sfdisk <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/8848>15:36
cachiomvo, I think those are comming from the base image15:37
mupPR snapd#8848 opened: tests: wait after creating partitions with sfdisk <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/8848>15:37
zygamvo: could you be getting a different region15:37
zygaand a different image in that region?15:37
mvozyga: ohhh, yes15:37
mvocachio: is that possible? what zyga said, that we get diffrent images for different regions that might be slightly different? so when I run from my local machine I get a different result than github actions?15:37
mvocachio: fwiw, I really want to run with the snaps because I would love to debug why this fails15:38
cachiomvo, ah, ok, makes sense15:38
cachiomvo, in parallel I need to check why are those snaps installed15:39
cachiogoogle:ubuntu-20.04-64-base .../tasks/google/start-instance# snap list15:39
cachioName              Version   Rev    Tracking         Publisher          Notes15:39
cachiocore18            20200427  1754   latest/stable    canonical✓         base15:39
cachiogoogle-cloud-sdk  296.0.0   135    latest/stable/…  google-cloud-sdk✓  classic15:39
cachiolxd               4.2       15457  latest/stable/…  canonical✓         -15:39
mvocachio: can you find out somehow ? i.e. is there a setting somewhere in the website for gce etc?15:39
cachiosnapd             2.45      7777   latest/stable    canonical✓         snapd15:39
mvocachio: nice! yeah, that's what I see in the logs15:39
cachiomvo, I just started an image which is not ours15:40
cachioit is from cloud team15:40
mvocachio: anyway, in a meeting let's talk a bit more later15:40
cachiomvo, sure, I am having lunch now15:40
kyrofajdstrand, can you help explain this denial? Log: apparmor="DENIED" operation="bind" profile="snap.nextcloud.occ" pid=19769 comm="loolwsd" family="unix" sock_type="stream" protocol=0 requested_mask="bind" denied_mask="bind" addr="@6C6F6F6C7773642D74634C4838366E3700000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"15:40
mvocachio: can I modify spread.yaml somehow to get that image? after lunch is fine :)15:40
jdstrandkyrofa: aa-decode 6C6F6F6C7773642D74634C4838366E370000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000015:42
jdstrandDecoded: loolwsd-tcLH86n715:42
jdstrandkyrofa: that abstract socket is not snap-specific15:42
cachiomvo, yes15:42
zygakyrofa: aa-decode is your friend15:42
cachiomvo, just use -> image: ubuntu-os-cloud/ubuntu-2004-lts15:42
kyrofaHuh, I've never used aa-decode before15:43
zygakyrofa: first time for everything15:43
jdstrandkyrofa: it should match: @snap.@{SNAP_INSTANCE_NAME}.**15:43
jdstrandit isn't a tool that you normally need, but when you do, it is handy :)15:43
kyrofajdstrand, what is SNAP_INSTANCE_NAME in this context?15:44
jdstrandkyrofa: 'nextcloud'15:45
kyrofaAh okay15:45
jdstrandie, they should use @snap.nextcloud.anything-they-want15:45
jdstrandif they want to support parallel installs, they should look at $SNAP_INSTANCE_NAME from the environment to obtain 'nextcloud'15:46
jdstrandthe snap probably isn't parallel-installable now for a bunch of reasons though15:47
jdstrand(eg, port binding)15:47
kyrofajdstrand, no it's not, for reasons like this: https://forum.snapcraft.io/t/run-configure-hook-of-nextcloud-1-snap-run-hook-configure-error-error-running-snapctl-unknown-service-nextcloud-apache/1142215:48
* cachio lucnh15:49
kyrofaBut yeah, ports would be problematic as well15:50
cachiomvo, I need to edit out scripts to clean preinstalled snaps15:50
mvocachio: let's talk after lunch/meeting15:50
* zyga looks at a conflict and sighs15:52
alazredzyga: The manjaro-arm team has already updated their snapd package to add the systemd.apparmor.service. Should be alright now! Thanks again for your help!!15:54
zygaalazred: wooot :)15:54
zygaalazred: how is the arm laptop?15:54
alazredzyga: Feel good still some quirks to go around since it's new hardware. But it is still a good piece of hardware! I've been playing around with it for a week and so far I'm really pleased with it !15:57
alazredzyga: The only really negative thing so far are the speakers... They are useless. you need to be in an almost silent room to hear anything. Other then that pretty good.16:02
zygayeah, speakers are often neglected in laptops16:02
alazredthose are probably the worst ;)  but really for the price it's hard to complain16:03
ograwho uses them anyway16:04
mvocachio: I changed the image for ubuntu-core-20-64 to ubuntu-os-cloud/ubuntu-2004-lts and still no snaps installed. so it looks like I still get a different image under some cistumstances :/16:07
zygamvo: echo the root password and IP address and run a PR16:08
zygamvo: then get int16:08
zygaand check cloud init data16:08
zygaand maybe the image16:08
zygayou may get cloud region info there too16:09
zygamvo: or jokes adise16:09
zygaspread tests are invoked from a machine we can ssh to16:09
zygaso ...16:09
zygajust saying16:09
zygaif you need I can help16:09
mvozyga: could init data is a good idea, let me try this now16:09
zygamvo: if you want I can just give you shell access16:09
zygaand you can spawn spread by hand16:09
zygajust like tests do16:09
zygaand debug away16:09
mvozyga: are you in an image with snaps pre-installed?16:10
zygamvo: I didn't try but if the theory is that location matters16:10
zygawell, I can give you location16:10
kyrofajdstrand, regarding that abstract socket denial, I wanted to make sure you knew that snappy-debug didn't give me any advice16:11
kyrofaIt just gave me the log16:11
mvozyga: let me first try from my location and check the cloud init data, in any case, we should not get totally differfent images so I really want to understand this16:12
kyrofaIn case that's something you wanted to support16:12
zygayeah, I'm +10 on getting to the bottom of it16:12
jdstrandkyrofa: there is a bug on that16:14
jdstranderr, a checklist item in a card16:15
jdstrandbut thanks16:15
jdstrandoh, actually, I fixed that in edge already16:16
jdstrandkyrofa: ^16:16
jdstrandI need to do some 2.45 policy updates for snappy-debug, when I do, the fix will be in stable16:17
kyrofajdstrand, nice!16:17
jdstrandkyrofa: actully, I think it needs a small tweak for the encoded path16:18
jdstrandkyrofa: but that fix will be in stable16:19
jdstrandwhen I push it16:19
mvozyga: meh, no information in /run/cloud-init about the image booted it seems16:22
zyganothing? when I was looking at the /home/ubuntu mystery I did see some data (more than I could understand)16:23
zygamaybe something in journald?16:23
* zyga limps out of bed to get some tea16:27
mvocachio, zyga things starting to make slightly more sense now - so the ubuntu-20.04-64 image has lxd/core18 installed. we have code in prepare-restore.sh to purge snapd. However as evident in 8845 this code is not always run or does not always work. so I think that is the issue, I'm on it, no need to change anything on the image - I strongly suspect the purge fails under some circumstances16:34
ijohnsonmvo if the image you booted was in GCE it should definitely have /run/cloud-init ?16:35
mvoijohnson: sorry, I was not clear. it does have /run/cloud-init just no information what the image name is16:35
ijohnsonah ok16:35
zygamvo: could it be that the sequence matters16:35
zygaand depending on which suite runs first16:35
zygamvo: try looking at the log you saw from the failure16:36
zygawhat was the first suite on that machine?16:36
mvozyga: we purge as part of --prepare-project so that should be ok16:37
zygaI see16:37
zygathat is weird then16:37
mvozyga: if it's an incomplete purge that explains also why the mount units are not there anymore16:37
* zyga returns to resolving conflicts16:40
pedronispstolowski: should I re-review #8812 ?16:40
mupPR #8812: o/snapstate: service-control task handler (4/N) <Needs Samuele review> <Services ⚙️> <Created by stolowski> <https://github.com/snapcore/snapd/pull/8812>16:40
mupPR snapd#8847 closed: tests: fail in setup_reflash_magic() if there are snaps already <Test Robustness> <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/8847>16:42
cachiomvo, ah, ok16:46
cachiowell, just let me know if we need to clean up the image before publish it16:46
mvocachio: please keep it for now16:48
cachiomvo, sure, np16:48
mvocachio: I pushed a new PR that hopefully helps figuring out what is going on16:48
mvocachio: pushed 8849 that hopefully catches this in the future16:49
zygamvo: do you know if the wait in 8848 needs to be reproduced in any production code?16:50
mupPR snapd#8848 closed: tests: wait after creating partitions with sfdisk <Test Robustness> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8848>16:52
mupPR snapd#8849 opened: tests: fail in setup_reflash_magic() if there is snapd state left <Test Robustness> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8849>16:52
mvozyga: I don't think we need this wait17:13
zygamvo: which one?17:13
mvozyga: the snap wait seed.loaded in setup_reflash_magic17:14
mvozyga: I think that's not needed, snapd is purged earlier so it will never seed17:14
mvozyga: in this code we also don't need to install core, we can just download it :)17:14
zygaheh, thanks for looking at this with fresh eyes17:14
mvozyga: but let's only change it after we found the root cause of the other issue17:14
* mvo really needs dinner now17:14
zygaI did knee-jerk reactions it seems17:14
=== KindTwo is now known as KindOne
zygajdstrand: hey17:30
zygaI'm going to EOW soon17:30
zygaI'd love a +1/-1 on this test tweak17:30
mupPR #8803: tests: port interfaces-many-core-provided to tests.session <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/8803>17:30
zygaso that I can end the week with no dbus-daemon leaking :)17:30
zygait's a +2 PR but you need to look before it lands or doesn't land17:30
pstolowskipedronis: yes17:33
=== ijohnson is now known as ijohnson|lunch
* cachio app with kinesiologist17:45
jdstrandzyga: I commented such that you should be unblocked. I did not approve since I didn't pore over it18:24
zygajdstrand: thank you :)18:26
zygait's +2 so that's good18:26
mupPR snapd#8803 closed: tests: port interfaces-many-core-provided to tests.session <Test Robustness> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8803>18:32
=== ijohnson|lunch is now known as ijohnson
hellsworthhey folks is there a uc20 image that is not signed that will work in kvm?19:26
roadmrwhy unsigned?19:27
hellsworthwell maybe i just need the signature :)19:27
hellsworthwhich of these *.gpg files would have been used to sign the image?19:28
hellsworthmd5sums, sha1sums, or sha256sums19:28
roadmrnone :)19:28
roadmrhellsworth: image signatures are contained in an assertion if I'm not mistaken, those are signed with a key that lives in the assertion service19:29
hellsworthhmm ok i just wanna boot uc20 in kvm..19:29
hellsworthi have a lovely qemu command that worked just fine for uc18 and not for uc2019:30
roadmrohh I see19:30
hellsworthany advice would be helpful19:30
roadmrbut you do have an uc20 image which you didn't build yourself?19:30
roadmrthat likely contains the correct signature19:30
roadmrwhy isn't it working? what do you see?19:30
hellsworthok here is what i see: https://drive.google.com/file/d/1LLXq8kBPaNAqPc26ldAheBL-SH938936/view?usp=sharing19:35
hellsworthand my command to launch it is: https://paste.ubuntu.com/p/PD6pwtsZm8/19:36
hellsworthalthough i'm happy to use any method to boot a uc20 vm in qemu19:38
hellsworthmaybe the answer is to just not use core20 in kvm and rely on the rpi..19:42
roadmrhellsworth: ohh I see, that doesn't look like a *snap* signature problem :/19:43
roadmrhellsworth: is it possible the uc20 image is somehow broken? is it a daily? ty using an older one?19:43
hellsworthi have19:43
hellsworthi've actually tried several images from the last month19:43
hellsworthall had the same problem19:44
hellsworthso i figured it was something *I* was doing19:44
ijohnsonhey hellsworth19:44
ijohnsonso you want to boot uc20 with qemu without tpm yeah ?19:45
ijohnsonhellsworth: the issue I see with your qemu cmd you posted is that you need to specify -bios to use uefi19:46
roadmrat last someone who knows what they're doing unlike me :)19:46
hellsworthroadmr: thanks so much for helping and learning with me though :)19:46
hellsworthokey dokey ijohnson i can toss that option in there..19:46
roadmrlet me know if it works :)19:46
ijohnsoninstall the ovmf pkg, then use OVMF_VARS.ms.fd from there19:47
hellsworthsorry but what's OVMF_VARS.ms.fd?19:47
hellsworth(ovmf is already installed)19:48
ijohnsonhellsworth: that's the uefi vars you need to specify on the cmdline19:48
ijohnsonsorry let me just show you the full cmdline19:48
hellsworththanks :)19:48
ijohnsonso since you don't have tpm drop the19:48
ijohnson    -chardev socket,id=chrtpm,path=/var/snap/swtpm-mvo/current/swtpm-sock -tpmdev emulator,id=tpm0,chardev=chrtpm -device tpm-tis,tpmdev=tpm0 \19:48
ijohnsonoh sorry also you will want to copy the OVMF_VARS.ms.fd from the /usr/share/OVMF to somewhere you can write to it19:49
ijohnsonI just copy it to the current dir and use $(pwd)/...19:49
ijohnsondoes that all make snse?19:49
hellsworthoh yeah it does19:50
ijohnsongreat let me know how it goes for you19:50
ijohnsonhellsworth: also see https://docs.ubuntu.com/core/en/releases/uc20?_ga=2.193753581.1178879775.1591618991-2086332007.156228758719:50
hellsworth/usr/share/OVMF/OVMF_VARS.ms.fd is a binary file so writing to it doesn't seem to be a thing19:51
ijohnsonhellsworth: no you don't write to it19:52
ijohnsonQemu writes to it19:52
cmatsuokaI think I have a somewhat simpler qemu command line somewhere, let me see...19:53
ijohnsonTo store uefi vars19:53
hellsworthijohnson: ah ok re qemu needing write perms19:53
ijohnsonRight that's why you copy it19:53
* ijohnson -> errands biab 19:53
cmatsuokahellsworth: if you don't need encryption, this one also should work: https://pastebin.ubuntu.com/p/PZXjnvNDJ3/19:56
hellsworthok neat thanks cmatsuoka ! i'm actaully trying the command that is in the docs under Testing UC20 https://docs.ubuntu.com/core/en/releases/uc2019:58
hellsworthgood that the documented command is working :)19:59
hellsworthroadmr: the answers to my questions seemed to be here https://docs.ubuntu.com/core/en/releases/uc2019:59
cmatsuokahellsworth: nice, I think the line from the uc20 page is the same command with parameters in a different order20:01
hellsworthwell it uses qemu-system-x86_64 instead of kvm directly20:01
cmatsuokaah ok, yes, that part is different20:01
hellsworthso uc20 doesn't have dpkg or apt.. how do i install stuff?20:12
cmatsuokasnap install20:12
hellsworthof course20:12
roadmrthat's the whole point, hellsworth !!!!! 😄20:13
hellsworthhaha yes yes :)20:13
hellsworthit's jsut my brain's defaults :)20:13
roadmrwhatever you do don't install crapsnap20:13
roadmr(I don't think I even have anything published on that one)20:13
hellsworthi'm just looking to install my locally built network-manager snap to test.. so this should be fine20:14
* cmatsuoka tries crapsnap20:14
hellsworththere is a crapsnap in stable and edge..20:14
cmatsuokastable is good enough for this particular snap, I guess20:15
cmatsuokaoh no docker20:16
hellsworthit's interesting to me that the uc20 vm has networking because i'm obviously able to ssh into it. but it doesn't have the network-manager snap installed... if i install a new network-manager snap, then how do i know that *it* is being used/tested20:17
roadmrhellsworth: did you ask cwayne_'s team? they do just that kind of thing :)20:18
hellsworthhmm no.. what room is best to find that team in?20:18
cmatsuokacachio: have you seen this error before? https://github.com/snapcore/snapd/pull/8824/checks?check_run_id=75908103920:21
mupPR #8824: many: move encryption and installer from snap-boostrap to gadget <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8824>20:21
cachiocmatsuoka, checking20:21
cachiocmatsuoka, no20:22
roadmrhellsworth: try #ce-certification-qa on canonical irc20:22
cachioI'll try to reprpoduce it20:22
hellsworththanks roadmr20:22
cmatsuokacachio: thanks! I'm not sure if it's deterministic or not20:22
kyrofaroadmr, I'm installing microk8s and I see that it's installing from a track that is not latest. Does that mean I can point latest at whatever I want now as a snap publisher? Any chance you know where the docs are for that?21:10
roadmrkyrofa: probably using "default" tracks21:11
roadmrkyrofa: yep, 1.18 has been marked as the default track, so trackless installs will install from (and follow) that track21:12
roadmrkyrofa: https://forum.snapcraft.io/t/behaviour-change-sticky-and-default-tracks/14970 for now I don't know if there's any more documentation21:13
kyrofaroadmr, how do you mark a track as default?21:13
kyrofaroadmr, ah, thank you21:13
roadmrkyrofa: it's a newish feature so if you do use it and have any comments/bugs please do let us know ;)21:13
kyrofaWill do :)21:17
=== KindTwo is now known as KindOne
cachiocmatsuoka, I couldn't reproduce the error21:57
cachiocmatsuoka, I'll try again21:57
cmatsuokacachio: I suspected it was random, I'll re-run the test21:57

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!