mupPR snapd#7832 closed: snap-bootstrap: support auto-detect device in create-partitions <UC20> <Created by mvo5> <Merged by cmatsuoka> <https://github.com/snapcore/snapd/pull/7832>01:07
mborzeckinfs-support failed on 19.10 https://paste.ubuntu.com/p/JvGPJxMxZx/06:29
sdhd-saschagood morning06:34
sdhd-saschamorzecki: how can i run this single test ? go test -check.f ... ?06:45
mborzeckisdhd-sascha: go test -check.f <testname> in the corresponding package or jsut passing the pacakge name eg. github.com/snapcore/snapd/cmd/snap06:46
mborzeckisdhd-sascha: writing import paths is tedious, so i just cd into the directory of the package06:47
sdhd-saschai tried:06:48
sdhd-sascha$ go test -check.f github.com/osutil/nfs_linux_test.go06:48
mborzeckisdhd-sascha: when in snapd tree, cd osutil ; go test -check.f nfsSuite06:51
mborzeckisdhd-sascha: nfsSuite because it's the test suite defined in nfs_linux_test.go file06:52
sdhd-saschamborzecki: thank you - it works06:53
sdhd-saschabut on my 19.10 it works06:54
sdhd-sascha go test -check.f nfsSuite06:54
sdhd-saschaOK: 1 passed06:54
sdhd-saschaok      github.com/snapcore/snapd/osutil        0.005s06:54
sdhd-saschaI have there a nfs-mount.06:54
sdhd-saschamborzecki: Or, is this a different nfs test in the paste from you? Error executing google:ubuntu-19.10-64:tests/main/nfs-support :06:56
sdhd-saschamborzecki: you mean the "spread" tests. sorry06:58
zygagood morning07:13
zygamborzecki: I have a question07:13
zygamborzecki: I was fighting snap security tag mess spread around the code and I think that's a separate cleanup07:15
mborzeckizyga:  hey07:15
zygamborzecki: do you think it would be ok to do just a regexp based validation on the supposed security tag07:15
zygamborzecki: and do nothing more in the sandbox/cgroup package for now?07:16
mborzeckizyga:  let me take a look at the pr07:17
zygamborzecki: I didn't push anything since yesterday07:18
zygamborzecki: because trying to make sense of how security tags are used and validated brings one headache07:18
mborzeckizyga: heh, indeed it does07:18
zygamborzecki: the best I can think of07:21
zygamborzecki: for now07:21
zygamborzecki: is that security tag becomes an interface that is implemented by a number of things07:21
zygamborzecki: pluse a few special cases that wrap a string07:21
mborzeckizyga: hm looks like the only code validating security tags is in s-c now07:21
mborzeckizyga: and it's a regex, so i guess it's fine07:23
zygait's a regex that says snap.$(snap name regex)(.hook)?.$(app name regex)07:25
zygaand one more weird variant, IIRC07:25
zygaon top of this I'd like to craft something that unifies all knowledge of security tags, documents it and puts validation functions that everything uses07:27
zygawe mostly always generate tags, not read them07:27
zygaso we didn't need much validation before07:27
zyga(in go, at least)07:27
sdhd-saschagood morning07:29
mborzeckizyga: sandbox/tags? or stuff that in snap/naming?07:30
zygahey, good morning sdhd-sascha07:30
zygamborzecki: I started snap/naming/sectag.go but not married to it07:31
zygaI think it's silly because of how we must be careful not to get import cycles07:31
zygaso eventual placement is really more related to where it compiles07:31
zygahey mvo07:39
mvohey zyga07:39
mborzeckimvo: hey07:44
mvohey mborzecki07:45
mvowhat did I miss this morning so far :) ?07:45
mvohow are tests looking?07:45
zygaI don't think you've missed much07:45
zygatests are green07:46
zygasome red but probably branch related07:46
zygamvo: schedule question, when do you cut07:49
zygamvo: and if you cut now, do you anticipate another cut before release or is that it?07:49
mvozyga: probably today07:53
mvozyga: and no worries, we can always cherry pick07:53
mvozyga: but once branched there will be no merge from master anymore07:54
mvozyga: like a full-one07:54
zygathat's fine07:54
zygajust wondering if everything you wanted in 2.43 is in master now07:54
zygaor is it likely we'll add more because there's known missing bits07:54
sdhd-saschazyga: didn't found the time these days to get thing working. Is difficult to explain, if you do it for free07:57
sdhd-saschazyga: Now i had another question07:57
sdhd-saschai just modified and build snap & snapd07:57
sdhd-saschaAnd i don't want to replace the original07:58
sdhd-saschaHow do you test the new installation on the same machine ?07:58
sdhd-saschaOr, do you use a vm ? I think about a user-systemd-service07:59
zygasdhd-sascha: it varies08:01
zygasdhd-sascha: it depends on what I changed08:01
sdhd-saschazyga: wait, i build a .deb package.08:01
zygasdhd-sascha: snapd vs snap-confine or other programs08:01
sdhd-saschai have enougt machines. and can replace it08:02
zygasdhd-sascha: I never build a distro package for that08:02
zygasdhd-sascha: most often than not I use "make hack" or I build snapd and run it manually08:02
sdhd-saschaI want to replace the snap cmdline and snapd08:02
zygasdhd-sascha: (just stop snapd.socket and snapd.service before)08:02
zygasdhd-sascha: for snap command you can just run it from the tree (go build ./cmd/snap and run it)08:02
sdhd-saschazyga: until now i never wrote "go" before. So i just copied "strace", remove the timings and add "ltrace" into osutils/ltrace08:05
zygasdhd-sascha: that's fine, go is easy to get into08:06
zygasdhd-sascha: there are remarkably few quirks08:06
sdhd-saschazyga: the unittests runs.08:06
zygasdhd-sascha: perhaps open a draft PR, it will be easier to give advice this way08:07
sdhd-saschazyga: if go had "tail-call optimization, then i would have started learning it earlier.08:09
zygasdhd-sascha: if it mattered, it would be there ;)08:10
sdhd-saschazyga: yes, i read about it. the solved anything by iteration. But the parser's api looks good at the first look ;-)08:11
mupPR snapd#7842 closed: tests: cache snaps also for ubuntu core and add new snaps to cache <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7842>08:12
zygacan a snap have an app that is just called "hook"?08:15
pedroniszyga: I don't think we have anything that would block that atm08:16
zygajust writing unit tests and got surprised08:16
zygagood morning pedronis :-)08:17
zygafound a bug already08:18
zygaValidateApp allows uppercase apps08:18
zygabut we don't allow that, do we?08:18
zygaat least, it was not allowed in the specs08:18
* zyga checks other implementations08:19
mupPR snapd#7775 closed: seed: support extra snaps on top of Core 20 dangerous models <UC20> <Created by pedronis> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/7775>08:20
zygacan app names have upper-cache characters?08:21
mvo\o/ another uc20 merged!08:21
zygaregexps say they do08:21
* mvo hugs mborzecki 08:21
zygamy memory says they don't08:21
pedroniszyga: no08:22
zygapedronis: we don't allow them?08:22
pedroniswe don't08:22
zygapedronis: then we need to fix a bunch of things first08:22
zygapedronis: I'll part my branch and address that first08:22
mborzeckiwe have ValidApp = regexp.MustCompile("^[a-zA-Z0-9](?:-?[a-zA-Z0-9])*$")08:23
pedroniszyga: you can look at snap/naming/validate.go08:23
zygayeah, I was there just now08:23
zygawriting tests and noticed one didn't fail as expected08:23
pedronisah, sorry, we do08:23
pedronisI was confused08:23
pedronisI read snap name08:24
pedronisno, we allow them08:24
pedroniswhat we don't allow is .08:24
zygapedronis: so name like package.FOO is okay?08:24
zyga(for an snap+app name)?08:25
pedronisseems so08:25
zygaok then08:25
zygaback to my branch08:25
pedronismborzecki: hi, what's the status of #7570 ?08:31
mupPR #7570: [RFC] snap-mount-dir: add mount unit for snap-mount-dir <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7570>08:31
mborzeckipedronis: i need to figure out the selinux errors that popped up there08:32
mborzecki#7840 is long but simple and could land easily08:37
mupPR #7840: tests: use test-snapd-sh snap instead of test-snapd-tools - Part 2 <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7840>08:37
pedronismborzecki: could you push your suggestions to #7555 when you have a slice of time08:50
mupPR #7555: tests: add a test demonstrating that snaps can't access the session agent socket <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/7555>08:50
mborzeckipedronis: sure, can do09:01
pedronismborzecki: thx09:01
zygamborzecki: pushed updates to https://github.com/snapcore/snapd/pull/782509:08
mupPR #7825: many: use transient scope for tracking apps and hooks <Created by zyga> <https://github.com/snapcore/snapd/pull/7825>09:08
zygamborzecki: have a look when you can, most of your comments can be resolved now, I think09:11
zygaI need to break for 30 minutes09:11
ackkmvo, hi, so I've  been testing a bit more with the maas snap, I do still see snapfuse using 100%cpu09:22
ackkmvo, I see maas-related processes keep being restarted, and I'm not sure if this is because they're responding too slowly because of the snapfuse issue or if it's unrelated09:23
pedronisackk: do you know if data in the snap is getting accessed all the time? naively I would expect the snapfuse cpu usage to match access, and I would expect less access once things are running. maybe it's really a bug in snapfuse not just a perf issue09:27
ackkpedronis, yeah there's a lot of I/O09:27
pedronisthat sounds strange in itself09:28
ackkpedronis, I'm going to compare with the same snap unpacked and installed with try09:28
mborzeckipedronis: left a note under 7555, the snippet i proposed doesn't work on arch after all, guess it's the socket mediation that's still missing from the mainline09:31
mborzeckiquick errand, back in 3009:33
pedronismborzecki: sounds like we should merge it as is then09:35
pedronismvo: is #7817 ready for re-review?09:36
mupPR #7817: boot,bootloader: make MakeBootable() uc20 aware <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/7817>09:36
ackkpedronis, yeah so same snap downloaded and unpacked works fine09:39
zygaackk: when you say there is lots of IO, can you explain what kind of data is maas reading after startup?09:40
zyga(from $SNAP)09:40
ackkzyga, there's a lot happening at maas startup, including reading a lot python files. I'm not sure if that's the cause, but what I see is that supervisord spawns the different processes (named dhcpd squid ...), no error in logs, then after a few seconds they get killed and respawned09:41
ackkthat doesn't happen if I unpack the snap09:41
mupPR snapd#7844 opened: ignore visual studio code directory <Created by sd-hd> <https://github.com/snapcore/snapd/pull/7844>09:42
zygaackk: ah09:42
zygaso it's essentially perpetual startup09:42
ackkzyga, correct09:42
zygaI was trying to understand if there's, normally, lots of IO after startup09:42
ackkzyga, no09:42
zygawhy do they crash?09:42
ackkzyga, I mean, there is if  your're synching imagaes09:42
ackkzyga, no idea, nothing in logs09:43
zygacan you see them via systemctl status09:43
mupPR snapd#7845 opened: add prefix to SYSTEMD_SYSTEM_GENERATOR_DIR <Created by sd-hd> <https://github.com/snapcore/snapd/pull/7845>09:43
zygaI guess no, because supervisord09:43
ackkzyga, correct09:43
zygawell, that's something to debug then09:43
ackkI'm going to try in a fresh container09:44
mupPR snapd#7837 closed: devicestate: implement creating partitions in "install" mode <UC20> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7837>09:46
mvopedronis: yes, I updated it this morning09:47
mvopedronis: once that is in I have a simple followup that adds this call to the install handler in devicestate09:47
ackkpedronis, zyga not sure if my message got through before the split10:01
=== igordhossegor is now known as 07EABDLLI
mupPR snapd#7846 opened: devicestate: add missing test for failing task setup-run-system <Simple πŸ˜ƒ> <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/7846>10:11
zygaackk: I didn't see it10:12
pstolowskiChipaca: ping10:23
Chipacapstolowski: pong10:24
pstolowskiChipaca: i've updated/tweaked your check-skeleton PR but for some reason coudn't push to your PR; mind merging my changes from https://github.com/stolowski/snapd/tree/check-skeleton-check-interfaces if you're ok with them?10:25
Chipacayou should be able to just push tho10:26
Chipacapstolowski: what error does it give you?10:26
pstolowskiChipaca: nvm! i had your remote with https:// ; worked with ssh://10:29
Chipacapstolowski: :-D yeah happens10:29
pedronispstolowski: thanks, I put it in my queue for re-review10:30
pstolowskipedronis: there is one remaining remark from you about moving ParseMountEntry to shared. no strong opinion about that, can do10:31
pedronispstolowski: I'll look again, is not a blocker either way10:31
pedronispstolowski: btw, I'm doing a pass again on some of your pre-seeding PRs10:37
pstolowskipedronis: ty, just saw your comments10:37
pstolowskimborzecki: do you have a moment to finish the review of https://github.com/snapcore/snapd/pull/7697 ?10:38
mupPR #7697: interfaces/apparmor: handle pre-seeding mode <Preseeding 🍞> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7697>10:38
mborzeckipstolowski: sure, will do10:39
mupPR snapd#7697 closed: interfaces/apparmor: handle pre-seeding mode <Preseeding 🍞> <Created by stolowski> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/7697>10:55
mborzeckipretty annoying how github doesn't let you know the PR you're commenting on has been updated11:02
pedronispstolowski: question in #732011:08
mupPR #7320: snap/pack, cmd_pack: 'snap pack --check-skeleton' checks interfaces <Needs Samuele review> <Created by chipaca> <https://github.com/snapcore/snapd/pull/7320>11:08
pedronispstolowski: thx11:18
mupPR snapd#7847 opened: Parse seed if either kernel or base are not mounted <Created by xnox> <https://github.com/snapcore/snapd/pull/7847>11:19
mborzeckisnap run --strace hanging on 16.04 too: https://paste.ubuntu.com/p/FGK8JysymF/11:22
mupPR snapd#7848 opened: Mount ubuntu-data tmpfs, in one go with kernel & base <Created by xnox> <https://github.com/snapcore/snapd/pull/7848>11:23
* Chipaca gets coffee11:27
mborzeckihas anyone noticed that when there's a change in progress, and it's currently downlaoding a snap, runnig snap change in a separate terminal takes really long to show anything?11:39
zygalock contention11:45
zygawe update the status a lot11:45
mborzeckiwonder if that's beacuse of re-refresh taking state lock too often11:45
zygaperhaps that's that, though I didn't debug it that deep11:45
mborzeckisince it basically tries to run all the time11:45
zygawe could use read locks11:45
zygaand write locks11:45
zygaperhaps that would help reading the status11:46
zyga(or, you know, a real database)11:46
mborzeckior use the tracing endpoints and find out whihc lock is hit most often :P11:46
pedronisit runs every 100ms, I wouldn't call it all the time11:47
pedronisbut could be11:47
pedronismborzecki: if there's a download going, there is the progress updates and polling as well11:48
mborzeckiiirc there were also some patches from mvo to make download updates more coarse11:48
pedronismborzecki: anyway my point is that before fixing we need to know what is happening exactly, I wouldn't make changes based on speculation11:50
zyga100ms is a bit excessive11:51
mvozyga: do you think you could check 7823 again? I fixed the bits you raised there11:52
mvomborzecki: yeah, it was really crazy in the past, it would update the progress for every packet basicly but that got fixed11:53
zygamvo: done11:54
mvothank you zyga !11:54
mupPR snapd#7823 closed: snap-bootstrap: implement "run" mode in snap-bootstrap initramfs-mounts <UC20> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7823>11:54
mupPR snapd#7844 closed: gitignore: ignore visual studio code directory <Created by sd-hd> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/7844>12:00
mborzeckipedronis: thanks for the feedback in #7821, applied it12:07
mupPR #7821: interfaces/seccomp: parallelize seccomp backend setup <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/7821>12:07
mborzeckioh and it needs a 2nd review12:07
pedronismvo: did you see my question whether 7817 can be re-reviewed ?12:08
Chipacazyga: why is cmd/snaplock a thing? why isn't it in osutil?12:10
Chipacawe've put some random stuff in cmd but only when it was used from daemon and client and there wasn't a clear place for it12:11
zygaChipaca: it was placed there after code review12:11
Chipacaand the imports were wrong for *util12:11
pedronisChipaca: it uses dirs,  it not a util12:12
Chipacahere the imports are simple, and it's used from cmd/foo but also from overlord/snapstate12:12
pedronisit's very specific12:12
Chipacapedronis: we already use dirs from osutil12:12
pedroniswe don't12:12
Chipacado so12:12
pedronisstrace then is also wrongly placed12:13
Chipacaputting libraries under cmd/ is not a nice thing12:13
pedronisChipaca: putting things in util that are very snap layout specific is worse12:14
Chipacamaybe we should set aside an afternoon at a sprint or something and do an on-paper reorg12:14
pedronisanyway strace is misplaced as is12:14
pedronisChipaca: snaplock is also there because the lock is also implemented in C similary for snap-confine12:15
pedronisChipaca: notice that we have C libraries in cmd12:15
pedronisChipaca: so basically as I see your point but it's more complicated, bleeding non utility things into *util is still worse in my book12:16
pedronisit means you cannot expect them to be isolated12:17
pedroniswhich as util they should12:17
zygapedronis: for the record12:17
zygapedronis: they are in cmd because there was aversion to a makefile in top level12:17
zygapedronis: otherwise they could be anywhere12:18
pedroniswell we could have a different directory and some more Makefile fun12:18
pedronisI'm personally not very offended either way12:18
pedronisI mean by libraries under cmd vs not12:18
zygaI think we could use a top-level makefile that cleans up various per-directory makefile things12:19
zyga(separate of the library discussion)12:19
zygait would allow us to reuse bits and not define the same things over inconsistently (e.g. libexecdir)12:19
mupPR snapd#7849 opened: osutil: cmd: add ltrace <Created by sd-hd> <https://github.com/snapcore/snapd/pull/7849>12:22
zyga15 minute break, kids home12:22
sdhd-saschazyga: just created PR for "ltrace"12:22
zygasdhd-sascha: cool, I'll check it out in a while12:23
sdhd-saschazyga: hope, i didn't make mistakes ... i really need to learn go first ;-)12:23
pedronismborzecki: thansk for the changes, they were not blockers, I made a follow up nitpick remark though12:25
pedronisChipaca: but yes I agree we need to reorg our code, there is too much things at top-level and some less than clean bits. OTOH can only be done when the open PRs are under control given the churn12:27
mborzeckihm we still have fwupd-refresh.service starting in debian-sid images12:29
mborzeckicachio: ^^ wasn't that explicitly disabled/removed at somep oint?12:29
cachiomborzecki, yes, the script has been updated12:31
cachiobut the image is autogenerated on monday12:31
cachioI can manuallly trigger the update now if it makes sense12:31
cachioI just tested the script is it is disabling correctly the service12:31
cachioon debian 9 and sid12:32
mborzeckicachio: please do, the tests/main/degraded test keeps failing on sid12:32
cachiomborzecki, ok, I'll do it12:32
mborzeckicachio: thanks!12:32
cachiomborzecki, np12:33
mupPR snapcraft#2831 closed: appstream extractor: add support for code <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2831>12:40
zygaback with hot tea12:43
zygasdhd-sascha: is sudo required there?12:43
sdhd-saschanot sure, on my test machine i runs without sudo12:44
zygasdhd-sascha: ok, I _suspect_ it may not be required, unlike strace12:44
sdhd-saschazyga: you mean "sudo snap run --ltrace ..." ?12:44
zygano, I mean, internally can it not use sudo and still work12:44
sdhd-saschazyga: most of the code is adapted from strace . And the commands are very similar12:45
zygayeah but the reason is different12:45
zygastrace uses ptrace12:45
sdhd-saschabut there is no static ltrace and no timings12:45
zygaltrace seems to only use it in a debug code12:45
zygait would be good to check if it's really required12:45
zygaone more thing12:45
zygacan you look at tests/main/snap-run12:45
zygaand there at task.yaml12:45
zygait exercises snap run --strace12:46
zygait would be good to extend that to ltrace12:46
zygaI can tell you how to run those tests12:46
sdhd-saschazyga:  HACKING.md: "### Running spread tests" ??12:47
zyganot sure how correct that is12:48
zygaI guess you will find out12:48
sdhd-saschazyga: sure -- it's little bit cold here ... good time to start some vm's on the server ;-) :-D12:49
pedroniszyga: where does snap-update-ns output finishes here:  https://github.com/snapcore/snapd/pull/7728/files#diff-e4c8d0c72c7b412376e0a04e6ad4db41 ?12:57
mupPR #7728: cmd: implement snap run --explain <Created by zyga> <https://github.com/snapcore/snapd/pull/7728>12:57
zygapedronis: because that example doesn't use a mount profile12:57
zygapedronis: there should be a [snap-confine (returning)] section or something like that12:58
zygaI'm looking at that no12:58
zygajust adding headers via << >> first12:58
pedroniswell I wanted to play with << >> first12:58
pedronisnot sure they are better12:58
zygapedronis: yeah, I added Header() so it's easy to change :)12:58
pedronisI wanted to give you a changed .in12:59
zygago ahead, I'll integrate it12:59
zygaI think we need to expand the test though, it doesn't handle enough of explain cases yet12:59
zygabut it's a good first start for  now12:59
* Chipaca wonders what the area under a patch is, to integrate it13:01
zygaChipaca: it's just len(bytes)  :D13:02
sdhd-saschazyga: this line in HACKING.md is scary: "curl https://niemeyer.s3.amazonaws.com/spread-amd64.tar.gz | tar -xz -C $GOPATH/bin"13:02
sdhd-saschai will build this from repo.13:02
zygayou can build it13:02
Chipacasdhd-sascha: or snap install spread13:03
Chipacaalthough that one's a little old13:03
* Chipaca promised to update it and has only found 1h to work on it so far13:03
sdhd-saschaChipaca: +113:03
mborzeckiijohnson: quite puzzled that SNAP_CONTEXT is not triggered when running run-checks13:04
pedroniszyga: my question was where would that snap-confine returning be placed ? After the whole Configuring mount namespace according to mount profile section?13:04
mborzeckiijohnson: can you bash -x run-check --unit maybe?13:04
ijohnsonmborzecki: sure after I finish breakfast :-)13:06
mborzeckiijohnson: hah sure ;) enjoy your breakfast13:06
sdhd-saschaChipaca: What's need to be updated there ?13:06
cmatsuokamvo: with #7837 landed, can #7762 be closed?13:06
mupPR #7837: devicestate: implement creating partitions in "install" mode <UC20> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/7837>13:06
mupPR #7762: devicestate: core 20 install mode handling <UC20> <β›” Blocked> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/7762>13:06
Chipacasdhd-sascha: the snap package has a few issues, that i'm wanting to fix13:07
Chipacasdhd-sascha: like not being able to access ~/.spread for example :)13:07
Chipacasdhd-sascha: and not having a runnable kvm in it either13:08
sdhd-saschaChipaca: ah, ok - i will try it ;-)13:09
zygapedronis: the returning message is in snap-confine13:10
zygapedronis: I added Header now but I'll wait for your call to change it to << >> or whatever else13:11
mvocmatsuoka: can be useful for reference, but it's fine to close it13:11
pedroniszyga: I'm trying with <<, not super happy though, mostly because they don't take up vertical space13:12
pedronisalso pastebin doesn't like my paste atm13:13
cmatsuokamvo: ok. I'll add a comment listing the split PRs13:13
zygaah, pastebin doesn't like <>13:14
zygaevil html javascript hackers13:14
Chipacause kannada13:14
mvocmatsuoka: thank you13:15
pedroniszyga: not sure, seems to have problems13:15
pedroniszyga: anyway I tried this:  https://gist.github.com/pedronis/93703067a720d6ddce49cf6351f44ca513:16
pedronisnotice that << is one change, the others are indentation  and I reformulated one thing13:16
zygalet me compare13:16
Chipacafwiw i meant ᐸᐳ instead of <> but was mostly joking (as is https://github.com/vasilevp/aboriginal )13:18
pedroniszyga: btw, do you remember why you picked [ ] initially ? they make me think of .ini files tbh13:19
zygapedronis: we just tried things until it looked good enough13:20
mupPR snapd#7762 closed: devicestate: core 20 install mode handling <UC20> <β›” Blocked> <Created by cmatsuoka> <Closed by cmatsuoka> <https://github.com/snapcore/snapd/pull/7762>13:20
pedroniszyga: anyway as I said, not super happy with << either13:20
pedronisneed to think a bit13:20
pstolowskipedronis:, Chipaca #7320 is green, ok to land?13:23
mupPR #7320: snap/pack, cmd_pack: 'snap pack --check-skeleton' checks interfaces <Needs Samuele review> <Created by chipaca> <https://github.com/snapcore/snapd/pull/7320>13:23
Chipacapstolowski: thank you13:24
pedronismborzecki: another comment in 7821, sorry13:25
mborzeckipedronis: haha i should get another coffee13:25
mupPR snapd#7320 closed: snap/pack, cmd_pack: 'snap pack --check-skeleton' checks interfaces <Needs Samuele review> <Created by chipaca> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/7320>13:25
=== ricab is now known as ricab|lunch
zygapedronis: applied, double checking if it passes and if I missed anything13:35
zygapedronis: I'll push when ready13:35
zygapedronis: as for headers, perhaps for explain we could use terminal features (colors, bold, etc)?13:36
pedroniszyga: possibly yes, maybe Chipaca has input on that13:39
Chipacamaybe :)13:41
Chipacazyga: what headers?13:41
zygaChipaca: please look at https://gist.github.com/pedronis/93703067a720d6ddce49cf6351f44ca513:41
zygaChipaca: headers are << foo >> lines13:41
Chipacado we want this to be parseable in any way?13:43
Chipacai don't know if the first line is a message to the user or a feature or a complaint :-)13:43
pedronisno, I don't think parseable is a requirement13:44
pedroniswe likely would do something very different if it was meant for machine consumption13:45
sdhd-saschaShould we remove *.deb packages ? because on master it isn't buildable13:47
Chipacasdhd-sascha: ?13:47
Chipacasdhd-sascha: master should be buildable to a .deb (and an .rpm)13:48
Chipacasdhd-sascha: in fact we build it to a deb every spread run13:48
Chipacaand i think we also build an rpm every test run but am not 100% sure13:48
sdhd-saschaChipaca: on thought it was on my local machine, but then i tested it on https://launchpad.net/~sdhd/+archive/ubuntu/snapcore/+build/1820698313:50
sdhd-saschait's master plus my small patches13:50
sdhd-saschaChipaca: i will look later, maybe something wrong with the recipe ?...13:52
ijohnsonmborzecki: so when I run with run-checks, it seems that all of the packages are listed in the cmdline to `go test ...` but if I just do `go test ./...` in the root of the git repo I hit that test failure every time13:52
mvopedronis: a (maybe high-level?) review of 7817 would be great13:52
jdstrandmborzecki: hey, fyi https://github.com/snapcore/snapd/pull/7555#discussion_r35432337613:52
mupPR #7555: tests: add a test demonstrating that snaps can't access the session agent socket <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/7555>13:52
mborzeckijdstrand: thanks for the comment, let me check that again, but there was no denial related to the session agent socket13:55
jdstrandmborzecki: there would not be if you went to forced devmode. check the profile13:56
mborzeckijdstrand: there was one related to dac override when i executed part of the test that runs curl as root13:56
mborzeckijdstrand: ok, let me start the test again13:57
ijohnsonmborzecki: actually it's that run-checks will sneakily find go 1.10 to run my tests with, even though I have go 1.13 installed and that specific test fails on go 1.13 but works on go 1.1013:57
jdstrandmborzecki: you may also want: sudo sysctl -w kernel.printk_ratelimit=013:57
pedronismvo: ok13:58
jdstrandmborzecki: to avoid kernel rate limiting. also, if you are ever thinking that the kernel is rate limiting, reload the profile and try again (even with the above sysctl, this is still needed with, for example, capability denials)13:59
niemeyer> i will build this from repo.14:01
niemeyersdhd-sascha: Make sure you read the source.. :)14:02
sdhd-saschaniemeyer: :-D14:02
jdstrandmborzecki: not sure what your local test was, but keep in mind, test -S won't trigger the denial.14:02
mborzeckijdstrand: the test is what's done in 755514:03
jdstrandmborzecki: ok, so if you ran spread, then look at test-snapd-curl.curl's profile to see if it is the classic template. if not, see if there are any matching rules14:05
mupBug #1617302 changed: Specifying a nonexistent plug does cause any errors <Snapcraft:Triaged> <snapd:Fix Committed by chipaca> <https://launchpad.net/bugs/1617302>14:09
mborzeckiijohnson: your go 1.13 is from a snap right? so SNAP_CONTEXT would be set when you run `go test ./...`14:11
ijohnsonmborzecki: yes14:11
ijohnsonmborzecki: it fails with go 1.10 as a snap too, so that's all it is14:13
mborzeckiijohnson: ok, easy to check, https://play.golang.org/p/_MvNzO2qf3i should print the actual context14:13
mborzeckiijohnson: if you go run it with 1.1314:13
jdstrandmborzecki: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/lsm_hooks.h#n76314:15
jdstrand"Whereas binding and connecting to sockets in the file name space is mediated by the typical file permissions (and caught by the mknod and permission hooks in inode_security_ops), ..."14:15
ijohnsonmborzecki: yes you're right it's just any go version from a snap will fail that test, I left a comment in the PR14:16
mborzeckijdstrand: nice, thanks for the link!14:16
Wouter0100Hey guys! I've spend my time today figuring out Ubuntu Core a bit, and seems awesome. Although, I have another question (the first one I asked on the forums, which got answered quickly, thanks for that! <3): how is the update from Ubuntu Core 16 to 18 planned for devices in the wild, and maybe later from 18 to 20? Will this also in a transactional14:18
Wouter0100manner? Or is this something that isn't going to be automated?14:18
Wouter0100I've seen a core16 and a core18 snap, and someone nearly succeeded in just upgrading this snap - but will there be some official procedure?14:18
ogra16 to 18 is kind of -> send someone out with USB key :)14:32
ogracore20 will have a new cool feature called re-modeling that will exactly cover switching to a new version ... but at the same time core20 will also come with a completely new partitioning scheme out of the box ... not sure if that will be covered ...14:33
ograpedronis, ^^^ do you know ?14:34
Wouter0100Oeh, that's fancy! Any release schema for core20? Because we are also still developing the Ubuntu Core setup, and when it'll be released in beginning of 2020, we should be able to wait for that just fine :).14:37
pedronisogra: remodeling is not core20 only14:37
Wouter0100And I think that the migration to Ubuntu Core will happen with sending someone out with a USB stick :P14:37
ograsure, but if you can keep the frequency low i bet thats better :)14:38
=== ricab|lunch is now known as ricab
ograpedronis, ah, so core18 will get it too ?14:38
ograthen i take back the USB statement :)14:39
ograthe beauty of rolling releases :)14:39
Wouter0100Indeed, the company I'm developing this software for does not really like the fact that they've to send an engineer throughout NL :-P. Luckily it's a small country, but still14:39
Wouter0100Oh, thats awesome! So then I'm just able to continue with core18 and when remodeling is released I will be able to upgrade to core20?! 😍14:40
ogratechnically yes ... practically there is that new partitioning scheme .... not sure how that will be handled during re-model14:40
Wouter0100I see14:41
ograi know mborzecki did an awesome job doing gadget upgrades (which holds the partitioning info) but i dont know if that will be integrateable with re-modeling14:41
mborzeckiogra: we do upgrade gadget during remodel, but the scheme must be compatible (the same to be exact)14:42
ograyeah, thats what i feared14:42
ograso 18 to 20 might be tricky14:42
ograunless we start soon with providing core18 images with the same partitioning scheme that core20 will have14:43
ogra(or at least example gadgets for customers to base on)14:44
mvoWouter0100: we plan to support 16->18 via the network, there shouldn't be any need for a usb stick here. it's a special case of the "remodel" feature that got recently added14:44
mvo18->20 is a bit more fuzzy at this point but I'm pretty sure we will support it, it may just be that you won't get some of the advanced features like recovery/encryption because the partition layout of 16/18 is not flexible enough for this14:44
pedronisogra: we are recommending/considering that for some of the recent core 18 projects, to start with a more compartible layout, at least to have the space14:45
ograpedronis, yeah, we should start pointing people to the example gadget.yaml so they know what partitions are needed14:46
pedronisdon't think we have plans to do that for the references images though14:46
ograyep, understood14:46
* zyga is in a car now14:47
zygapedronis: please have a look at https://github.com/snapcore/snapd/pull/7728 again,14:47
mupPR #7728: cmd: implement snap run --explain <Created by zyga> <https://github.com/snapcore/snapd/pull/7728>14:47
zygaI think the expected.in file is now what you specified14:47
mupPR snapd#7850 opened: apparmor: allow 'r' /sys/kernel/mm/transparent_hugepage/hpage_pmd_size <Created by jdstrand> <https://github.com/snapcore/snapd/pull/7850>14:48
Wouter0100I see, awesome. Then i'll have a look if I'm able to use the new partition layout for 18. And encryption? Is that something coming in 20? As I've discussed that with my customer that I'm developing this for, and at first said it wasn't possible - but if that would be possible in 20 - some additional leverage to get Ubuntu Core through :)14:49
mvoWouter0100: it will come with UC20 (encryption)14:51
Chipacabut later in 20, right?14:52
mupPR snapd#7851 opened: tests: use test-snapd-sh snap instead of test-snapd-tools - Part 3 <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/7851>14:52
Wouter0100mvo that's awesome14:55
mvoChipaca: well, potentially later, we aim still to release with it but it will be a challenge :/14:56
Wouter0100When will UC20 release then? :O14:56
Wouter0100Or no ETA planned?14:57
* cachio lunch14:57
zygaWouter0100: 20.04 AFAIR14:57
mvoWouter0100: roughly at the same time as 20.04 is what we aim for. no exact date set yet though. and we want to have the encryption ready by that date but it's a complex feature that we need to get right so this part *might* slip14:58
mvobut we *really* want to have it as part of the first uc20 release and work very hard to make this happen14:59
pedronis20.04 is for amd6415:01
mvopedronis: indeed15:02
mvosorry, did I miss some context and this was about non amd64? the non-amd64 will come a bit later, not april 202015:02
mborzeckijdstrand: this is weird https://paste.ubuntu.com/p/8FWf5S8rcv/15:03
pedronismvo: I'm not sure, just making clear that 20.04 is not for all archs15:03
jdstrandmborzecki: is the curl command actually talking to the right thing? you can snap run --shell and try the head and the curl15:07
jdstrandmborzecki: fyi, for hpage_pmd_size: https://github.com/snapcore/snapd/pull/785015:08
mupPR #7850: apparmor: allow 'r' /sys/kernel/mm/transparent_hugepage/hpage_pmd_size <Simple πŸ˜ƒ> <Created by jdstrand> <https://github.com/snapcore/snapd/pull/7850>15:08
mborzeckijdstrand: this is even more interesting https://paste.ubuntu.com/p/Wk4GHjfQKJ/15:08
mborzeckijdstrand: and it's using the same path15:10
jdstrandmborzecki: hmm. so, seems like there is something fishy going on in the snap run/snap-confine areas15:10
jdstrandI think an strace is in order to see how snaap-confine is being invoked15:11
mborzeckijdstrand: but the last paste is without s-c, just system curl vs head under the same profile, curl works, but head is blocked15:11
Wouter0100mvo i see, thanks!15:11
jdstrandmborzecki: oh, I misread that. need to strace that first one15:12
mborzeckijdstrand: it's doing connect https://paste.ubuntu.com/p/nZvBJjJtW3/ maybe that isn't mediated as per file hooks?15:13
pedronispstolowski: it's not hookstate importing ifacestate, it's hookstate/ctlcmd right?15:16
Wouter0100Is there any way for a snap to postpone a refresh by a couple of minutes or houes? The snap I develop sometimes requires to be not interrupted. I could schedule15:17
Wouter0100I could schedule the refreshes for in the night, but can't guarantee that it'll be possible then. I've also seen a feature for network manager to postpone a refresh when on metered connection, but no hook or other ability to integrate this in my snap15:18
pedronisWouter0100: not yet, but is a planned feature15:18
ijohnsonhmm is there not a "AsDesigned" status for lp bugs? I seem to remember there was but perhaps that was another old bug tracker15:18
ijohnsonWontFix seems the closest to this15:19
Chipacaijohnson: nope, no asdesigned on launchpad15:19
pedronisyea, it needs to be WontFix15:19
ijohnsonyeah must have been another bug tracker I'm thinking of15:19
Chipacaijohnson: no LolNo either :-(15:19
ijohnsonChipaca: what about OMFGYASSS ?15:19
Wouter0100pedronis awesome! Do you know the issue for that? Can't seem to find it. I might try to contribute such thing myself - if that's allowed.15:20
mborzeckijdstrand: ok, using the same code for both scenarios and path is used explicitly, https://paste.ubuntu.com/p/GKTxkfH2GY/15:20
ograWouter0100, just work with the various channels ... leave your install on stable (or beta) and do all your work in edge ... only push it up to the next channel when you think it is ready15:21
Chipacaijohnson: is that a Status, or an Importance?15:21
ijohnsonboth imho15:21
* ijohnson stops distracting people15:21
* ijohnson for now15:21
pedronisWouter0100: no issue yet, it needs to interact with permissioning because we dont want to allow arbitrary delays. Also usually a snap needs to stop refreshes of more than itself15:22
Wouter0100Yeah, that's correct.. also for some networking snaps possibly. And I see, a more complicated matter.15:23
pstolowskipedronis: yes yes15:24
pedronispstolowski: that's totally fine15:24
pedronisctclmd is slightly oddly placed itself but that's different matter15:25
pedronispstolowski: ifacestate imports hookstate so that import would not have worked15:25
mupPR snapd#7852 opened: devicestate: call boot.MakeRunnable() in devicestate <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/7852>15:26
pstolowskipedronis: right15:29
=== Son_Goku is now known as Conan_Kudo
=== Conan_Kudo is now known as Son_Goku
zygajdstrand, mborzecki: what's going on?15:29
pedronispstolowski: +1  with two comments, one is really for a follow up at this point15:30
mborzeckizyga: see my last paste, there's a /run/user/12345/snapd-session-agent.socket, the test in #7555 checks that the snap cannot access it, the check fails on arch15:30
mupPR #7555: tests: add a test demonstrating that snaps can't access the session agent socket <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/7555>15:30
zygamborzecki: do you have denials?15:31
zygawhat's the permission on the socket?15:31
zygamborzecki: did we rule out DAC?15:31
jdstrandzyga: it is failing open15:31
zygamborzecki: ah, wait, so the test fails because you *can* access it?15:32
zygajdstrand: right15:32
mborzeckizyga: connecting using python (no s-c or snaps involved), while running with the profile, can access it as the user (unexpected), but not as root (expected)15:32
zygamborzecki: can you pastebin the apparmor profile for the snap please15:32
zygaI'm in a car but I can look15:32
zyga(not driving)15:32
jdstrandmborzecki: I'm in two different conversations and will circle back to this in a few minutes15:33
mborzeckijdstrand: sure15:33
zygajdstrand: hey, can you try to review snap-explain patches, just the snap-confine C changes15:33
zygajdstrand: there are two printing functions there, some getenv15:33
zygajdstrand: but you need to look if we hope of having it for 2.4315:33
mborzeckizyga: https://paste.ubuntu.com/p/N2r2n5T26b/15:34
mborzeckizyga: oh, and it's on arch15:34
jdstrandzyga: it is on the list. there are a pile of store reviews I need to tend to though first15:34
zygajdstrand: ack, no rush, it's not a priority item, just a nice-to-have15:34
jdstrandzyga: I'll try later today/tomorrow15:34
zygamborzecki: what's the owner of the socket?15:35
mborzeckizyga: the test user, it's a session service15:35
pstolowskipedronis: thanks15:35
zygamborzecki: owned by 12345?15:35
mborzeckizyga: yup15:36
zygamborzecki: can you try to run apparmor parser15:36
zygamborzecki: and get a pre-processed profile15:36
zygamborzecki: with all includes resolved15:36
zygamborzecki: it could be an arch bug in the include statements15:36
zygamborzecki: (in the things that get included)15:36
zygaI'd tell you the command but I'm on fedora now15:36
zygamborzecki: if you get that please pastebin the result15:37
mborzeckizyga: this shoudl be it https://paste.ubuntu.com/p/yNTxCKq5Wh/15:38
zygamborzecki: nothing, it's good :/15:39
mborzeckiit's 5.4.1 kernel, in case that matters15:39
pedronismvo: I did a pass on 781715:47
zygamborzecki: is this a regression?15:48
mborzeckihmmm, good question, there's isn't much of an older kernel in arch to try out ;)15:59
mvopedronis: THANK YOU15:59
mvopedronis: :)15:59
mvopedronis: in a meeting now, but will jump to it right after15:59
zygamborzecki: is this a new test?16:00
mborzeckizyga: yes, #755516:01
mupPR #7555: tests: add a test demonstrating that snaps can't access the session agent socket <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/7555>16:01
zygaDoes it pass on tumbleweed?16:02
mborzeckizyga: it's supposed to run on ssytems with strict confinement, i recommended looking at specific features, jdstrand pointed me to relevant part of kernel where unix socket connect should be mediated as file operations, but it's failing (?)16:02
mborzeckizyga: i think we degrade the apparmor profile on TW16:02
zygaWe do not16:04
pedronismvo: I don't understand why we need MakeRunnable now that we pass options to MakeBootable16:04
pedroniswell, we don't but we could/should16:09
ijohnsonpstolowski: mborzecki: does this PR description look ok now? https://github.com/snapcore/snapd/pull/7843#issue-34907442316:12
mupPR #7843: tests/cmd/snapctl: unset SNAP_CONTEXT for the suite <Simple πŸ˜ƒ> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/7843>16:12
pstolowskiijohnson: yes16:14
mborzeckiijohnson: yeah, land it16:15
pedroniszyga: what is the purpose of decode-mounts-opts ? I don't see it invoked anywhere16:18
mupPR snapd#7843 closed: tests/cmd/snapctl: unset SNAP_CONTEXT for the suite <Simple πŸ˜ƒ> <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/7843>16:18
mborzeckipedronis: iirc the pr using that is still on review16:19
mvopedronis: I'm happy to change it so that we can use MakeBootable() for that we need a way to say if it's from PrepareImageTime or from install time. I could add something to "BootsWith" or I could add "opts *bootloader.Options" to MakeBootable (but then we would import bootloader in higher level code :/16:20
mvopedronis: anyway, open for suggestions16:20
mvopedronis: also I should probably reply in the PR instead of here - let me do that - sorry16:21
mborzeckizyga: jdstrand: behaves the same on tumbleweed with 5.3 kernel16:21
jdstrandmborzecki: ack, I'm looking at this now. I'm going to distill this down and then talk to jj about it when he comes online16:22
pedronismvo: I'm confused because the other PRs has TODOs like:  I don't understand why we need MakeRunnable now that we pass options to MakeBootable16:22
pedronismvo: oops16:22
pedronismvo: TODOs like: // XXX: allow to override this16:23
jdstrandmborzecki: I suggest moving on for the time being until I circle back16:23
mborzeckijdstrand: ack, need to take my son to the swimming pool, so eod'ing anyway16:23
mvopedronis: my original idea was to (re)use MakeBootable but I'm not sure it's the best fit anymore. But happy to chnage the PR accordingly. we just need to decide how the options are passed16:24
mborzeckijdstrand: the preprocessed apparmor profile from arch https://paste.ubuntu.com/p/yNTxCKq5Wh/16:25
pedronismvo: I'm also not sure I understand why we need to pass RecoverySystem16:25
pedronismvo: yes, we need to agree on what we want16:26
mvopedronis: I can leave this out for now, it will become clearer late, we will need to write the "modeenv" file on the real run16:26
mvopedronis: the modeenv will contain the recovery_system because when the "run" system boots it will also seed itself and it needs to know what to seed (or am I missing something?)16:27
pedronismvo: ok16:27
mvopedronis: sorry, maybe I'm slicing it too much :/ I can make a bigger PR that includes more maybe?16:28
pedronismvo: I think I'm confused because we have two many half done things in this area16:28
pedronisand they start not to match16:28
mvopedronis: yeah, maybe it's better to create a big(ger) PR?16:28
pedronismvo: or try to go step by step16:28
mvopedronis: that implements most of the MakeBootable/MakeRunnable16:29
pedronisinstead of rush ahead16:29
pedroniswe haven't even set up the install system full correctly or not?16:29
mvopedronis: you mean, wait until e.g. 7817 is merged before opening the next one?16:29
pedronismvo: do we have the boot until install fully working?16:29
pedronisthere is quite a few XXX in 781716:30
pedronismvo: how do we find the kernel?16:30
mvopedronis: I think with 7817 we should have a working boot into install mode16:30
mvopedronis: kernel is found via grub.cfg from the recovery16:31
mvopedronis: it will scan the disk and pick the kernel it finds16:31
pedronismaybe the issue is also that the grub cfg are still in the gadget16:31
pedronisand I haven't looked at them in a bit16:31
pedronismvo: anyway meeting time16:31
mvopedronis: indeed16:32
pstolowskihmmm travis is taking ages..17:12
pedronis7768 needs a 2nd review17:44
pstolowskitravis is spinning on 'job received', and not progressing17:48
ijohnsonpstolowski: I've seen that before it for some reason always seems to happen after EOD for EU timezones :-/ my working guess is that the nightly jobs are configured to run after EOD for someone in EU and so PR's submitted during that time have to wait for those nightly jobs17:52
ijohnsoncachio: do you know when the nightly spread-cron tasks are configured to run ?17:52
ijohnsonpedronis: I can do a review of #7768 in my PM I think17:53
cachiowhich task ?17:53
pedronisijohnson: great17:53
mupPR #7768: interfaces: add raw-volume interface for access to partitions <Created by jdstrand> <https://github.com/snapcore/snapd/pull/7768>17:53
pstolowskicachio: https://github.com/snapcore/snapd/pull/777117:53
ijohnsoncachio: any of them?17:53
mupPR #7771: o/hookstate/ctlcmd: snapctl is-connected command <Needs Samuele review> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7771>17:53
cachioijohnson, this is the main project17:53
cachiothen in any branch there is a task17:53
ijohnsoncachio: right but when do the tasks run?17:53
cachioijohnson, it is defined in a file, this is an ecxample https://github.com/snapcore/spread-cron/blob/gce-update-centos-7/options17:54
pstolowskiijohnson, cachio may i ask either of you to retry https://github.com/snapcore/snapd/pull/7771 some time later and land if it's green?17:54
mupPR #7771: o/hookstate/ctlcmd: snapctl is-connected command <Needs Samuele review> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7771>17:54
cachioijohnson, this is another example https://github.com/snapcore/spread-cron/blob/snapd-trusty-sru/options17:54
ijohnsonpstolowski: sure17:55
pstolowskiijohnson: thanks. i'm eod'ing\17:55
cachiopstolowski, sure17:55
ijohnsoncachio: so that first one you posted has `$(date +%H%M) \< "1100"` is that evaluated on travis ?17:55
cachioijohnson, no17:56
cachiothis is evaluated in a vm in an internal env17:56
cachioijohnson, it evaluates every 10 minutes aprox any branch17:56
cachioand executes the run-checks fiel17:56
ijohnsoncachio: any idea what TZ that internal VM is on?17:57
cachioijohnson, no17:58
cachioI can check17:58
cachiogive me 2 minutes17:58
ograa virtual one perhaps17:58
cachioijohnson, https://paste.ubuntu.com/p/NVSdkZJyk8/18:00
ijohnsoncachio: thanks, so it seems that if some of the tests are generally written to run after EOD on that machine they will run right around now18:03
cachioijohnson, nightly suite is configured like pattern_extractor="if [ $(date +%H%M) \< "0200" ]; then echo ""; else date +%Y%m%d; fi"18:04
cachiothe rest of the branches which run tests are configured based on other triggers18:05
cachiofor example18:05
cachiothe core snap was promoted to candidate18:05
cachioso tests run using the new core snap18:05
ijohnsoncachio: sorry was at lunch, but so does that pattern mean that the job only runs when it is later than 0200 UTC? or it only runs when it is before 0200 UTC?19:17
cachioijohnson, it prints the date and it is executed when [ $(date +%H%M) \< "0200" ] is not true19:20
cachioso it runs after 2am19:20
cachioafter 2am it prints the date19:20
ijohnsonso anytime later than 0200 is when the test gets executed?19:20
cachioand it is compared with the previous date non empry19:20
cachioand if it is different then the job is executed19:21
cachiofirst try after 2am it will be executed19:21
ijohnsonbut what controls when this check is performed then?19:22
cachioijohnson, the code where this is controlled is https://github.com/snapcore/spread-cron/blob/master/cron.sh19:23
cachiowe have a snap which runs this constantly19:24
ijohnsonI see, so it's not really deterministic when these jobs actually run, the pattern there really just defines when this agent shouldn't run the job19:25
cachioijohnson, it runs at some poing after 2am19:26
cachiousually before 023019:26
cachiowhen the scirpt processs that branch19:27
ijohnsoncachio: ok thanks for explaining19:27
ijohnsonis that the only job that has a time of the day trigger? all other jobs are event based?19:27
cachioijohnson, why?19:27
cachiodo you need any change?19:27
ijohnsoncachio: I'm wondering because it seems somewhat frequently that we have the issue Pawel has this morning where Travis is stuck in the "job received" state sometimes for a couple hours19:29
ijohnsonI was thinking this was causing that somehow19:29
=== ricab_ is now known as ricab
roadmrjdstrand: hi there! is there any way to constrain kernel-module-control to only be able to load one specific module?19:32
cachioall the jobs we use for update images are time base19:32
cachiobase are triggered on monday19:32
cachioijohnson, you can see here when the jobs ran https://travis-ci.org/snapcore/spread-cron/branches19:32
cachioijohnson, I jsut retrigger debian 919:34
ijohnsonthanks for that cachio, it seems a bit like that isn't the cause of the "job received" issue19:34
cachioijohnson, perhaps it is caused because too many jobs running for the same project19:35
cachioperhaps there is a limit19:35
cachionot sure19:35
jdstrandroadmr: no. but we have the kmod backend that is meant to load modules on behalf of the snap. what module?19:39
ijohnsoncachio: could be, anyways it seems Pawel's PR failed on Ubuntu due to mount ns changes, the cgroup mount now seems to be ro instead of rw as expected? was there recently an ubuntu 16.04 image change recently?19:40
roadmrjdstrand: I don't know, but I can ask. The snap in question is domotzpro-agent-publicstore. I can point them to examples of using kmod to load the module they need, if you point me to some, in turn :)19:40
jdstrandroadmr: well, it doesn't work quite like that. if we know the module, we can consider adding it to a new or existing interface19:41
jdstrandroadmr: but it would be hardcoded in snapd, not something that they declare19:41
roadmrright... I see19:41
jdstrandroadmr: typically a brand store will do this via their kernel. there may be a way to do it via the gadget, but idk otoh19:42
roadmrjdstrand: right, this is intended to be in the public store :/ so that's the boggle here19:43
roadmrthey said they'd look at options on their side including making do without the module loading capability, but were asking if we had something else to use19:43
pedronisroadmr: we really need to understand a bit more what they are trying to do if they want to target the public store19:43
roadmrjdstrand: fwiw this is a customer (domotz/fingbox) and they do have a brand store, but because reasons, they want this particular snap to be on the global store19:44
jdstrandroadmr: it is hard to recommend something without knowing the module in question19:44
pedronisjust saying we need to load modules is too vague for that space19:44
roadmrjdstrand, pedronis : ok, let me ask them for more details and get back to you19:44
jdstrandpedronis: it is possible for use to have a generic interface that lists the modules to load as attributes and shove that into the kmod backend. not suggesting that, but this conversation made me think of that19:45
jdstrandwe protect those attributes via snap decl like with personal-files/system-files19:45
jdstrands/those attributes/that attribute/19:46
pedronisjdstrand: yes, that's an approach, but we would still like to know whether a more specific or an addition to an existing interface would make more sense I think19:46
jdstrandanyway, putting it out there for background thought19:46
pedronisanyway even for that case we know the list of modules19:46
pedronis*we need to know19:46
jdstrandfor sure. need to know the modules :)19:46
roadmrI've asked, will wait for a reply from them folks :)19:47
jdstrandroadmr: you might not suggest my otoh comment there at the end. I don't know if we want to go there19:48
jdstrandI was more just mentioning that to pedronis as food for thought19:48
roadmrjdstrand: I haven't offered anything concrete until I know more, just told them to get us more info so we can give them options that suit their use case19:50
ijohnsonjdstrand: quick question, do you think layouts should be allowed for new top-level directories in "/" ? I tried creating one and it failed because snap-update-ns isn't allowed to create a mimic on top of "/" and I'm not sure if this is intentional or a bug19:53
ijohnsonjdstrand: denial looks like type=1400 audit(1575575534.977:339): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="snap-update-ns.my-snap-name" name="/tmp/.snap/" pid=39112 comm="7" srcname="/" flags="rw, rbind"19:53
pedronisijohnson: it's a limitation of the current implementation19:58
ijohnsonpedronis: ah ok19:58
pedronisin principle we could allow some subsets of layout at that level carefully19:58
pedronisbut needs implementation changes19:58
ijohnsonso it could work someday and is kind of a bug that it doesn't work today19:58
pedronisit's kind of a bug, but given is top-level and doesn't work we haven't decided the exact policy19:59
pedronisI mean in general is a limitation, specific instances might still not work19:59
ijohnsonoh actually if I had read the docs the doc does say that you can't do that19:59
pedronisbecause of security/policy reasons once implemented19:59
ijohnsonperhaps what we should do is add a warning or fail during install/validation of a snap.yaml declaring those19:59
ijohnsonuntil the implementation could support it20:00
pedronisijohnson: yes, it is known not to work, so not surprised it is documented as such20:00
ijohnsonyes my fault for not reading docs closely enough :-)20:00
pedronisijohnson: saw your comment, did we hit another instance of the baseline of the mountspace checks changing behind our backs?20:01
ijohnsonpedronis: yes I think so20:03
pedronisthis is getting a bit tedious20:03
pedronismaybe we should rethink how we compute the baseline20:03
ijohnsonpedronis: unless somehow snapctl is remounting cgroups /o\20:10
pedronisI hope not20:10
pedronisijohnson: it's the kernel again20:11
pedroniswe decide to stick with the old one20:11
pedronisbut didn't20:11
pedronisor something like that20:12
pedroniscachio: did we re-build new images with the wrong kernel ^ ?20:12
cachiopedronis, in bionic?20:15
cachiopedronis, we should be using 4.1520:16
pedroniscachio: seems xenial now20:16
cachiopedronis, ah, I just updated bionic20:16
cachiopedronis, I can update xenial as well20:16
pedroniscachio: let me point to the failing PR20:17
cachioI'll make the change20:17
pedroniscachio: the issue is here:  https://github.com/snapcore/snapd/pull/7771#issuecomment-56228390720:17
mupPR #7771: o/hookstate/ctlcmd: snapctl is-connected command <Needs Samuele review> <Created by stolowski> <https://github.com/snapcore/snapd/pull/7771>20:17
cachiopedronis, thanks20:19
cachioI'll take a look to the image and I'll make the change to see if that fixes the issue20:19
cachiopedronis, it is pretty similar to the issue we saw in bionib20:19
pedroniscachio: yes, that's why I asked about the kernel, it seems similar20:20
cachiopedronis, xenial has 4.15.020:26
cachiothe problem could be something else20:26
cachioijohnson, the problem in bionic was caused by the kernel v5.020:28
cachioin this case in xenial the kernel is 4.1520:28
cachioso I am not sure if the problem could be different20:28
cachioboth mount-ns tests failed? or just 1?20:28
mupPR snapcraft#2832 opened: appstream extractor: take xml comments into account <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2832>20:29
NickZIs there a way to get a snapcraft package to stage packages from a third-party repo?20:30
ijohnsoncachio: sorry multiple tests failed I don't remember all of the test names20:31
cachioijohnson, np, lets wait for the results again20:32
cachioijohnson, 2019-12-05 20:28:10 Aborted tasks: 020:46
cachio2019-12-05 20:28:10 Failed tasks: 120:46
cachio    - google:ubuntu-16.04-64:tests/main/mount-ns:inherit20:46
cachiojust one failed20:46
cachiothe other passed20:46
cachioI think this is something new20:46
cachioijohnson, the images was updated  4 days ago20:47
cachioso this is some new package or a change in the code20:47
ijohnsonhmm this does seem different from the bionic issue20:47
cachioijohnson, my PR also failed on that test20:54
ijohnsonwell something updated it just seems a question of what updated20:55
mupPR snapd#7853 opened: [RFC] boot,bootlaoder: setup the snapd_recovery_kernel grubenv <Created by mvo5> <https://github.com/snapcore/snapd/pull/7853>21:02
ijohnsonwell after looking into it some more it seems like /sys/fs/cgroup _should_ be mounted ro, I'm not sure why our tests have it as being mounted rw21:04
ijohnsonI can submit a PR in a little bit updating the tests I think to unbreak master at least21:04
ijohnsonwould be good to understand why it was measured as being mounted rw however21:05
cachioijohnson, perhaps we could review it with zyga tomorrow21:08
cachioit is weird21:08
zygaI'm back but too sleepy to work21:29
zygasee you tomorrow21:29
zygaijohnson: I saw the failure21:29
stgraberso recent core (snapd) now prevents installing lxd inside a lxd container, breaking our daily CI and probably a bunch of systems out there22:26
stgraberjdstrand: as the only snap person who may still be awake, any idea of what changed in that last core (and snapd) update that could cause issues?22:27
roadmrinception 🀯22:27
stgraberas far as I can tell, it only affects the LXD snap and it may be apparmor related though I'm not getting any record denials22:28
stgraberthe culprit is snap-confine getting spawned and being told no by the kernel (looks like)22:28
stgraber2686  execve("/snap/core/8213/usr/lib/snapd/snap-confine", ["/snap/core/8213/usr/lib/snapd/sn"..., "snap.lxd.activate", "/usr/lib/snapd/snap-exec", "lxd.activate"], 0xc4203272c0 /* 19 vars */ <unfinished ...>22:28
stgraber2676  pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=20000}, NULL <unfinished ...>22:28
ijohnsonstgraber: I'm afk for a little bit but I'll be around a little later to discuss more22:28
stgraber2686  <... execve resumed> )            = -1 EACCES (Permission denied)22:28
ijohnsonstgraber: man page says that could also be due to the filesystem being mounted noexec, is that possible?22:34
ijohnsonstgraber: Could you post a full strace log?22:35
stgraberpastebin rejects it, too large22:36
ijohnsonHow large is the log? Could you email it?22:39
roadmraiee folks use one of the shared servers!22:39
roadmryou both have access :)22:40
stgraberijohnson: http://paste.ubuntu.com/p/rDRxqrdXn9/22:40
stgraberijohnson: un-b64 and un-gz to get the original22:40
roadmrclever solution :)22:40
stgraberroadmr: pretty used to having to extract crap from random user systems behind a bunch of firewalls and such :)22:41
ijohnsonstgraber: thanks I'll let you know if I figure anything else out, and make sure that the rest of the Snapd folks take a look in their morning22:44
stgraberijohnson: do you know what version of snapd used to be in core?22:45
stgraberwould make a local bisect a bit faster :)22:45
ijohnsonstgraber: when?22:46
stgraberlatest core updated snapd to 2.42.4, I'd like to know what the snapd version was prior to that in core, if it was 2.42.3 that'd make the diff pretty short22:46
mupPR snapd#7854 opened: snap-confine: raise egid before calling setup_private_mount() <Created by jdstrand> <https://github.com/snapcore/snapd/pull/7854>22:47
stgraberlooks like it was 2.42.1 looking at the previous version of the core snap on disk22:48
stgraberthat's unfortunate22:48
mupPR snapd#7835 closed: gadget: add missing test for duplicate detection of roles <Simple πŸ˜ƒ> <UC20> <Created by mvo5> <Merged by cmatsuoka> <https://github.com/snapcore/snapd/pull/7835>22:48
roadmrstgraber: confirmed, 16-2.42.1 was the previous on stable (e.g. revision 8039 for amd64)22:48
roadmrstgraber: I looked at my release history for stable22:48
roadmrcurrent rev is 821322:49
ijohnsonSorry looks like roadmr beat me to it22:51
roadmrijohnson: stgraber beat us both to it :)22:51
stgrabersnap-confine apparmor change introduces a "deny unix," which may cause silent denials, not sure why it would trip on an execve though22:51
stgraberbut that's the main suspicious thing I'm seeing, so lets try reverting that first22:52
stgraberroot@c1:~# snap install lxd22:54
stgraberlxd 3.18 from Canonicalβœ“ installed22:54
ijohnsonHmm that's odd22:54
ijohnsonAlso odd that our tests didn't catch this we have lxd inside lxd tests22:54
ijohnson(that supposedly work)22:55
stgrabergoing to re-introduce that line to confirm that it fails again if I do so, just to make sure I didn't disable too much of apparmor by accident22:55
stgrabernope, that's clearly the issue, re-introducing that line keeping everything the same causes the failure22:56
stgraberit could be an apparmor parser bug, a kernel bug, ... but "deny unix," is clearly denying too much somehow22:57
jdstrandstgraber: that explicit deny rule in snap-confine was always denied. that shouldn't change anything22:58
stgraberjdstrand: it sure does somehow, and only in containers, so likely causing something interesting to happen in the kernel22:59
jdstrandie, it isn't a new thing being denied. it is just a new thing not being logged22:59
jdstrandthat is very surprising22:59
jdstrandsnap-confine had no unix rules to begin with22:59
jdstrandapparmor_parser -p /var/lib/snapd/apparmor/profiles/snap-confine.core.8213 |grep unix23:00
jdstrand    deny unix,23:00
jdstrandstgraber: can you change that to an 'audit deny unix,'? or 'audit unix,'?23:01
stgraberjdstrand: "lxc launch ubuntu:18.04 c1 -c security.nesting=true" and then "lxc exec c1 -- snap install lxd" will reproduce it23:02
stgraberjdstrand: sure, I was actually looking for the syntax for that23:02
stgraberjdstrand: really smells like a kernel bug, "audit deny unix," doesn't log anything but things now behave23:04
stgraberI only see profile_load STATUS messages, no sign of anything else in there23:04
stgraberremoving the audit prefix and back to it failing23:05
NickZfor the record, to stage packages from a third-party repository, do this: https://gist.github.com/NickZ/828be7ad23cd9d9c88b9ab0b3624bf4423:06
ijohnsonstgraber: so interesting thing I don't think our tests test snaps inside lxd inside lxd, we only test lxd inside lxd. I will make sure we have a regression test for this23:07
ijohnson(well also snaps inside lxd directly)23:07
jdstrandstgraber: where are you adjusting the rule, int he container?23:08
NickZi've heard that snaps have been broken on bionic lxd for a while now23:11
jdstrandstgraber: basically, I can see things are broken, but can't fix it cause when I snap install in the container, the snap-confine profile is overritten and my changes are undone23:11
jdstrandstgraber: fixing it on the host has no effect. what did you do?23:13
stgraberijohnson: the problem is installing lxd inside a lxd container, so in theory your test is already doing that23:14
stgraberjdstrand: it's the profile in the container you need to fix23:14
jdstrandright, and as soon as I do snap install, the fix is reverted23:14
ijohnsonstgraber: this is the test https://github.com/snapcore/snapd/blob/master/tests/main/lxd/task.yaml23:14
stgraberjdstrand: I did: "cp /snap/core/8213/etc/apparmor.d/usr.lib.snapd.snap-confine.real ." then "vim" and finally "mount -o bind usr.lib.snapd.snap-confine.real /snap/core/8213/etc/apparmor.d/usr.lib.snapd.snap-confine.real", followed by a teardown to force a reload of the profiles23:14
stgraberijohnson: the problem I suspect is that you're testing the new snapd on the host23:15
stgraberijohnson: the issue happens when you run the new snapd in the container23:15
stgraberhmm, no, you're installing the new snapd in there23:15
jdstrandthat's... weird. I wouldn't expect .real to have anything to do with it, but you gave me an idea. I can just copy /var/lib/snapd/apparmor/profiles/snap-confine... aside, modify it and load it23:16
stgraberijohnson: ah, the problem is that you're never actually installing lxd in lxd23:17
stgraberijohnson: your test creates a lxd container, updates snapd inside it and then installs test-snapd-sh23:17
stgraberijohnson: if you were to snap install lxd instead of test-snapd-sh on line 100, you'd have gotten the failure23:17
stgraberjdstrand: I went with a naive "grep -r deny.*unix /snap/core", found the file that has the offending string and over-mounted it with a version which doesn't :)23:18
jdstrandstgraber: what host are you on?23:18
stgraber18.04 LXD VM23:19
jdstrandon an eoan host, it isn't working at all23:21
stgraberOh? As in broken on the host too?23:24
jdstrandstgraber: no, it is fine on the host, but I can't get the nesting to work23:25
jdstrandI'm trying in a vm23:25
* jdstrand tried with his lxd that was already configured23:25
NickZjust curious, has anyone here ever successfully got mono to run in containment?23:30
jdstrandstgraber: what is the output of aa-status in the container?23:40
jdstrandok, finally23:42
jdstrandijohnson: I downgrading to 8159 (2.42.2) and it does not have the unix rule. I then added to /var/lib/snapd/apparmor/snap-confine/foo 'deny unix,', the did apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap-confine* then tried to install lxd and it failed23:44
jdstrandijohnson: we should revert that rule. I'll need to investigate to see why it is failing23:44
jdstrandijohnson: this indeed is a kernel bug23:45
stgraberjdstrand: how fast can we get a that line reverted in the stable snap?23:47
jdstrandstgraber: that's an mvo question23:47
stgraberOk, I'll email mvo to make sure he's aware when he starts the day23:48
jdstrandstgraber: please cc me and feel free to paste my comments above about downgrading and that I confirmed this is a kernel bug23:49
jdstrandstgraber: you might cc jj as well23:49
stgraberWill do, thanks23:50
jdstrandit could be a parser bug too...23:50
jdstrandanyhoo, thanks23:50
ijohnsonstgraber: jdstrand: I'll make sure mvo is aware23:52
jdstrandstgraber: fyi, this is a more specific rule (ie, to  address the thing that prompted the PR in the first place): deny unix (receive, send) type=stream addr=none peer=(addr=none), and it also causes lxd to fail23:53
jdstrandstgraber: I can pass that to jj if you already sent the email or don't want to integrate that detail23:54
stgraberHaving dinner will send email after and include that in23:54

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!