mupPR snapcraft#3329 opened: cli: update revisions to use releases API <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/3329>00:21
mupPR snapd#9476 closed:  many: have install return encryption keys for data and save, improve tests <Run nested> <UC20> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9476>06:06
mborzeckimvo: hey06:37
mvohey mborzecki06:38
mvogood morning06:38
mborzeckipstolowski: hey07:02
mvogood morning pstolowski07:05
pstolowskihey mvo07:05
mvomborzecki: anything I should review for you? I'm doing zygas 9446 right now07:05
mborzeckimvo: no thanks, the reelvant prs have landed and i haven't opened anything new yet07:06
mvomborzecki: cool07:06
zygagood morning everyone07:10
mvozyga: goooood morning07:14
zygaindeed, it's a morning to be good on07:14
mborzeckizyga: heya07:15
zygahey hey07:15
zygaI'm looking at tumbleweed failures07:15
zyga<kill-timeout reached>07:15
mborzeckizyga: more of those?07:15
zygaon quiet govendor sync07:15
zygaI'll debug interactively in a moment07:16
mborzeckizyga: that most likely is the kernel panic again07:16
zygamaybe kernel modules and kernel itself are skewed?07:16
mborzeckizyga: idk, last time the panic susgested problems with fs07:25
zygaI'll know soon07:26
zygamborzecki passed without hanging or crashing07:46
mborzeckizyga: make sure there are no packages in the vendor directory (whcih should be the default with the changes to spread.yaml in master)07:47
zygaoh, good point07:47
zygarunning again07:48
pstolowskihuh, i think the download stalling issue is simply due to missing timeout, we create http.Client with default timeout 0 there, for reference https://golang.org/src/net/http/client.go?s=3300:3996#L9708:18
mvopstolowski: hm, timeout is not easy because we just don't know how long it might take :/08:21
pedronismvo: pstolowski: will have to think08:24
pstolowskimvo, pedronis i'd hope that this is smart enough to timeout of no response from transport, and otherwise be not affected by actual download time; func (b *cancelTimerBody) Read(p []byte) might be relevant there08:25
mvozyga: 9446 has a review08:25
pedronispstolowski: no, see the definition of timeout in the pkg, at least that timeout is not like that08:26
mvopstolowski, pedronis I guess we would need something like "timeout-for-no-new-data-from-the-remote". not sure if that is available in go(?)08:26
pedronismvo: pstolowski: we need to think a bit08:27
pstolowskipedronis: sorry, timeout in which pkg?08:27
pedronispstolowski: I mean, look at the doc of Client.Timeout08:27
zygamvo: I'll go over the feedback after uboot08:28
pstolowskifwtw I found retry logic fine and tried with separate standalone code that uses same combo of strategies as snapd, and hits limit on every strategy correctly. so getting stuck in io.Copy is my main suspect now08:29
pstolowskiah, "... and will interrupt reading of the Response.Body". ok, I think i misunderstood the meaning of this08:30
pstolowskipedronis: some ideas here https://medium.com/@simonfrey/go-as-in-golang-standard-net-http-config-will-break-your-production-environment-1360871cb72b - see Variable timeout section, with ltime.AfterFunc, a loop around CopyN and timer.Reset that pushes timer back08:36
pedronispstolowski: yea, let's discuss after standup or something, basically the issue is that using io.Copy for something that can take 10s of minutes was/is probably a bit naive08:38
mupPR snapd#9525 opened: packaging: make sure that static binaries are indeed static, fix openSUSE <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9525>08:42
mvopedronis: a peek at 9373 would be good for uc1.0. it's a bit of a extra but it would be nice if console-conf could have the ability to show the recovery key. but your earlier comments made me wonder if this should be a proper API in snpad instead of a client command.08:50
mupPR snapd#9523 closed: store, snap-repair: fix use of retry strategies <Bug> <⛔ Blocked> <Created by stolowski> <Closed by stolowski> <https://github.com/snapcore/snapd/pull/9523>08:52
pedronismvo: the blocker there is more that we need a bit more progress on save to know where exactly things are08:52
mvopedronis: oh, good point08:53
mupPR snapd#9525 closed: packaging: make sure that static binaries are indeed static, fix openSUSE <Simple 😃> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9525>09:57
mupPR snapd#9526 opened: snapshotstate: add clenup of abandonded snapshot imports <Created by mvo5> <https://github.com/snapcore/snapd/pull/9526>10:32
mborzeckipedronis: mvo: we may need to have larger ubuntu-save to be able to encrypt it10:52
mborzeckicryptsetup open complains about `Requested offset is beyond real size of device /dev/vda4`, probably something about the backup header10:53
pedronismborzecki: yes, moderm LUKS default use quite a bit of space11:03
mborzeckipedronis: ran luksFormat with --debug-json, if i'm not mistaken, just the keyslot area is 7MB11:04
zygamvo I've managed to build uboot from the old tree11:11
zygawith the patch11:11
zygagoing to boot it now11:11
pedronismborzecki: there's  --luks2-metadata-size and --luks2-keyslot-size , but the doc are not very clear what are reasonable values11:14
mborzeckipedronis: the docs aren't very clear at all :P11:14
mborzeckipedronis: this is what i'm looking at currently: https://gitlab.com/cryptsetup/LUKS2-docs/blob/master/luks2_doc_wip.pdf11:14
pedronismborzecki: afaik that one hasn't been updated to the larger defaults11:15
pedronisin 2.1 they: The default size of the LUKS2 header is increased to 16 MB.11:16
pedronisbut that doc still uses the smaller old values iirc11:16
mupPR snapd#9527 opened: o/snapstate: implement undo handler for unlink-snap <Needs Samuele review> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9527>11:22
zygapstolowski tiny feedback on your branch11:30
pstolowskizyga: yes, saw that, thanks!11:47
pstolowskizyga: hmm, did you mean boot.Participant ?11:50
zygapstolowski no no11:51
zygasorry I meant link participant11:51
zygapstolowski merge master, there's a new call in all link* methods11:51
zyganear the end11:51
pstolowskizyga: righ, I can see it now11:52
zygamvo: I was thinking, snapd could perform this test as a part of seeding, at least on pi, we could try to reboot and see the bootloader increment the boot counter11:54
zygamvo then we'd known instantly11:54
zygaand not at the next rare kernel update11:54
zygaor base update11:54
pedroniswon't the kernel cache get in the way of naive testing?12:05
pedronisah, you said try to reboot12:06
zygapedronis there's one bit missing that is essential12:06
zygapedronis I had a call with mvo where I described an extension to our boot protocol12:07
zygapedronis where apart from setting snappy_mode, the compatible bootloader would operate on a snapd-provided cookie12:07
zygapedronis the goal would be to be able to detect that bootloader can or cannot write to storage even on rollback12:08
zygapedronis so snappy_write_cookie=pong on snappy_write_cookie=ping12:08
zygaor something similar12:08
zygawe could be smarter but something extremely basic like that12:08
zygathe bootloader script could contain another hint that this is supported at all12:08
zygaand if we detect that it is supported but bootloader could not write, we could report to the store12:09
zygaI've booted my build of uboot!12:10
pedroniszyga: to me, it's a bit unclear what's our goal here12:16
zygapedronis to detect that bootloader cannot write to the storage and cannot ever upgrade certain snap types that require such writes12:16
zygapedronis so that we can report this and get statistics about it, right now we are in the blind12:17
pedroniszyga: but we don't have a way to report statistics12:17
zygaone thing at a time12:17
zygawe could send error reports12:17
zygait's better than nothing12:18
zygawe could send MMC card information as well12:18
mupPR snapd#9528 opened: cmd/snap-bootstrap: mount ubuntu-save during boot if present <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9528>12:18
zygamvo: it's fixed12:26
pedronismborzecki: question/comment in #952812:27
mupPR #9528: cmd/snap-bootstrap: mount ubuntu-save during boot if present <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9528>12:27
zygamvo: refreshed core18 just now12:31
zygano write issues12:31
zygarefreshing the rest (kernel) to check if it's good in general12:32
zygamvo: heh, updated kernel doesn't boot :D12:34
zygahangs after starting kernel12:34
zygapi and pi kernel well both following 18-pi/stable12:34
mvozyga: oh no! why is it hanging?12:35
zygamvo no idea, it rolls back the kernel on reboot12:42
zygamvo the only message I get is "Starting kernel"12:42
zygamy hunch is as followw12:42
zygaif you build a new image using stable 18 everything12:42
zygait will work12:42
zygabut upgrades are hitting dtb skew12:43
zygaand fail12:43
zygaI can check that easily by upgrading manually12:43
zyga(I'm running u-boot with fixed SD write support but otherwise matching revision 17)12:43
zygarefreshed gadget assets by hand, re-trying kernel update12:46
zygaso, the latest uboot we ship still has the write error12:51
zygaso it cannot make progress past that boot cycle12:51
zygaI'll build update u-boot with the patch and verify12:51
zygabut this is also interesting, it means that _no_ current uboot we have is good on this12:52
zygaand we can make images, and test them, that will just fail later12:52
ograzyga, hangs after starting kernel usually points to mistmatch of ctb12:58
zygaI'm sure that's the problem12:59
ograwell, you can easily test (on pre UC20 at least) by simply replacing it and moving the original one to *.bak12:59
zygayeah :)12:59
zygajust one sec, no network ;)13:00
zygastandup first13:00
pedroniszyga: what's the gadget and original kernel are you testing? pi2 pi3 ?13:50
zygapedronis: this is a pi3b+ with the snaps listed in https://bugs.launchpad.net/snapd/+bug/1900693 -- pi-kernel and pi (gadget) from 18-pi/stable tracks13:51
mupBug #1900693: snapd cannot refresh on some SD cards due to uboot bug <snapd:New> <https://launchpad.net/bugs/1900693>13:51
zygapedronis the particular affected revisions are listed ion the bug13:52
mborzeckithis is unexpected `2020-10-21 13:54:01 Cannot allocate google-nested:ubuntu-18.04-64: cannot find any Google image matching "ubuntu-1804-64-virt-enabled" on project "computeengine" or "ubuntu-os-cloud"`13:54
zygamvo: https://github.com/snapcore/snapd/pull/9149#pullrequestreview-51377543414:58
mupPR #9149: gadget: provide new gadget.ResolveContentPaths() (2/N) <Created by mvo5> <https://github.com/snapcore/snapd/pull/9149>14:58
mvocachio: could you please check https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1895929 and sru validate it?15:13
mupBug #1895929: [SRU] 2.47 <verification-needed> <verification-needed-bionic> <verification-needed-focal> <verification-needed-xenial> <snapd (Ubuntu):Fix Released>15:13
mup<snapd (Ubuntu Xenial):Fix Committed> <snapd (Ubuntu Bionic):Fix Committed> <snapd (Ubuntu Focal):Fix Committed> <https://launchpad.net/bugs/1895929>15:13
* zyga breaks for now, more later15:14
zyganeed to look after Lucy15:14
cachiomvo, sure15:28
* cachio lunch15:34
mupPR snapcraft#3330 opened: package repositories: fix case where formats is empty <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3330>18:19
mupPR snapcraft#3331 opened: spread tests: move package-repositories test snaps into own dir <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3331>18:29
mupPR snapcraft#3324 closed: set ROS_PYTHON_VERSION for rosdep <Created by artivis> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3324>19:19
mupPR snapcraft#3326 closed: lxd unit tests: simplify command checking pattern <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3326>19:19
cachiocmatsuoka, ijohnson hey20:11
cachioI see the test uc20-create-partitions failing in 20.0420:11
ijohnsoncachio: do you have logs?20:11
cachioijohnson, yes20:11
cachioijohnson, https://paste.ubuntu.com/p/qC5RvpCdVv/20:12
RingtailedFoxsince snapd isn't available for my distro, can i compile it from source?  i don't know if the fedora RPMs would work well with it...20:13
RingtailedFoxi know you can have snapd install flatpak on ubuntu.. is there a reverse way of doing that? (using flatpak to install snapd)20:14
ijohnsoncachio: the test needs to be updated, there was an update to the gadget snap20:14
ijohnsonRingtailedFox: what distro are you on ?20:14
cachioijohnson, ah, nice, do you know which update does it need?20:14
RingtailedFoxMageia Linux (aa descendent of Mandrake, later, Mandriva)20:14
ijohnsoncachio: let me take a quick look20:15
RingtailedFoxit's related to fellow forks ROSA, OpenMandriva and PCLinuxOS20:15
ijohnsonRingtailedFox: does your distro use systemd as it's init system ?20:15
RingtailedFoxit has since the days of Mandriva 2011.0 :)20:15
RingtailedFoxi sometimes miss sysvinit, but systemd has matured well enough to no longer suck :P20:16
ijohnsoncachio: yes I know what to change, it's a simple 1-liner I will file a PR shortly20:16
cachioijohnson, nice, thanks a lot20:16
ijohnsonRingtailedFox: then you should be able to compile snapd from source if you want to, not sure what kind of package management that distro uses, you mentioned something about rpms ?20:16
RingtailedFoxyeah it uses RPM and dnf20:17
RingtailedFoxalong with urpmi20:17
RingtailedFoxgithub.com/snapcore/snapd/releases/tag/2.47.1/ shows three different versions (no-vendor, only-vendor, and vendor).. which would i use?20:17
RingtailedFoxor should i just download the source tarball and try my luck with that?20:17
ijohnsonyou should be able to just compile/build the snapd rpm on your machine, you probably want the vendor version20:17
RingtailedFoxi never had much luck with compiling from source to RPM...20:17
ijohnsonhmm I'm not actually sure which version fedora builds from regarding vendorization ...20:17
ijohnsonyeah I'm not an expert at building rpms at all either tbh20:18
RingtailedFoxi have gotten some programs/libraries to work in the past by using fedora RPMs, but i try not to do that unless i have no other options20:18
* ijohnson is barely been able to build debs :-)20:18
RingtailedFoxi do have alien installed, so i can try to import a deb and convert it to rpm20:18
ijohnsonmaybe that works better? I dunno I have not uesed alien either20:19
ijohnsonsorry I can't be more helpful20:19
RingtailedFoxno no, you're still helpful20:19
RingtailedFoxi'll try to compile the vendor version and see if that works. if not, compile from source. if not that... then try to see if someone has a flatpak containing snapd :P20:19
RingtailedFoxi still find it hilarious i was able to get flatpak onto ubuntu through snap lol20:19
RingtailedFoxok. so snapd-2.47.1 is extracted... it looks like it's already coimpiled into binary format20:20
ijohnsoncachio: see 952920:22
cachioijohnson, checking20:22
mupPR snapd#9529 opened: tests/uc20-create-partitions: don't check for grub.cfg <Simple 😃> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9529>20:25
ijohnsonthanks cachio20:31
RingtailedFoxso what do i do with snapd-2.47.1? shove it in /usr/bin/ or /opt/?20:40
cmatsuokaijohnson: in core, what systemd unit file ultimately launches snapd?21:29
ijohnsoncmatsuoka: which release of ubuntu core21:29
ijohnsoncmatsuoka: so after the system has been seeded that will be snapd.service which is shipped by the snapd snap21:29
ijohnsonbefore that point, there is a systemd service file in the core20 snap which unpacks the snapd snap to run initially21:29
ijohnsonlet me find the name of that21:30
ijohnsoncmatsuoka: it's `core.start-snapd.service`21:30
cmatsuokauuhm ok, thanks, I'll have a look21:31
cmatsuokait seems that the systemd keyring mode setting failed for some reason21:31
ijohnsoncmatsuoka: is this for install mode or run mode?21:32
cmatsuokait's run mode21:32
cmatsuokaand it seemed that snapd was executed by some other way and snapd.service was not used21:32
ijohnsoncmatsuoka: well initially snapd in run mode for first seeding will be run from that `core.start-snapd.service` unit21:33
ijohnsonso if you need to modify snapd.service, you may need to adjust that one too21:33
ijohnsonbut the core service should do minimal amount of work to set up the real snapd snap service, then re-exec there21:34
cmatsuokathanks, I'll see what would be the best approach here21:34
ijohnsoncmatsuoka: is the issue that you need to read something from systemd upon startup to save it?21:34
ijohnsoncmatsuoka: if so, you might want your startup method to recognize if it's being run from core.start-snapd.service and if so then just skip that, and let the real snapd read that data21:35
cmatsuokaijohnson: yes, so if this first execution doesn't attempt to reseal anything, I won't try to read the auth key21:35
cmatsuokaijohnson: are you aware of a good method to detect that?21:35
ijohnsoncmatsuoka: let me take a look21:35
cmatsuokathe way snapd bootstraps isn't exactly straightforward I see21:36
ijohnsoncmatsuoka: what part of snapd uses the data that you are getting from systemd ?21:42
ijohnsonin other words, what depends on it?21:43
ijohnsoncmatsuoka: because what might be easiest is instead of a StartUp method on the devicemgr is to instead add a ensureSomething() that  gets called as part of the devicemgr.Ensure(), and that ensureSomething checks to make sure that the device is seeded, and then gets the data it needs from systemd, because the ephemeral snapd that is started by the core.start-snapd won't ever finish seeding21:44
cmatsuokaI collect the data in the device manager startup and use it when resealing21:44
ijohnsonpedronis may disagree with me on this however, not sure how important it is that we get this data from systemd asap after starting up21:44
ijohnsoncmatsuoka: ok, so we wouldn't reseal until at least after seeding is done21:45
ijohnsonso my proposal doesn't sound too terrible21:45
cmatsuokaOk, it sounds interesting, let's try that21:45
cmatsuokathanks Ian21:45
ijohnsonnp, let me know if you need any help, but for example you coudl take a look at how ensureCloudInitRestricted is implemented, that is essentially doing the same thing, waiting for snapd to be seeded then doing something21:46
cmatsuokaI think the idea of doing it asap is to block access to the keyring before we run too much stuff21:46
ijohnsonyeah that might be the reason why my ensure proposal isn't a good idea21:47
ijohnsoncmatsuoka: so after getting this data we essentially destroy it or otherwise prevent ourselves from accessing it from snapd again ?21:47
cmatsuokaso you didn't find a good way to determine where snapd was started from?21:48
cmatsuokaijohnson: yes, after reading it the user keyring is unlinked from the systemd session keyring21:48
cmatsuokaso it can't be accessed again21:48
mupPR snapd#9529 closed: tests/uc20-create-partitions: don't check for grub.cfg <Simple 😃> <Test Robustness> <Created by anonymouse64> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9529>21:50
ijohnsoncmatsuoka: hmm so the other thing you could do in StartUp is to check what systemd service you are running as, the ephemeral one will be under "snapd-seeding" instead of snapd21:51
cmatsuokaah interesting21:51
* ijohnson EODs22:04
mupPR snapcraft#3330 closed: package repositories: fix case where formats is empty <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3330>22:19
mupPR snapcraft#3329 closed: cli: update revisions to use releases API <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3329>23:04
mupPR snapd#9519 closed: tests: moving the lib section from systems.sh helper to os.query tool <Simple 😃> <Created by sergiocazzolato> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9519>23:16

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!