[00:04] ijohnson: I'll update the data tables to be correct and push [00:06] zyga great but also feel free to wait until tomorrow :-) [00:07] ijohnson: small review on renameat2 https://github.com/snapcore/snapd/pull/8044#pullrequestreview-347688118 [00:08] PR #8044: grub: support atomically renaming kernel symlinks [00:12] zyga: thanks for the review, tldr is that it's not a big deal if the random source is poor, as it's just being super extra super safe anyways [00:14] ijohnson: and the renameat part? [00:15] zyga: the paths are absolute so the fds are ignored [00:15] hmm [00:16] I would have used rename2 but the only way to get atomicity is to use renameat2 [00:16] (with the RENAME_EXCHANGE flag) [00:18] ijohnson: I think that's okay but I would still use the special value, technically the 0 descriptor is valid and means something but the code that ensures arguments are absolute is not close to the call [00:18] * zyga nods [00:19] zyga I'll add a comment where it's used [00:20] I think using the special value is preferred [00:20] even with a comment [00:20] it's more like what you wanted [00:25] Okay I'll use that then [00:27] ijohnson: double check with jdstrand but I think that's safer [00:28] Okay, the man page says the FD is supposed to be ignored when it's an absolute path [00:30] yes, that's true [00:30] my point is not that this is incorrect [00:30] it's just that it can be safer [00:31] Sure [06:44] morning [06:58] PR snapcraft#2889 closed: meta: always generate snapcraft-runner to workaround classic PATH bug [07:03] zyga: interesting forum topic https://forum.snapcraft.io/t/permission-denied-in-general-ubuntu-19-10-snap-2-42-5/15161 [07:10] hmm not sure anything happend on the desktop front, snaps still render with boxes instead of actual fonts: https://i.imgur.com/V3mKUt8.png [07:54] <__chip__> 👋 [08:00] People : hi ! I have 9 machines (ubuntu LTS 18.04 server) that are split into 3 lxd clusters. Yesterday, at 15:13 all nine decided to update the snap lxd package. They are all stuck at task "Copying datas from the snap lxd package". [08:01] So, since yesterday, the "lxc" commands provided by this snap are not available anymore, and I can't do anything. I don't want (and can't) lose all my lxd containers. What can I do to have snap lxd package up-to-date ? [08:05] Hey [08:05] Outside [08:05] Back soon [08:05] zyga: __chip__: hey [08:05] geodb27: can you collect the output of `snap change --last=refresh` from one of the machines? [08:05] morning [08:05] pstolowski: hey [08:05] Thanks for your answer mborzecki. I'll check this right now. [08:07] Well, it remains the same as "snap changes" followed by "snap tasks nn" with nn the id of the change. And this leads to the step that is waiting since yesterday 15:13 as I said before. [08:11] I've just checked all my 9 machines. 6 are OK. But 3 (the three of one same cluster) are in the same state. Stuck at : [08:11] Doing yesterday at 15:13 CET - Copier les données du paquet Snap "lxd" [08:14] geodb27: weird, iirc lxd keeps container data under the location shared by all revisions [08:15] geodb27: can you check whether there any any files under /var/snap/lxd/current/ ? [08:16] there is no /var/snap/lxd/current. All I have is /var/snap/lxd/{12317,12631,common} [08:16] (On the first machine). [08:16] geodb27: try /var/snap/lxd/12317/ [08:16] re [08:17] On the two others I have /var/snap/lxd/{12317,12631,13073,common). But the last two are stuck at "Doing yesterday at 15:12 CET - Démarrer les services du paquet Snap "lxd" (13073)" [08:17] ha [08:17] geodb27: anything in there? can you run `du -sch /var/snap/lxd/12317/` ? [08:17] I saw this [08:17] last night [08:17] mborzecki: lxd cannot be stopped [08:17] it sticks around forever [08:18] I noticed this yesterday when rebooting [08:18] systemd eventually kills it after 10 minutes and just reboots [08:18] zyga: w8, so it hits all the timeouts, systemd still doesn't stop it? [08:18] but when I noticed the messages it was not able to unmount cleanly [08:18] zyga: but sunce we moved to the next task, systemd must have reported it stopped [08:18] mborzecki: it seems there are no timeouts for lxd [08:18] maybe it's something special it does [08:18] On the first machine, the /var/snap/lxd/12317 folder is empty. [08:18] mborzecki: for me the reproducer case was ubuntu 19.10 (now 20.04) with lxd and a container [08:19] stopping lxd doesn't work [08:19] geodb27: right, as zyga mentioend, maybe it's related to lxd not stopping [08:19] geodb27: can you ps -ef|grep lxd ? [08:19] Indeed, it could be. Let me check. [08:19] root 1518 1 1 janv.10 ? 05:36:05 [lxd] [08:20] zyga: ^^ [08:20] ha [08:21] What now ? If I reboot, will lxd complete it's upgrade and will I get all my containers back up and running ? [08:22] geodb27: I'd not test in production, cannot say for sure [08:22] geodb27: I suspect that snapd will recover [08:22] geodb27: and carry on with the update [08:22] geodb27: but depending on how your time works I'd evaluate this separately [08:23] Thanks for your help zyga. Anyhow, I have backups of my machines (fortunately, the containers are hosted on a ceph cluster)... I'll give it a try and give a feedback here as the reboot completes. [08:24] geodb27: thank you [08:24] mborzecki: we should check this out [08:24] zyga: any clue whether stgraber and his team noticed this too? [08:24] no idea [08:27] hmm `Instance name is: on-jackass` [08:28] is there a way to wait for a change? [08:28] from command line [08:28] snap watch --last [08:28] ah [08:29] zyga: if you're addressing the commnents in your PR then retry-tool probably makes more sense in case the change does not complete [08:29] ish [08:29] I must be doing it wrong [08:29] while snap changes | grep Doing; do sleep 1; done [08:29] this failed just now [08:29] with snap changes showing a change that is Doing [08:29] wtf? [08:30] https://www.irccloud.com/pastebin/pc7zuZZX/ [08:30] but I'm very suspect of the test itself [08:30] it happily REBOOTS [08:30] without any synchronization on async snapd activity [08:30] I suspect it need to have snap watch --last=... in each stage [08:30] as otherwise nothing is certain [08:33] Well... Fyi : I rebooted only one machine of the lxd cluster... It took some time (the reboot process had to kill a process related to filesystem mounted)... And... Tada : all three machines are up-to-date and running fine. Thanks for your kind help ! [08:34] I do not have time (and it is anyhow too late) to investigate further to find out what was hanging the system. One could suspect some lock (maybe on the cephfs side or on the lxd database at some point) that was holding things stalled. [08:43] we should write down the reasons for avoiding golang.org/x/sys/unix [09:04] mborzecki: I'll grab some food and make tea [09:04] mborzecki: I'm iterating on the wait/reboot problem [09:15] zyga: heh any ideas why /etc/apparmor.d/usr.bin.snap might exist? [09:21] zyga: meh, reading renameat2 implementation, not sure whether we need to sync the directory too [09:22] mborzecki: yes [09:22] mborzecki: I read about a case where someone had it [09:23] mborzecki: _something_ definitely creates it [09:23] mborzecki: not snapd [09:23] mborzecki: I cannot recall the details of the case I remember but it was probably on the forum too, let me check [09:23] hmm, not on the forum [09:24] zyga: so this guy got both snap and usr.bin.snap in apparmor profiles in case you missed it https://forum.snapcraft.io/t/permission-denied-in-general-ubuntu-19-10-snap-2-42-5/15161 [09:24] yeah [09:24] I read [09:43] + snap watch --last=revert [09:43] error: no changes of type "revert" found [09:43] + snap changes [09:43] 9 Done today at 09:39 UTC today at 09:41 UTC Revert "core" snap [09:43] sigh [09:43] that's such a crappy behavior [09:44] ah, it's called revert-snap [09:44] oh well [09:47] PR snapcraft#2891 opened: meta: always generate snapcraft-runner to workaround classic (#2889) [09:52] and I found a small bug [09:52] https://www.irccloud.com/pastebin/JBOzGPsz/ [09:53] ok, one more run [09:53] brrr [09:53] cooold [10:05] iteration on this is painful [10:09] mborzecki: I wonder if that denial was from snap-store [10:09] maybe it does something weird [10:10] zyga: that diff loos like we could use a helper of sorts [10:15] mborzecki: I changed https://github.com/snapcore/snapd/pull/8039 [10:15] PR #8039: tests: remove revision leaking from ubuntu-core-refresh [10:15] pstolowski: ^ can you look again [10:15] k [10:15] I mainly added the snap watch --last=... [10:16] as otherwise we just race [10:18] I'm running a small loop of this to see if it is stable [10:45] back with coffee [10:45] tests going [10:45] still good [10:50] passed [10:50] ok onto more branches [11:03] guys, please don't merge any of the test fixes yet [11:03] I'm iterating on comments and even though some are green and +2 I'll adjust them [11:27] heh [11:27] weird failure [11:27] https://www.irccloud.com/pastebin/SIB7x6zG/ [11:27] snapd is not installable [11:33] PR snapd#8045 opened: osutil: add helpers for creating symlinks and renaming in an atomic manner [11:36] mborzecki: ^ I think AtomicRename without replace/not-replace is unsafe [11:36] as it's easy to use it to nuke stuff [11:36] zyga: isn't rename the same? [11:36] yes, but that's my point [11:37] PR snapd#8046 opened: many, tests: integrate all preseed bits and add spread tests [11:37] it depends on how it is used obviously [11:48] hmm [11:49] why is kernel remodeling so slooooow [11:49] it's 10 per test [11:49] 10 minutes [11:49] unexpected failure https://paste.ubuntu.com/p/D6n2g4MfwD/ [11:50] hmmm [11:50] no idea, [11:50] I am really wondering if we do things right [11:50] we pull the rug from under snapd in teardown [11:50] not wait for things [11:53] <__chip__> zyga: the kernel is very prima donna about its hair, so remodeling takes ages [11:53] * __chip__ in Friday mode [11:53] __chip__: we should ... shave a few minutes [11:54] <__chip__> zyga: rimshot [11:54] <__chip__> zyga: google 'rimshot pirates gif' for the right thing [11:54] <__chip__> mah interwebs are slow so i can only do meta-gifs [11:55] __chip__: how's Friday? [11:55] __chip__: I was super happy yesterday [11:55] <__chip__> zyga: what happened yesterday? (let's do it more!) [11:55] __chip__: I fixed the three tests leaking stuff on core167 [11:55] some lessons learned as well [11:56] __chip__: now mount-ns can run on core [11:56] _ages_ [11:56] <__chip__> is that core based on ubuntu 2167.04? [11:56] it took years to do this [11:56] hahaha [11:56] star trek [11:56] the next remodel [11:56] these are the changes of the packaging systems [11:56] to explore ... channels [11:56] yada yada [11:59] <__chip__> zyga: sudo apt install xscreensaver-gl-extra, and then /usr/lib/xscreensaver/starwars -program 'cat stufftxt' [12:08] * zyga iterates on another PR [12:22] mborzecki: #8045 failed on a travis test with a random store error so I restarted it [12:22] PR #8045: osutil: add helpers for creating symlinks and renaming in an atomic manner [12:22] cmatsuoka: thanks! === ricab is now known as ricab|lunch [12:51] hmm [12:51] is the store all right [12:51] my tests are timing out on super slow traffice [12:52] downloading core takes close to an hour [12:53] 10KB/s [12:53] 20-60 [12:57] brb [12:59] re [13:41] degville: did you find anything interesting in the content interface? [13:43] zyga: I'm still working on it - but I did get my own slot/plug snaps working with it, and I've mostly added the details you suggested/linked to. [13:43] cool! :) [13:44] I think the thing I've learned is that simple stuff tends to work [13:44] but is surprising when you start being very creative [13:45] ok, sounds sensible. I'll make a note of it. Certainly, my own snaps are simple, but they were just for my own understanding. [13:45] morning folks [13:45] - Download snap "ubuntu-image" (161) from channel "edge" (received an unexpected http response code (504) when trying to download https://canonical-lcy01.cdn.snapcraft.io/download-origin/canonical-lgw01/4RW78vIax8JW5S8HkYsa8lNbv68uPaYX_161.snap?token=1579885200_4dbf2a807d9b9a8d56cb263d6e7dcc6e61a45952) [13:45] ijohnson: hey [13:45] hey [13:45] ijohnson: store is wonky [13:45] does that mean I can just go back to bed [13:45] :-) [13:45] it is :/ [13:46] error: cannot perform the following tasks: [13:46] - Download snap "core" (8268) from channel "stable" (Get https://canonical-bos01.cdn.snapcraft.io/download-origin/canonical-lgw01/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_8268.snap?token=1579881600_d8a5fae185f0cf6884a96c15c566fc6876922512: EOF) [13:46] ijohnson: or play in the snow [13:46] yeah [13:46] mborzecki: so what's the story behind this AtomicRename ? [13:46] mborzecki: is my implementation flawed? I see your comment about the dir syncing [13:46] ijohnson: want to talk before the standup? [13:47] sure, like now? [13:47] or after/or during :P doesn't matter, it's only 5 of us today [13:47] btw, I'm next to lucy [13:47] she's sleeping [13:47] I may join standup and listen on very quiet level [13:47] either I swap with my wife or it's silent standup for me :) [13:47] ijohnson: zyga: let's do it [13:47] no no [13:47] I mean leater [13:47] I'm working on stuff now [13:48] go chat! [13:48] mborzecki: I'll join in just a minute === ricab|lunch is now known as ricab [14:35] zyga: why use `snap watch` instead of `retry-tool` ? [14:36] ijohnson: because snap watch does exactly what I need [14:36] and retry tool would have to do snap changes and grep [14:36] I'm still just a little worried that it will block forever [14:36] exactly what snap watch does inside [14:36] I wish we had a way to make retry-tool ask snapd something directly [14:37] retry-tool snap debug settled [14:37] or something [14:37] I see your point, it's just lesser evil right now (out of two choices this one is more robust in being precise) [14:37] not that the other would just grep Doing [14:37] which doesn't catch Undoing [14:38] and is fooled by "Done Doing stuff" [14:45] zyga: ok, it's fine for now then I guess [14:45] I'll approve in a little bit [14:45] I think "snap debug settle" would be good [14:45] I'll ask about it next week [14:45] also the shell code after reboot-causing activity [14:45] that scans logs [14:45] to know when it can REBOOT [14:45] yuck [14:45] we should have a better way [14:46] not calling REBOOT is not good because then spread was not anticipating it [14:46] it would be good to have ANTICIPATE_REBOOT [14:46] and then just do what normally happens [14:46] but there is no channel to send that information today [14:46] (sadly) [14:47] Afk for little bit [14:47] - Download snap "core20" (331) from channel "edge" (Get https://canonical-bos01.cdn.snapcraft.io/download-origin/canonical-lgw01/DLqre5XGLbDqg9jPtiAhRRjDuPVa5X1q_331.snap?token=1579888800_379b23e2fe4d0bd8d7152c92943200e95a92deb0: EOF) [14:47] I think today is early EOD land [14:48] I will do reviews [14:59] mborzecki, ijohnson: -1 on https://github.com/snapcore/snapd/pull/8045#pullrequestreview-348005564 [14:59] PR #8045: osutil: add helpers for creating symlinks and renaming in an atomic manner [15:02] ijohnson: i joined tgif the second you left [15:03] pstolowski: sorry be there in a minute [15:03] irl thing [15:03] no worries [15:05] ijohnson: what does zsys stand for? [15:07] ijohnson: https://github.com/snapcore/snapd/pull/8043#pullrequestreview-348015574 + but please check my comment [15:07] PR #8043: osutil: add Renameat2() [15:32] zyga: zsys is just what the go sources use for those definitions [15:32] i.e. https://github.com/golang/sys/blob/1a3b71a79e4aff00d87e69e0744746d7d67a3f8f/unix/zsysnum_linux_arm.go [15:33] zyga: what do you think about my implementation vs mborzecki ? [15:37] zyga: mborzecki: I think I will push to 8045 a change to do try creating the symlink in a loop and then refactor my grub PR to use mborzecki's implementation and drop the osutil.Renameat2. we can always refactor to use that later if we find os.Rename it's not safe enough [15:40] ijohnson: I think the looping should be in maciek's pr but otherwise I agree [15:40] ijohnson: ah [15:40] ok [15:40] (about zsys) [15:40] - Download snap "core18" (1668) from channel "edge" (received an unexpected http response code (404) when trying to download https://canonical-lcy01.cdn.snapcraft.io/download-origin/canonical-lgw01/CSO04Jhav2yK0uz97cr0ipQRyqg0qQL6_1668.snap?token=1579892400_a96a7203408107730307613e718e90209628f548) [15:40] store is so down [16:02] zyga: okay sounds good [16:03] zyga: wait 404? [16:03] zyga: is that reproducible? [16:03] ijohnson: pushing in a bit [16:03] it should NOT 404 (no found) on you - I might expect a 50x but not a 404 [16:03] mborzecki: ah okay if you're already doing that I'll wait for you [16:04] zyga: wfm fwiw [16:11] ijohnson: zyga: pushed [16:12] time to wrap it up [16:17] PR snapd#8043 closed: osutil: add Renameat2() [16:24] re [16:42] zyga: are your leaky mount ns PR fixes all ready? if so I can merge them this PM after I review them [16:42] trying to finish UC20 things this morning [16:47] hi, sorry to mention you. I already ask at #openstack and #ubuntu-kernel. [16:47] Should a bond which is member of a bridge has a mac-address ? And if so, should it be the same mac ? (netplan use automatically the same mac) [17:08] sdhd-sascha: I don't know the answer myself, but there's also the #netplan channel [17:14] ijohnson: thank you :-) meanwhile i found #lxc-dev, too. [17:14] sdhd-sascha: glad to help [17:42] ijohnson: no, not ready [17:42] ijohnson: close but not yet, I'm wrapping up another branch and will return to get through feedback and do tests [17:42] zyga: ack, np [17:42] ijohnson: but I'm still up here [17:43] so I'll try to merge them if possible by EOD [17:43] zyga: sure np [18:05] Is it possible to approve this classic-snap : https://build.snapcraft.io/user/sd-hd/termite-snap . (I already asked in the forum for a `classicmode` track ) [18:05] sdhd-sascha: does it have required votes? [18:06] zyga: just talk today, per email, with someone who has requested support from me. [18:06] What are required votes ? [18:07] sdhd-sascha: it must go through the review process [18:07] sdhd-sascha: and get enough votes [18:07] I published it as strict before, with ssh and byobu. But didn't had the time to test and use it in production. [18:08] zyga: Ok, understood. No problem. I told the reviewer, that he could build the snap himself. (He knows how to strace it ;-) ) [18:10] zyga: is this enough: ? https://forum.snapcraft.io/t/termite-classicmode-track/15169 [18:11] no [18:11] please look at the process requirements [18:11] it's documented on the forum [18:11] you must justify it [18:11] there are requirements and it's not always granted [18:11] it also takes some time so don't expect instant replies [18:16] zyga: hmm [18:31] zyga: https://forum.snapcraft.io/t/termite-classicmode-track/15169/2 [18:37] sdhd-sascha: I'm sorry it's too late for me to focus on that [18:37] I'm trying to wrap up some fixes and spend some time away from work [18:38] zyga: no problem - i just try to look at github actions... maybe i can create it there, for now [18:38] sdhd-sascha: someone recently tweeted about using github actions for snaps [18:39] but I'm off now, need to take the dog out [18:39] i know... christmas presend ;-) i remember ;-) [18:39] no problem ;-) [18:55] Hey folks - did you ever had a remote tree-tag on github. But no branch. How can i fetch this ? [18:55] /me i know, i wish too, i had had people in the past to ask ;-) [19:22] ijohnson: updated differential PR [19:23] switching to the fixes [19:23] zyga: ack thanks [20:03] * zyga checks on tests [20:22] markstos: welcome :-) [20:26] markstos: well, termite on sway launched on my system [20:27] Yes, it runs on strict ... But you didn't have access on outher applications. Except you use some remote login, to another system [20:28] markstos: the most folks, here are gone, and be on gmt+3 back ;-) [20:28] or, so [20:28] But, if i say something wrong, than they can criticism me [20:29] If i understand you, then, you would like to have some terminal which is strict protected with apparmor and seccomp. But its usable ? [20:30] Ah, so it's good for SSH, but not for accessing the filesystem. That's a lot less useful. [20:30] I'm not surprised Sway works, since Sway uses Wayland and Wayland seems to work as the default for Termite. [20:32] :-) [20:32] No, no -- the most guys said to me, that `classic mode` with full filesystem (not only home) makes more sense [20:32] My primary interest was simply to have a package that I could install on Ubuntu. [20:32] I read, in your mail signature, that you have a security company - or you are freelance ? [20:33] hmm - ... maybe it was the easiest, if i create a `ppa` for you ? and termite plus libvte-ng [20:33] then there is no snap needed ? [20:34] Firstly, i thought you would like to have a snap. [20:35] markstos: What OS are you use ? [20:36] I'm using Ubuntu 18.04, but plan to soon convert a new persona laptop laptop to Ubuntu 20.04 alpha. I'll run containers to support 18.04 apps until 20.04 is released. [20:36] Oh, you just said them. I'm sorry [20:36] * sdhd-sascha just remember [20:37] * sdhd-sascha also have to try to use slack.... (thanks for the tip) [20:38] Yes, a PPA would also be welcome. As much as possible I'm trying to automate my environment with Ansible, so whichever format it is, I'll automate installing and next time upgrades will just about as easy either way. [20:42] hey, markstos - my wife is come back. And yes, i can create a ppa the next days ;-) [20:42] But mention, that libvte-ng and libvte can be different version. (i try to do my best to resolve this ;-) ) [20:43] see us - tomorrow ;-) [20:43] Thanks! [20:44] markstos: of course, i'm glad to help (sometimes, i didn't have the time, for an one-person ... ;-) ) [22:10] PR snapcraft#2892 opened: Add snapcraft plugin for Qt Build Suite (qbs) [22:24] * zyga alters a test and spawns another run [22:42] ijohnson: https://github.com/snapcore/snapd/pull/8038 updated [22:42] PR #8038: tests: fix gadget-update-pc test leaking snaps [22:42] 4 is enough [22:44] * zyga notices LXD test is broken [22:44] eh [22:44] I guess that's not a good way to end the week [22:44] something to fight on Monday [22:44] ttyl [22:44] * zyga eods [22:45] Have a good weekend zyga! [22:45] thank you, you too [22:46] (lxd is not broken, installing snapd in lxd doesn't pass through dpkg, some deps are wrong, perhaps the test starts to suffer from the build-on-A-and-install-on-B problem) [22:46] * zyga goes to watch Picard pilot