=== ads20000_ is now known as ads20000 [06:15] morning [06:38] PR snapd#4901 opened: cmd/snap-confine: nvidia: add tls/libnvidia-tls.so* glob === chihchun_afk is now known as chihchun [07:08] good morning [07:09] zyga: hey [07:09] zyga: take a look at 4901 please [07:09] ohhh [07:09] I see now [07:09] man, is that it? [07:09] yeah [07:09] 'magic' [07:10] man :) [07:10] that's insane [07:10] zyga: left a note here https://forum.snapcraft.io/t/nvidia-acceleration-on-chrome-and-firefox/4532/16 , there's 2 copies of these libraries [07:10] but also brittle on our part, I wonder how can we make sure to pick the right set in general [07:10] I think that should be a snap [07:10] and that snapcraft should prevent people from shipping those [07:10] * zyga reads [07:10] we already pick up libnvidia-tls.so*, bu there's another one under tls/libnvidia-tls.so*, no clue how the second one gets loaded [07:11] are they identical? [07:11] yup [07:12] aah w8, [07:12] no, damn, looked wrong [07:12] they're different [07:12] need to correct the post [07:13] Ah [07:13] This makes more sense now [07:14] Ok [07:14] Still +1 [07:14] note to self, make sure to sort the inputs [07:14] Are they coming out of nvidia private build? [07:14] Or out of libglvnd [07:14] zyga: https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/nvidia-utils#n114 nvidia [07:15] zyga: it even works the arch glvnd libs now [07:16] Ok [07:17] Can you please do a small experiment [07:17] Take the nvidia build from their website [07:17] Unpack it [07:17] And correlate the files inside with our patterns [07:17] Are we missing anything [07:27] mborzecki: I'll review and perhaps land the makefile change today [07:28] but can you please look at this: https://github.com/snapcore/snapd/pull/4891 [07:28] PR #4891: tests: add support for phased prepare-restore logic [07:28] I have some nice follow-ups [07:28] and I wanted to get the foundation in place [07:31] zyga: hm i think we might be doing something silly with those globs === chihchun is now known as chihchun_afk [07:32] on the host I have both paths /usr/lib/libnvidia-tls.so.390.42 and /usr/lib/tls/libnvidia-tls.so.390.42, but in the mount ns only one appears [07:33] then readme in the drivers states this: The nvidia-tls libraries (/usr/lib/libnvidia-tls.so.390.42 and /usr/lib/tls/libnvidia-tls.so.390.42); these files provide thread local storage support for the NVIDIA OpenGL libraries (libGL, libnvidia-glcore, and libglx). Each nvidia-tls library provides support for a particular thread local storage model (such as ELF TLS), and the one appropriate for your system will [07:33] be loaded at run time. [07:35] mborzecki: look at the snap-confine code there, we probably don't look at tls/* at all [07:36] mborzecki: my idea is to convert the nvidia tarball into a snap [07:36] and mount it [07:36] ignore anything the host does [07:36] you'd need a tarball for each version of the drivers [07:37] we only need to support each family, not each version [07:37] there are about 3 at most [07:37] and yes, I agree [07:37] although I don't know if all families support libglvnd [07:38] what is the layout of the working setup at runtime, what is in /var/lib/snapd/lib/gl [07:38] (can you ls -lR there please?) [07:39] just a sec, got some changes in place [07:39] kk [07:39] great work btw! this is very promising [07:42] zyga: https://paste.ubuntu.com/p/jHSFhjbjV9/ [07:43] zyga: we have this: libnvidia-tls.so.390.42 -> /var/lib/snapd/hostfs/usr/lib/tls/libnvidia-tls.so.390.42, but the symlink should be to var/lib/snapd/hostfs/usr/lib/libnvidia-tls.so.390.42 and we should have directory tls with symlink to respecive libs inside [07:43] same for vdpau [07:44] hmm [07:44] this will require small changes there [07:44] but yeah [07:45] and can you pastebin the tree of files that is in the nvidia tarball? [07:45] zyga: the correct layout https://paste.ubuntu.com/p/NHHHpNCrVY/, seems to work fine too [07:47] hmm [07:47] a bit magic [07:47] since you said those libnvidia-tls.so files are not the same [07:48] zyga: files in the driver package pulled from nvidia's site: https://paste.ubuntu.com/p/zDFygPwSQ2/ [07:49] yeah the readme says: 'the one appropriate for your system will be loaded at run time.' [07:53] drwxr-xr-x 2 maciek maciek 4096 03-03 14:03 tls [07:53] they have a tls dir [07:53] I need to run [07:53] my son forgot his homework (yeah) [07:53] and just called me [07:53] AFK [07:54] and then I will work from a coffee shop because I planned to move anyway [07:54] mvo: good morning, I'll be back soon, please ping me if urgent [07:55] zyga: ok [08:01] PR snapd#4901 closed: cmd/snap-confine: nvidia: add tls/libnvidia-tls.so* glob [08:02] mvo: can you cherrypick it to 2.32? [08:02] mborzecki: already done [08:02] mborzecki: thank you [08:02] mvo: thanks [08:12] moin moin [08:16] morning [08:46] PR snapd#4902 opened: cmd/snap-confine: nvidia: preserve globbed file prefix [08:46] zyga: ^^ https://paste.ubuntu.com/p/NSWPHDrNBd/ [09:04] Almost back [09:11] re [09:11] sorry for being late [09:12] mvo: hey, I'm ready to assist in any way I can [09:12] mvo: I was thinking about the trespassing bug [09:12] mvo: and I think we can postpone that for 2.33 [09:12] mvo: since the impact is not serious and this is a beta feature [09:12] mvo: I will work on cleaning it up to have proper data path from main all the way down [09:13] mvo: and this will lessen the impact on the release [09:13] mvo: I would only like to merge the symlink fixes [09:13] mvo: and do more more more testing to see if we can reproduce the issue again [09:13] zyga: that sounds great. I'm working right now on the snap run fixes we talked about yesterday [09:13] mvo: I also have a tiny PR to add (I can add it to the symlink PR) to add something to debug: there [09:13] mvo: thank you sounds good [09:13] mvo: I moved downtown to meet with an old colleague [09:14] super smart guy, I wonder if he would consider joining our team :) [09:15] zyga: heh, ok [09:15] he's an old lisp hacker, not sure if he'd want to use go ;-) [09:17] zyga: ha! a sage [09:17] I think he needs to upgrade his beard a few times for that ;D [09:18] zyga: CE testing found that 2.32 would not work correct on caracalla because the security profile re-generation. I am curretnly trying to write up how this happens, do you remember in what way we modify the security profiles between 2.31->2.32 that would trigger this? [09:18] yes [09:18] zyga: please tell me then :) [09:18] mvo: we now require a new profile snap-update-ns.$SNAP_NAME on startup [09:19] in the past whenever we changed profiles and the changes were not huge we would just start with the old profile [09:19] and replace the profile in place as soon as snapd woke up [09:19] zyga: what requires this? snap-confine? [09:19] zyga: please take a look at 4902 when you can [09:19] now we cannot start apps because w ecannot complete the profile transition from snap-confine to snap-update-ns [09:19] mborzecki: ack [09:19] mvo: in 2.31 we had one profile for snap-update-ns [09:19] one that was very open because of layouts [09:19] zyga: that is excellent, this also means that without system-key this would not even possible? [09:20] i'll be looking at 4891 in a while and then will try nvidia & bionic [09:20] zyga: is it snap-confine that loads these extra profiles? [09:20] in 2.32 we have one profile per snap, that is tailored to construct the layout [09:20] mvo: no, it's not snap-confine [09:20] zyga: what is loading those? [09:20] mvo: snap confine doesn't even check if they exist, it just requests transition on the next exec [09:20] mvo: normally they are loaded by apparmor init script on early boot [09:20] mvo: those are stored just like snap application profiles, in /var/lib/snapd/apparmor/profiles [09:21] zyga: hm, I'm puzzled then, so part of the new profile (that require these extra things) are on disk already? [09:21] mvo: no [09:21] mvo: they will be created by snapd after core reboots to 2.32 [09:21] zyga: what am I missing :) [09:21] mvo: in 2.31 they were not a thing [09:21] but in 2.31 -> 2.32 transition they will not be there until after reboot and start of snapd, no? [09:21] mvo: so we hit a hard transition when a profile is missing [09:21] yes pedronis, exactly right [09:22] mvo: this will become less of an issue once we have snapd.snap [09:22] and snapd can restart itself without rebooting the box [09:22] zyga: what bit falls over? I mean, if the old profiles are there until we run snapd one would assume that network-manager would run happily with the old profiles? [09:22] then the profiles will show up in phase 2 [09:22] mvo: you miss one fact [09:22] mvo: it's a brand new profile [09:22] not one that existed in 2.31 at all [09:23] mvo: well after reboot there's a profile that requires a profile that is not there [09:23] not quite sure how the profile transition work [09:23] mvo: the name is snap-update-ns.network-manager [09:23] mvo: we never had that profile in 2.31 [09:23] mvo: note that this can happen even if we take snap-update-ns out of the picture [09:23] zyga,pedronis: why the mismatch, I mean, if there is a new profile after reboot then the other profile should also be written. and if its an old profile it does not reference the new profile. what am I missing :) ? [09:24] zyga: sorry for being a bit slow today [09:24] mvo: 2.31 doesn't write snap-update-ns.network-manager [09:24] mvo: 2.32 does and requires it to start network-manager process [09:24] zyga: but does 2.31 references this profile in any way? [09:24] mvo: no [09:25] mvo: do you want to HO to have me explain this in person? [09:25] zyga: yes [09:25] kk [09:25] one sec [09:25] sure, I'm sure there is some tiny details I'm missing, but I don't know yet what [09:25] * zyga hopes background coffee shop music won't be an issue [09:26] zyga: I'm in the stadnup HO [09:26] mvo: one moment, I don't have chrome here [09:26] normally I HO from my desktop [09:26] * mvo nods [09:26] 2 minutes to download [09:27] uh, I should visit that telco store and get a modem for my sim :( [09:28] zyga: we could also have a old-fashioned phone call if you want :) [09:29] installing chrome now [09:29] ok [09:29] if this fails, yeah [09:29] I'll call you [09:37] pedronis: can you read Laney's comments on https://bugs.launchpad.net/snapd/+bug/1723094 and reply on the bug? [09:38] Bug #1723094: Live images should be able to turn off Snap updates [09:44] mwhudson: I tried to answer, does it make sense? [09:45] pedronis: it makes sense to me at least :) [09:46] thanks! [09:54] mborzecki: done [10:05] * Chipaca afk for a bit [10:06] mvo: can we merge https://github.com/snapcore/snapd/pull/4899 [10:06] PR #4899: many: backported fixes for layouts and symlinks (2.32) [10:06] when i'm running bionic from usb stick, are the changes preserver accross reboots? [10:06] mborzecki: dunno [10:06] it used to be (some) [10:06] but not sure [10:10] * Chipaca really afk now [10:10] o/ [10:10] mborzecki: it depends on whether, when you created the thing, you told it to leave some rw space [10:11] Chipaca: I dd'ed iso to usb stick [10:12] hmm, i can't see where on usb-creator-gtk there was that option, but i'm sure it was there before [10:13] I think it may have been removed now [10:13] Chipaca: dd is the only usb-creator i ever use :) [10:13] because that code evolved a lot [10:13] zyga: let me look at it. in the meantime: 4882 is updated [10:13] mborzecki: install ubuntu on some spare hdd [10:13] mvo: thank you, looking [10:14] usb-creator (0.3.0) xenial; urgency=medium [10:14] [ Marc Deslauriers ] [10:14] * Rework the whole imaging process for writing to devices: [10:14] - Use an equivalent of dd to make an exact copy of the image to the device [10:14] - This also breaks persistence. [10:14] that last line there? boom. no more persistence. [10:14] damn [10:14] godd! [10:14] just saying :) [10:14] mvo: does that do persistence [10:14] :-) [10:15] its persistently nice, if that is what you mean ;) [10:15] mborzecki: get your hands on usb-creator-gtk 0.2.68 :-) [10:15] mvo: dunno, seems unmaintained, i proposed a pr ages ago and got no feedback [10:15] * Chipaca runs [10:15] Chipaca: i'll try installing to a usb stick (while running the installer from another usb stick) [10:16] mborzecki: usb-creator-gtk from trusty should do it [10:16] mborzecki: https://packages.ubuntu.com/trusty-updates/amd64/usb-creator-gtk/download [10:18] Chipaca, mborzecki: I would not trust that :) [10:18] zyga: 'course you can trust it, it's got 'trusty' in the name! [10:18] * Chipaca really needs to afk for a bit now [10:20] hahaah [10:20] :-) [10:20] * zyga bakes a few more patches in google [10:20] while on the go, google is a nice place to test [10:20] we should update the reference tag to get smaller deltas [10:21] zyga: hm, one test missing, sorry for this, will push a new 4882 in some minutes [10:21] ok [10:22] mvo: do we know if for some reasons network-manager needs to start before snapd on those machines? (I suppose not, we write the all the relevant units and there's no such dep) [10:23] pedronis: I think that on those machines n-m manages all networking [10:23] pedronis: and it just starts as early as it can [10:23] it's not strictly "before snapd" but not explicitly linked to through dependencies [10:23] pedronis: and if we go and change the units now we'd have issues in the field as there's no easy way to change any systemd units [10:23] that's fine as long as snapd doesn't wait for that, and the timeout on starting that service is within the time it takes for snapd to do its job [10:23] (we don't have equivalent of ensure there) [10:24] in terms of profile writing [10:24] hm the installer does something funny, there's 'Erase disk and install Ubuntu' option, but I have 5 disks plugged in at the moment, so which one will it erase? [10:24] pedronis: that's a good point [10:24] mborzecki: choose manual partitioning [10:24] mborzecki: 18.04 or 16.04? [10:24] I'm used to 16.04 installer but I think that 18 changed a lot [10:25] 18.04 [10:25] mborzecki: one more "hint" detach other disks :D [10:26] it's surprisingly low tech robust solution for this problem [10:26] also helps if you install windows or other OS with silly installer that just overwrites everything agressively [10:26] i'm clicking, but franly i'm minutes away from debootstrapping the thing [10:29] mborzecki: you can do it, just a bit more patience :) [10:41] slightly worrying, wonder wher grub efi will get installed, i've made a partition for efi and set the bootloader to be installed to that disk [10:44] aaannd it installed grub to the wrong drive [10:55] inistaller finished, now i can start fixing the system :/ [10:57] mborzecki: is it booting? [10:57] what's up? [10:57] mborzecki: sorry :/ [10:59] zyga: the installer hijacked the ESP on the main nvme drive, heh so the system i got now is ubuntu/efi on nvme and rootfs on usb stick [10:59] got to install ubuntu's grub in efi partition of usb stick and make arch boot again [11:00] mborzecki: are you having fun yet [11:04] PR snapd#4899 closed: many: backported fixes for layouts and symlinks (2.32) [11:04] woot [11:04] thanks mvo [11:04] mvo: I forgot to push that debug branch so I'll add a debug PR to 2.32 as well [11:05] mvo: in 15 minutes, busy with some topics [11:05] zyga: 4882 is ready to review. this and your debug pr are the last bits, right? [11:05] yes [11:05] well [11:05] 4882 not sure [11:05] I forgot which PR that was [11:05] zyga: snap run --system-key [11:05] ay [11:06] +1 [11:06] yes [11:06] I left the debug bits at home so I'll just make a new PR [11:09] zyga: fwiw bionic boots now [11:10] and from the right drive [11:10] mborzecki: cool :) [11:12] did you have to update bootloader layout? [11:15] zyga: yes, mount proper partiion under /boot/efi, edit fstab to make it work after reboot, install grub x86_64-efi [11:16] i'll deal with arch later, i have a slightly convoluted luks+lvm there [11:29] PR snapd#4903 opened: tests: split prepare-restore.sh into prepare-restore.d modules [11:33] hey mborzecki-ubuntu :) [11:33] Chipaca, shouldn't whatever needs /var/cache/snapd/commands.db create it if it is none-existent ? (see https://forum.snapcraft.io/t/var-cache-snapd-commands-db-permission-denied/4590 ) [11:33] zyga: hey [11:33] ogra_: if it doesn't exist there's nothing to do, so it should just move on [11:34] well, it doesn't [11:34] ogra_: IKR [11:38] ogra_: I think at least two things need fixing: one, which i think mvo is already on, is to make _command_not_found hide stderr from snap advise-snap (or maybe make snap advise-snap not error on no db) [11:38] ogra_: the other is to make the classic snap grab the commands.db from host :-) [11:39] yeah, obviously [11:39] i only noticed after i wrote the first comment that it seems to be created on first boot [11:39] ogra_: if you've got classic installed somewhere, could you see if you could just cp it from hostfs? [11:39] so we need a cp in the classic setup script [11:39] err, yes, see my last post there [11:40] ogra_: no, from _inside_ [11:40] ogra_: maybe it's fixable inside :-) [11:40] ah, nop, that wont work [11:40] aw [11:40] we're in a chroot [11:40] ogra_: i thought the classic snap had crazy privs [11:40] and dont do fancy hostfs bind mounting or anything [11:40] its a simple chroot :) [11:41] but it is no prob to simply add it to the chroot setup script [11:41] ogra_: given core can be size constrained, maybe ln || cp (or is there a better way to do that) [11:41] * Chipaca knows nothing [11:42] mvo: 2.32 debug PR https://github.com/snapcore/snapd/pull/4904 [11:42] PR #4904: tests: change debug for layout test (2.32) [11:42] PR snapd#4904 opened: tests: change debug for layout test (2.32) [11:43] zyga: ta [11:44] Chipaca, linking wont work (and bind mounting adds complexity i'd like to avoid) ... cp will do [11:44] well, nvidia doesn't work in snaps on bionic [11:44] i'll try my branch [11:46] OMG!!!! git checkout - [11:46] is tehre a command i can use to install all build deps listed in debian/control? [11:46] my favourite command [11:46] zyga: yeah, it's like cd - ;) [11:46] mborzecki-ubuntu: sudo apt build-dep ./ [11:46] yes [11:46] I just complained I wish there was something like that [11:46] mborzecki-ubuntu: if you are inside the unpacked deb [11:46] and suddenly my colleague shows me this [11:47] mvo: ta [11:49] oops, pushed to wrong PR [11:55] PRs corrected [11:55] mborzecki-ubuntu: https://github.com/snapcore/snapd/pull/4903 [11:55] PR #4903: tests: [WIP] split prepare-restore.sh into prepare-restore.d modules [11:55] no need to review yet before the other one lands [11:55] but this is what I was thinking about [11:56] there's lots of room to improve from this state [11:56] but it's already much more approachable than the big one we have now [11:57] is there a spread variable that holds the name of the current test? [11:58] zyga: meh, the system-key stuff is actually more work because snap run needs to read the build-id of snapd not of snap [11:58] oohh [11:58] yeah [11:58] and great catch :/ [11:58] mvo: and now it needs to know which snapd to read [11:59] mvo: /o\ [11:59] zyga: make the whole thing more complicated. exactly, re-exec and all that :( [11:59] mvo: maybe [11:59] mvo: maybe we drop the tailored profiles [11:59] mvo: use the old profile for 2.32 [11:59] zyga: that sounds wise [11:59] mvo: and at least 2.32 ships with the new profile on disk [11:59] mvo: and 2.33 won't have massive issues [11:59] zyga: how much work is it? [11:59] not a solution much but yeah [11:59] mvo: drop a few lines of C from snap-confine [11:59] zyga: \o/ [12:00] zyga: lets do that [12:00] mvo: restore child profile for snap-confine.apparmor.in [12:00] mvo: and it _should_ work [12:00] zyga: much safer and we look at this problem again with more time and less presure [12:00] mvo: we'll still make the snap-update-ns.$SNAP_NAME profiles and load them but they will be unused [12:00] yes [12:00] mvo: I'll prepare a PR [12:00] mvo: and you can get fewer gray hair :) [12:00] zyga: heh, indeed [12:00] zyga: I have way too many of those already [12:01] mvo: ok, let me do this quickly [12:01] mvo, https://github.com/snapcore/classic-snap/pull/16 [12:01] PR classic-snap#16: fix breakage of command-not-found [12:03] * Chipaca switches tasks, from improving test coverage to improving lunch coverage [12:03] hahaha [12:03] * zyga hugs Chipaca [12:03] zyga: how're your breaks coming? [12:04] * zyga looks the other way to avoid eye contact [12:04] I'm not at home [12:04] that's an improvement [12:04] ogra_: thank you [12:04] but relaease rush [12:04] so ... you know [12:04] *release even [12:04] yeah [12:14] mvo: ok, running locally now [12:15] mvo: https://github.com/snapcore/snapd/pull/4905 [12:15] PR #4905: cmd/snap-confine: don't use per-snap s-u-n profile [12:15] PR snapd#4905 opened: cmd/snap-confine: don't use per-snap s-u-n profile [12:15] not tested yet [12:17] zyga: failed to create prefix path: /tmp/snap.rootfs_jW5tpa/var/lib/snapd/lib/vulkan/icd.d: Permission denied [12:19] a bit of fun: "how to make package managers cry" https://www.youtube.com/watch?v=kguJ1ihOyV8 [12:25] mborzecki-ubuntu: dmesg | grep DENIED [12:25] I suspect we need some apparmor love there [12:26]