=== Unit193 is now known as HeadlessHorseman === RAOF is now known as Guest50892 === Guest50892 is now known as RAOF [08:05] tumbleweed: Around? [08:12] what's the procedure for building custom ubuntu core images that can be installed on servers, without having to go through snapcraft? I'm curious if ubuntu core is an alternative to other "container OS", but the documentation is lacking. [08:12] (or rather geared towards IoT deployments)\ [08:41] kklimonda: i use debootstrap [08:47] LtWorf: I don't think you can build Ubuntu Core images using debootstrap, given how different they are from a normal ubuntu distribution [08:48] i think it's the same, just lacking systemd [08:48] and very minimal [08:49] are we both talking about https://www.ubuntu.com/core ? [08:49] kklimonda: ubuntu-image is the thing that makes ubuntu core images [08:50] kklimonda: what would you want to customize in your image? [08:55] mwhudson: different kernel, specific packages, container runtimes, at this point nothing and everything - I'm just looking into various available options, thinking about future upgrades of the servers we have. [08:56] at this point I'm trying to build a model of how all the ubuntu core parts fit together [08:56] kklimonda: well for the most part, adding packages is just a matter of installing more snaps, having them baked into the image is mostly an optimization [08:57] the kernel would be different though [09:01] tsimonq2: is LP: #1660108 still an issue with GCC 8? [09:01] Launchpad bug 1660108 in gcc-6 (Debian) "Since 6.3.0-3ubuntu1 some acc autotests fail with "atomic_base.h:390:7: error: inlining failed in call to always_inline"" [Undecided,New] https://launchpad.net/bugs/1660108 === RAOF is now known as Guest36026 [09:18] mwhudson: different, as in unsupported and frowned upon, or just poorly documented? [09:18] kklimonda: just different [09:18] mwhudson: is core even meant to compete with project atomic, rancheros etc.? [09:18] kklimonda: i certainly know nothing about how the kernel snap is maintaineed [09:18] mhm [11:13] seb128, https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/locutusofborg-ppa/+packages [11:14] I uploaded the new poppler if you want to grab it :) [11:14] I syncd it with the latest debian fixes, so now the delta is about two patches and one line in rules file [11:14] LocutusOfBorg, thx, you need sponsoring or...? [11:21] seb128, I need somebody who does upload it :) if you want to do the transition, please go ahead :) [11:22] I think we need some symbols refresh [11:25] LocutusOfBorg, I can have a look, is D open to upload yet? [11:28] no [11:36] doko, your "no" made his connection drop! [11:54] seb128, no, it is not open yet, but we should try to push when it opens :) [11:54] LocutusOfBorg, right, I can do that, thx for the work! [11:54] we can manually sync rdeps from debian, since they have lots of 0.68 fixes pushed already [11:55] and fix new failures together [11:55] I asked pochu to refresh symbols files and push the new version in experimental too [11:56] thx === M_hc is now known as _hc [15:16] @apw, @cking, we've noticed a roughly 30% increased cpu usage of our web apps when moving from 4.17->4.18 *(mainline kernels), have you guys had any similar reports? [15:16] Error: "apw," is not a valid command. [15:16] apw, cking, we've noticed a roughly 30% increased cpu usage of our web apps when moving from 4.17->4.18 *(mainline kernels), have you guys had any similar reports? [15:18] chiluk, i have not indeed sforshee ^ ? [15:18] basically we discovered it when one particular cgroup cpu bound app started being throttled when we moved to 4.18... [15:18] chiluk: first I've heard of anything like that [15:19] yeah a performance regression was expected 4.14->4.15 due to kpti/meltdown/spectre fun... but we're definitely hitting something 4.17->4.18... anyhow.. ping me if you guys hear anything.. [15:22] chiluk: the only thing that comes to mind recently that might have caused anything like that is l1tf mitigations [15:22] yeah I saw that.. I'll check llc hit % later today. [15:22] <- somehow became a performance guy in the last few weeks. [15:23] sforshee: wasn't l1tf pushed into 4.19? did that get backported onto the stable trees? [15:24] chiluk: I'd guess it got backported assuming the stable was maintained at that point. I don't remember exactly when all of that landed [15:24] yeah I'll go look it up. [15:24] thanks for the idea sforshee [15:26] chiluk: https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html [15:35] chiluk: L1TF first landed in a mainline release in 4.19 but they landed in -stable trees, as well [15:35] interesting.. afaict whatever I'm hitting hit us during the 4.18 development cycle so it's unlikely to be l1tf. [15:36] chiluk: additionally, you should only see a perf hit from L1TF mitigations if these web apps are running inside of VMs [15:36] chiluk: the bare metal mitigation for L1TF was very simple and should have a negligible perf hit [15:37] yeah we're baremetal with cgroup containers in mesos... so yeah.. I knew there was a reason I ignored l1tf... [15:39] well if you guys hear anything with the recent release of cosmic let me know, and I'll be sure to return the favor. [16:10] xnox: hey, small question, do you know where is the ubiquity translation installed on disk for all languages? (I'm particularly interested in the minimal translation we ship for languages we don't fully ship on the iso; I guess it's only ubiquity ones?) [16:11] didrocks, ubiquity assembles it's own strings, a few stock strings, and d-i strings. and yes ships it itself. [16:12] xnox: any particular files they are in? Like looking for "mhr" on disk only shows the slideshow files + some generic example-content and such [16:13] (but ok, the main info I was interested in was "everything related/we ship for those are part of ubiquity, and nothing else") [16:13] hmmmmm [16:13] Most of it should be in /var/lib/dpkg/info/ubiquity.templates IIRC [16:14] Or somewhere similar (and then loaded into the debconf db of course) [16:14] didrocks, yes, that! [16:14] oh right, they are coming from debconf… [16:14] * xnox just found it =) [16:14] thanks cjwatson & xnox :) [16:14] * xnox is slow =) [16:14] it's 15M templates file [16:15] unsurprisingly :p [16:34] xnox: in subiquity, how do you do anything special with the multi-layer/union fs system? Like creating the multi-layers, rebuilding the dpkg database in the end (depending on the amount of layers you touched) and anything related? Also, any doc about those? :) [16:36] didrocks, there is nothing special about that. [16:36] hum, you have certainly a way to build upon the multi-layer system that you told was available at FOSDEM? [16:36] (when we talked about the minimal installation) [16:37] didrocks, it's just perfectly stacked. as in bootstrap minimal squashfs, copy, mount overlay install more, unmount & copy aside, mount again (multi-lower), install more, unmount etc. [16:37] didrocks, let me point you at the code [16:37] thus each layer has it's own dpkg db, cause it got update. [16:37] thus each layer has it's own dpkg db, cause it got updated. [16:37] ah [16:37] hum [16:38] but we are going to have a lot of layers [16:38] like for langpacks [16:38] https://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/trunk/view/head:/live-build/ubuntu-server/hooks/030-root-squashfs.binary [16:38] so, it means a lot of combinations [16:38] https://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/trunk/view/head:/live-build/ubuntu-server/hooks/031-maas-squashfs.binary [16:38] https://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/trunk/view/head:/live-build/ubuntu-server/hooks/032-installer-squashfs.binary [16:38] didrocks, i did say, it's not gonna work to out of order slot in language packs.... [16:39] didrocks, so when discussed with e.g. seb128 we did say, one would have to pre-install the 6 language packs into the installer layer (live-boot only), and install the only required language pack, in target, from the .deb from the pool. [16:39] it does mean, that effectively all lang packs are shipped twice. [16:39] not really nice [16:39] unless one can magically reconstruct a deb out of livefs [16:40] can we somehow keep the pacakge names for a given layer and register them in dpkg without unpacking/running things? [16:40] didrocks, the layers that this works for is "minimal", "full", "installer". [16:40] should be quite safe for langpaks [16:41] or just generate the now (12) top combinations [16:41] didrocks, locales regenerated? [16:41] you do call [16:41] langpacks are going to have to be the top layer, so it can be peeled off before installation, and we'll have to just reinstall the deb. [16:41] yeah [16:41] /usr/share/locales/install-language-pack "en" "" "$2" || true [16:41] Anything else is a bit too magic. [16:41] postinst [16:42] that's exactly where I'm going to head to [16:42] wasn't in my previous diagram, but as a top layer, for the languages we fully ship, doesn't sound that crazy [16:42] infinity, for old ddebs we did have magic to reconstruct a deb, out of the filesystem, no? [16:42] (12 because 6*2: minimal + langpacks) [16:42] full* [16:42] no, we will not do that [16:43] xnox: Reconstructing debs is a big ick. [16:43] ok [16:43] It's a bit of a shame to ship them twice, but it's not really a big deal. [16:43] minimal, full, installer (with ubiquity and langpacks in it); and then pool needs to have: ubiquity, langpacks again. [16:43] (because oem-config, and langpacks) [16:44] We can maybe come up with some vaguely "elegant" way to stitch together dpkg status snippets, but I wouldn't want to consider that for a first cut. [16:44] also pool logic needs to change, not to exclude things that are already shipped.... and somehow maybe based that off the full squashfs, rather than the minimal one [16:44] it would be nice if langpacks were innert; without maintainer scripts. [16:44] or minimal, full, installer (with only trads for installer) and lang-minimal-desktop-en, lang-minimal-desktop-fr, … lang-full-desktop-en… [16:44] and like have triggers to trigger things at most. [16:44] then -{lang} are for the 6 we fully ship [16:45] and lang-full-desktop-en stack on top of lang-minimal-desktop-en, [16:45] as infinity said, at the top of the whole layer stack [16:45] xnox: Changing langpacks to declarative with triggers would be easy, especially since we control all of them. [16:45] xnox: But you still have the status DB to contend with. [16:45] infinity, that's the rpm command to rebuilddb?! =) [16:46] * xnox giggles [16:46] didrocks: Err, why are you doing en and fr stacks? [16:46] didrocks: That implies a user making a choice *at boot* before you stack the FSes. [16:46] infinity: ah, for the live session… [16:46] didrocks: Surely, you just want all the langpacks running in the live env. [16:47] infinity, cause didrocks wants to ship each langpack 3 times, clearly ;-) [16:47] well, I don't think we want to install them at boot time from the pool either… [16:47] xnox: no, they would only ship once that way [16:47] didrocks: No, you want installer+languages in the top stack. [16:47] didrocks, how would live session have them all? [16:47] didrocks: And then langpack debs in the pool to install the user's selected language in the target. [16:47] didrocks, or you want to stack live session, with a broken dpkg database? [16:48] didrocks, ubiquity can install the langpacks during installation; the same way it currently removes redundand ones. [16:48] didrocks, and i think installing the one you want is quicker than removing 5 you don't. [16:48] didrocks: I get what you're trying to do, but it's not going to work. [16:48] yeah, but one of the goal would have been quicker installation time :p [16:48] didrocks, installing one lang pack is quick. removing 5 is not. [16:48] but hem, yeah, live session… [16:48] xnox: Weeeeell. [16:48] xnox: not only the langpack, there are a lot of packages [16:48] xnox: Installing langpacks calls localegen, that's not quick. [16:49] can we preship that too? or not? [16:49] and some of the issues we currently have in ubiquity is that it's not in sync with locale-chooser on what to installer for instance :p [16:49] That said, we could change the langpacks to ship pre-compiled locales. [16:49] if clean-install copy this, otherwise regen. [16:49] Not sure why we don't actually. [16:49] didrocks, when you say "not only the langpacks" what else do you mean? [16:50] didrocks, current minimal install is borked and we know that. but we will have minimal squashfs to copy from and desktop (full) squashfs to copy from. [16:50] xnox: see the live seed, dictionaries, libreoffice-something-{lang}, thunderbird-* [16:50] then the "extra" packages to install is the langpacks + bootloader. [16:50] And some languages pull in input methods, etc. [16:50] yep [16:50] although imho we should preinstall bootloaders packages in minimal too, and and just pre-install both grub-pc & grub-efi [16:50] Anyhow, there's no "clean" solution to that right now. [16:51] yeah, sounds like it [16:51] The dpkg database being a single flat file makes it tricksy. [16:51] Yes, we could write something to deal with that. [16:51] there is no db.d/ true [16:51] But I don't want that to be a blocker or even a near-future goal for stacked installers. [16:52] didrocks, i think having minimal + desktop + installer squashfs is a win, despite the langpacks mess. [16:52] ... [16:52] infinity, didrocks - why do we not preinstall 6 languages by default?! [16:52] xnox: yeah, I wanted to go the extra mile and have something really nice, but doesn't sound like it's doable in the short term [16:52] because live session? [16:52] i mean for target [16:52] hum, it's quite large [16:53] we are already taking a non negligeable amount of spaces for other things [16:53] xnox: langpacks aren't small. It's kinda why we have them. [16:53] otherwise, we would ship them in the initial debian binary package :p [16:53] and yeah, especially considering input methods, dictionaries… [16:53] en is 6M [16:53] and we need en always, no? [16:54] sorry 7M [16:54] counting all extra packages? [16:54] no, talking to Gunnahr, we don't [16:54] and one of the goal is to stop installing it for everyone [16:54] maybe not, this is pack-en-base gnome-en-base [16:54] and dictionaries… [16:54] look at my seed reorg, it should make clear what we install for each language [16:54] also, there are some dep chains IIRC [16:54] didrocks, just make them snaps [16:55] like if we install libreoffice-{lang} unconditionnally [16:55] this pulls back libreoffice [16:55] contradicting minimal :p [16:56] xnox: sure, note that I said "we are already taking a non negligeable amount of spaces for *other* things" ;) [16:56] didrocks, infinity - i think there are two options. Preinstall langpacks in minimal squashfs; stack desktop; live -> remove langpacks (existing codepaths really) [16:57] or have clean minimal, clean desktop, and preinstall langpacks in live; and install langpacks from /pool to /target. [16:57] I guess installing what's needed is what makes the most sense, as we are going to rerun locale-gen anyway in both cases? [16:58] and preinstall in minimal squashfs, is actually the path of least resistance / least change, imho. [16:58] why not in the live one? sounds like we want to keep it on top if we install what needed afterwards [16:59] remember that we can't install everything langpack-related in minimal, due to deps on libreoffice, thunderbird… [16:59] or even, keeping a "default language" stack, maybe on top of everything? dunno yet [17:00] anyway, at least, I guess my dream is broken :p but thanks for the info and code pointer xnox & infinity [17:01] didrocks, preinstall language-pack-base-[6 langs] in minimal; preinstall thunderbird-locale-[6 langs] in desktop; live has no new langpacks. [17:01] xnox: why it it better than its own layer on top? [17:01] didrocks, then ubiquity removes un-needed langpacks as it did before. [17:01] ah, for removing [17:01] didrocks, because you don't need to ship a second copy in the pool, this way. [17:01] hum, unsure if it's better than installing [17:01] true [17:02] but 5 locale-gen calls potentially instead of one [17:02] and more packages to remove [17:02] didrocks, and minimal, is slightly faster than now, as it only removes langpacks; and doesn't need to remove e.g. thunderbird packs. [17:02] yep [17:02] it should be just one locale-gen call. [17:02] cause it should be trigger based, on remove; just like kernels.... [17:02] is it really a trigger, not posinst? I remember to have seen it running multiple times on upgrade [17:03] ok, need to think about it anyway, but good food for thoughts :) [17:03] if it runs multiple times, we can fix that, we have triggers tech to make it a single call per transaction. [17:03] minor bug [17:03] I wonder how we could keep clear definition of what's needed langpack-wise [17:03] and share that with check-local-support [17:03] (as now, the definition of a complete locale is going to be split in 2 places) [17:04] didrocks, also need to check if delta-squashfs can be actually generated for all cases; and how big they are. [17:05] didrocks, by doing reverse-stacking; e.g. make the big-fat all in one squashfs; and generate delta squashfs with things removed. [17:05] as in negate 5 langauges; times 6; [17:05] times 6, not times 2 ? [17:05] (minimal and full) [17:06] and again negate desktop down to minimal and negate 5 languages; times 6 [17:06] but it's a good idea :) [17:06] no idea if it works [17:06] sarnold: (responding to your message from last week) You say you're skeptical of userspace nfs ... I'd love to have it so I can mount to my NAS on demand through Deja Dup :) [17:06] cause overlayfs squashfs for removals, i wasn't sure it was working right. [17:06] sarnold: or through nautilus [17:06] xnox: sounds like an interesting things to try out [17:07] doesn't impact us on doing the langpacks in minimal + full stack as decided [17:07] rfleming: and 'mount -t nfs -osoft,rw nas:/exports/home/fleming /home/rfleming/nas' doesn't do the trick? [17:07] and replace the removal part with this if we see it works [17:08] sarnold: it does... but I have to do it. duplicity/deja-dup does backups whenever it feels like it. I'm also using it for my laptop :) [17:09] sarnold: maybe I'm just salty because Windows CIFS is better implemented in nautilus than NFS is :) [17:09] didrocks, "as decided" *eyebrow* [17:09] didrocks, deciding things without a feasible implementation?! interesting [17:10] xnox: ? it's exactly what you wrote: "preinstall language-pack-base-[6 langs] in minimal; preinstall thunderbird-locale-[6 langs] in desktop; live has no new langpacks" [17:10] this is shipping "the langpacks in minimal and in full" stacks [17:11] didrocks, so far, this is all madness =) [17:11] why would that be different to what is currently in server? [17:11] (with subiquity) [17:11] didrocks, server has no langpacks [17:11] you are that afraid by the size of each stack? [17:11] no [17:12] it's just genuinely not useful [17:12] we do locale, and keyboard setup. [17:12] that's it. [17:12] yeah, but this is not "without a feasible implementation" then [17:13] (I'm only talking about that part here, then a second step is to look at your negative langpacks squashfs delta idea) [17:13] didrocks, langpacks have never been stacked to-date. the only stacking we did was strict supersets, in order, not this tree of things. [17:13] it's going still to be strict supersets? [17:13] live [17:13] full desktpo [17:13] minimal desktop [17:13] strict supersets should work, yes. [17:13] with full & minimal having langpacks for the 6 shipped langs [17:14] didrocks, also, let's stop calling it live; but call it `installer` [17:14] yeah, that's the part I set to "decided" :p [17:14] cause that matches e.g. subiquity iso. [17:14] agreed [17:15] didrocks, and installing langpacks from /pool is imho "cleaner" [17:15] but time/disk-space [17:15] but time/image-space [17:15] yeah [17:16] let's see once this "default stacks" are here if we can look at the negative delta and also their impact on disk size, it's just an install time optimization in the end as it's the "same" than removing debs, but pre-generated [20:15] hey hey, anyone know why adding a library path (/usr/lib/aarch64-linux-gnu/mali-egl/) to /etc/ld.so.conf.d/01_mali.conf doesn't work? [20:16] when I have lrwxrwxrwx 1 root root 13 Jun 30 2015 /usr/lib/aarch64-linux-gnu/mali-egl/libEGL.so.1 -> libEGL.so.1.4 [20:16] ( all the paths are valid fwiw ) [20:17] namei -l on that path look fine? [20:17] how are you determining that it didn't work? [20:35] shadeslayer: did you run ldconfig after making that change? [20:40] ahasenack: yes [20:40] sarnold: I determined it didn't work because ldconfig -p still points libEGL.so.1 to /usr/lib/aarch64-linux-gnu/libEGL.so.1 [20:46] sarnold: https://paste.ubuntu.com/p/JtsSnc78V9/ [20:49] shadeslayer: can you try ldd on a binary that is linked with that lib? [20:49] and do you have only one libEGL in the output of ldconfig -p? [21:03] ahasenack: yeah, ldd says libGLESv2.so.2 => /usr/lib/aarch64-linux-gnu/libGLESv2.so.2 (0x0000007fb6f8b000) [21:07] ahasenack: sarnold so if I export LD_LIBRARY_PATH to the extra path, then it works [21:08] shadeslayer: does your /etc/ld.so.conf have an include directive for /etc/ld.so.conf.d/*.conf ? [21:08] ahasenack: yes [21:08] and I've added the 01_mali.conf in /etc/ld.so.conf.d/ [21:08] can you try adding the path directly to /etc/ld.so.conf instead of the file inside the .d directory? [21:09] shadeslayer: /usr/lib/aarch64-linux-gnu/libEGL.so.1 is from libglvnd and I doubt you can override that other than by using a diversion [21:11] tjaalton: but I also have the exact same issue with GLESv2.so [21:11] same thing [21:11] ah damn [21:11] tjaalton: why is that the case? [21:11] tjaalton: I essentially worked around it by setting LD_LIBRARY_PATH and setting my mali-egl dir in front of everything [21:11] the ld.so.conf trick only worked when both were in a subdir [21:11] I see [21:12] so it works? [21:12] tjaalton: with the env variable [21:12] ok [21:12] nvidia-340 migrated to diversions because of this [21:12] too bad only newer nvidia blobs support glvnd [21:13] not amdgpu-pro, not any of the arm stuff [21:20] tjaalton: I'm still confused as to *why* this is a issue :) [21:24] shadeslayer: ldconfig, it used to work when mesa & blobs put their libs in a separate directory, and then that directory was added to a ld.so.conf.d snippet via alternatives [21:24] depending on which one you chose, the correct set of libs was loaded [21:25] and now it prefers everything in /usr/lib/arch-tripet over everything in a subdir? [21:26] yes [21:26] I see! [21:26] tjaalton: thanks for the info :) [21:27] yw [21:28] I find it incredibly frustrating though [21:28] oh well [21:31] which package installs stuff in /usr/lib/../mali-egl? [21:31] tjaalton: I'm making my own [21:31] tjaalton: it's essentially this https://github.com/netrunner-pine64/pine64-mali-x11 [21:31] but for a new board on Ubuntu 18.04 [21:32] so make it use dpkg diversions [21:32] ack [22:24] shadeslayer,tjaalton, aha :) thanks for the information