[08:40] <SvenKieske> good morning from europe. So I'm really not sure if I'm supposed to ask end user questions in here or if this is a dev only channel (the wiki seems to say it's okay to ask here?). But I want to restate my question from yesterday: why do we not get the latest security update for the focal hwe edge kernel via metapackage "linux-generic-hwe-20.04-edge"?
[08:41] <SvenKieske> is this metapackage in the process of being updated to the next LTS Kernel (22.04) and thus security updates are stopped for the moment? :/
[08:59] <JanC> who says it doesn't get security updates?
[09:12] <SvenKieske> to be specific, it is still missing this update, and the "edge" kernel was also not mentioned in the associated USN Security Announcement: https://ubuntu.com/security/CVE-2022-0847
[09:15] <SvenKieske> and going back through our announcement archive, the "hwe-edge" kernel almost never get's mentioned in these announcements. I just want to know if we maybe misuse this kernel and want to double check that we get security updates asap, or if we should maybe use "hwe" kernel without edge?
[09:16] <SvenKieske> this is for datacenter/ private/public cloud workloads
[09:16] <SvenKieske> JanC: any input would be appreciated :)
[09:17] <JanC> if it is a security issue, it might also be useful to mention it in #ubuntu-security
[09:18] <JanC> IIRC there was a discussion there about various kernels there being vulnerable or not
[09:18] <SvenKieske> thanks for the pointer, until yesterday I wasn't really active in the ubuntu irc community, will do.
[09:19] <JanC> also, did you check if there is a new kernel in -proposed maybe?
[09:25] <SvenKieske> yeah, but in "proposed, there is 5.15 instead of the current 5.13, my guess is that has maybe todo with the upcoming next LTS GA of ubuntu? but it's just a guess. I'm not deeply familiar with the ubuntu dev cycle, coming from fedora land originally.
[09:26] <SvenKieske> iirc I saw a fixed 5.13 in the official hwe lts git branch tagged, that's why I wondered why it wasn't released..wait a second..
[09:27] <klebers> SvenKieske, hi! The -edge kernels are not guaranteed to received security fixes asap
[09:27] <klebers> they are not supported kernels
[09:28] <klebers> the hwe-edge kernel version (in this case 5.15) is a test kernel until it gets stable enough 
[09:29] <klebers> this kernel is the backport of the kernel from 22.04 LTS which is still under development
[09:29] <SvenKieske> klebers: hi, then why some "hwe" kernel get patches via security announce, and some do not? is this best effort community stuff? :) 
[09:29] <SvenKieske> the patch is in git, just not released, it seems? https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal/commit/?h=hwe-5.13
[09:29] <SvenKieske> sorry, this commit: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal/commit/?h=hwe-5.13&id=438da6e5c849ffe553fc15379471bf331346c3d2
[09:29] <klebers> the hwe kernel is supported and receives security updates, but not hwe-edge
[09:30] <klebers> so if you need a supported kernel you need to use hwe and not hwe-edge
[09:30] <SvenKieske> klebers: very good to know! I feared as much! is this documented somewhere? if yes, it's well hidden, because 3 engineers didn't find that information yesterday :D
[09:32] <SvenKieske> thank you very, very much! that's very important information for me. if I can help out communicate that somehow better (the edge kernel seems somehow underdocumented in general), let me know :)
[09:32] <klebers> We have this document: https://ubuntu.com/kernel/variants
[09:32] <klebers> "Provides early access to the next generic-hwe kernel."
[09:33] <klebers> though it's not clear about the support and security updates. I'm looking for some other doc
[09:33] <SvenKieske> yeah, I read that page, and several others, up and down..
[09:34] <SvenKieske> it might be improved :) but thanks for letting me know! :)
[09:36] <klebers> SvenKieske, thanks for pointing that out! I might be missing something but it seems we really don't have an official documentation making it clear. I'll get in touch with the people responsible for this page so we can have it expanded  
[09:38] <SvenKieske> klebers: very much appreciated! better documentation is always a win, hope it helps someone else in the future :)
[09:39] <JanC> maybe the -edge kernel shouldn't be in 'main' if it is not officially supported...
[09:43] <klebers> they are canonical maintained kernels, just not supported in the sense that they are test kernels and we don't make an effort to fix bugs as quickly as the stable kernels
[09:44] <klebers> so they are not suitable for production environments 
[09:46] <SvenKieske> remembering..I guess we deduced that if they are in main they get updates..but that was some years ago, so my memory might be wrong
[09:46] <SvenKieske> we thought they where just newer than "hwe" I guess.
[09:48] <SvenKieske> I think it's fine as is, but it should just be clearly documented somewhere (easy to find). this is still a large problem for most software out in the world, anyway.
[09:49] <SvenKieske> just yesterday I rummaged through go-yaml just to find out they do no security announcements at all, you find those in kubernetes repositories instead..ah well, we still live in the dark age of computing.
[09:53] <SvenKieske> whoopsie, that's also developed by canonical I guess. not trying to step on anyone's toes, it's all hard work and oss, after all :)
[10:05] <JanC> klebers: that really sounds like they don't belong in 'main' to me...
[10:06] <JanC> other stuff in universe is maintained by Canonical employees too
[10:07] <JanC> just not officially or with guarantees
[10:20] <xnox> JanC:  hwe-edge used to be in PPA only, but it made it very impractical to test. From time to time, we do one-off spins of Ubuntu Desktop .iso with hwe-edge kernel, instead of hwe one. To create offline media for Vendors to test. Ubuntu Desktop .iso are built with main suite only. Ditto Ubuntu Server isos and cloud images. Thus to allow wider, pre-release, testing of the next hwe kernel, we
[10:20] <xnox> have to put it as hwe-edge into main.
[10:20] <xnox> all of our attempts at having it in a ppa, or keep it stuck in focal-proposed, ended up being futile efforts and the target users were unable to install and test it.
[10:21] <JanC> I don't see why that would require 'main' instead of 'universe'
[10:21] <xnox> that is true. but that's how those images are built for Ubuntu product today, they force disable universe, and keep only main enabled.
[10:21] <SvenKieske> mhm, this is maybe not the right channel, but maybe you know people who could fix this: when downloading source packages from packages.ubuntu.com all download links to .xz files are downloads via "http" instead of "https" and new releases of firefox outright block downloads via http..
[10:22] <xnox> SvenKieske:  cute, there should be a link to file a bug report at the bottom of the packages website.
[10:22] <JanC> well, fix how those images are built then  :)
[10:23] <xnox> JanC:  also it iis not true that universe doesn't get security support.... because on some LTS releases one can purchase that from Canonical as an Ubuntu Pro entitelment. It's just hwe-edge has a carve out.
[10:23] <SvenKieske> xnox: will do, thanks!
[10:23] <JanC> or find some way to make sure that hwe images always override hwe-edge images
[10:24] <xnox> SvenKieske:  maybe say something like "urls could point at launchpad librarian / launchpad https download instead of http archive"
[10:24] <xnox> JanC:  ideally we wanted not to ship hwe-edge in the archive, and have it just in a PPA. But that hill prooved too much to climb =(
[10:25] <JanC> xnox: I just said that lots of universe gets Canonical support, but it's not guaranteed the same way as for 'main'
[10:25] <xnox> .... but also main
[10:25] <xnox> literarly everything is case by case basis =)
[10:25] <JanC> which would fit hwe-edge
[10:26] <xnox> cause if you squint hard enough, you will see remains of EOL v5.6 v5.8 v5.11 kernels in focal-updates too that stopped receiving security suport.
[10:26] <xnox> from all the various -edge types of kernels (hwe-edge is not the only edge kernel variant)
[10:27] <xnox> we used to have Supported: field on the package metadata that we were able to tweak
[10:27] <xnox> but i don't have a way to declare stuff as carve-outs in the metadata automatically for things that are not supported.
[10:27] <JanC> move everything that is not supposed to be supported to universe?
[10:27] <JanC> eh?
[10:28] <JanC> so, you need to fix that regression  ;)
[10:28] <xnox> main/universe split is based on whether things are seeded in ubuntu products
[10:28] <xnox> rather than what canonical provides commercial support for, or not.
[10:28] <xnox> and security support is orthogonal to that.
[10:28] <xnox> things that get through MIR process and get seeded are in main.
[10:29] <xnox> the source packages of hwe-edge fit that bill, because eventually that same source package starts to produce -edge packages, and then eventually stops again, as we roll on and off it.
[10:29] <xnox> so it must / will be in main proper for a period of time.
[10:30] <xnox> the roll-on/roll-off is messy though.
[10:30] <JanC> commercial support, you can handle in separate contracts, but other people really assume stuff in main gets security support
[10:30] <JanC> that's how it was originally communicated...
[10:31] <SvenKieske> I agree, that this should at least be better documented, what is in "main". I'm not even talking about desktop users, those tend to have a false sense of security what get's updated most of the time anyway.
[10:32] <SvenKieske> honestly I don't know a single distro who really gets this right, you can easily check this by mapping upstream vulns to distro sec announcements. maybe redhat for their base distro is close, but they miss tons of kernel vulns as well.
[10:33] <JanC> some kernel issues might be so minor or hard to exploit that they don't bother also
[10:34] <SvenKieske> still, I think honest and open communication is key, so it would be good to get some kind of easy to identify indicator on a per package basis how up to date it is, the debian security tracker is somewhat decent at this (but also not always up to date itself).
[10:34] <xnox> about current situation...... have you installed and running hwe-edge kernel and did something tell you to do that?
[10:35] <xnox> and if one is monitoring USN / CVE notices, it should have clear indication which kernels are EOL, and which ones are still to be patched. And it should be easy to see that information.
[10:35] <xnox> i would want to start with simple accurante and critical information being clear first.
[10:35] <SvenKieske> well, if you run servers, that is, new server hardware, you want those new drivers in newer kernels, so you need hwe, or hwe-edge.. we had some internal discussion which one to choose, unfortunately I don't find any minutes from that meeting atm..
[10:36] <JanC> xnox: there is no clear warning in the package descriptions either
[10:36] <JanC> for those -edge kernels
[10:36] <SvenKieske> I remember being in favor of the "hwe" kernel, without the "edge"..honestly, if it where my decision alone I would go with upstream latest LTS release (so kernel.org kernels).
[10:37] <xnox> so ahead of 20.04.2 release we take 20.10 kernel and package it as hwe-edge and test it. if all is good and no regressions are identified we roll out that new kernel as both hwe & hwe-edge flavours, until we start preparing for 20.04.3
[10:37] <SvenKieske> what I remember is, that the idea was: if we use "hwe-edge" we can already test if we have issues with the next lts release. because we want to upgrade from lts to lts
[10:38] <JanC> yeah, hwe seems to be fine, but there seems to be no proper explanation about hwe-edge in the packages, so it would be easy to be mislead 
[10:38] <SvenKieske> to not blow things out of proportion: I think some docs on the kernel variants page, or the old hwe kernel wiki page or in the package description would've been enough, we checked all those :)
[10:39] <xnox> so hwe-edge is ahead of hwe, only for a short period of time (i.e. 1-2 months) every 6 months, for about 18 months, from the 6month lifemark of an LTS
[10:39] <xnox> SvenKieske:  hwe-edge doesn't not give you that, hwe does. Because eventually, hwe will roll upto the next LTS kernel and park on that.
[10:39] <JanC> if it really is a temporary cludge, and should be indicated as such in its package descriptions...
[10:40] <xnox> hwe-edge just offers that 1-2 months earlier, still not fully tested and with regressions potentially.
[10:40] <xnox> SvenKieske:  please don't run *-edge kernels in production.
[10:40] <xnox> JanC:  Agree with that.
[10:40] <JanC> "do not install this until explicitly asked to for testing"  :)
[10:40] <JanC> and move it to universe really
[10:40] <SvenKieske> xnox: I'm about to change that :) it's still pre production, so not too late :D 
[10:41] <SvenKieske> xnox: some clear messaging somewhere would be really good, I myself don't even bother where it's written down (well, maybe not in a kernel source comment), as long as I can find it. we spent the whole afternoon with 3 ppl yesterday looking for information when to use which kernel
[10:42] <SvenKieske> because I already feared that the missing security update was a sign we use the wrong kernel
[10:42] <SvenKieske> I guess most people wouldn't even notice
[10:42] <SvenKieske> need to head to my "daily", see you in a bit
[10:42] <xnox> SvenKieske: Ubuntu Server installer iso defaults to GA kernel always, and offers hwe$ flavour as an option (if one must). My personal opinion for servers is that first two years of the LTS it is best to use generic. After first two years, it makes sense to switch to hwe, because it is the next-lts's generic, which gives one features and stability of the next-lts kernel with ease of upgrading
[10:42] <xnox> to it, as one is already running almost the same kernel.
[10:43] <xnox> SvenKieske:  most people are not smart enough to find/locate/install hwe-edge kernel. It is a very manual task!
[10:43] <SvenKieske> xnox: agreed :) I think I also said as much in the meeting deciding about kernels, unfortunately that was some years ago, and I don't find the minutes.. anyway we are now better informed :)
[10:44] <JanC> well, depends if you need the hardware support
[10:44] <SvenKieske> xnox: you find hwe kernel fast, if your hardware doesn't boot ;)
[10:44] <xnox> and that. yes, one may be forced to use hwe even during the first 24 months.
[10:44] <xnox> SvenKieske:  desktop.iso should always boot (it defaults to hwe)
[10:45] <JanC> even if it boots, you might still need it for optimal performance
[10:45] <xnox> SvenKieske:  and server iso should offer it as an option..... hopefully that correctly makes people find hwe$ (not -edge)
[10:45] <SvenKieske> xnox: well I certainly don't boot any GUI on my 10k servers ;)
[10:46] <xnox> how dare you not =) 
[10:46] <JanC> it shows how bad documentaion can confuse even an experienced admin...?
[10:47] <xnox> >_< =)
[10:47] <xnox> i think it is still true that one cannot certify server java install without X & gtk
[10:49] <SvenKieske> yeah, would call me rather experienced after roughly 10 years of linux :)
[10:50] <JanC> it's also still true that most admins don't care about certified stuff
[10:51] <JanC> I remember a friend who got bitten by that
[10:51] <SvenKieske> beside elasticsearch we fortunately don't run java :)
[10:56] <xnox> SvenKieske:  it makes sense for prod to have hwe, but testing environment to have hwe-edge. That way, if there are upcoming kernel roll being prepared, one will notice breakage in test 1-2 months before it hits production.
[10:59] <xnox> https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/1964467
[11:01] <JanC> that friend used a paid distro because it was the only officially supported one for a commercial groupware solution they also paid for; then a distro upgrade broke their mail/calendaring/etc.; after 2 days of support requests but no solutions (and his users getting very angry) he also asked around on Debian IRC (he used Debian personally), he got an answer explaining the problem with that paid distro, and an assurance the groupware still ran fine on Debian; 
[11:01] <JanC> he never looked back...  :P
[11:04] <xnox> i dream to run all of distro CI whilst pulling vanilla rc kernel; git tip of systemd; git tip of glibc; git tip of gcc; etc.
[11:05] <xnox> but nobody does that. simultaniously. at most people only test one new thing, not all new things together all the time.
[11:05] <JanC> well, obviously, that's why you use distros...
[11:06]  * xnox is distro developer, and thinks that's how distro should ci itself, but doesn't
[11:07] <JanC> well, you should test things separately and together both
[11:08] <JanC> one of the problems in FOSS is the mismatch between upstream & distro schedules
[11:26] <SvenKieske> xnox: afaik facebook does that, somewhat, they use centos and backports from fedora for systemd, kernel etc. check out their github, pretty awesome
[11:28] <JanC> facebook have more resources than almost any distro
[11:34] <xnox> SvenKieske:  as far as i understand for systemd they just run the one from main branch in production. by taking snapshot of it, running it through their dev environment, then test, then to prod. so like stuff landing in main can be deployed in fb production within like 1-2 weeks time.
[11:35] <xnox> based on the chats we had with them at systemd io in berlin.
[11:35] <xnox> i don't know what they do with kernels.
[21:37] <ricotz> hello :), I am curious if there is an ETA for kernel update for jammy?