=== mhcerri8 is now known as mhcerri
SvenKieskegood morning from europe. So I'm really not sure if I'm supposed to ask end user questions in here or if this is a dev only channel (the wiki seems to say it's okay to ask here?). But I want to restate my question from yesterday: why do we not get the latest security update for the focal hwe edge kernel via metapackage "linux-generic-hwe-20.04-edge"?08:40
SvenKieskeis this metapackage in the process of being updated to the next LTS Kernel (22.04) and thus security updates are stopped for the moment? :/08:41
JanCwho says it doesn't get security updates?08:59
SvenKiesketo be specific, it is still missing this update, and the "edge" kernel was also not mentioned in the associated USN Security Announcement: https://ubuntu.com/security/CVE-2022-084709:12
ubottuA flaw was found in the way the "flags" member of the new pipe buffer structure was lacking proper initialization in copy_page_to_iter_pipe and push_pipe functions in the Linux kernel and could thus contain stale values. An unprivileged local user could use this flaw to write to pages in the page cache backed by read only files and as such escalate their privileges on the syste... <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-0847>09:12
SvenKieskeand going back through our announcement archive, the "hwe-edge" kernel almost never get's mentioned in these announcements. I just want to know if we maybe misuse this kernel and want to double check that we get security updates asap, or if we should maybe use "hwe" kernel without edge?09:15
SvenKieskethis is for datacenter/ private/public cloud workloads09:16
SvenKieskeJanC: any input would be appreciated :)09:16
JanCif it is a security issue, it might also be useful to mention it in #ubuntu-security09:17
JanCIIRC there was a discussion there about various kernels there being vulnerable or not09:18
SvenKieskethanks for the pointer, until yesterday I wasn't really active in the ubuntu irc community, will do.09:18
JanCalso, did you check if there is a new kernel in -proposed maybe?09:19
SvenKieskeyeah, but in "proposed, there is 5.15 instead of the current 5.13, my guess is that has maybe todo with the upcoming next LTS GA of ubuntu? but it's just a guess. I'm not deeply familiar with the ubuntu dev cycle, coming from fedora land originally.09:25
SvenKieskeiirc I saw a fixed 5.13 in the official hwe lts git branch tagged, that's why I wondered why it wasn't released..wait a second..09:26
klebersSvenKieske, hi! The -edge kernels are not guaranteed to received security fixes asap09:27
klebersthey are not supported kernels09:27
klebersthe hwe-edge kernel version (in this case 5.15) is a test kernel until it gets stable enough 09:28
klebersthis kernel is the backport of the kernel from 22.04 LTS which is still under development09:29
SvenKieskeklebers: hi, then why some "hwe" kernel get patches via security announce, and some do not? is this best effort community stuff? :) 09:29
SvenKieskethe patch is in git, just not released, it seems? https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal/commit/?h=hwe-5.1309:29
SvenKieskesorry, this commit: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal/commit/?h=hwe-5.13&id=438da6e5c849ffe553fc15379471bf331346c3d209:29
ubottuCommit 438da6e in ~ubuntu-kernel/ubuntu/+source/linux/+git/focal "UBUNTU: Ubuntu-hwe-5.13-5.13.0-35.40~20.04.1 Ubuntu-hwe-5.13-5.13.0-35.40_20.04.1 hwe-5.13"09:29
klebersthe hwe kernel is supported and receives security updates, but not hwe-edge09:29
klebersso if you need a supported kernel you need to use hwe and not hwe-edge09:30
SvenKieskeklebers: very good to know! I feared as much! is this documented somewhere? if yes, it's well hidden, because 3 engineers didn't find that information yesterday :D09:30
SvenKieskethank you very, very much! that's very important information for me. if I can help out communicate that somehow better (the edge kernel seems somehow underdocumented in general), let me know :)09:32
klebersWe have this document: https://ubuntu.com/kernel/variants09:32
klebers"Provides early access to the next generic-hwe kernel."09:32
klebersthough it's not clear about the support and security updates. I'm looking for some other doc09:33
SvenKieskeyeah, I read that page, and several others, up and down..09:33
SvenKieskeit might be improved :) but thanks for letting me know! :)09:34
klebersSvenKieske, thanks for pointing that out! I might be missing something but it seems we really don't have an official documentation making it clear. I'll get in touch with the people responsible for this page so we can have it expanded  09:36
SvenKieskeklebers: very much appreciated! better documentation is always a win, hope it helps someone else in the future :)09:38
JanCmaybe the -edge kernel shouldn't be in 'main' if it is not officially supported...09:39
klebersthey are canonical maintained kernels, just not supported in the sense that they are test kernels and we don't make an effort to fix bugs as quickly as the stable kernels09:43
klebersso they are not suitable for production environments 09:44
SvenKieskeremembering..I guess we deduced that if they are in main they get updates..but that was some years ago, so my memory might be wrong09:46
SvenKieskewe thought they where just newer than "hwe" I guess.09:46
SvenKieskeI think it's fine as is, but it should just be clearly documented somewhere (easy to find). this is still a large problem for most software out in the world, anyway.09:48
SvenKieskejust yesterday I rummaged through go-yaml just to find out they do no security announcements at all, you find those in kubernetes repositories instead..ah well, we still live in the dark age of computing.09:49
SvenKieskewhoopsie, that's also developed by canonical I guess. not trying to step on anyone's toes, it's all hard work and oss, after all :)09:53
JanCklebers: that really sounds like they don't belong in 'main' to me...10:05
JanCother stuff in universe is maintained by Canonical employees too10:06
JanCjust not officially or with guarantees10:07
xnoxJanC:  hwe-edge used to be in PPA only, but it made it very impractical to test. From time to time, we do one-off spins of Ubuntu Desktop .iso with hwe-edge kernel, instead of hwe one. To create offline media for Vendors to test. Ubuntu Desktop .iso are built with main suite only. Ditto Ubuntu Server isos and cloud images. Thus to allow wider, pre-release, testing of the next hwe kernel, we10:20
xnoxhave to put it as hwe-edge into main.10:20
xnoxall of our attempts at having it in a ppa, or keep it stuck in focal-proposed, ended up being futile efforts and the target users were unable to install and test it.10:20
JanCI don't see why that would require 'main' instead of 'universe'10:21
xnoxthat is true. but that's how those images are built for Ubuntu product today, they force disable universe, and keep only main enabled.10:21
SvenKieskemhm, this is maybe not the right channel, but maybe you know people who could fix this: when downloading source packages from packages.ubuntu.com all download links to .xz files are downloads via "http" instead of "https" and new releases of firefox outright block downloads via http..10:21
xnoxSvenKieske:  cute, there should be a link to file a bug report at the bottom of the packages website.10:22
JanCwell, fix how those images are built then  :)10:22
xnoxJanC:  also it iis not true that universe doesn't get security support.... because on some LTS releases one can purchase that from Canonical as an Ubuntu Pro entitelment. It's just hwe-edge has a carve out.10:23
SvenKieskexnox: will do, thanks!10:23
JanCor find some way to make sure that hwe images always override hwe-edge images10:23
xnoxSvenKieske:  maybe say something like "urls could point at launchpad librarian / launchpad https download instead of http archive"10:24
xnoxJanC:  ideally we wanted not to ship hwe-edge in the archive, and have it just in a PPA. But that hill prooved too much to climb =(10:24
JanCxnox: I just said that lots of universe gets Canonical support, but it's not guaranteed the same way as for 'main'10:25
xnox.... but also main10:25
xnoxliterarly everything is case by case basis =)10:25
JanCwhich would fit hwe-edge10:25
xnoxcause if you squint hard enough, you will see remains of EOL v5.6 v5.8 v5.11 kernels in focal-updates too that stopped receiving security suport.10:26
xnoxfrom all the various -edge types of kernels (hwe-edge is not the only edge kernel variant)10:26
xnoxwe used to have Supported: field on the package metadata that we were able to tweak10:27
xnoxbut i don't have a way to declare stuff as carve-outs in the metadata automatically for things that are not supported.10:27
JanCmove everything that is not supposed to be supported to universe?10:27
JanCso, you need to fix that regression  ;)10:28
xnoxmain/universe split is based on whether things are seeded in ubuntu products10:28
xnoxrather than what canonical provides commercial support for, or not.10:28
xnoxand security support is orthogonal to that.10:28
xnoxthings that get through MIR process and get seeded are in main.10:28
xnoxthe source packages of hwe-edge fit that bill, because eventually that same source package starts to produce -edge packages, and then eventually stops again, as we roll on and off it.10:29
xnoxso it must / will be in main proper for a period of time.10:29
xnoxthe roll-on/roll-off is messy though.10:30
JanCcommercial support, you can handle in separate contracts, but other people really assume stuff in main gets security support10:30
JanCthat's how it was originally communicated...10:30
SvenKieskeI agree, that this should at least be better documented, what is in "main". I'm not even talking about desktop users, those tend to have a false sense of security what get's updated most of the time anyway.10:31
SvenKieskehonestly I don't know a single distro who really gets this right, you can easily check this by mapping upstream vulns to distro sec announcements. maybe redhat for their base distro is close, but they miss tons of kernel vulns as well.10:32
JanCsome kernel issues might be so minor or hard to exploit that they don't bother also10:33
SvenKieskestill, I think honest and open communication is key, so it would be good to get some kind of easy to identify indicator on a per package basis how up to date it is, the debian security tracker is somewhat decent at this (but also not always up to date itself).10:34
xnoxabout current situation...... have you installed and running hwe-edge kernel and did something tell you to do that?10:34
xnoxand if one is monitoring USN / CVE notices, it should have clear indication which kernels are EOL, and which ones are still to be patched. And it should be easy to see that information.10:35
xnoxi would want to start with simple accurante and critical information being clear first.10:35
SvenKieskewell, if you run servers, that is, new server hardware, you want those new drivers in newer kernels, so you need hwe, or hwe-edge.. we had some internal discussion which one to choose, unfortunately I don't find any minutes from that meeting atm..10:35
JanCxnox: there is no clear warning in the package descriptions either10:36
JanCfor those -edge kernels10:36
SvenKieskeI remember being in favor of the "hwe" kernel, without the "edge"..honestly, if it where my decision alone I would go with upstream latest LTS release (so kernel.org kernels).10:36
xnoxso ahead of 20.04.2 release we take 20.10 kernel and package it as hwe-edge and test it. if all is good and no regressions are identified we roll out that new kernel as both hwe & hwe-edge flavours, until we start preparing for 20.04.310:37
SvenKieskewhat I remember is, that the idea was: if we use "hwe-edge" we can already test if we have issues with the next lts release. because we want to upgrade from lts to lts10:37
JanCyeah, hwe seems to be fine, but there seems to be no proper explanation about hwe-edge in the packages, so it would be easy to be mislead 10:38
SvenKiesketo not blow things out of proportion: I think some docs on the kernel variants page, or the old hwe kernel wiki page or in the package description would've been enough, we checked all those :)10:38
xnoxso hwe-edge is ahead of hwe, only for a short period of time (i.e. 1-2 months) every 6 months, for about 18 months, from the 6month lifemark of an LTS10:39
xnoxSvenKieske:  hwe-edge doesn't not give you that, hwe does. Because eventually, hwe will roll upto the next LTS kernel and park on that.10:39
JanCif it really is a temporary cludge, and should be indicated as such in its package descriptions...10:39
xnoxhwe-edge just offers that 1-2 months earlier, still not fully tested and with regressions potentially.10:40
xnoxSvenKieske:  please don't run *-edge kernels in production.10:40
xnoxJanC:  Agree with that.10:40
JanC"do not install this until explicitly asked to for testing"  :)10:40
JanCand move it to universe really10:40
SvenKieskexnox: I'm about to change that :) it's still pre production, so not too late :D 10:40
SvenKieskexnox: some clear messaging somewhere would be really good, I myself don't even bother where it's written down (well, maybe not in a kernel source comment), as long as I can find it. we spent the whole afternoon with 3 ppl yesterday looking for information when to use which kernel10:41
SvenKieskebecause I already feared that the missing security update was a sign we use the wrong kernel10:42
SvenKieskeI guess most people wouldn't even notice10:42
SvenKieskeneed to head to my "daily", see you in a bit10:42
xnoxSvenKieske: Ubuntu Server installer iso defaults to GA kernel always, and offers hwe$ flavour as an option (if one must). My personal opinion for servers is that first two years of the LTS it is best to use generic. After first two years, it makes sense to switch to hwe, because it is the next-lts's generic, which gives one features and stability of the next-lts kernel with ease of upgrading10:42
xnoxto it, as one is already running almost the same kernel.10:42
xnoxSvenKieske:  most people are not smart enough to find/locate/install hwe-edge kernel. It is a very manual task!10:43
SvenKieskexnox: agreed :) I think I also said as much in the meeting deciding about kernels, unfortunately that was some years ago, and I don't find the minutes.. anyway we are now better informed :)10:43
JanCwell, depends if you need the hardware support10:44
SvenKieskexnox: you find hwe kernel fast, if your hardware doesn't boot ;)10:44
xnoxand that. yes, one may be forced to use hwe even during the first 24 months.10:44
xnoxSvenKieske:  desktop.iso should always boot (it defaults to hwe)10:44
JanCeven if it boots, you might still need it for optimal performance10:45
xnoxSvenKieske:  and server iso should offer it as an option..... hopefully that correctly makes people find hwe$ (not -edge)10:45
SvenKieskexnox: well I certainly don't boot any GUI on my 10k servers ;)10:45
xnoxhow dare you not =) 10:46
JanCit shows how bad documentaion can confuse even an experienced admin...?10:46
xnox>_< =)10:47
xnoxi think it is still true that one cannot certify server java install without X & gtk10:47
SvenKieskeyeah, would call me rather experienced after roughly 10 years of linux :)10:49
JanCit's also still true that most admins don't care about certified stuff10:50
JanCI remember a friend who got bitten by that10:51
SvenKieskebeside elasticsearch we fortunately don't run java :)10:51
xnoxSvenKieske:  it makes sense for prod to have hwe, but testing environment to have hwe-edge. That way, if there are upcoming kernel roll being prepared, one will notice breakage in test 1-2 months before it hits production.10:56
ubottuLaunchpad bug 1964467 in linux-meta (Ubuntu) "edge variants are not obvious" [Undecided, New]10:59
JanCthat friend used a paid distro because it was the only officially supported one for a commercial groupware solution they also paid for; then a distro upgrade broke their mail/calendaring/etc.; after 2 days of support requests but no solutions (and his users getting very angry) he also asked around on Debian IRC (he used Debian personally), he got an answer explaining the problem with that paid distro, and an assurance the groupware still ran fine on Debian; 11:01
JanChe never looked back...  :P11:01
xnoxi dream to run all of distro CI whilst pulling vanilla rc kernel; git tip of systemd; git tip of glibc; git tip of gcc; etc.11:04
xnoxbut nobody does that. simultaniously. at most people only test one new thing, not all new things together all the time.11:05
JanCwell, obviously, that's why you use distros...11:05
* xnox is distro developer, and thinks that's how distro should ci itself, but doesn't11:06
JanCwell, you should test things separately and together both11:07
JanCone of the problems in FOSS is the mismatch between upstream & distro schedules11:08
SvenKieskexnox: afaik facebook does that, somewhat, they use centos and backports from fedora for systemd, kernel etc. check out their github, pretty awesome11:26
JanCfacebook have more resources than almost any distro11:28
xnoxSvenKieske:  as far as i understand for systemd they just run the one from main branch in production. by taking snapshot of it, running it through their dev environment, then test, then to prod. so like stuff landing in main can be deployed in fb production within like 1-2 weeks time.11:34
xnoxbased on the chats we had with them at systemd io in berlin.11:35
xnoxi don't know what they do with kernels.11:35
ricotzhello :), I am curious if there is an ETA for kernel update for jammy?21:37

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!