-queuebot:#ubuntu-release- Builds: Xubuntu Desktop amd64 [Lunar Daily] has been updated (20230216)02:01
-queuebot:#ubuntu-release- Builds: Xubuntu Desktop amd64 [Jammy Daily] has been updated (20230216)02:08
-queuebot:#ubuntu-release- Builds: Xubuntu Minimal amd64 [Lunar Daily] has been updated (20230216.1)02:25
=== chris14_ is now known as chris14
-queuebot:#ubuntu-release- Builds: Ubuntu MATE Desktop amd64 [Jammy Daily] has been updated (20230216)03:19
-queuebot:#ubuntu-release- Builds: Ubuntu Base amd64 [Jammy Daily] has been updated (20230216)05:35
-queuebot:#ubuntu-release- Builds: Ubuntu Base amd64 [Lunar Daily] has been updated (20230216)05:35
-queuebot:#ubuntu-release- Builds: Ubuntu Base s390x [Lunar Daily] has been updated (20230216)05:35
-queuebot:#ubuntu-release- Unapproved: py-macaroon-bakery (jammy-proposed/main) [1.3.1-2 => 1.3.1-2ubuntu0.1] (ubuntu-desktop)05:52
-queuebot:#ubuntu-release- Unapproved: py-macaroon-bakery (kinetic-proposed/universe) [1.3.1-3 => 1.3.1-3ubuntu0.1] (ubuntu-desktop)05:52
-queuebot:#ubuntu-release- Builds: Kubuntu Desktop amd64 [Jammy Daily] has been updated (20230216)05:55
arraybolt3In case it maters, I was able to retry the build failures for the package I was working on earlier (lxqt-policykit) that was failing due to python3, and the builds are succeeding this time, so it looks like things are at least closer to resolved, if not resolved already.06:24
-queuebot:#ubuntu-release- New: accepted procps [amd64] (lunar-proposed) [2:4.0.2-3ubuntu1]06:36
-queuebot:#ubuntu-release- New: accepted procps [armhf] (lunar-proposed) [2:4.0.2-3ubuntu1]06:36
-queuebot:#ubuntu-release- New: accepted procps [ppc64el] (lunar-proposed) [2:4.0.2-3ubuntu1]06:36
-queuebot:#ubuntu-release- New: accepted procps [s390x] (lunar-proposed) [2:4.0.2-3ubuntu1]06:36
-queuebot:#ubuntu-release- New: accepted procps [arm64] (lunar-proposed) [2:4.0.2-3ubuntu1]06:36
-queuebot:#ubuntu-release- New: accepted procps [riscv64] (lunar-proposed) [2:4.0.2-3ubuntu1]06:36
-queuebot:#ubuntu-release- New: accepted procps [i386] (lunar-proposed) [2:4.0.2-3ubuntu1]06:36
-queuebot:#ubuntu-release- New: accepted budgie-wallpapers [amd64] (lunar-proposed) [23.04.1]06:36
-queuebot:#ubuntu-release- New: accepted retroarch [amd64] (lunar-proposed) [1.14.0+dfsg-1]06:36
-queuebot:#ubuntu-release- New: accepted ghostscript [amd64] (lunar-proposed) [10.0.0~dfsg1-0ubuntu1]06:37
-queuebot:#ubuntu-release- New: accepted ghostscript [armhf] (lunar-proposed) [10.0.0~dfsg1-0ubuntu1]06:37
-queuebot:#ubuntu-release- New: accepted ghostscript [ppc64el] (lunar-proposed) [10.0.0~dfsg1-0ubuntu1]06:37
-queuebot:#ubuntu-release- New: accepted ghostscript [s390x] (lunar-proposed) [10.0.0~dfsg1-0ubuntu1]06:37
-queuebot:#ubuntu-release- New: accepted ghostscript [arm64] (lunar-proposed) [10.0.0~dfsg1-0ubuntu1]06:37
-queuebot:#ubuntu-release- New: accepted ghostscript [riscv64] (lunar-proposed) [10.0.0~dfsg1-0ubuntu1]06:37
-queuebot:#ubuntu-release- New: accepted ghostscript [i386] (lunar-proposed) [10.0.0~dfsg1-0ubuntu1]06:37
-queuebot:#ubuntu-release- Builds: Ubuntu Server amd64 [Lunar Daily] has been updated (20230216)06:57
-queuebot:#ubuntu-release- Builds: Ubuntu Server arm64+raspi [Lunar Daily] has been updated (20230216)06:57
-queuebot:#ubuntu-release- Builds: Ubuntu Server arm64 [Lunar Daily] has been updated (20230216)06:57
-queuebot:#ubuntu-release- Builds: Ubuntu Server armhf+raspi [Lunar Daily] has been updated (20230216)06:57
-queuebot:#ubuntu-release- Builds: Ubuntu Server riscv64+licheerv [Lunar Daily] has been updated (20230216)06:57
-queuebot:#ubuntu-release- Builds: Ubuntu Server riscv64+nezha [Lunar Daily] has been updated (20230216)06:57
-queuebot:#ubuntu-release- Builds: Ubuntu Server riscv64+unmatched [Lunar Daily] has been updated (20230216)06:57
-queuebot:#ubuntu-release- Builds: Ubuntu Server riscv64+visionfive [Lunar Daily] has been updated (20230216)06:57
-queuebot:#ubuntu-release- New binary: rcheevos [amd64] (lunar-proposed/universe) [10.6.0-1] (no packageset)07:35
-queuebot:#ubuntu-release- New binary: rcheevos [s390x] (lunar-proposed/universe) [10.6.0-1] (no packageset)07:35
-queuebot:#ubuntu-release- New binary: rcheevos [armhf] (lunar-proposed/universe) [10.6.0-1] (no packageset)07:35
-queuebot:#ubuntu-release- New binary: rcheevos [arm64] (lunar-proposed/universe) [10.6.0-1] (no packageset)07:36
-queuebot:#ubuntu-release- New binary: rcheevos [ppc64el] (lunar-proposed/universe) [10.6.0-1] (no packageset)07:36
-queuebot:#ubuntu-release- New: accepted rcheevos [ppc64el] (lunar-proposed) [10.6.0-1]07:40
-queuebot:#ubuntu-release- New: accepted rcheevos [amd64] (lunar-proposed) [10.6.0-1]07:41
-queuebot:#ubuntu-release- New: accepted rcheevos [armhf] (lunar-proposed) [10.6.0-1]07:41
-queuebot:#ubuntu-release- New: accepted rcheevos [arm64] (lunar-proposed) [10.6.0-1]07:41
-queuebot:#ubuntu-release- New: accepted rcheevos [s390x] (lunar-proposed) [10.6.0-1]07:41
-queuebot:#ubuntu-release- New binary: rcheevos [riscv64] (lunar-proposed/universe) [10.6.0-1] (no packageset)07:45
-queuebot:#ubuntu-release- Builds: Ubuntu Server amd64 [Jammy Daily] has been updated (20230216)07:46
-queuebot:#ubuntu-release- Builds: Ubuntu Server arm64+raspi [Jammy Daily] has been updated (20230216)07:46
-queuebot:#ubuntu-release- Builds: Ubuntu Server arm64 [Jammy Daily] has been updated (20230216)07:46
-queuebot:#ubuntu-release- Builds: Ubuntu Server armhf+raspi [Jammy Daily] has been updated (20230216)07:46
-queuebot:#ubuntu-release- Builds: Ubuntu Server riscv64+licheerv [Jammy Daily] has been updated (20230216)07:46
-queuebot:#ubuntu-release- Builds: Ubuntu Server riscv64+nezha [Jammy Daily] has been updated (20230216)07:46
-queuebot:#ubuntu-release- Builds: Ubuntu Server riscv64+visionfive [Jammy Daily] has been updated (20230216)07:46
=== Eickmeyer5 is now known as Eickmeyer
vorlonLocutusOfBorg: so what do we do with llvm-toolchain-14 failing its autopkgtests?08:19
vorlon(and blocking NBS removals)08:20
-queuebot:#ubuntu-release- Builds: Ubuntu Desktop amd64 [Lunar Daily] has been updated (20230216)08:23
-queuebot:#ubuntu-release- Builds: Ubuntu Server Subiquity amd64 [Lunar Daily] has been updated (20230216)08:28
-queuebot:#ubuntu-release- Builds: Ubuntu Server Subiquity arm64 [Lunar Daily] has been updated (20230216)08:28
-queuebot:#ubuntu-release- Builds: Ubuntu Server Subiquity ppc64el [Lunar Daily] has been updated (20230216)08:28
-queuebot:#ubuntu-release- Builds: Ubuntu Server Subiquity s390x [Lunar Daily] has been updated (20230216)08:28
LocutusOfBorgvorlon, hint them? :)08:29
LocutusOfBorgthey are new tests, tracked in debian bug: #102701308:30
-ubottu:#ubuntu-release- Debian bug 1027013 in src:llvm-toolchain-14 "llvm-toolchain-14 autopkg test fails" [Serious, Open] https://bugs.debian.org/102701308:30
vorlonLocutusOfBorg: or remove the broken tests to not regress autopkgtest coverage?08:37
vorlonicingaweb2-module-pdfexport Depends: chromium, wtf²08:37
-queuebot:#ubuntu-release- Builds: Ubuntu Core amd64 edge [Lunar Daily] has been updated (20230216)08:40
LocutusOfBorgvorlon, it takes days to build...08:42
LocutusOfBorgand I would like to fix in Debian and then sync08:42
vorlonwell, when it's fixed in Debian, great08:42
vorlonin the meantime I've uploaded to disable the broken autopkgtest08:43
vorlonso that it's migratable and I can drop libgrpc1008:43
LocutusOfBorgmigratable, if autopkgtests wants to run08:44
vorlonthey are running; just have a backlog and the infra is still requiring handholding08:44
LocutusOfBorgoh... you dropped too much08:44
vorlondid I?08:44
LocutusOfBorgthat qualify-clang.sh was not newly introduced, some content inside that script was :D08:44
vorlonhmm well08:45
vorlonif you want to fix it up I can cancel builds for -10ubuntu208:45
LocutusOfBorgyeah please lets fix08:46
LocutusOfBorgproperly I mean08:46
LocutusOfBorgI'm doing it after some scrum daily meeting08:46
-queuebot:#ubuntu-release- Packageset: Added llvm-toolchain-16 to i386-whitelist in lunar09:04
-queuebot:#ubuntu-release- Builds: Ubuntu Desktop amd64 [Jammy Daily] has been updated (20230216)09:07
-queuebot:#ubuntu-release- Builds: Ubuntu Desktop arm64 [Jammy Daily] has been updated (20230216)09:07
-queuebot:#ubuntu-release- New: accepted rcheevos [riscv64] (lunar-proposed) [10.6.0-1]09:07
-queuebot:#ubuntu-release- Builds: Ubuntu Desktop arm64+raspi [Lunar Daily] has been updated (20230216)09:22
LocutusOfBorgUploading llvm-toolchain-14_14.0.6-11_source.changes09:45
LocutusOfBorgI let only amd64 and arm64 build09:50
-queuebot:#ubuntu-release- Builds: Ubuntu Desktop arm64+raspi [Jammy Daily] has been updated (20230216)10:07
sil2100hmmm, why is the daily milestone reporting status here..?10:08
-queuebot:#ubuntu-release- Builds: 49 entries have been added, updated or disabled10:09
-queuebot:#ubuntu-release- Builds: 52 entries have been added, updated or disabled10:13
sil2100juliank: hey! I see the shims in proposed have block-proposed tags10:24
sil2100juliank: does that mean we still want to wait with the release of those?10:24
juliankthey have block tags for the kernel waiting10:24
juliankso I guess they can be removed now10:24
julianksil2100: blocks removed10:26
sil2100klebers: hey! Just to be entirely sure: the new-key-signed kernels have landed in kinetic-updates already, right?10:27
sil2100juliank: btw. on https://bugs.launchpad.net/oem-priority/+bug/1842320 I see someone mentioning that the bug isn't fixed for them..?10:29
-ubottu:#ubuntu-release- Launchpad bug 1842320 in OEM Priority Project "Can't boot: 'error: out of memory.' immediately after the grub menu" [Critical, Triaged]10:29
juliankwell yes they didn't install the fixed version10:29
sil2100Ok, releasing in that case10:30
sil2100I guess we can also unblock and mark the cd-boot-images as good, right? Did you test some of the newer jammy dailies?10:37
sil2100juliank: ^10:41
kleberssil2100, hey! Do you mean jammy-updates?10:43
sil2100klebers: no no, kinetic-updates. Since jammy-updates I know we already resolved yesteday, but I wanted to release the new shims for kinetic and others too10:44
kleberssil2100, I see! Yes, all kinetic kernels that we have signed with the new key are already in kinetic-updates10:45
klebersyw! :)10:46
sil2100klebers: also, just to double-make-sure (since I see some kernels in jammy-proposed, but I suppose those are the 'next cycle') - are all the cloud kernels for jammy with the new keys already in jammy-updates?10:50
sil2100I wouldn't want to break someone by releasing shim suddenly. But I think we were only waiting for hwe to have the complete set, so others were already there, right?10:51
kleberssil2100, we still have two private kernels that were not released to jammy-updates yet (jammy/linux-fips and jammy/linux-intel-iot-realtime). I'm not sure though whether they are signed with the new key, I'll double check with the team10:58
sil2100Ouch, would that block the shim release?11:11
sil2100Since I already released it for jammy11:11
-queuebot:#ubuntu-release- New binary: kodi-game-libretro [amd64] (lunar-proposed/universe) [20.2.2-2] (no packageset)11:22
-queuebot:#ubuntu-release- New binary: kodi-game-libretro [arm64] (lunar-proposed/universe) [20.2.2-2] (no packageset)11:23
-queuebot:#ubuntu-release- New binary: kodi-game-libretro [ppc64el] (lunar-proposed/universe) [20.2.2-2] (no packageset)11:23
-queuebot:#ubuntu-release- New binary: kodi-game-libretro [armhf] (lunar-proposed/universe) [20.2.2-2] (no packageset)11:25
xnoxsil2100:  jammy is not ready for new shim yet!11:35
xnoxsil2100:  KINETIC is ready and you released in kinetic.11:36
xnoxsil2100:  lunar and jammy are not ready yet11:36
-queuebot:#ubuntu-release- New binary: kodi-game-libretro [s390x] (lunar-proposed/universe) [20.2.2-2] (no packageset)11:39
-queuebot:#ubuntu-release- New binary: kodi-game-libretro [riscv64] (lunar-proposed/universe) [20.2.2-2] (no packageset)11:40
sil2100xnox: that is not what I have been told11:42
sil2100xnox: juliank unblocked shims in jammy, I've been told that the hwe kernels were the last thing blocking the shim updates11:43
sil2100Do I need to revert?11:43
sil2100juliank: ^11:43
sil2100xnox: what are we waiting for in jammy still? I need to have the new shim there for the point release11:44
xnoxsil2100:  where/where you saw juliank speak about jammy? date-time timestamps?11:44
juliankxnox: what's missing in jammy now?11:44
sil2100xnox: https://bugs.launchpad.net/ubuntu/+source/shim/+bug/1996503/comments/2511:44
-ubottu:#ubuntu-release- Launchpad bug 1996503 in shim-signed (Ubuntu) "shim 15.7-0ubuntu1" [Undecided, In Progress]11:44
juliankxnox: you said we're waiting for generic and 5.19 hwe, both are released.11:44
xnoxjuliank:  kernels are not considered out, until they are replicated to mirrors (mirror dwell) and until they are copied to security. it is being rolled out but it is not out.11:46
sil2100I need to know ASAP if I need to remove jammy-updates shim and get the old one back11:46
xnoxjuliank:  see our dashboard which doesn't say that hwe-5.19 and lowlatancy-hwe-5.19 is fully out yet.11:46
xnoxsil2100:  set phasing to zero %11:47
juliankxnox: but security and mirrors are irrelevant to us11:47
xnoxjuliank:  they are not.11:47
juliankwe only waited for the kernels for image building11:47
xnoxjuliank:  the kernel is not out yet.11:47
juliankit is in the -updates pocket, where the images are being built from11:47
xnoxjuliank:  you can brick people11:47
juliankit's irrelevant to the image building if it is on the mirrors or in security yet11:47
xnoxthat's not good.11:47
xnoxjuliank:  i don't care about your image people =)11:48
juliankyou cannot brick installed systems11:48
xnoxhave you met our users?! =)11:48
juliankbecause they only install the new shim if their booted kernel and any newer kernel is not revoked11:48
sil2100I phased it to 0 for now, but IIRC from my converstaions with juliank, he made sure that out-of-order upgrades should be handled11:49
juliankdid you miss the entire bit about how it installs the old kernels and sets up an alternative to switch them?11:49
xnox...... we have a tonne of people who don't use that to choose what to boot with what11:49
xnoxjuliank:  you are not kernel team to decide when all the kernels are out =)11:50
juliankxnox: it is out to users, is all I say11:52
xnoxjuliank: it isn't.11:52
juliankand users can install shims and kernels in any order and get safe results11:52
xnoxsil2100:  please revert shim and shim-upgrade in focal11:53
xnoxfocal is not ready11:53
xnoxjuliank:  dude11:54
sil2100xnox: eek, ok11:54
xnoxsil2100:  juliank: yesterday it was asked which series are good to go, and the answer was only kinetic. things have not changed dramatically since. so i am very surprised why a non-sru team member, who is not kernel-team member deciding to release shim to jammy and focal.11:56
xnoxthat's not ok.11:56
sil2100juliank was the one preparing and verifying the shim updates, so it was obvious to me to ask him about the readiness of the SRUs. I was not aware of having to wait additionally for the jammy kernels, the only information I got was: "we need jammy kernels to be released". The state 'jammy released' is not automatically same to 'fully synced on mirrors'11:58
xnoxsil2100:  also i'm not sure if kinetic is fine w.r.t. image building given grub is phased at 59% yet.11:58
juliankWhen sil2100 asked me this morning I double checked that the kernels were in the updates pocket publishing11:58
juliankxnox: It does not matter11:58
juliankxnox: please understand that image building ignores phasing and that the shim also breaks the old grubs and fwupd-signed, so you can't end up with unbootable images there no matter what you do11:59
xnoxjuliank:  our image building ignores phasing, sure. but not everyone elses.12:00
juliankthat's where the 2nd part comes in12:00
juliankalso you have to put in a lot of effort to build an image with phasing12:00
sil2100I know it might be 'risky', but as juliank said, users should not be able to brick their device just because they get the new shim without the new kernel12:01
xnoxnot really, we have people snapshot live system and do apt download from the live system to force unpack stuff.12:01
juliankphasing just isn't active in containers, chroots, only systems with an init12:01
juliankmore or less12:01
xnoxthat's where people do get versions of installed packges and unapck them.12:01
sil2100Anyway, reverting focal right now12:01
juliankthen all work is futile, we delay the point release another month and do two kernel SRU cycles12:02
juliankI mean that's the only realistic way you can ensure safety there by having the kernels out for enough time for everyone to have them12:02
juliankand that's precisely what we're trying to avoid here12:03
juliankin the normal world we would have had the signing PPAs ready in October, signed new kernels last year and then went ahead with the shim update in February and everyone would have been happy12:03
juliankbut we had basically 3 months of delays after generating the signing keys and then push to have the shim out ASAP12:05
sil2100Also, as for asking for status and who is to be asked for go/no-go - juliank is the maintainer of shim and was coordinating the release of the new shim, so he feels like the best person to ask for readiness. I could have asked for a really explicit +1 from the kernel team as well, but I did not get a hold-the-line signal at any point up until now after we released the kernels yesterday12:05
xnoxsil2100:  yesterday i was asked which kernels were ready, and the answer was only kinetic.12:06
xnoxthat's it.12:06
sil2100I have raised the concern of users upgrading things out of order with juliank when he started preparing the shim uploads, and he made sure to do it the best way he could12:06
julianksil2100: I mean I thought you were aware of the kernel releases, saw they were all in updates and pinged me asking if anything else is blocking them12:07
xnoxwe previously agreed to have kernels in -security pocket, before releasing shim to updates12:07
xnoxi don't see why we are backtracking on that agreement.12:07
sil2100xnox: who was it agreed with?12:07
juliankxnox: I am not aware of any such agreement12:07
sil2100Then it was not communicated to me12:07
sil2100And I am the lead for 22.04.212:07
sil2100Today is the planned date of when we NEED to have everything for .2 ready in -updates12:08
juliankmy awareness regarding kernels is to wait until they are released, which for purposes of SRU ordering, means accepted into -updates.12:08
juliankit's not like if you accept shim after kernels that they can appear on the mirror before the kernels12:08
ahasenacksil2100: hi, seeing talks of .2, is there an SRU freeze in place perhaps that I have missed?12:10
juliankwhen we originally planned this I was saying we should have two kernel SRU cycles completed before releasing shim but that didn't work12:10
juliankahasenack: I think it was completely forgotten to announce any kind of SRU freeze for jammy, tbh12:10
sil2100klebers, xnox: ok, now I need an explicit information: can the kernels be ready enough for us to have shim fully released today? Since if we're waiting for security as a requirement, this means 'no' and this introduces a serious issue in our timeline12:10
sil2100ahasenack: I'm almost done with the e-mail12:10
ahasenackI knew a dot release was/is coming up, and that its date was pushed once or twice, so I've been expecting that sru freeze email, but none so far12:10
sil2100juliank: it was not forgotten12:10
sil2100juliank: we only send that today12:11
xnoxsil2100:  focal is out of scope, i hope that's clear.12:11
ahasenacksil2100: tldr accepting into proposed is still fine, just not updates, is that it?12:11
sil2100I don't care about focal, my bad that I released it. That's entirely on me12:11
sil2100But the jammy situation agitates me a lot12:12
xnoxnow jammy12:12
xnoxboth hwe-5.19 and lowlatency-hwe-5.19 are ready to be copied to security12:12
xnoxcan you do that?12:12
-ubottu:#ubuntu-release- Launchpad bug 2004157 in Kernel SRU Workflow "jammy/linux-hwe-5.19: 5.19.0-32.33~22.04.1 -proposed tracker" [Medium, In Progress]12:12
-ubottu:#ubuntu-release- Launchpad bug 2006747 in Kernel SRU Workflow "jammy/linux-lowlatency-hwe-5.19: 5.19.0-1017.18~22.04.1 -proposed tracker" [Medium, In Progress]12:12
juliankxnox: so will you commit to removing the block-proposed tags on https://bugs.launchpad.net/ubuntu/+source/shim/+bug/1996503 as necessary then?12:14
-ubottu:#ubuntu-release- Launchpad bug 1996503 in shim-signed (Ubuntu) "shim 15.7-0ubuntu1" [Undecided, In Progress]12:14
juliankxnox: because I can't guess when you're happy, sorry12:14
xnoxjuliank:  yes i will commit to removing block-proposed tags12:14
-queuebot:#ubuntu-release- Unapproved: shim-signed (focal-updates/main) [1.40.9 => 1.40.7] (core) (sync)12:15
-queuebot:#ubuntu-release- Unapproved: shim (focal-updates/main) [15.7-0ubuntu1 => 15.4-0ubuntu9] (core) (sync)12:16
xnoxklebers:  re: jammy:linux-fips => it's ok, as it fails to boot by itself anyway =)12:16
-queuebot:#ubuntu-release- Unapproved: accepted shim-signed [sync] (focal-updates) [1.40.7]12:19
-queuebot:#ubuntu-release- Unapproved: accepted shim [sync] (focal-updates) [15.4-0ubuntu9]12:19
xnoxklebers:  re:linux-intel-iot-realtime it is up to date12:19
xnoxwhere it needs to be12:19
xnoxi see one image build that will be broken for private kernel12:21
sil2100I wasn't aware of those being a blocker. I only knew hwe and ga kernels were still blocking shim12:22
juliankthat was what was communicated, yes12:23
juliankbasically "we need to wait for kernels used in images", "what are those", "ga and hwe"12:24
xnoxthe expectation was that one would be have been out by 1st of february but it didn't go out as far as i can see12:24
sil2100xnox: I can release the kernels to -security indeed! But I thought there was always a some-day-ish delay between updates and security due to bandwidth?12:26
sil2100Or is that not a concern anymore?12:26
xnoxsil2100:  the dwelling delay has passed, our report says mirrors are sufficiently up to date, so it is ready for security12:27
xnoxsil2100:  see https://kernel.ubuntu.com/sru/dashboards/web/kernel-stable-board.html?cycle=2023.01.0212:28
xnoxpreviously it said "holding release to security, waiting for replication dwell" or some such12:29
sil2100Ok, then ugh, running the manual release commands!12:30
juliankxnox: at least we are learning now for when we need to do the whole dance for NX support againb12:33
seb128sil2100, adding that as a note for after the current fires, but you probably want to release bug #1990368 to J otherwise the ISO will probably start to fail to build once you switch to use updates instead of proposed since that new package is a depends of software-properties-gtk now12:35
-ubottu:#ubuntu-release- Bug 1990368 in ubuntu-advantage-desktop-daemon (Ubuntu Jammy) "Rename Ubuntu Advantage to Ubuntu Pro" [Undecided, Fix Committed] https://launchpad.net/bugs/199036812:35
sil2100xnox: ok, just to confirm - since I see on the tracking bug "promote-to-security: []", I should simply release the set of packages that are in promote-to-updates, right? They're really ready-ready?12:36
juliankxnox: because all our kernels need whatever the current version of https://lore.kernel.org/linux-kernel/cover.1668958803.git.baskov@ispras.ru/ is for the next shim, sigh.12:37
seb128sil2100, sorry, I didn't notice that the previous version was  moved to updates, so probably not an ISO build issue but still would be nice to get. It was verified ages ago but seems the tag editing failed :/12:37
xnoxsil2100:  it is confusing but yes it is all of them.12:38
xnoxjuliank:  =)))))))))))))))))))))))))))))))))))))))))))) yeah, nx stuff will be fun.12:38
juliankxnox: And I need to figure out if the kernel has the patchset from the binary in a bash script so I can adjust my script for revoked kernels to check for non-NX kernels12:39
xnoxsil2100:  juliank: i copied a contingency for the remaining private kernel. so once hwe-5.19 and lowlatency-hwe-5.19 are in security, i am happy to have shim / shim-signed in jammy to be phasing normally.12:39
juliankxnox: or maybe I don't, I don't know if there is firmware out there with NX that boots the shim without NX now12:39
xnoxmeanwhile checking focal state12:40
sil2100Whoops on focal ;p12:40
julianklike maybe the NX shim doesn't need to be ordered after any other NX bits because the firmware that uses the NX bit in the shim wouldn't boot the old one either12:40
sil2100I removed the shims and tried copying in the old ones12:40
sil2100Not sure if it all didn't overly confuse the publisher12:41
sil2100I did phase them to 0 just-in-case before doing the revert dance12:41
xnoxjuliank:  sil2100: btw there are multiple escalations on snapd side of things, as people have been using shim from proposed / ubuntu archive to build their gadgets with out dated kernel snaps and fail. Especially since the snapstore emailed them saying their stuff is out of date =)12:41
xnoxsil2100:  let's wait for a publisher cycle and check with rmadision12:41
juliankfrom proposed -> lol12:41
xnoxi know12:41
juliankupdates? it's only been out 2 hours12:42
xnoxthat's why i say, our users are amazing12:42
xnoxthe details are fuzzy, but they did break themselves12:42
juliankwhy would they get alerts that quickly12:42
xnoxi don't trust they are doing anything right12:42
xnoxsil2100:  the publishing records look good.12:43
sil2100seb128: phew, I'll get to that eventually! Thanks!12:44
juliankxnox: Just write them "Thank you for testing our pre-release software. Problems like this can be expected. We value your feedback"12:46
sil2100eh, need to use copy-package directly since the kernels got already removed from -proposed12:47
xnoxjuliank:  i like it. are you learning your customer phasing writting skills with help of chatgpt?12:48
juliankxnox: yes12:48
xnoxsil2100:  yeah -updates to -security12:48
juliankxnox: But I should pass it through it.12:49
juliankI asked ChatGPT:12:50
juliank> Customers had issues installing new 'shim' package. But this is pre-release software. Write a reply saying that we thank them for testing pre-release software and problems are expected.12:50
juliankxnox: this is the chatgpt version https://paste.ubuntu.com/p/G7jkKDM2yC/12:50
xnoxlove it12:51
juliankbasically I write a bug reply, I jsut tell ChatGPT to reformulate it12:51
-queuebot:#ubuntu-release- Unapproved: linux-lowlatency-hwe-5.19 (jammy-security/main) [5.19.0-1017.18~22.04.1 => 5.19.0-1017.18~22.04.1] (no packageset) (sync)12:51
-queuebot:#ubuntu-release- Unapproved: linux-restricted-modules-lowlatency-hwe-5.19 (jammy-security/restricted) [5.19.0-1017.18~22.04.1+1 => 5.19.0-1017.18~22.04.1+1] (no packageset) (sync)12:51
-queuebot:#ubuntu-release- Unapproved: linux-meta-lowlatency-hwe-5.19 (jammy-security/main) [ =>] (no packageset) (sync)12:51
-queuebot:#ubuntu-release- Unapproved: linux-signed-lowlatency-hwe-5.19 (jammy-security/main) [5.19.0-1017.18~22.04.1 => 5.19.0-1017.18~22.04.1] (no packageset) (sync)12:51
juliankCan we integrate chatgpt into launchpad?12:52
juliank"Suggested rewording"12:52
-queuebot:#ubuntu-release- Unapproved: linux-restricted-signatures-lowlatency-hwe-5.19 (jammy-security/restricted) [5.19.0-1017.18~22.04.1+1 => 5.19.0-1017.18~22.04.1+1] (no packageset) (sync)12:52
juliankYou can give it one statement, it transforms it into a knowledgebase entry, a blog post, and a press release with fake c-level quotes.12:53
sil2100xnox: I guess those look fine ^ ?12:53
xnoxit is there in the queue12:54
xnoxi guess queubot is slow12:54
juliankYummy kernels12:54
xnoxsil2100:  yes please approve those syncs, they look good12:54
-queuebot:#ubuntu-release- Unapproved: accepted linux-lowlatency-hwe-5.19 [sync] (jammy-security) [5.19.0-1017.18~22.04.1]12:55
-queuebot:#ubuntu-release- Unapproved: accepted linux-restricted-modules-lowlatency-hwe-5.19 [sync] (jammy-security) [5.19.0-1017.18~22.04.1+1]12:55
-queuebot:#ubuntu-release- Unapproved: accepted linux-signed-lowlatency-hwe-5.19 [sync] (jammy-security) [5.19.0-1017.18~22.04.1]12:55
-queuebot:#ubuntu-release- Unapproved: accepted linux-meta-lowlatency-hwe-5.19 [sync] (jammy-security) []12:55
-queuebot:#ubuntu-release- Unapproved: accepted linux-restricted-signatures-lowlatency-hwe-5.19 [sync] (jammy-security) [5.19.0-1017.18~22.04.1+1]12:55
juliankfwiw, I only see linux-generic
juliankon archive.ubuntu.com from my hetzner vps12:55
xnoxabove was only lowlatency stuff12:56
xnoxhwe-5.19 is separate12:56
juliankjust src:linux12:57
juliankI confuse that and the 66 in proposed is the next cycle12:57
juliankI think I should take a break :D12:58
juliankGet some espresso into me12:58
-queuebot:#ubuntu-release- Unapproved: linux-hwe-5.19 (jammy-security/main) [5.19.0-32.33~22.04.1 => 5.19.0-32.33~22.04.1] (no packageset) (sync)13:02
-queuebot:#ubuntu-release- Unapproved: linux-restricted-modules-hwe-5.19 (jammy-security/restricted) [5.19.0-32.33~22.04.1 => 5.19.0-32.33~22.04.1] (no packageset) (sync)13:02
-queuebot:#ubuntu-release- Unapproved: linux-meta-hwe-5.19 (jammy-security/main) [ =>] (no packageset) (sync)13:02
-queuebot:#ubuntu-release- Unapproved: linux-restricted-signatures-hwe-5.19 (jammy-security/restricted) [5.19.0-32.33~22.04.1 => 5.19.0-32.33~22.04.1] (no packageset) (sync)13:02
xnoxsil2100:  our dashboard is happy with jammy/linux-lowlatency-hwe-5.19 it is now marked as phase: Complete13:02
xnoxsil2100:  the hwe-5.19 copies look good but i don't see linux-signed-hwe-5.1913:03
sil2100Should be there, maybe it lagged a bit13:05
juliankHeh proposed -> lol from me is strong as I have src:linux src:linux-signed src:linux-meta src:apt src:python-apt pinned up for proposed :D13:05
-queuebot:#ubuntu-release- Unapproved: linux-signed-hwe-5.19 (jammy-security/main) [5.19.0-32.33~22.04.1 => 5.19.0-32.33~22.04.1] (no packageset) (sync)13:05
juliankSome days I run kernels directly from  o=LP-PPA-canonical-kernel-team-unstable13:06
juliankwhat about cd-boot-images?13:06
-queuebot:#ubuntu-release- Unapproved: accepted linux-hwe-5.19 [sync] (jammy-security) [5.19.0-32.33~22.04.1]13:06
-queuebot:#ubuntu-release- Unapproved: accepted linux-restricted-modules-hwe-5.19 [sync] (jammy-security) [5.19.0-32.33~22.04.1]13:06
-queuebot:#ubuntu-release- Unapproved: accepted linux-signed-hwe-5.19 [sync] (jammy-security) [5.19.0-32.33~22.04.1]13:06
-queuebot:#ubuntu-release- Unapproved: accepted linux-meta-hwe-5.19 [sync] (jammy-security) []13:06
-queuebot:#ubuntu-release- Unapproved: accepted linux-restricted-signatures-hwe-5.19 [sync] (jammy-security) [5.19.0-32.33~22.04.1]13:06
juliankcan they be released?13:07
xnoxsil2100:  signing tarballs should be rejected13:08
sil2100I think those should be fine, I don't think anyone besides our cdimage/debian-cd infra uses those13:08
xnoxsil2100: nothing may use them =) as they will contiain invalid broken signatures ;-)13:08
juliankI'm sitting around with half my issues blocked and people looking at me sadly because they haven't moved pulse after pulse13:08
xnoxsil2100:  pending publications all look good.13:09
sil2100xnox: I meant cd-boot-images!13:09
sil2100xnox: the signing tarballs I'll reject indeed ;p13:10
juliankI haven't set verfication-done on those, but they are verifying themselves basically13:10
xnoxyes cd-boot-images is fine to release in jammy and kinetic13:10
sil2100I didn't test any of the images on secure-boot though13:10
sil2100But as long as they're built, I guess they're good13:11
juliankthey're the same binaries as in the main archive13:11
juliankwell you know what I mean13:11
juliank(for secure boot anyway)13:11
xnoxwell the cd one is like totally different (smirk)13:12
juliankAnyhow we'll only find out when we build images13:12
juliankNormally images build with -proposed enabled but I guess they don't do right now?13:12
juliankWhat I know is that they are not revoked13:13
sil2100All the recent images were built with -proposed, yes13:13
xnoxarchive should publish quicker13:14
juliankwell then the packages were tested by iso testing13:14
xnoxstill pending13:14
coreycbhello sru team, I'd like to see if we can get a review of aodh in jammy unapproved and neutron in focal unapproved please13:17
xnoxcoreycb:   seeded-in-ubuntu aodh13:18
xnoxaodh-doc (from aodh) is seeded in:13:18
xnox  kubuntu: supported13:18
xnox  lubuntu: supported13:18
xnox  ubuntu-budgie: supported13:18
xnox  ubuntu-mate: supported13:18
xnox  ubuntu: supported13:18
xnox  ubuntukylin: supported13:18
xnox  xubuntu: supported13:18
xnoxdoes not affect images.13:18
xnoxbut busy with point release stuff.13:18
xnoxafk to lunch13:19
coreycbxnox: ah, sorry.. is that soft freeze related for jammy?13:20
coreycbI assume so13:20
sil2100New livecd-rootfs incoming, will self-accept it like a terrible person13:28
-queuebot:#ubuntu-release- Unapproved: livecd-rootfs (jammy-proposed/main) [2.765.15 => 2.765.16] (desktop-core, i386-whitelist)13:28
-queuebot:#ubuntu-release- Unapproved: accepted livecd-rootfs [source] (jammy-proposed) [2.765.16]13:33
xnoxlinux-hwe-5.19 phase: Complete13:50
xnoxsil2100:  juliank: https://bugs.launchpad.net/ubuntu/+source/shim/+bug/1996503/comments/3313:51
-ubottu:#ubuntu-release- Launchpad bug 1996503 in shim-signed (Ubuntu) "shim 15.7-0ubuntu1" [Undecided, In Progress]13:51
-queuebot:#ubuntu-release- Unapproved: accepted net-snmp [source] (kinetic-proposed) [5.9.3+dfsg-1ubuntu1.3]14:06
-queuebot:#ubuntu-release- Unapproved: accepted net-snmp [source] (jammy-proposed) [5.9.1+dfsg-1ubuntu2.5]14:07
sil2100I'll resume phasing soon in that case14:20
Eickmeyersil2100: I see Studio is still very broken due to linux-lowlatency.15:17
sil2100Eickmeyer: it should be good now!15:19
sil2100Eickmeyer: I missed one line in livecd-rootfs, re running the build just now as I see the package is now published15:19
Eickmeyersil2100: Ok, good to know. I'll keep an eye. I was running out of tears to cry XD15:20
sil2100Ok, requested build, fingers crossed!15:26
* Eickmeyer watches while sipping coffee15:28
juliankxnox: shenanigans meet?15:34
-queuebot:#ubuntu-release- Unapproved: shim-signed (focal-updates/main) [1.40.9 => 1.40.7] (core) (sync)15:55
-queuebot:#ubuntu-release- Unapproved: shim (focal-updates/main) [15.7-0ubuntu1 => 15.4-0ubuntu9] (core) (sync)15:56
sil2100vorlon: I guess those look correct? https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=shim15:56
xnoxsil2100:  that is correct please accept16:01
vorlonsil2100: you put it in unapproved? :)  yeah that looks correct to me16:01
sil2100I am extra cautious today ;)16:01
-queuebot:#ubuntu-release- Unapproved: accepted shim-signed [sync] (focal-updates) [1.40.7]16:02
-queuebot:#ubuntu-release- Unapproved: accepted shim [sync] (focal-updates) [15.4-0ubuntu9]16:02
-queuebot:#ubuntu-release- Unapproved: base-files (jammy-proposed/main) [12ubuntu4.2 => 12ubuntu4.3] (core, i386-whitelist)16:14
sil2100vorlon, bdmurray: can one of you take a look at base-files? Just pushed it to unapproved ^16:15
sil2100Eickmeyer: http://cdimage.ubuntu.com/ubuntustudio/jammy/dvd/20230216.1/ \o/16:16
EickmeyerGonna sync that biznezz when I get back from taking my son to school this morning.16:16
sil2100Thanks o/16:16
xnoxvorlon:  short ubuntu queue is running tests from 8 days ago. is there a shorter than short queue please?16:19
xnoxi need kernels, dkms, nvidia out of proposed to start uploading new ones, as the -release pocket has useless stuff at this point.16:20
xnoxalso i need dkms to migrate, which does have regression w.r.t. zfs test case handling, so i am thinking to upload dkms again since it shouldn't matter since all the tests are still not running.16:35
juliankQA team can hack kernel stuff to the top of the queue16:35
juliankIt's a bit annoying16:35
juliankdump queue in file, requeue the kernel things, and then requeue the others again16:36
juliankbdmurray: autopkgtest.u.c should use priority queues that would be easier, then we can bump priorities around; and you could penalize failing tests by requueing them with lower priority16:37
juliank-> https://www.rabbitmq.com/priority.html16:37
juliankI don't know I might have created the queues priority ready16:38
juliankthere should be like priority = some-function(number of retries, age, base priority) + bump16:39
juliankah no the queues had to be recreated from scrach with priorities I don't think I did that16:40
juliankxnox: we still aob meeting16:40
xnoxwe need to reupload all lrm because of change of nvidia; we need to upload all kernels due to signing change; we need to reupload dkms because it has error codes regressions making zfs-linux false negative. all of the above triggers hundreds of tests each in an explosive matrix.16:41
xnoxand it is useless, as we have tested these things by hand locally, because of the testing lag, hence it would be nice to release dkms things to with skip test16:41
juliankyeah it sucks that we are time based and don't have priorities right now16:42
xnoxjuliank:  should i gate crash?16:43
juliankwe're closing16:43
xnoxvorlon:  come back =)16:43
xnoxi really want to put dkms into sync blacklist16:50
vorlonxnox: why?  It is currently in sync with Debian; do you want to freeze it there?16:52
vorlonginggs: scipy-triggered tests down from 3170 to 1626 and dropping16:54
ginggsvorlon: thanks!16:54
xnoxvorlon:  because 7 in a row uploads syncs from debian caused regressions across all of our dkms and autopkgtest infrastructure generating over 1000+ failed autopkgtests builds in a row, and it is still broken.16:54
=== andypandy_ is now known as andypandy
xnoxliterary new upstream release; and debian revision updates have been a dumpster fire in both debian and ubuntu autopkgtest infra16:54
xnoxand it is still bad16:55
-queuebot:#ubuntu-release- Unapproved: rejected base-files [source] (jammy-proposed) [12ubuntu4.3]16:55
xnoxand somehow debian maintainers are trigger happy to sync dkms package to ubuntu, which sets back kernel team trying to migrate a kernel.16:55
xnoxLocutusOfBorg:  can you stop uploading dkms to ubuntu please?16:55
xnoxas at this point, i sort of on the verge of reverting back to what we currently have in lunar-release.16:56
bdmurrayI'm in another meeting16:57
-queuebot:#ubuntu-release- Unapproved: thermald (kinetic-proposed/main) [2.5.1-1 => 2.5.1-1ubuntu1] (core)17:15
vorlonxnox: if dkms has regressions, please do upload it to fix; we can filter stale test requests out of the queue17:16
vorlonxnox: how does a new dkms package in -proposed which is broken and won't pass autopkgtests impact the kernel team's migration?17:17
-queuebot:#ubuntu-release- Unapproved: accepted base-files [source] (jammy-proposed) [12ubuntu4.3]17:18
vorlonbdmurray: why does your python-apt/jammy have regressed autopkgtests for apport and aptdaemon?17:18
bdmurrayHow could it have completed tests? ;-)17:19
vorlonbdmurray: because the jammy queue is shorter17:19
vorlonbdmurray: previous successful autopkgtests in jammy were Jan 18, so not an issue of long-term bitrot in stable release17:20
vorlonbdmurray: I'm retrying them with --no-proposed anyway17:21
bdmurraySystemError: E:Failed to fetch http://ddebs.ubuntu.com/dists/focal/main/binary-amd64/Packages.xz  File has unexpected size (514024 != 513848). Mirror sync in progress? [IP: 80]17:22
bdmurrayHashes of expected file:17:22
bdmurrayJust retry it17:22
bdmurrayOr that was apport17:22
bdmurrayLoL that dbus namehasnoowner error appears in the aptdaemon autopkgtest17:23
bdmurrayAnyway, I'd just retry both of them.17:24
LocutusOfBorgxnox, it was buggy autopkgtests for itself.17:25
LocutusOfBorgand it was uploaded in Debian 3 days ago17:25
LocutusOfBorgvorlon, please sfepy/i386 hint please?17:27
LocutusOfBorgalso dkms/i386 please... (I'm asking about autopkgtests being uninstallable there)17:30
arraybolt3Why on earth does lxqt-policykit trigger autopkgtests for riseup-vpn and update-manager? I seriously doubt either depends on lxqt-policykit directly or indirectly, and similarly doubt that it depends on either one of those.18:20
LocutusOfBorgmigrating python3-distutils/3.11.2-2/amd64 to testing makes python3.10-full/3.10.9-1/amd64 uninstallable18:31
LocutusOfBorgdoko, ^^ something didn't work in version relax?18:31
LocutusOfBorgor maybe it just needs an hint...18:32
bdmurrayarraybolt3: update-manger does have an alternate dep on lxqt-policykit - verified using apt-cache show update-manager18:42
arraybolt3Gah, I didn't think about that, because it needs a polkit agent. And so does Riseup it looks like. Sorry.18:44
bdmurrayI'm keen not to run unnecessary tests so keep your eyes peeled!18:53
vorlonLocutusOfBorg: dkms/i386 badtested.  going to look at sfepy19:45
-queuebot:#ubuntu-release- Unapproved: dovecot (jammy-proposed/main) [1:2.3.16+dfsg1-3ubuntu2.1 => 1:2.3.16+dfsg1-3ubuntu2.2] (ubuntu-server)19:45
-queuebot:#ubuntu-release- Unapproved: accepted dnsmasq [source] (jammy-proposed) [2.86-1.1ubuntu0.2]19:54
-queuebot:#ubuntu-release- Unapproved: accepted tcpdump [source] (kinetic-proposed) [4.99.1-4ubuntu0.1]20:15
-queuebot:#ubuntu-release- Unapproved: accepted tcpdump [source] (jammy-proposed) [4.99.1-3ubuntu0.1]20:16
-queuebot:#ubuntu-release- Unapproved: software-properties (bionic-proposed/main) [ =>] (desktop-core, ubuntu-server)20:16
-queuebot:#ubuntu-release- Unapproved: accepted tcpdump [source] (focal-proposed) [4.9.3-4ubuntu0.2]20:17
-queuebot:#ubuntu-release- Unapproved: accepted tcpdump [source] (bionic-proposed) [4.9.3-0ubuntu0.18.04.3]20:18
-queuebot:#ubuntu-release- Unapproved: rejected rabbitmq-server [source] (focal-proposed) [3.8.2-0ubuntu1.4]20:19
vorlonbdmurray: aha I think I just figured it out, I've been running amqp-filter in a loop for the huge queues, but due to britney funnypants aggregation of triggers there are a number of tests in the non-huge queues; so I'll iterate over those now too20:24
vorlonbdmurray: in any event, this doesn't *directly* account for rabbit's reported number of queued tests going up and down, though it may be a side effect where rabbit just doesn't do a good job of consistently reporting its queue while things are in flux20:25
-queuebot:#ubuntu-release- New binary: mosdepth [amd64] (lunar-proposed/universe) [0.3.3+ds-2] (no packageset)20:44
vorlonbdmurray, ginggs: looks like we've finally converged (more or less) - 842 scipy-triggered tests remain20:50
-queuebot:#ubuntu-release- Unapproved: accepted nmap [source] (jammy-proposed) [7.91+dfsg1+really7.80+dfsg1-2ubuntu0.1]20:54
-queuebot:#ubuntu-release- Unapproved: accepted nmap [source] (focal-proposed) [7.80+dfsg1-2ubuntu0.1]20:54
-queuebot:#ubuntu-release- New binary: mosdepth [arm64] (lunar-proposed/universe) [0.3.3+ds-2] (no packageset)21:08
-queuebot:#ubuntu-release- New binary: mosdepth [armhf] (lunar-proposed/universe) [0.3.3+ds-2] (no packageset)21:10
-queuebot:#ubuntu-release- New binary: mosdepth [ppc64el] (lunar-proposed/universe) [0.3.3+ds-2] (no packageset)21:10
-queuebot:#ubuntu-release- New binary: mosdepth [riscv64] (lunar-proposed/universe) [0.3.3+ds-2] (no packageset)21:26
-queuebot:#ubuntu-release- Unapproved: ubuntustudio-default-settings (jammy-proposed/universe) [ =>] (ubuntustudio)22:48
xnoxvorlon:  because it appears that dkms is always pulled from proposed. as triggers are getting collated, and i cannot get my dkms and kernels to migrate without using dkms from proposed it seems.22:52
xnoxvorlon:  6611 number of enqued tests on i386 looks wrong, as surely we don't have that many source packages on i386, nor even that many arch:all packages usable on i386?22:55
xnoxwe have roughly 40,000 source packages, and the queues in total are for  roughly a quarter of them. are we still doing migration reference tests?22:56
LocutusOfBorgvorlon, sfepy ping? its not built on i386 at all...23:03
vorlonxnox: you know there's a glibc that still has tests in the queue, right23:04
xnoxand how useful is that?23:04
xnoxit's not like any of them will catch a glibc bug23:05
vorlonnot interested in you bikeshedding the release policy around testing in response to a service outage23:06
vorlonthe glibc testing DOES catch regressions23:06
vorlonyou can consider them glibc bugs or not, but they show real package breakage23:06
vorlonxnox: is there a bug ref for the current round of dkms breakage in -proposed? "breaks kernel testing" is sufficient reason to remove it from -proposed rather than letting in cook there, but a bug pointer is useful for documentation23:11
vorlonxnox: I see no open bugs in LP or Debian for breakage of dkms 3.0.10-6.  update_excuses shows most tests 'in progress' but passing on armhf23:24
vorlondoing some queue analysis, the current greatest offenders are in fact: numpy, python3-stdlib-extensions, python3.11, nodejs, link-grammar (some of these are spurious because of britney's weird and buggy trigger aggregation)23:43
vorlonpystemd ok this package name is terrible23:45
vorlonbdmurray, ginggs: there's no more obvious low-hanging fruit for stale test triggers.  it would be easier to do this analysis if britney consistently set sane triggers in the first place instead of what it does now23:51
bdmurrayvorlon: ack, thanks for looking23:53
bdmurrayThe autopkgtest cloud workers seem to be in better shape so we should be back to the state we were at before the infrastructure failure23:53

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!