[08:46] <LocutusOfBorg> doko, https://launchpadlibrarian.net/715376454/buildlog_ubuntu-noble-ppc64el.libunivalue_1.1.1+20191112-2_BUILDING.txt.gz looks like libunivalue needs some more optional symbols
[09:11] <schopin> @pilot in
[13:06] <dviererbe> @ahasenack I have seen the discussion and am currently trying to get the test suite to work. I already added a hello world test for the cgi handler. I have also seen that doku uploaded my WIP version. I will ask for a sonsor of an updated version once I get the test suite running and passing.
[13:12] <ahasenack> dviererbe: sounds great
[13:12] <ahasenack> dviererbe: would you like to perhaps create an upstream PR, linked to the upstream bug you created?
[13:12] <ahasenack> then it's also easier to comment
[13:13] <dviererbe> I will do that, just a minute...
[13:13] <ahasenack> thx
[13:22] <dviererbe> ahasenack: you can the PR here https://github.com/grisha/mod_python/pull/131
[13:22] -ubottu:#ubuntu-devel- Pull 131 in grisha/mod_python "Refactor patch for Python 3.12" [Open]
[13:23] <ahasenack> awesome
[13:26] <dviererbe> I also added you as a contributer to the repo, so you can push commits (if you want)
[13:26] <ahasenack> oh, you are upstream?
[13:26] <ahasenack> ah, to your fork
[13:26] <ahasenack> works too :)
[13:27] <dviererbe> yes
[13:27] <dviererbe> Could you help me figuring out how to get the test suite working? I get
[13:27] <dviererbe> Bad value for mod_python.version.HTTPD :
[13:27] <dviererbe> Bad value for mod_python.version.TESTHOME : <undefined>
[13:27] <dviererbe> Bad value for mod_python.version.TEST_MOD_PYTHON_SO : <undefined>
[13:27] <dviererbe> Please check your mod_python/version.py file
[13:27] <dviererbe> The proble is there is no mod_python/version.py file
[13:29] <ahasenack> is this run at build time?
[13:29] <ahasenack> pkg build time, I mean
[13:29] <ahasenack> and maybe make the PR against the fork, not against that patch, otherwise the diff looks small
[13:30] <ahasenack> as it only shows the find_spec/spec_from_file_location change
[13:33] <dviererbe> I did not see the tests in the build log, but my limited understanding of dh_make tells me that the tests should be run during build time
[13:33] <ahasenack> I'll check
[13:33] <dviererbe> I also reset the base for the fork
[13:35] <dviererbe> I see it in the log now "Cannot import mod_python.version. "
[13:36] <dviererbe> but still, I should be able to run it as an dep8 test
[13:38] <ahasenack> I see it
[13:38] <ahasenack> maybe it's complaining that the file doesn't exist, that's why the variables it's showing just before are empty
[13:40] <ahasenack> setup.py.in has a def generate_version_py() to generate it
[13:40] <ahasenack> :q
[13:46] <dbungert> qa-help: can you help me understand what's going on with the test retries on jupyter-notebook ?  When we explicitly trigger as jupyter-notebook/6.4.12-2.2ubuntu1 it's selecting jupyter-notebook/6.4.12-2.2 anyhow
[13:47] <Skia> good question, I've no idea right now
[13:47] <ahasenack> dbungert: saw that too, asked internally (in a slightly different way)
[13:47] <ahasenack> dbungert: before it was because 2.2ubuntu1 was FTBFS, but now it's correct
[13:48] <dbungert> right, I wonder if that old FTBFS info is cached or something and it refuses to use that version yet
[13:49] <ahasenack> and even with a trigger it's selecting the old version?
[13:49] <dbungert> yes
[13:49] <ahasenack> I don't remember if I saw that attempt
[13:49] <ahasenack> ok
[13:50] <dbungert> multiple test retries have been attempted on https://autopkgtest.ubuntu.com/packages/j/jupyter-notebook/noble/amd64, some explicitly with ubuntu1, and the few I have spot checked are all using 6.4.12-2.2
[13:51] <Skia> andersson1234: maybe you can help here ^ ?
[13:51] <ahasenack> armhf used the correct version sometimes: https://autopkgtest.ubuntu.com/packages/j/jupyter-notebook/noble/armhf
[13:51] <ahasenack> still failed, what's up this time...
[13:55] <dbungert> the NotebookUnixSocketTests failure was fixed with python3-requests-unixsocket
[14:04] <ahasenack> dviererbe: so two things
[14:04] <ahasenack> dviererbe: a) right after package build, lib/python/mod_python/version.py  exists and has correct content
[14:04] <ahasenack> dviererbe: b) somehow, that is stripped when the file is copied into the package. That copy has empty values
[14:04] <dviererbe> that is weird
[14:04] <ahasenack> dviererbe: c) (ok, 3 things) the test uses the version.py file from the installed package, not from the build directory. So it finds the empty values, and complains
[14:08] <ahasenack> and I didn't find yet what actually populates that file with values. The function in setup.py only generates a blank one, with only the module version populated, but not the rest
[14:09] <adrien> bryyce: hey, when I use ppa wait, it clears my terminal and I find it inconvenient; is that a known thing and is there a way to disable this behavior?
[14:10] <dviererbe> ahasenack: I do not think that setup.py is called during build time, because distutils is also removed with python 3.12 and would fail the build
[14:10] <ahasenack> it is called, according to deb build logs
[14:30] <adrien> bryyce: I tried various things; "| cat -v" helped but the output is slightly ugly, "| ansifilter" can help too but is not installed everywhere, "| col -b" worked and the output was less ugly than cat -v but still had traces of control chars; the best solution so far as been to set the TERM env var: TERM=dumb
[14:30] <ahasenack> dviererbe: I don't think these bundled tests are ok, at first glance. For once, they require write access to the python lib
[14:30] <ahasenack>         # the rest of this test requires write perms to site-packages/mod_python
[14:30] <ahasenack>         if os.access(version_path, os.W_OK):
[14:31] <ahasenack> although that might just be skipped
[14:31] <ahasenack> :q
[14:31] <ahasenack> ops
[14:31] <ahasenack> I'm jsut checking now what actually populates version.py
[14:31] <dviererbe> ahasenack: I am currently trying to get these tests to run as roo. I don't think that write permissions are an issue
[14:32] <dviererbe> *root
[14:32] <ahasenack> for dep8 then
[14:36] <ahasenack> doko: will we eventually remove bin:python3-distutils 3.11.5?
[14:39] <schopin> @pilot out
[14:43] <ahasenack> dbungert: got a pass with this trigger https://autopkgtest.ubuntu.com/packages/jupyter-notebook/noble/armhf
[14:43] <ahasenack> I'll repeat it for the other architectures
[14:43] <dbungert> outstanding
[14:43] <ahasenack> hmpf, would be useful it the link was ready to paste, but I have to construct it again
[14:44] <dviererbe> ahasenack: I am not sure how we can replace distutils. The recommended migration is setuptools and setuptools was removed with lunar See LP: #1998970
[14:44] -ubottu:#ubuntu-devel- Launchpad bug 1998970 in rawkit (Ubuntu) "Remove pypy from lunar" [Undecided, Fix Released] https://launchpad.net/bugs/1998970
[14:46] <ahasenack> dviererbe: I was thinking about a dep8 test like the smoke one, but for cgi. the bundled test suite might be too much effort for now
[14:46] <ahasenack> that version.py file being populated seems to be a distutils thing too
[14:46] <dviererbe> Well i already have a working hello world test
[14:46] <dviererbe> for the cgi handler
[14:46] <ahasenack> add a print of the environment to it, I'm curious if we are leaking something
[14:47] <ahasenack> I'll shelve this build-time-test pursuit for now
[15:01] <dviererbe> ahasenack: printing sys.modules gives me this https://paste.ubuntu.com/p/Pg4QPvRsRV/plain/
[16:47] <bryyce> adrien, yes use `ppa wait --log`.  Sorry that wasn't more discoverable, it's the last item in `ppa wait --help`.  Where did you first look to see how to disable that?  I can see about adding better docs at that spot.
[16:48] <adrien> oh, I was assuming --log would want to write to a file
[16:48] <adrien> and I most probably skipped reading the description in full
[16:49] <adrien> but also, besides the screen clearing, isn't printing the ongoing status to console the default behavior anyway?
[16:49] <bryyce> likely you should be able to set 'wait_logging' to false in your config file to change that behavior permanently locally
[16:50] <adrien> also, I expected that piping the output to another program would disable that behavior
[16:50] <adrien> I need to leave to prepare dinner unfortunately but I'll be back a bit later
[16:50] <bryyce> yes it used to be, but other users found _that_ to be inconvenient and at the time I didn't know if anyone would prefer the current behavior (but I left this config option just in case)
[16:52] <adrien>  :D
[16:52] <adrien> users are convenient
[16:52] <bryyce> <3 users
[16:53] <bryyce> I'm not sure on use cases for saving log output to a file but that's doable via `ppa wait --log | tee ...` so that's as far as my thinking got there
[16:54] <bryyce> adrien, I'm glad you asked about this though, I was curious if the non-clearing function was useful for some people
[16:54] <adrien> I'm not saying I would use such an option, rather than it exists in that form in other apps and was expecting something similar here
[16:55] <adrien> tbh, I don't see how clearing is useful to some :)
[16:55] <adrien> clearly, the people who think or do differently than I do are monsters
[16:55] <bryyce> haha
[16:56] <bryyce> well the argument was that one would leave it running in a terminal to visually note changes
[16:56] <adrien> but also, I'm wondering if I get the same behavior: I end up with an empty screen which I can't scroll up
[16:57] <bryyce> yeah that's a "feature" of the clear screen system call
[16:58] <adrien> I might try some output reworking; there's something I'd like to try (but I have to leave now :) )
[16:59] <bryyce> adrien cool, I look forward to hearing more
[18:18] <doko> ahasenack: yes, when 3.11 is removed from python3-defaults as supported
[18:18] <ahasenack> we might have a new influx of bugs with import failures then
[18:18] <ahasenack> right now, some are still "working" because distutils is there, from 3.11
[18:19] <ahasenack> but that isn't importable from python 3.12 anyway, so shouldn't be many
[18:27] <doko> if we really need, we could do something like a python3-zombie-distutils, but I would like to avoid that
[18:52] <bdmurray> dbungert: your jupyter-notebook issues should be sorted now e.g. feel fre to kick off a test again
[18:53] <dbungert> bdmurray: thanks.  Are tests that are already queued OK?
[18:53] <bdmurray> yes
[19:28] <dbungert> bdmurray: test run c6bf2236-3559-4bfb-b936-32b20551a907 was queued after this conversation, but is still using the wrong version of jupyter-notebook in spite of triggers.
[19:34] <bdmurray> dbungert: I don't know how to search for a uuid yet what arch release package?
[19:34] <dbungert> https://autopkgtest.ubuntu.com/packages/j/jupyter-notebook/noble/arm64
[19:35] <dbungert> top entry
[19:37] <bdmurray> dbungert: I don't see jupyter-notebook in those triggers
[19:37] <dbungert> thank you
[19:37] <bdmurray> that's an unexpected response
[19:38] <dbungert> I will requeue with a fixed trigger
[19:38] <bdmurray> getting warmer ;-)
[19:39] <dbungert> I was wrong and Brian was wright
[19:39] <dbungert> right
[20:01] <ahasenack> so besides jupyter,
[20:01] <ahasenack> remaining issues seem to be ansible, one libapache2-mod-python (ppc64el, runners were odd there, new runs are queued)
[20:01] <ahasenack> and I see one update-motd armhf-only red
[20:02] <ahasenack> my "favorite" arch
[20:02] <ahasenack> "fcntl() invalid argument"
[20:02] <ahasenack> huh
[20:05] <doko> yes, I looked at that already, same "huh"
[20:05] <doko> ansible/armhf is also an armhf-only issue
[20:07] <ahasenack> it's hard to replicate the armhf environment that LP uses
[20:07] <ahasenack> I have this in my notes: https://git.launchpad.net/autopkgtest-cloud/tree/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/armhf-lxd.userdata#n76
[20:07] <ahasenack> on how autopkgtests launch armhf containers
[20:08] <ahasenack> bdmurray: do you know what the host is for the armhf autopkgtest containers? focal? jammy? running kernel?
[20:09] <bdmurray> ahasenack: focal
[20:12] <dbungert> running kernel is visible near the top of the test log
[20:12] <ahasenack> I'll try with my mantic pi4 first (using an armhf container launched like in the git url above)
[20:13] <ahasenack> if that doesn't show the error, then I'll try a bare metal arm64 running focal deployed with maas, but that will take much longer
[20:13] <ahasenack> lxc launch ubuntu-daily:noble/armhf pi4:n-armhf-lp -c raw.lxc="apparmor.profile=unconfined" -c raw.lxc="seccomp.profile=" -c security.nesting=true
[20:13] <ahasenack> trying that first
[20:15] <waveform> FYI, you can run focal on the pi4 (if you want something as close as possible to the autopkgtest setup)
[20:17] <bdmurray> Let me know if there is anything I can do to help
[20:21] <ahasenack> waveform: yep, but this is a "production" pi4 :)
[20:21] <ahasenack> don't want to boot it on another OS right now
[20:22] <ahasenack> actually, let me in parallel find that nice arm64 system that can spawn a armhf container
[20:22] <ahasenack> or, oh, maybe canonistack is working well enough
[20:29] <ahasenack> hm, no go for canonistack
[20:29] <ahasenack> | 88479cf3-ff4b-4eac-91be-e87c8f5dfd20 | n-arm64      | ERROR  | -          | NOSTATE
[20:46] <ahasenack> it passed locally, with the mantic kernel
[20:59] <ahasenack> we could debian/tests/wait-for-apt-update-to-finish || :
[20:59] <ahasenack> that script is just to wait for an apt lock in case there is an apt running
[20:59] <ahasenack> it's to help with making the test more resilient, ironically
[21:00] <ahasenack> the actual test is calling update-motd after that
[21:00] <ahasenack> that being said,
[21:00] <ahasenack> it's only failing on python 3.12
[21:00] <ahasenack> migration-reference/0 works (with py3.11)
[21:00] <ahasenack> and that is a python script
[21:00] <ahasenack> ok, let's check what changed in 3.11 -> 3.12 in what that script does
[21:09]  * ahasenack has a hunch
[21:12] <ahasenack> despite the multiple reruns and triggers, all tests from https://autopkgtest.ubuntu.com/packages/update-motd/noble/armhf were against python 3.11, not python 3.12
[21:12] <ahasenack> so it looks like it was that issue that just got fixed
[21:21] <ahasenack> ah, no, 3.11 is the update-motd version, not python
[21:21] <ahasenack> hunch busted
[21:31] <ahasenack> ok, at least I can reproduce it
[21:32] <ahasenack> works with python 3.11, fails with 3.12: https://pastebin.ubuntu.com/p/Ccr7SB4sRN/
[21:32] <ahasenack> I'll file a bug to get this recorded, since it's eod
[21:32] <ahasenack> ah, there is one, excellent
[21:41] <ahasenack> doko: about update-motd, I added some interesting info to https://bugs.launchpad.net/ubuntu/+source/update-motd/+bug/2054369
[21:41] -ubottu:#ubuntu-devel- Launchpad bug 2054369 in update-motd (Ubuntu) "update-motd fails autopkg test on armhf in noble, blocking python3-defaults" [Undecided, Confirmed]
[21:41] <ahasenack> but I'm eod now, can continue tomorrow
[21:41] <ahasenack> I have an instance up on canonistack where this is happening, if somebody wants to take a look while I'm gone
[21:42] <ahasenack> I'm trying to udnerstand where the magic struct.pack('hhllhh', fcntl.F_WRLCK, 0, 0, 0, 0, 0) comes from
[21:42] <mwhudson> ahasenack: that strace output is pretty yikes
[21:42] <ahasenack> something is being packed incorrectly
[21:51] <doko> kanashiro: please forward the swig issue to debian
[21:52] <kanashiro> I'll forward all the patches I am adding to debian
[23:06] <waveform> ahasenack, just had a bit of a google around and ran across the fcntl module tests in python (https://github.com/python/cpython/blob/main/Lib/test/test_fcntl.py) -- of particular interest is get_lockdata from line 53
[23:07] <waveform> following that down we can see that on linux it'll use the struct 'hh'+start_len+'hh'; start_len is ll on platforms without os.O_LARGEFILE, but qq on platforms with (which would suggest the struct in use is too short on armhf?)
[23:08] <waveform> (assuming armhf has os.O_LARGEFILE -- I'll try and check that in a bit)
[23:16] <waveform> yes, armhf has os.O_LARGEFILE so it'll be using a struct that's too short
[23:35] <ahasenack> waveform: why does the call pass in python 3.11 I wonder?
[23:36] <waveform> dumb luck would be my guess; the fcntl call will be reading too much from the struct and it just *happens* to run across a load of garbage at the end in 3.12 while it didn't in 3.11
[23:37] <ahasenack> but yeah, changing ll to qq passes with python 3.12
[23:37] <ahasenack> this all sounds like a VERY CONVOLUTED way to check a lockfile :D
[23:37] <waveform> yes ... I'm wondering if it can just be replaced with a call to flock instead ...
[23:38] <waveform> the test is also ... a bit weird. It's setting NONBLOCK on the fd, then does an explicit blocking wait for the lock ...
[23:38] <waveform> (not to mention it's testing for -1 on the fcntl return, which is also wrong -- if fcntl fails it just raises OSError; not that that makes any difference, the script will still bail with rc 1)
[23:39] <waveform> still, it all kinda screams "re-write me nicely"
[23:40] <ahasenack> hm, on my amd64 mantic laptop, python 3.11,
[23:40] <ahasenack> $ python3 -c "import os; print(os.O_LARGEFILE)"
[23:40] <ahasenack> 0
[23:40] <ahasenack> on armhf (host arm64),
[23:40] <ahasenack> # python3.11 -c "import os;print(os.O_LARGEFILE)"
[23:40] <ahasenack> 131072
[23:40] <ahasenack> same with py 3.12 there
[23:40] <waveform> yeah, but it's just testing for the *existence* of O_LARGEFILE, not for its value
[23:41] <ahasenack> oh, true
[23:41] <ahasenack> so it should be qq always?
[23:41] <ahasenack> in our cases
[23:41] <waveform> on the platforms we care about, apparently yes
[23:42] <ahasenack> apparently this check was added because of freedombox
[23:42] <ahasenack> which was now removed d/t/control
[23:43] <ahasenack> https://git.launchpad.net/ubuntu/+source/update-motd/commit/?id=c060dd002473f5700d8db3916bd8da122ec94fc3
[23:43] -ubottu:#ubuntu-devel- Commit c060dd0 in ubuntu/+source/update-motd "3.6-0ubuntu9 (patches unapplied) import/3.6-0ubuntu9"
[23:44] <ahasenack> and freedombox was removed: https://git.launchpad.net/ubuntu/+source/update-motd/commit/?id=1fb924d7cb573df5202a16d93ebcae1e20d62e2a
[23:44] -ubottu:#ubuntu-devel- Commit 1fb924d in ubuntu/+source/update-motd "3.11 (patches unapplied) HEAD import/3.11 ubuntu/noble-proposed ubuntu/noble-devel ubuntu/noble ubuntu/devel"
[23:44] <ahasenack> (why was it a test dep for update-motd anyway??)
[23:44] <ahasenack> weird
[23:45] <waveform> yeah, everything about this is bizarre ... I'm just looking at the fcntl.flock/lockf code in CPython and I'm *fairly* sure all this could just be replaced with one of those calls
[23:45] <waveform> (assuming we even want to keep the test)
[23:46] <ahasenack> I'll setup a run without it
[23:46] <ahasenack> or, for something safer(?), switch to qq
[23:46] <waveform> that would be the minimal safe change
[23:53] <ahasenack> oh, freaking native packages