/srv/irclogs.ubuntu.com/2021/05/04/#ubuntu-release.txt

dokovorlon: did you only reueue the ones for focal, or everything? just seeing the ones in impish triggered for gcc-7/gcc-8 being in progress as well04:00
RikMillslooks like just uploaded netplan.io is the latest thing to hit the missing riscv64 build problem. I guess what was said in #launchpad is the latest on this?08:38
slyonRikMills: Yeah. I was wondering why the riscv build is missing..08:39
RikMillsif you hadn't seen, for some reason the builders are not doing riscv64 builds for arch: linux-any08:40
slyonokay.08:50
RikMillsexcuses is seeing these as missing build, so blocking migration. so I hope we get a fix soonish :/08:55
-queuebot:#ubuntu-release- Unapproved: shim-signed (hirsute-proposed/main) [1.46 => 1.47] (core) (sync)09:11
-queuebot:#ubuntu-release- Unapproved: shim (hirsute-proposed/main) [15.4-0ubuntu1 => 15.4-0ubuntu2] (core) (sync)09:11
-queuebot:#ubuntu-release- Unapproved: shim-signed (groovy-proposed/main) [1.45 => 1.47] (core) (sync)09:11
-queuebot:#ubuntu-release- Unapproved: shim (groovy-proposed/main) [15+1552672080.a4a1fbe-0ubuntu2 => 15.4-0ubuntu2] (core) (sync)09:11
* enyc meows09:38
enychrrm, hirsute stuff appearing in -release channel ....  does this channel remain 21.04hirsute  related for a while now...?09:38
xnoxenyc:  all releases always appear here =)10:16
xnoxenyc: so you see mentions for SRUs fro focal bionic here too.10:16
enycxnox: aaah, i see.10:18
enycxnox: hrrm, new puzzle -- why did a hueeeeeeeg list of  price-   changes all appear all at ance earlier?10:18
xnoxenyc: precise has had ESM for a few years now. However, that is now stopping. It looks like all the precise ESM updates are now being made public, such that they end up in the public old-releases.ubuntu.com archive as part of precise complete sunset (end of ESM).10:34
xnoxcause those who have paid for ESM in the past, need to still be able to find sources and binaries forever.10:35
enycxnox: hrrm that mkes sense! =).10:35
xnoxwhich will then be archived onto tape and taken to cold offline storage too.10:35
tseliot_vorlon, the i386-whitelist doesn't seem to apply to the nvidia 465 driver in this PPA (soon to be in the archive too) in Groovy and Focal (Hirsute and Bionic are covered): https://launchpad.net/~oem-solutions-group/+archive/ubuntu/nvidia-driver-staging/+packages12:13
rbalintdoko, python3-lib2to3 in impish-proposed needs python3:any (>= 3.9.5-0~), i think a python3-defaults upload is missing13:43
rbalintdoko, it makes systemd ftbfs13:43
dokook, uploading13:44
dokohmm, better I soften the requirement13:45
rbalintyes, possibly13:50
juliankLaney: did you just break autopkgtest.u.c?14:00
juliankLaney: now it's back but it just had server error, going to read logs14:00
juliankstill takes 16s to load the page in chrome14:02
juliankthe / page14:02
juliankbut my curl needs 2s, so ...14:03
juliankah now it got a 10s one too14:03
julianktime to rewrite autopkgtest-web in Go :D14:04
Laneyheh14:46
Laneydon't accuse me >:(14:46
Laneyjuliank: I bet some of the sql queries could be optimised, that's what it spends it time doing really14:50
Laneyor the db itself14:50
julianksqlite was a suboptimal choice maybe14:50
juliankbut worth profiling the queries14:51
-queuebot:#ubuntu-release- Unapproved: neutron (bionic-proposed/main) [2:12.1.1-0ubuntu4 => 2:12.1.1-0ubuntu7] (openstack, ubuntu-server)15:18
-queuebot:#ubuntu-release- Unapproved: neutron (bionic-proposed/main) [2:12.1.1-0ubuntu4 => 2:12.1.1-0ubuntu4.1] (openstack, ubuntu-server)16:03
ddstreeti just uploaded this neutron, who keeps uploading the increasing version numbers?16:04
ddstreetcoreycb are you uploading neutron, specifically the ...ubuntu7 version?16:04
ddstreetcoreycb looks like your gpg key, are you ok if i ask for deletion of the ubuntu6 and ubuntu7 uploads, so they can review my ubuntu4.1 upload? it's properly versioned and includes the correct changelog in the changes file16:09
ddstreetbdmurray if you're on sru shift already today, could you delete the neutron uploads in bionic with versions ...ubuntu6 and ...ubuntu7, and review the upload in bionic with version ...ubuntu4.116:09
ddstreetthe ubuntu4.1 and ubuntu7 uploads have the same patch content, the only diffs are version suffix and i put dep3 info into the added patches16:10
waveformjuliank, yes -- the (single as far as I can tell?) query on the front-page would benefit from the addition of an index on result(run_id desc)16:14
juliankwaveform: we can play with that!16:15
waveformjuliank, it makes a fair difference to the explain output: https://paste.ubuntu.com/p/3PCznCdnKF/16:18
juliankwaveform: I'll do benchmark16:19
juliankWhat I think though is that we need some output caching16:19
juliankI don't know if Apache has that, I use nginx on my site and configure fastcgi output caching there16:20
waveformI'm always rather wary of caching when I can wring a bit more out of the (poor, usually horribly abused!) database16:20
juliankI don't know how many requests we get, but like a 30s cache of /running and / could be useful16:22
juliankHuh but running just opens a json file and dumps it, why is it so slow16:23
waveformhow big's the json?16:24
waveformhmm, it is using the built-in the json parser too so if it's particularly large that might be another cheap win (switching to simplejson or some such)16:25
Laneylooking forward to merge requests ;-)16:26
Laneysome well selected indexes would be great16:27
waveformLaney, switching the json import I could certainly MP but fiddling with the database structure ... I can happily suggest things but I've yet to figure out where its structure is defined (unless it's just publish-db?)16:28
juliankI have code!16:29
Laneywaveform: init_db() in utils.py or publish-db depending on which table you're after (yeah ...)16:31
Laneysounds like juliank is saving you though16:31
juliankLaney: Should be it https://code.launchpad.net/~juliank/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/40221016:33
juliankLaney: eek, I pushed it to main repo16:33
juliankLaney: I have wrong remote set up :(16:34
Laneygit revert!16:34
Laneynah it's ok, let's just try it16:35
juliankLaney: :)16:36
Laneyfix that though 😬16:36
juliankLaney: I have now :)16:36
* waveform peeks through his fingers to see if his inadvertantly pushed suggestion blows up autopkgtest ...16:36
Laneylol16:37
Laneywe have staging for testing such things16:37
Laneyusually not things directly pushed to tha main branch though :p16:37
Laneyhttps://autopkgtest.staging.ubuntu.com/ I mean, not broken ...16:38
juliankLaney: What changes is publish-db and download-{all-,}results really :D16:40
juliankLaney: What do I need to do aside from pushing, build the charm locally and then deploy it from wendigo?16:40
Laneyjuliank: I did it already, but https://autopkgtest-cloud.readthedocs.io/en/latest/deploying.html#update-the-code16:41
juliankLaney: ah ok, I have do the charm publishing locally and then run mojo run -m manifest-upgrade on wendigo, though for future things?16:41
Laneypushing to the charm store is the main bit which makes the manifest able to see it16:41
juliankLaney: Can we auto-publish main with some git hook from launchpad?16:42
juliankSounds dangerous, need to remove commit privileges and setup a merge bot :D16:42
Laneywe could lock it and require merge proposals16:43
LaneyI don't know how to set that latter part up though16:43
Laneyanyway that is in production now!16:44
juliankOr we make it publish tags, and tag when we want to publish, and continously deploy main to staging16:44
* Laney whispers "github actions"16:44
waveformhmm, is publish-db generally a bottleneck? Just reading through it and there's quite a few changes I could suggest there too (would also depend on the version of python3 on the host though -- some stuff like using the SQLite backup API to copy stuff instead of messing with flocks would depend on it being >=3.7)16:45
juliankIt doesn't run in the web frontend thread, but it sure eats up like 40% CPU when it runs16:45
Laneywe're on focal there16:46
waveformokay - I can certainly understand why. Any idea what python version is on the host?16:46
waveformokay, that's good16:46
Laneybe nice to move off sqlite completely really16:47
juliankLaney: Still have to create a sqlite db for publishing?16:47
juliankWell I guess we could go to another interchange format for britney and friends16:47
waveformwhile I'm more of a postgres type, sqlite really is pretty capable -- just need to treat it right :)16:48
juliankLaney: I think download-results.service needs a restart to make it create the index, which mojo didn't od16:48
juliankI'd run a NoSQL file based database :D16:48
* waveform shudders16:48
Laneyheh16:48
Laneyjuliank: ah right, I thought browse.cgi called init_db too, do it16:49
Laneytwo autopkgtest-webs don't forget16:50
juliankLaney: I think what we should do is split the result downloading and publishing into its own server, and then download the db on the web workers, this would avoid the spikes every couple minutes in CPU usage that drive request times up to 17s16:51
juliankLaney: waveform / now returns in like 0.8s best case, nice.16:52
Laneycould do16:52
juliankLaney: or we switch to dqlite :D16:53
Laneycheap thing to do quickly would be to apply resource limits to it, but maybe waveform's fixes will sort it all out16:53
waveformgood to hear -- I'll run some experiments tonight. I'm reasonably confident some fairly big gains could be made in publish-db though I'm still not clear on some bits of this architecture. Is there "one true" SQLite db (which stuff writes to) that gets periodically copied by publish-db for publishing?16:53
juliankwaveform: yes, well each web server has its own db they built independently and then copy them16:54
Laneyyeah there's a "read only" one16:54
Laneyit's basically building that16:54
juliankwaveform: So the JSON is 368K large16:56
waveformat that size it's probably worth benchmarking the internal json vs simplejson assuming adding dependencies like that isn't a major no-no16:57
juliankLaney: I think what would help would be second CPUs since we run at like 60% quite a lot16:58
juliankMost overhead is cache-amqp really16:58
juliankand amqp-status-collector and publish-db16:59
juliankBut yeah, CPU limits on background services would help too17:01
LaneyI guess we can either move the load off or beef up the machines or optimise the code17:01
Laneywell, any combination of thoes17:01
Laneyright I'm off, see ya later17:01
coreycbddstreet: I'm fine with my uploads getting rejected, let's just make sure the stable/queens git branch is updated17:02
coreycblet me know if you don't have access to push17:02
ddstreetcoreycb so the pkg can match what's already in the ubuntu-openstack-dev git tree, we should probably ask for my upload to be rejected, but i think you also need to rebuild your upload using the -v param so that it includes the changelogs from ...ubuntu5 and ...ubuntu6, since those versions were never accepted17:26
ddstreeteither way is fine with me17:26
RikMillsbritney runs are are not completing due to failure to fetch results. not had a successful run and update of excuses since ~11am17:49
RikMillsjuliank: is that related to what you were doing, or something else?17:49
juliankno!17:50
juliankRikMills: Well, 11am UTC?17:50
RikMillsyep17:50
juliankRight no, not done anything before 15:0017:50
RikMillscurrent excuses: Generated: 2021.05.04 10:42:04 +000017:51
juliankum 14:00 :D17:51
juliankThat's when I saw the error on autopkgtestubuntu.com17:51
juliankI'll leave britney stuff to Laney and sil2100, I don't have insight there17:51
juliankor I guess someone in the US17:51
RikMillsruns are crashing with things like: https://paste.ubuntu.com/p/n53fXB6PvN/17:51
juliankhmm works for me17:52
juliankThat url17:52
RikMillshttps://people.canonical.com/~ubuntu-archive/proposed-migration/log/impish/2021-05-04/17:26:13.log17:53
RikMillsIt fetches result fine for a while, then boom17:53
juliankRikMills: ack, I see it in haproxy log17:53
RikMillsaha. anyway, as long as someone is now aware :)17:54
RikMillswhich you highlight should have taken care of ;)17:56
RikMills*your17:56
juliankRikMills: Hmm, so this is just proxying to swift storage18:06
juliankRikMills: I have forwarded that to IS18:15
RikMillsjuliank: cheers!18:15
juliankRikMills: The error seems to be May  4 17:41:17 juju-4d1272-prod-proposed-migration-6 haproxy[3496355]: [WARNING] 123/174117 (3496355) : Server https_service/apache2-0-80 is DOWN, reason: Layer7 wrong status, code: 500, info: "Internal Server Error", check duration: 585ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.18:58
Laneywe should back off better, but I also think the proxying should move to haproxy, there's no need for it to have to hit the backends I think18:58
juliankLaney: ^18:58
Laneyjuliank: both died or what?18:58
Laneythat's half the point of load balancing them18:59
juliankLaney: Yes, both die regularly on / with database error (inconsistent image or whatever) and return 50018:59
juliank10.15.102.10 - - [04/May/2021:17:41:16 +0000] "HEAD / HTTP/1.0" 500 172 "-" "-"18:59
Laneyat the same time though18:59
Laneysad18:59
juliankLaney: I think it won't retry immediately19:00
juliankLaney: Shifting britney cron job back 5 minutes might fix it...19:00
Laneyanyway, let's move it off the web nodes19:00
Laneybritney just runs whenever it can19:00
Laneydo you speak haproxy?19:00
juliankLaney: I just read some docs where they did that, but no19:01
juliankThis is the error fwiw, sqlite3.DatabaseError: database disk image is malformed: /var/lib/juju/agents/unit-autopkgtest-web-0/charm/webcontrol/browse.cgi19:01
juliankNot sure why it prints the browse.cgi script as the database name19:02
Laneywhat is it, a race with publish-db?19:02
Laneybad bad bad19:02
juliankMaybe?19:02
juliankfor haproxy, probably    use_backend swift_server if { path_beg /results/ }19:03
juliankand setup backend19:03
Laneylets fiddle on staging19:03
juliankLaney: I didn't understand how the image can be inconsistent, but I guess now it might be a problem with the journal19:11
juliankno?19:11
juliankIt shouldn't be because the journal is a write ahead one that gets merged with the db?19:11
juliankand we rename the new db atomically19:12
juliankhow can that race?19:12
LaneyI don't get it, we should see either the old or the new one as we do it atomically19:13
Laneyhttps://autopkgtest.staging.ubuntu.com/results/autopkgtest-impish/impish/armhf/g/gzip/20210428_154557_d7576@/log.gz <- that is proxying @ haproxy now19:13
Laneynow I need to translate from haproxy.cfg to the charm config19:13
juliankThere might not be a way to atomically copy the .db while the wal is being merged19:13
juliankBy copying the file19:13
juliankwaveform's changes to make it use sqlite backup stuff should fix that I suppose19:14
juliankLaney: Or, we are supposed to open the r/w database with sqlite, acquire a shared lock using it, and _then_ we can copy the file19:15
juliankso instead of the top, we should do19:17
juliankcon = sqlite3.connect(config["web"]["database"])19:17
juliankbck = sqlite3.connect(target_new)19:17
juliankwith bck:19:17
juliank    con.backup(bck, pages=1, progress=progress)19:17
juliankcon.close()19:18
juliankThat should fix the race I suppose if there is one19:18
juliankPotentially we might want to use VACUUM INTO instead, which makes the published database smaller19:19
juliankbut costs cycles19:19
juliankso con.execute("VACCUM INTO {}", target_new)19:20
juliankum VACUUM INTO ?19:20
julianklet's run a quick bench19:23
juliankbackup API makes more sense19:28
LaneySURE does!19:30
juliankLaney: They both have the same speed on our DB, though; but since we add data afterwards anyway it makes sense to use backup API19:34
juliankLaney: https://code.launchpad.net/~juliank/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/40221619:41
juliankLaney: Also backup is actually 1/3-50% faster19:41
juliankLaney: We ought to do it incrementally with sleep I guess as https://www.sqlite.org/backup.html says such that the lock can be released in between19:44
juliankAh Python sleeps automatically for us19:45
juliankAh no, we need to specify a pages=5 or something19:46
juliankNow it does online backup19:49
juliankwith lock releasing to not block other things from writing to it for 2s :D19:49
juliankTesting now on staging sort of20:00
-queuebot:#ubuntu-release- Unapproved: neutron (bionic-proposed/main) [2:12.1.1-0ubuntu4 => 2:12.1.1-0ubuntu7] (openstack, ubuntu-server)20:04
coreycbddstreet: alright, I've uploaded a new version. thanks for catching that.20:05
coreycbplease can someone reject all but the most recent upload of neutron (2:12.1.1-0ubuntu7) from the bionic unapproved queue? and if someone from the SRU time has cycles we'd like to see if we can get a review of that last upload.20:06
waveformjuliank, yup -- backup API is definitely the way to do -- don't worry too much about vacuuming; there's generally very few scenarios where it's actually worthwhile (although I'm not familiar enough with autopkgtest to say it definitely isn't one of those, my general rule of thumb is that it's not :)20:09
Laneywaveform: maybe you can review that MR if juliank stole your work? :-)20:09
waveformLaney, sure thing :)20:09
waveformoh, while I'm here could someone reject linux-firmware-raspi2 from focal-proposed -- was checking something earlier and realized I'd missed something when reviewing it last week (it shouldn't remove the Breaks line in d/control)20:11
juliankLaney, waveform I have run the current state of the branch successfully on staging instance fwiw20:11
juliankwaveform: Sorry for stealing your stuff :D20:12
waveformjuliank, no prob -- steal away :)20:12
juliankwaveform: I'm not sure if I want to increase #pages or decrease the sleep (python defaults to 250ms)20:13
juliankLaney: Can I charm push this to edge channel?20:16
juliankLaney: I only curled the file and ran it manually :D20:17
juliankBut I want to play with pushing to channel properly again20:17
juliankbut there's your change in there20:17
juliank(I suppose)20:17
juliankLaney: eek, no write access to the autopkgtest-web charm anyway20:18
Laneyone second20:18
Laneyah yeah you're not in ubuntu-release ;-)20:18
Laneylet me remember how to grant that20:19
* juliank wants to get britney running again before going to bed :D20:19
Laneyok that should work20:20
Laneyjuliank: can you check out wip/haproxy-proxy too20:20
juliankLaney: Shall I rebase on that?20:20
juliankmerge them locally I guess20:20
Laneyas you wish, I don't mind merge commits really20:21
waveformjuliank, the changes in the merge look sane but I'm probably being horribly thick in not really understanding the *three* databases involved here (not that that's a change you've made -- I get that's always the way it's been, but I'm not grokking the purpose of them all)20:21
juliankwaveform: The r/o database has extra data on top of the r/w for consumers, so we first copy the r/w one in; then the extra data from the old r/o one20:22
juliankprobably should open the old r/o in ?mode=ro20:22
juliankLaney: can't push still :(20:22
Laneywhat20:23
juliankLaney:20:23
juliank$ charm push /tmp/charm-builds/autopkgtest-web cs:~ubuntu-release/autopkgtest-web20:23
juliankERROR cannot post archive: access denied for user "juliank"20:23
* juliank sad20:23
Laneylet me read docs I guess20:24
waveformjuliank, brief note on pages/sleep on the backup API -- I might be tempted to bump the pages up a bit -- with the default page-size (4KB) that'll mean it's copying 4MB then sleeping a quarter second, rinse'n'repeat. Given a typical SSD can manage quite a bit more than that I'd be tempted to bump it up to doing ~1s worth of writing before its 1/4 sec pause (which is reasonable)20:24
juliankLaney: Your change looks good20:25
juliankwaveform: Right, but it only sleeps if there are writers waiting?20:25
juliankIt certainly did not sleep for me, but I had no other connections open20:25
waveformjuliank, good question, don't know -- I would *hope* that's the case but it's not a feature I'm terribly familiar with (one of those new things I've briefly played with and gone "cool!" but not dug into deeply yet)20:26
juliankwaveform: I'll raise to 128k pages20:28
Laneylaney@dev> charm grant --acl write cs:\~ubuntu-release/autopkgtest-cloud-worker juliank20:28
Laneylaney@dev>20:28
LaneyI don't know20:28
Laneywait web not cloud-worker20:29
juliankLaney: yeah web :D20:29
Laneyok did that20:29
Laneyit returned 0 ...20:29
juliankwaveform: Updated to 128k pages20:29
juliankLaney: doesn't work yet, maybe there's a delay until it takes effect20:31
juliankLaney: Anyhow, I manually tried the change by curling the file and running it20:31
Laneywill have to ask the juju people I guess :(20:32
* Laney joins #juju20:32
* juliank joins as well20:33
LaneyI merged my branch20:34
Laneynow /results/ is not hitting the backends20:35
Laneyhopefully that gets proposed-migration working20:35
juliankLaney: possibly, want to get mine in too to make sure though20:36
juliank:D20:36
Laneyyeah for sure20:36
Laneymight have to sort the ACL stuff out tomorrow20:36
Laneyneed to do duolingo and play animal crossing20:36
juliankI need to go to sleep soon :/20:37
juliankwaveform: could you write me an ACK on the merge? :D20:39
julianklate night infra fixing :)20:42
waveformsure, jsut a mo20:42
waveformdone20:43
juliankLaney: please just merge, publish, and deploy that :D20:45
juliankcharm playing on staging I can do tomorrow :D20:45
juliankThis worked last year (two years ago?)20:45
Laneywaveform: I looked for that pkg to reject btw and didn't see it20:52
Laneycoreycb: I'll do yours tomorrow if someone doesn't beat me20:53
Laneynn!20:53
coreycbthanks Laney20:53
waveformLaney, weird -- I can see it in the queue. Anyway, it can wait -- go have fun :)20:54
Laneymaybe I'm looking in the wrong queue20:54
Laneyshare a link and i'll do it tomrrow too20:54
waveformhttps://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=&memo=30&start=30 linux-firmware-raspi2 second from the top on that page (multiverse, misc)20:55
waveformversion 4-0ubuntu0~20.04.120:55
juliankreleased!20:56
waveformah https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=linux-firmware-raspi2 might be more useful! (doh)20:56
juliankRikMills: We believe that britney should be working again now20:57
waveformoh, it's not in -proposed yet -- that's why you can't see it. Anyway, doubly not to worry then!20:57
juliankwaveform: heh, publish-db with backup API now takes 2 minutes instead of 30s, but the corruption is gone, so not super unhappy21:06
juliankA bit odd maybe, maybe it needs a lot of retrying due to lots of writes21:07
waveformthat could well be the case -- it was always going to be longer than a straight file-copy anyway, and add a bit extra on top for any writes getting in the way -- that's not too bad. Can probably gain a bit back with some other optimizations -- will see if I can have another look tomorrow21:15
-queuebot:#ubuntu-release- Unapproved: pi-bluetooth (focal-proposed/multiverse) [0.1.10ubuntu6 => 0.1.15ubuntu0~20.04.1] (raspi)21:28
-queuebot:#ubuntu-release- New binary: pipewire [amd64] (impish-proposed/main) [0.3.26-1] (desktop-core)22:54

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!