=== JanC is now known as Guest7392 === JanC is now known as Guest7756 [08:08] LP git still having issues [08:08] ruinedyourlife: ^^ [09:10] very very very slow [09:11] can we have an update please? [09:11] this does affect ubuntu development work === JanC is now known as Guest2617 [10:10] is it still affecting you? [10:15] RikMills im not having issues atm and not seeing any specific load spike [10:17] ruinedyourlife: pushing 67 git repos lets see how that does [10:18] earlier today that took over an hour, whereas usually it would take a few minutes [10:18] oh wow... [10:20] 10 pushed already. so much better now [10:20] maybe still a little slow, but workable [10:22] when you are packaging things like KDE for Ubuntu which involves hundreds of git repos sometimes, you really notice when things are slow [10:23] later today I will be wanting to do KDE gear new release. that is 235 repos for 235 source packages [10:23] and a few extra for metadata etc [10:33] at 57/67 [10:34] ok its making good progress so far, maybe a bit slow yeah [10:34] yes, this is still quite slow [10:34] but not disastrously so [10:34] however, would appreciate it being looked into [10:35] yeah i have already brought it up during our daily this morning, but im not the most versed in this domain so it might have to wait for our sprint to be done (next week) [10:36] im looking at our grafana and ganesha+turnip look fine, so im not sure if the issue is even there [10:37] last time a SRE checked the perfs when i noticed slowness and it was at 40% load too, but we not be thinking correctly and checking the correct places [10:38] we might* not [10:45] ruinedyourlife: ok, fair enough, and thanks! [10:45] yw === JanC is now known as Guest4195 [13:04] hello [13:05] nteodosi, are you aware of the fact that riscv64 builders are taking forever to build simple packages? to me looks like some kernel/infra issue [13:05] specially haskell packages [13:05] look e.g. https://launchpad.net/ubuntu/+archive/test-rebuild-20250314-plucky/+build/30560182 [13:06] setting up ghc since 8 days [13:06] https://launchpad.net/ubuntu/+source/mediawiki2latex/8.28-1/+build/30672695 configuring since 20h [13:06] people keep canceling and rescheduling, and they always get stuck at the same point [13:07] just wondering if anybody can ssh to some machine and check what is wrong [13:21] LocutusOfBorg, I was not aware and passed the information on. Thank you for raising it. === JanC is now known as Guest9769 [13:45] thanks! [13:45] I was wondering if it was already known or not [14:15] we have seen slowness in the riscv builders, but it was because all the builders are taken so that people could not get their builds going, this seems extreme tho [14:15] ive asked colleagues that might know more than i do [14:26] ruinedyourlife, yes, but this is now about long queue [14:27] yup, these builds are very very slow [14:27] btw nteodosi just FYI I did pbuilder-dist riscv64 create of both plucky and questing, building haskell-zstd was successful, on amd64 VM [14:28] ive checked with colleagues, and apparently, riscv emulation is just that slow on amd that it could take 8 days? ive asked an SRE if we could check the builder itself so we know if anything is running or if its dead [14:29] it took 15m on my amd64 VM, can't take 8 days... [14:29] really :) [14:29] even unpacking ghc is dead [14:29] ok thats very weird [14:30] I suspect kernel is the culprit [14:30] some kernel slowliness [14:30] can't proove [14:31] I'm really surprised anything at all can build in 15 min in emulated RiscV. Even logging in was slow for me. [14:31] ^ [14:32] by kernel, do you mean the kernel version of the builders might have a bug that makes it that slow? [14:33] time pbuilder-dist questing riscv64 build haskell-zstd_*.dsc [14:33] I'll report once finished [14:33] yes or something similar that broke when questing opened [14:33] new compiler, new foo [14:34] I would try a build of the same tool on plucky [14:34] https://launchpad.net/ubuntu/+archive/test-rebuild-20250314-plucky/+build/30560182 [14:34] well, this one explains that the builders are buggy not the baseline of the software [14:34] this is plucky [14:35] thanks for the info [14:35] I'm building with pbuilder from a noble amd64 VM (vitualbox= [14:35] nothing too fancy [14:35] 5 cores shared inside the VM [14:35] did builder farm got upgraded recently before opening? [14:36] not that im aware of [14:38] I'm diffing the latest build of plucky on riscv64 and one recent [14:40] strace on ghc's pid seems to show that stuff is happening [14:40] doesnt look dead === dkarakasilis1 is now known as dkarakasilis [15:09] 13m full build [15:13] yeah i really dont know whats happening there [15:13] we cant fully compare LP builds to local builds, but thats still insane to me [15:17] ill raise that with the team, see if we find what the issue is === JanC is now known as Guest1504 === graber1 is now known as graber === JanC is now known as Guest9872 === JanC is now known as Guest456