| === Eickmeyer is now known as Eickmeyer[m] | ||
| === E_Eickmeyer is now known as Eickmeyer | ||
| -lugito:#lubuntu-devel- [rMANUAL22cc7c26f5ed: Update Readme for LTS] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUAL22cc7c26f5ed | 02:20 | |
| -lugito:#lubuntu-devel- [rMANUALd5c0c61c6fb5: Improve wording] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUALd5c0c61c6fb5 | 02:22 | |
| tsimonq2 | I had to play with the SSH keys in Phabricator a little bit. | 03:01 |
|---|---|---|
| tsimonq2 | Nothing was mirroring to Launchpad and that was a problem. | 03:01 |
| tsimonq2 | After a few hours in the hole, the issue ended up being that Launchpad needs a separate SSH key pair. It can be identical to the previous pair, but it just needs a different username. | 03:02 |
| tsimonq2 | So there's now one for Launchpad mirrors with the username Lugito and another for the GitHub mirrors with the username git. | 03:02 |
| tsimonq2 | I ran all of the mirror commands manually on the server with the CLI. There are maybe three minor errors but that's about it. | 03:04 |
| tsimonq2 | https://discourse.phabricator-community.org/t/wrong-column-type-everywhere-other-errors-on-fresh-db-upgrade/3872/9?u=tsimonq2 | 03:33 |
| tsimonq2 | Spent some time on that. | 03:33 |
| tsimonq2 | I tried to dig into that but unfortunately migrating several major versions of MySQL is over my head. | 04:09 |
| tsimonq2 | I hope to learn something when Evan takes a look. | 04:09 |
| tsimonq2 | In the meantime, Jenkins and Phab are on 20.04. | 04:10 |
| tsimonq2 | I hope to work on the other ones as time permits. | 04:10 |
| tsimonq2 | In between that, I'm still testing that Lintian patch for the builds | 04:10 |
| tsimonq2 | It failed on groovy_unstable_qterminal because I did the versioning wrong for the command. Should be working now but I have to wait for a successful build + publish to confirm. | 04:12 |
| tsimonq2 | The reason why it hasn't been unstable, and it's been marked as failed, is because in the Jenkins config for the job I marked an exit status of 1 as unstable. Any other non-0 exit status just falls through to a failure. | 04:13 |
| tsimonq2 | This works with Lintian because: | 04:13 |
| tsimonq2 | EXIT STATUS | 04:13 |
| tsimonq2 | 0 No policy violations or major errors detected. (There may have been warnings, though.) | 04:13 |
| tsimonq2 | 1 Policy violations or major errors detected. | 04:13 |
| tsimonq2 | 2 Lintian run-time error. An error message is sent to stderr. | 04:14 |
| tsimonq2 | I want to get this handled and run a nightly before the LP publisher maintenance starts. | 04:14 |
| tsimonq2 | Tested that qterminal will actually pass Lintian locally. It does. | 04:19 |
| tsimonq2 | The manual site will be going down shortly for a little less than an hour for the 20.04 upgrade. This should probably be the most harmless upgrade of them all. | 04:29 |
| tsimonq2 | I'll also see if we can actually get it to point to Phab this time instead of having to use the GitHub mirror. | 04:29 |
| tsimonq2 | (Especially since mirroring can sometimes break and it's not the lowercase-c canonical source.) | 04:30 |
| tsimonq2 | Finished: SUCCESS | 04:34 |
| tsimonq2 | Nice. | 04:34 |
| tsimonq2 | I'll deploy that to all the jobs then. | 04:35 |
| -lugito:#lubuntu-devel- [rCI082ee862b0b7: Run Lintian on built binaries if the package build succeeds.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCI082ee862b0b7 | 04:41 | |
| tsimonq2 | I don't know if I'll catch this in time for publisher maintenance. :/ | 04:42 |
| tsimonq2 | I might. | 04:42 |
| -lugito:#lubuntu-devel- [rCI050cf6cb124e: Bump the total lp_check allowed time to 6 hours from 2, to account for…] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCI050cf6cb124e | 04:45 | |
| tsimonq2 | There. I'll revisit things when the nightly is done, and if it gets caught up in publisher maintenance, it shouldn't fail all of the jobs. | 04:46 |
| tsimonq2 | (To be fair, we probably could have gotten away with 4, so 6 hours is more than enough, but meh, I decided to intentionally overshoot on that one. Worst case scenario, someone can just kill the job.) | 04:47 |
| -lugito:#lubuntu-devel- [rCIbf8edaaf9dc6: Make sure we don't fail when loading the non-data-using release-mgmt template.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCIbf8edaaf9dc6 | 04:57 | |
| tsimonq2 | If my Python is correct, that should make jobgenerator pass again. | 04:57 |
| -lugito:#lubuntu-devel- [rCI5fb09fda6db9: Slightly clean that up.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCI5fb09fda6db9 | 04:59 | |
| * tsimonq2 wonders why LCI is timing out even though the nightly didn't even run at midnight O_o | 05:28 | |
| lubot | <teward001> @lubuntu_bot [*tsimonq2: wonders why LCI is timing out even though the nightly didn't even run …], because SOMETHING you have running pegged the host system | 05:31 |
| lubot | <teward001> load avg 150 for > 30 minutes | 05:31 |
| tsimonq2 | Probably the nightly. | 05:31 |
| tsimonq2 | I need to reduce the amount of concurrent jobs that can run. | 05:31 |
| lubot | <teward001> yes, you do | 05:31 |
| lubot | <teward001> except it seized up the entire host system | 05:32 |
| tsimonq2 | I'll do that as soon as it's back up. ;) | 05:32 |
| tsimonq2 | Oh, nice. | 05:32 |
| tsimonq2 | Kick it? | 05:32 |
| lubot | <teward001> vCenter just triggered the notification right now | 05:32 |
| lubot | <teward001> and force rebooted the system | 05:32 |
| lubot | <teward001> hardreset | 05:32 |
| lubot | <teward001> yep that's happened | 05:32 |
| tsimonq2 | To be fair, I don't think the nightly has run on this server yet. | 05:32 |
| lubot | <teward001> probably not :P | 05:32 |
| lubot | <teward001> BUT it explains why it seized up | 05:32 |
| lubot | <teward001> and why vCenter's "Unstable System" script I wrote kicked in | 05:32 |
| lubot | <teward001> @tsimonq2 vCenter failed in guest OS restart, so it hard-reset the box, should be up shortly | 05:35 |
| lubot | <teward001> wow really you pegged it again | 05:36 |
| tsimonq2 | O_o | 05:36 |
| tsimonq2 | No, I haven't touched it dude. | 05:36 |
| tsimonq2 | I can at least SSH in now. | 05:37 |
| lubot | <teward001> seriously, 32 concurrent Jenkins runs? | 05:37 |
| lubot | <teward001> that just started 32 concurrent jobs | 05:37 |
| tsimonq2 | Oh, and the Web UI is working now too. | 05:37 |
| tsimonq2 | O_O | 05:37 |
| tsimonq2 | Yeah, that should be reduced. | 05:37 |
| lubot | <teward001> yep | 05:37 |
| tsimonq2 | I'm on it. | 05:37 |
| lubot | <teward001> you forget I have host node access to see *all* systems :P | 05:37 |
| tsimonq2 | :P | 05:37 |
| lubot | <teward001> yep htop on the host VM can see container processes which helps to identify things :P | 05:38 |
| tsimonq2 | :P | 05:38 |
| lubot | <teward001> 'cept I had to count the running processes manually :P | 05:38 |
| tsimonq2 | There, reduced to 30. It was at 100 available before, which is probably why it instantly filled 1/3 of the spots, heh. | 05:39 |
| lubot | <teward001> :P | 05:39 |
| lubot | <teward001> well it spiked load to 30 which is bad on a 4CPU system | 05:39 |
| tsimonq2 | 30 what? | 05:39 |
| lubot | <teward001> you do know that load averages indicate pressure on the threads/CPUs right? | 05:39 |
| tsimonq2 | Please further educate me. | 05:40 |
| lubot | <teward001> load of 30 means that processes are demanding 7 times the load of available CPUs. so 600% overloaded | 05:41 |
| tsimonq2 | ohhh | 05:41 |
| tsimonq2 | That's very fun. Heh. | 05:41 |
| tsimonq2 | AKA UNLEASHING THE KRAKEN | 05:41 |
| tsimonq2 | *shots* | 05:41 |
| lubot | <teward001> 30 - 4 = 26. SO anything over a load of 4.0 over the averages indicates overload | 05:41 |
| lubot | <teward001> 650% overloaded | 05:42 |
| lubot | <teward001> USUALLY that's not an issue in the instantaneous and smaller averages | 05:42 |
| lubot | <teward001> but when that persists over a LONG period it gets concerning | 05:42 |
| lubot | <teward001> it spiked down now of course | 05:43 |
| lubot | <teward001> so instant average is 0.46 | 05:43 |
| lubot | <teward001> 1min = 0.46, 5min = 4.15, 15min = 4.02 | 05:43 |
| lubot | <teward001> so that's actually OK | 05:43 |
| lubot | <teward001> (load average of 4 means 4 CPUs are needed) | 05:43 |
| lubot | <teward001> so it looks stable now but it instantaneously spiked to 30 :P | 05:44 |
| lubot | <teward001> which slows things down | 05:44 |
| lubot | <teward001> ESPECIALLY when the load is something like 150 :P | 05:44 |
| tsimonq2 | :P | 05:45 |
| tsimonq2 | Okay, so. | 05:45 |
| tsimonq2 | Jenkins apparently has load monitoring built in. | 05:45 |
| tsimonq2 | And I can confirm what you've been saying. | 05:45 |
| tsimonq2 | What would you consider a *max* load if I wanted to tweak this over time? | 05:45 |
| tsimonq2 | Like a load that can be handled for 5-10 mins. | 05:46 |
| tsimonq2 | Give me a number and I'll tweak to get there. | 05:46 |
| tsimonq2 | (I'm curious why you chose it too but meh.) | 05:46 |
| lubot | <teward001> well 150 is obviously overload | 05:46 |
| tsimonq2 | True. | 05:46 |
| lubot | <teward001> but I usually say 4 - 6 times CPU is usually oK. We have 4 CPUs on the host VM so... *maths* | 05:47 |
| lubot | <teward001> i'd say cap load at 32 if you want to do some lockdown | 05:47 |
| lubot | <teward001> i'll increase the CPUs as well if you can shut down all running jobs (or I'll just hardreboot the thing again) | 05:47 |
| -lugito:#lubuntu-devel- [T161: Fix latest errors in manual building on prod] tsimonq2 (Simon Quigley) just created this task: https://phab.lubuntu.me/T161 | 05:53 | |
| tsimonq2 | @teward001: I'll set it to shut down in Jenkins so that it doesn't let any more jobs execute. Once that's taken care of (might be a while because of the aforementioned LP publishing) then go for it. | 05:54 |
| tsimonq2 | Unless you want to table it until later when the nightly is done and there are no jobs. | 05:54 |
| tsimonq2 | Your call. | 05:55 |
| lubot | <teward001> i'll wait until later :P | 05:56 |
| lubot | <teward001> i'm headed to sleep now | 05:56 |
| tsimonq2 | Okay. | 05:56 |
| tsimonq2 | Have fun. | 05:56 |
| lubot | <teward001> let me know tomorrow when you've shut it :P | 05:56 |
| tsimonq2 | I'll be up until the early morning. | 05:56 |
| tsimonq2 | When you wake up just say something here and hard-reboot. | 05:56 |
| lubot | <teward001> @tsimonq2 saw that it stopped all builds and JFDI and upped its CPUs to 8. That lets each CPU have 4 things scheduled. | 06:06 |
| tsimonq2 | Thanks. | 06:07 |
| lubot | <teward001> with a load of 32 cap | 06:07 |
| tsimonq2 | Sweeeet. | 06:07 |
| lubot | <teward001> it's hardrebooted now | 06:07 |
| tsimonq2 | Much appreciated. | 06:07 |
| -lugito:#lubuntu-devel- [rMANUAL0ef331fea445: Fix warnings sorry should have checked] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUAL0ef331fea445 | 06:16 | |
| -lugito:#lubuntu-devel- [rMANUAL08d47c21b1bd: Fix warnings sorry should have checked] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUAL08d47c21b1bd | 06:16 | |
| tsimonq2 | Thanks Lyn, no worries. :) | 06:16 |
| -lugito:#lubuntu-devel- [rMANUAL522d0ef82ec9: Fix warnings sorry should have checked] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUAL522d0ef82ec9 | 06:17 | |
| -lugito:#lubuntu-devel- [rMANUAL5253f9fdf7d2: Fix table alignment finishes fixes for T161] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUAL5253f9fdf7d2 | 06:26 | |
| -lugito:#lubuntu-devel- [rMANUALb39e0ff5b109: Fix table alignment finishes fixes for T161] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUALb39e0ff5b109 | 06:29 | |
| -lugito:#lubuntu-devel- [rMANUALce90991c8aa1: Fix table alignment finishes fixes for T161] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUALce90991c8aa1 | 06:29 | |
| tsimonq2 | I've done a very large amount of work over the last five to six hours on CI tooling. | 11:07 |
| tsimonq2 | We have a better way now to see what exactly takes how long for jobgenerator. | 11:08 |
| tsimonq2 | I had to fix the jobgenerator tool because it apparently wasn't updating the config files for jobs. Whoops. | 11:09 |
| tsimonq2 | I did just kick off a nightly, so we should have a good idea as to what our Lintian failures are pretty soon. | 11:10 |
| tsimonq2 | So if you see any | 11:10 |
| tsimonq2 | ..."unstable" jobs, that's why. | 11:10 |
| tsimonq2 | I also did a lot of work on overhaul for Phab permissions. | 11:11 |
| tsimonq2 | Since wxl messed things up in summer of last year where we now can't add new people to the Developers project, I decided it was about time to create several sub-projects to delegate things better. | 11:11 |
| tsimonq2 | For example, Erich probably doesn't need commit access to our default settings but he could use commit access to Calamares. | 11:12 |
| tsimonq2 | That's a little bit more fine-tuned now. | 11:12 |
| -lugito:#lubuntu-devel- [rCI28c7f2b39705: Replace some timer calls.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCI28c7f2b39705 | 11:29 | |
| -lugito:#lubuntu-devel- [rCI1b502ee4e435: Add a decorator to TimerMetrics and wrap most of the functions.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCI1b502ee4e435 | 11:29 | |
| -lugito:#lubuntu-devel- [rCIc6f4117ab7f5: The removed timers were redundant.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCIc6f4117ab7f5 | 11:29 | |
| -lugito:#lubuntu-devel- [rPPABRITNEY363622addf6e: Update ubuntu-archive-tools.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rPPABRITNEY363622addf6e | 11:37 | |
| -lugito:#lubuntu-devel- [rPPABRITNEY7973af9bbbd2: Update britney2-ubuntu.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rPPABRITNEY7973af9bbbd2 | 11:37 | |
| tsimonq2 | It seems like a lot of the builds previously failed on one of the most recent nightlies due to the build environment not being cleaned up correctly. | 11:51 |
| tsimonq2 | Those builds should be retried. | 11:52 |
| tsimonq2 | Also, on the bright side, seems like the Lintian stuff is working. | 11:52 |
| tsimonq2 | SDDM builds are now being marked as unstable. | 11:52 |
| tsimonq2 | Fun. | 11:52 |
| -lugito:#lubuntu-devel- [rLIBLXQTPACKAGING683e8fa79f9d: Update symbols from amd64 build logs.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rLIBLXQTPACKAGING683e8fa79f9d | 12:38 | |
| -lugito:#lubuntu-devel- [T131: Version Number in Plymouth?] apt-ghetto (apt-ghetto) commented on the task: https://phab.lubuntu.me/T131#3482 | 17:08 | |
| -lugito:#lubuntu-devel- [T162: Openbox standalone] hmollercl (Hans P. Möller) just created this task: https://phab.lubuntu.me/T162 | 20:16 | |
| lubot | <tsimonq2> Hey @lynorian this might be worth looking into: https://www.reddit.com/r/Lubuntu/comments/gl3iya/lubuntu_2004_sensors_widget_set_crit_temp_for/?utm_medium=android_app&utm_source=share | 22:16 |
| lubot | <tsimonq2> Thanks for your really quick turnaround on that task btw | 22:16 |
| lubot | <tsimonq2> Looks like the usual builds are still FTBFS, but we now have some "unstable" builds if anyone wants to get some practice cleaning up Lintian errors | 22:17 |
| lubot | <tsimonq2> I might actually make Lintian exit 1 on non-critical | 22:18 |
| lubot | <tsimonq2> But I feel like a better use of my time might just be making something like a status page that highlights the number of Lintian flags are raised and on which packages | 22:18 |
| lubot | <RikMills> Qt 5.14.2 landing in proposed, FYI | 22:51 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!