=== Eickmeyer is now known as Eickmeyer[m]
=== E_Eickmeyer is now known as Eickmeyer
[02:20] -lugito:#lubuntu-devel- [rMANUAL22cc7c26f5ed: Update Readme for LTS] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUAL22cc7c26f5ed
[02:22] -lugito:#lubuntu-devel- [rMANUALd5c0c61c6fb5: Improve wording] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUALd5c0c61c6fb5
[03:01] <tsimonq2> I had to play with the SSH keys in Phabricator a little bit.
[03:01] <tsimonq2> Nothing was mirroring to Launchpad and that was a problem.
[03:02] <tsimonq2> After a few hours in the hole, the issue ended up being that Launchpad needs a separate SSH key pair. It can be identical to the previous pair, but it just needs a different username.
[03:02] <tsimonq2> So there's now one for Launchpad mirrors with the username Lugito and another for the GitHub mirrors with the username git.
[03:04] <tsimonq2> I ran all of the mirror commands manually on the server with the CLI. There are maybe three minor errors but that's about it.
[03:33] <tsimonq2> https://discourse.phabricator-community.org/t/wrong-column-type-everywhere-other-errors-on-fresh-db-upgrade/3872/9?u=tsimonq2 
[03:33] <tsimonq2> Spent some time on that.
[04:09] <tsimonq2> I tried to dig into that but unfortunately migrating several major versions of MySQL is over my head.
[04:09] <tsimonq2> I hope to learn something when Evan takes a look.
[04:10] <tsimonq2> In the meantime, Jenkins and Phab are on 20.04.
[04:10] <tsimonq2> I hope to work on the other ones as time permits.
[04:10] <tsimonq2> In between that, I'm still testing that Lintian patch for the builds
[04:12] <tsimonq2> It failed on groovy_unstable_qterminal because I did the versioning wrong for the command. Should be working now but I have to wait for a successful build + publish to confirm.
[04:13] <tsimonq2> The reason why it hasn't been unstable, and it's been marked as failed, is because in the Jenkins config for the job I marked an exit status of 1 as unstable. Any other non-0 exit status just falls through to a failure.
[04:13] <tsimonq2> This works with Lintian because:
[04:13] <tsimonq2> EXIT STATUS
[04:13] <tsimonq2>        0   No policy violations or major errors detected.  (There may have been warnings, though.)
[04:13] <tsimonq2>        1   Policy violations or major errors detected.
[04:14] <tsimonq2>        2   Lintian run-time error. An error message is sent to stderr.
[04:14] <tsimonq2> I want to get this handled and run a nightly before the LP publisher maintenance starts.
[04:19] <tsimonq2> Tested that qterminal will actually pass Lintian locally. It does.
[04:29] <tsimonq2> The manual site will be going down shortly for a little less than an hour for the 20.04 upgrade. This should probably be the most harmless upgrade of them all.
[04:29] <tsimonq2> I'll also see if we can actually get it to point to Phab this time instead of having to use the GitHub mirror.
[04:30] <tsimonq2> (Especially since mirroring can sometimes break and it's not the lowercase-c canonical source.)
[04:34] <tsimonq2> Finished: SUCCESS
[04:34] <tsimonq2> Nice.
[04:35] <tsimonq2> I'll deploy that to all the jobs then.
[04:41] -lugito:#lubuntu-devel- [rCI082ee862b0b7: Run Lintian on built binaries if the package build succeeds.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCI082ee862b0b7
[04:42] <tsimonq2> I don't know if I'll catch this in time for publisher maintenance. :/
[04:42] <tsimonq2> I might.
[04:45] -lugito:#lubuntu-devel- [rCI050cf6cb124e: Bump the total lp_check allowed time to 6 hours from 2, to account for…] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCI050cf6cb124e
[04:46] <tsimonq2> There. I'll revisit things when the nightly is done, and if it gets caught up in publisher maintenance, it shouldn't fail all of the jobs.
[04:47] <tsimonq2> (To be fair, we probably could have gotten away with 4, so 6 hours is more than enough, but meh, I decided to intentionally overshoot on that one. Worst case scenario, someone can just kill the job.)
[04:57] -lugito:#lubuntu-devel- [rCIbf8edaaf9dc6: Make sure we don't fail when loading the non-data-using release-mgmt template.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCIbf8edaaf9dc6
[04:57] <tsimonq2> If my Python is correct, that should make jobgenerator pass again.
[04:59] -lugito:#lubuntu-devel- [rCI5fb09fda6db9: Slightly clean that up.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCI5fb09fda6db9
[05:28]  * tsimonq2 wonders why LCI is timing out even though the nightly didn't even run at midnight O_o
[05:31] <lubot> <teward001> @lubuntu_bot [*tsimonq2: wonders why LCI is timing out even though the nightly didn't even run …], because SOMETHING you have running pegged the host system
[05:31] <lubot> <teward001> load avg 150 for > 30 minutes
[05:31] <tsimonq2> Probably the nightly.
[05:31] <tsimonq2> I need to reduce the amount of concurrent jobs that can run.
[05:31] <lubot> <teward001> yes, you do
[05:32] <lubot> <teward001> except it seized up the entire host system
[05:32] <tsimonq2> I'll do that as soon as it's back up. ;)
[05:32] <tsimonq2> Oh, nice.
[05:32] <tsimonq2> Kick it?
[05:32] <lubot> <teward001> vCenter just triggered the notification right now
[05:32] <lubot> <teward001> and force rebooted the system
[05:32] <lubot> <teward001> hardreset
[05:32] <lubot> <teward001> yep that's happened
[05:32] <tsimonq2> To be fair, I don't think the nightly has run on this server yet.
[05:32] <lubot> <teward001> probably not :P
[05:32] <lubot> <teward001> BUT it explains why it seized up
[05:32] <lubot> <teward001> and why vCenter's "Unstable System" script I wrote kicked in
[05:35] <lubot> <teward001> @tsimonq2 vCenter failed in guest OS restart, so it hard-reset the box, should be up shortly
[05:36] <lubot> <teward001> wow really you pegged it again
[05:36] <tsimonq2> O_o
[05:36] <tsimonq2> No, I haven't touched it dude.
[05:37] <tsimonq2> I can at least SSH in now.
[05:37] <lubot> <teward001> seriously, 32 concurrent Jenkins runs?
[05:37] <lubot> <teward001> that just started 32 concurrent jobs
[05:37] <tsimonq2> Oh, and the Web UI is working now too.
[05:37] <tsimonq2> O_O
[05:37] <tsimonq2> Yeah, that should be reduced.
[05:37] <lubot> <teward001> yep
[05:37] <tsimonq2> I'm on it.
[05:37] <lubot> <teward001> you forget I have host node access to see *all* systems :P
[05:37] <tsimonq2> :P
[05:38] <lubot> <teward001> yep htop on the host VM can see container processes which helps to identify things :P
[05:38] <tsimonq2> :P
[05:38] <lubot> <teward001> 'cept I had to count the running processes manually :P
[05:39] <tsimonq2> There, reduced to 30. It was at 100 available before, which is probably why it instantly filled 1/3 of the spots, heh.
[05:39] <lubot> <teward001> :P
[05:39] <lubot> <teward001> well it spiked load to 30 which is bad on a 4CPU system
[05:39] <tsimonq2> 30 what?
[05:39] <lubot> <teward001> you do know that load averages indicate pressure on the threads/CPUs right?
[05:40] <tsimonq2> Please further educate me.
[05:41] <lubot> <teward001> load of 30 means that processes are demanding 7 times the load of available CPUs.  so 600% overloaded
[05:41] <tsimonq2> ohhh
[05:41] <tsimonq2> That's very fun. Heh.
[05:41] <tsimonq2> AKA UNLEASHING THE KRAKEN
[05:41] <tsimonq2> *shots*
[05:41] <lubot> <teward001> 30 - 4 = 26.  SO anything over a load of 4.0 over the averages indicates overload
[05:42] <lubot> <teward001> 650% overloaded
[05:42] <lubot> <teward001> USUALLY that's not an issue in the instantaneous and smaller averages
[05:42] <lubot> <teward001> but when that persists over a LONG period it gets concerning
[05:43] <lubot> <teward001> it spiked down now of course
[05:43] <lubot> <teward001> so instant average is 0.46
[05:43] <lubot> <teward001> 1min = 0.46, 5min = 4.15, 15min = 4.02
[05:43] <lubot> <teward001> so that's actually OK
[05:43] <lubot> <teward001> (load average of 4 means 4 CPUs are needed)
[05:44] <lubot> <teward001> so it looks stable now but it instantaneously spiked to 30 :P
[05:44] <lubot> <teward001> which slows things down
[05:44] <lubot> <teward001> ESPECIALLY when the load is something like 150 :P
[05:45] <tsimonq2> :P
[05:45] <tsimonq2> Okay, so.
[05:45] <tsimonq2> Jenkins apparently has load monitoring built in.
[05:45] <tsimonq2> And I can confirm what you've been saying.
[05:45] <tsimonq2> What would you consider a *max* load if I wanted to tweak this over time?
[05:46] <tsimonq2> Like a load that can be handled for 5-10 mins.
[05:46] <tsimonq2> Give me a number and I'll tweak to get there.
[05:46] <tsimonq2> (I'm curious why you chose it too but meh.)
[05:46] <lubot> <teward001> well 150 is obviously overload
[05:46] <tsimonq2> True.
[05:47] <lubot> <teward001> but I usually say 4 - 6 times CPU is usually oK.  We have 4 CPUs on the host VM so... *maths*
[05:47] <lubot> <teward001> i'd say cap load at 32 if you want to do some lockdown
[05:47] <lubot> <teward001> i'll increase the CPUs as well if you can shut down all running jobs (or I'll just hardreboot the thing again)
[05:53] -lugito:#lubuntu-devel- [T161: Fix latest errors in manual building on prod] tsimonq2 (Simon Quigley) just created this task: https://phab.lubuntu.me/T161
[05:54] <tsimonq2> @teward001: I'll set it to shut down in Jenkins so that it doesn't let any more jobs execute. Once that's taken care of (might be a while because of the aforementioned LP publishing) then go for it.
[05:54] <tsimonq2> Unless you want to table it until later when the nightly is done and there are no jobs.
[05:55] <tsimonq2> Your call.
[05:56] <lubot> <teward001> i'll wait until later :P
[05:56] <lubot> <teward001> i'm headed to sleep now
[05:56] <tsimonq2> Okay.
[05:56] <tsimonq2> Have fun.
[05:56] <lubot> <teward001> let me know tomorrow when you've shut it :P
[05:56] <tsimonq2> I'll be up until the early morning.
[05:56] <tsimonq2> When you wake up just say something here and hard-reboot.
[06:06] <lubot> <teward001> @tsimonq2 saw that it stopped all builds and JFDI and upped its CPUs to 8.  That lets each CPU have 4 things scheduled.
[06:07] <tsimonq2> Thanks.
[06:07] <lubot> <teward001> with a load of 32 cap
[06:07] <tsimonq2> Sweeeet.
[06:07] <lubot> <teward001> it's hardrebooted now
[06:07] <tsimonq2> Much appreciated.
[06:16] -lugito:#lubuntu-devel- [rMANUAL0ef331fea445: Fix warnings sorry should have checked] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUAL0ef331fea445
[06:16] -lugito:#lubuntu-devel- [rMANUAL08d47c21b1bd: Fix warnings sorry should have checked] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUAL08d47c21b1bd
[06:16] <tsimonq2> Thanks Lyn, no worries. :)
[06:17] -lugito:#lubuntu-devel- [rMANUAL522d0ef82ec9: Fix warnings sorry should have checked] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUAL522d0ef82ec9
[06:26] -lugito:#lubuntu-devel- [rMANUAL5253f9fdf7d2: Fix table alignment finishes fixes for T161] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUAL5253f9fdf7d2
[06:29] -lugito:#lubuntu-devel- [rMANUALb39e0ff5b109: Fix table alignment finishes fixes for T161] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUALb39e0ff5b109
[06:29] -lugito:#lubuntu-devel- [rMANUALce90991c8aa1: Fix table alignment finishes fixes for T161] lynorian (Lyn Perrine) committed: https://phab.lubuntu.me/rMANUALce90991c8aa1
[11:07] <tsimonq2> I've done a very large amount of work over the last five to six hours on CI tooling.
[11:08] <tsimonq2> We have a better way now to see what exactly takes how long for jobgenerator.
[11:09] <tsimonq2> I had to fix the jobgenerator tool because it apparently wasn't updating the config files for jobs. Whoops.
[11:10] <tsimonq2> I did just kick off a nightly, so we should have a good idea as to what our Lintian failures are pretty soon.
[11:10] <tsimonq2> So if you see any 
[11:10] <tsimonq2> ..."unstable" jobs, that's why.
[11:11] <tsimonq2> I also did a lot of work on overhaul for Phab permissions.
[11:11] <tsimonq2> Since wxl messed things up in summer of last year where we now can't add new people to the Developers project, I decided it was about time to create several sub-projects to delegate things better.
[11:12] <tsimonq2> For example, Erich probably doesn't need commit access to our default settings but he could use commit access to Calamares.
[11:12] <tsimonq2> That's a little bit more fine-tuned now.
[11:29] -lugito:#lubuntu-devel- [rCI28c7f2b39705: Replace some timer calls.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCI28c7f2b39705
[11:29] -lugito:#lubuntu-devel- [rCI1b502ee4e435: Add a decorator to TimerMetrics and wrap most of the functions.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCI1b502ee4e435
[11:29] -lugito:#lubuntu-devel- [rCIc6f4117ab7f5: The removed timers were redundant.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rCIc6f4117ab7f5
[11:37] -lugito:#lubuntu-devel- [rPPABRITNEY363622addf6e: Update ubuntu-archive-tools.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rPPABRITNEY363622addf6e
[11:37] -lugito:#lubuntu-devel- [rPPABRITNEY7973af9bbbd2: Update britney2-ubuntu.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rPPABRITNEY7973af9bbbd2
[11:51] <tsimonq2> It seems like a lot of the builds previously failed on one of the most recent nightlies due to the build environment not being cleaned up correctly.
[11:52] <tsimonq2> Those builds should be retried.
[11:52] <tsimonq2> Also, on the bright side, seems like the Lintian stuff is working.
[11:52] <tsimonq2> SDDM builds are now being marked as unstable.
[11:52] <tsimonq2> Fun.
[12:38] -lugito:#lubuntu-devel- [rLIBLXQTPACKAGING683e8fa79f9d: Update symbols from amd64 build logs.] tsimonq2 (Simon Quigley) committed: https://phab.lubuntu.me/rLIBLXQTPACKAGING683e8fa79f9d
[17:08] -lugito:#lubuntu-devel- [T131: Version Number in Plymouth?] apt-ghetto (apt-ghetto) commented on the task: https://phab.lubuntu.me/T131#3482
[20:16] -lugito:#lubuntu-devel- [T162: Openbox standalone] hmollercl (Hans P. Möller) just created this task: https://phab.lubuntu.me/T162
[22:16] <lubot> <tsimonq2> Hey @lynorian this might be worth looking into: https://www.reddit.com/r/Lubuntu/comments/gl3iya/lubuntu_2004_sensors_widget_set_crit_temp_for/?utm_medium=android_app&utm_source=share
[22:16] <lubot> <tsimonq2> Thanks for your really quick turnaround on that task btw
[22:17] <lubot> <tsimonq2> Looks like the usual builds are still FTBFS, but we now have some "unstable" builds if anyone wants to get some practice cleaning up Lintian errors
[22:18] <lubot> <tsimonq2> I might actually make Lintian exit 1 on non-critical
[22:18] <lubot> <tsimonq2> But I feel like a better use of my time might just be making something like a status page that highlights the number of Lintian flags are raised and on which packages
[22:51] <lubot> <RikMills> Qt 5.14.2 landing in proposed, FYI