[00:42] <phyphor> tomreyn: "on an affected Ubuntu 22.04 system" I mean, I can install a 22.04 VM and not install squid if that'll help
[00:54] <phyphor> oh, "affected" - give me a moment
=== xenial is now known as Guest3819
[01:34] <phyphor> rbasak: just saw your post in the bug - unfortunately we can't provide the network traffic but I had hoped that it being a known bug in squid would be sufficient
[01:35] <phyphor> I can spin up an additional proxy in our set-up with any proposed fix and keep an eye on it, if that helps
[01:47] <phyphor> tomreyn: submitted - thanks for the assist and apologies for misreading earlier (in my defence it's 2am here)
[01:54] <tomreyn> phyphor: thanks for adding the extra info.
[01:57] <phyphor> foolishly I assumed you could just throw up a later version of squid
[01:58]  * phyphor isn't very wise to this sort of thing and it's been a few years since I hung out with any Canonical people to have asked them in person
[08:30] <rbasak> phyphor: unfortunately just throwing up a newer version isn't acceptable in a stable release. We need to take some steps to make sure we aren't inadvertenly regressing users. We'd like to verify that the bug is actually fixed, and also ensure that we aren't regressing the package for anyone else. If we can provide steps to reproduce in the bug, that would be the easiest path.
[08:31] <rbasak> Like for example here it might work to have a script that opens and closes tunnels in the way that causes the leak. Then we'd be able to demonstrate that running it many times reduces memory consumption after the patch is applied.
[09:00] <phyphor> I would argue that having 5.2 in the stable release did regress users but it's easy for someone outside the process to criticise without understanding
[09:01] <phyphor> so, I'll see if I can knock together a script but if I can't I'm happy to run an additional proxy in production with whatever fix is applied to see if it survives
=== xenial is now known as Guest3142
[10:59] <rbasak> phyphor: indeed it did apparently regress users who took a deliberate action to upgrade. But if, in trying to fix it, we inadvertently regress users who aren't even upgrading to a new release, that would be even worse!
[15:11] <phyphor> I must be missing something as I thought putting a later version of package in the repo for Jammy would only afect those who upgraded to Jammy. But luckily I don't work anywhere that my ignorance can cause problems :D
[15:13] <rbasak> phyphor: the package update will be recommended to all Jammy users of the squid package. So we need to make sure that dosen't break them.
[15:14] <phyphor> Oh, right, but the problem affects a fresh install of Jammy - squid 5.2 is broken
[15:14] <rbasak> Sure, but only for people who use tunnels in a particular way.
[15:14] <phyphor> true
[15:14] <rbasak> Lots of users use squid in other ways, and aren't currently broken.
[15:14] <rbasak> We need to make sure our changes don't affect them.
[15:14] <phyphor> ah, and Ubuntu can't trust that squid didn't add a bug
[15:14] <phyphor> gotcha
[15:15] <rbasak> Right. So we're happy to add this patch in principle, but we have to do it carefully!
[15:15] <phyphor> I was confused by the point  
[15:15] <phyphor> "who aren't even upgrading to a new release"
[15:15] <phyphor> but I think we were talking past each other - and I'm grateful for you taking the time to explain
[15:16] <rbasak> Thanks! So we want steps to reproduce the bug, to think about how the code changes might inadvertently affect other use cases, etc.
[15:16] <rbasak> That way we make sure we really fix the bug with the patch, and don't need to risk disruption to unaffected users a second time, too.
[15:16] <rbasak> I think reproduction steps should be possible here? Because the fix is known, I think the conditions to reproduce the issue are also known.
[15:17] <rbasak> Just need a load generator to open and terminate tunnels in such a way as to demonstrate the leak.
[15:18] <rbasak> It's not a hard requirement, but it does help give us confidence in the fix. If it's really not possible then we can mitigate risk in other ways, but a reproducer is easier if that can be done.
[15:20] <phyphor> I've spun up an additional proxy in one of our locations, running Jammy, and can put it into service to test
[15:20] <phyphor> load generation from actual use :D
[15:24] <rbasak> That will be very valuable when we're verifying the fix - thanks!
[15:24] <rbasak> The baseline expectation though is that the bug explains to everyone how to reproduce the problem.
[15:25] <rbasak> But verifying the fix (once the final proposed binary is available for testing) by saying that you've been testing it in a production environment that reproduces the issue and verified that it fixes the issue would be very helpful
[15:30] <phyphor> the steps to reproduce were simply "set up squid as a proxy, as part of a highly available set up, for several thousand VMs"
[15:30] <rbasak> It's possible to come up with a much more minimal way of tickling the bug
[15:31] <phyphor> sure, I just didn't dive into it once I found that squid had identified and fixed it
[15:31] <phyphor> let's see how the bug was riased with squid
=== Eickmeyer is now known as NotEickmeyer
[16:08] <phyphor> looks like that all used real world use-cases