| guruprasad | cjwatson, do you have a few spare minutes to answer some questions about buildbot? I see that you have been contributing to the maintenance of the Debian buildbot package recently. I wanted to confirm that the package still doesn't have the plugins and web bits that lpbuildbot uses, right? We are evaluating upgrading to a model similar to the LP charms where we install the vendored deps using pip and wanted to confirm. Also, if you have any | 08:03 |
|---|---|---|
| guruprasad | advice, please let us know. Thanks! | 08:03 |
| guruprasad | We want to get off the manual set up that we have now for lpbuildbot soon | 08:04 |
| cjwatson | guruprasad: Most of the remaining bits in lpbuildbot were pretty custom for LP. I don't think any of them went upstream, though I haven't checked in any detail and there could be odds and ends that are simplified by newer versions. | 09:57 |
| cjwatson | guruprasad: You definitely need some kind of pinning (whether that's via choice of Ubuntu version with associated system packages, or pip), since some of the bits in lpbuildbot are very version-delicate. If I were trying to port to a new version then I'd be particularly careful about things such as GitPoller which involved cloning-and-hacking of bits of buildbot to change particular behaviours - | 10:00 |
| cjwatson | you'd want to compare against the version they were designed for, and then see which changes from buildbot you need to incorporate into the lpbuildbot code. | 10:00 |
| cjwatson | guruprasad: Though actually it looks as though the need for that cloned-and-hacked GitPoller might go away with buildbot 4.1.0. That'd be nice. | 10:02 |
| cjwatson | guruprasad: If you can get authentication working (I think either Tom or I tried but failed for some reason I forget), then some other things can probably be simplified - you might be able to use buildbot's own support for forcing builds rather than the hack that's currently in lpbuildbot for that. | 10:05 |
| cjwatson | guruprasad: buildbot seems to have support for subunit observing nowadays, and it may be worth seeing whether LP can use that rather than injecting its own (I _think_ the code in lpbuildbot dates from before buildbot supported this). | 10:06 |
| cjwatson | guruprasad: The only other bit of any complexity is AggregatingScheduler. I've long since paged out the details of that so you're on your own there :) | 10:08 |
| guruprasad | Thanks cjwatson. This is helpful. | 10:21 |
| guruprasad | I had a lot of trouble reading and understanding the AggregatingScheduler code but at least resigned to understand what it tries to do at a high level instead of trying to understand every line of Twisted code in it. | 10:22 |
| guruprasad | I also wonder if we can drop that and live with the default scheduler and its drawbacks in comparison to the AggregatingScheduler if porting it ends up being too difficult. | 10:23 |
| guruprasad | Since we do not understand a lot of what most of the custom code does, there is also a thought about starting with vanilla buildbot, finding out what doesn't work well and look into customizing those. | 10:24 |
| guruprasad | Instead of assuming that all the lpbuildbot customizations are needed and trying to port them forward. | 10:24 |
| cjwatson | I would not recommend that approach ... | 10:27 |
| cjwatson | It's not so much custom code that it's impossible to understand. | 10:27 |
| cjwatson | You could drop the AggregatingScheduler if you're OK with the consequence, which is that if you land several changes in close succession then buildbot will get backed up because it will do a test run for each in turn. | 10:28 |
| guruprasad | I think our challenges are two-fold - lack of prior Twisted knowledge (this can be bridged) and lack of understanding of buildbot itself and its patterns (this a lot more tough) | 10:28 |
| cjwatson | That might be not quite so bad now that it's faster than it used to be. | 10:28 |
| guruprasad | Yeah, I managed to understand what the AggregatingScheduler does. | 10:29 |
| cjwatson | I don't _think_ the default scheduler will do a run for each individual commit introduced by a merge commit, but you'd probably want to rig up something to test that if you were going to drop AggregatingScheduler. | 10:29 |
| guruprasad | The builds run in ~1 hour these days. So it might be okay to just rely on the treeStableCount used by the default scheduler and see how well it works/doesn't work | 10:30 |
| cjwatson | Honestly I think if you got onto 4.1.0, dropped GitPoller, maybe switched to its built-in subunit observing, and maybe were able to use the default scheduler, the rest of lpbuildbot would not remotely be a problem to maintain | 10:30 |
| cjwatson | (by which I mean >=4.1.0, not specifically that version) | 10:30 |
| guruprasad | Do you know if there is a path forward at all to get all the plugins in the upstream deb package? It has been many years and releases. | 10:30 |
| guruprasad | And I do not see any information pointing to the upstream issues about this getting resolved. | 10:31 |
| cjwatson | No idea, sorry. I've been doing basic package maintenance but only really superficial stuff. | 10:31 |
| guruprasad | No problem, thanks. | 10:31 |
| guruprasad | This is so helpful already. | 10:31 |
| cjwatson | https://bugs.debian.org/1004344 looks like the main bug report about this. | 10:32 |
| cjwatson | But my appetite for digging through JS packaging stuff is fairly low. | 10:32 |
| cjwatson | I can do Python! | 10:32 |
| cjwatson | Re challenges: if one of your problems is lack of understanding of buildbot itself etc., then exactly what you _don't_ want to do is to throw away code written by previous people who've beaten their heads against this without first working out what it does. | 10:34 |
| cjwatson | If you were all buildbot experts, then I'd absolutely say you could start with vanilla and it would be a quick job to figure out the necessary customizations. | 10:35 |
| cjwatson | But when you have a knowledge gap, IMO that's backwards. | 10:35 |
| guruprasad | Okay, that makes sense | 10:36 |
| guruprasad | Our only concern with not throwing away the existing code is that it is for a buildbot version so old that a lot of those things may not be needed any more. | 10:37 |
| cjwatson | By all means once you upgrade and find that something is superfluous _then_ throw it away :) | 10:37 |
| guruprasad | But I guess we have to start somewhere and looking at the existing customizations and the version they were written for is a good place. | 10:37 |
| cjwatson | I forget what you're on at the moment. xenial? Or was the buildbot manager still stuck on precise or something? | 10:38 |
| cjwatson | I thought we got it onto xenial. | 10:38 |
| guruprasad | bionic with a hand-ported, manually setup package that you installed somehow. | 10:39 |
| cjwatson | Oh, that's not so bad. | 10:39 |
| cjwatson | What buildbot version? | 10:39 |
| guruprasad | I think it is the xenial package that you forward ported. | 10:39 |
| guruprasad | 0.8.12 iirc | 10:39 |
| cjwatson | Did I happen to leave records of why? I wonder if that was because we still had to support xenial builders at the time. | 10:40 |
| cjwatson | And we only dropped xenial builders shortly before I left | 10:40 |
| guruprasad | I think one of the reasons I remember reading you mention somewhere was that the xenial version was the last one that had all the plugins and web stuff bundled in. | 10:41 |
| guruprasad | *the xenial version of the buildbot package | 10:41 |
| cjwatson | Hm. OK, I don't remember that but I guess it's possible. | 10:41 |
| guruprasad | From bionic onwards, they were dropped from the debian package, I think | 10:41 |
| cjwatson | I suppose that's the Debian bug above then. | 10:41 |
| guruprasad | Yes | 10:41 |
| cjwatson | So yeah, might not be silly to see if you can get an equivalent setup with pip, and then it should be easy enough to try upgrading gradually. | 10:42 |
| cjwatson | I might get round to looking at that Debian bug eventually but it's unlikely to be any time soon - at the moment I don't have a need for it | 10:43 |
| cjwatson | Or a customer who cares | 10:43 |
| guruprasad | Yeah, I understand | 10:45 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!