| guruprasad | cjwatson, icymi, my colleague Tushar had asked a question - https://irclogs.ubuntu.com/2024/12/05/%23launchpad-dev.txt. When you get some spare time, can you share your thoughts on that please? Thanks! | 13:24 |
|---|---|---|
| cjwatson | guruprasad: I don't remember the fine details. In general it was something that I thought might be worthwhile provided that the data sets involved were completely disjoint, both at the database level and at the filesystem level, so that we wouldn't have to worry about locking. | 13:26 |
| cjwatson | guruprasad: It should be possible, but I never went through and exhaustively checked for odd corner cases. | 13:26 |
| guruprasad | Thanks. We are looking at things at 2 different levels - the first about just parallelizing each grouped invocation (debian, ubuntu, derived distros + public & private PPAs) | 13:27 |
| guruprasad | and the next about how in one invocation the publisher just iterates over the archives in a for loop. | 13:27 |
| guruprasad | Iiuc, Tushar's question was about the former. | 13:27 |
| cjwatson | guruprasad: It's pretty much the same fundamental question either way, anyway. | 13:27 |
| guruprasad | We suspect that there shouldn't any inter-archive dependencies in the PPA publishing process and this information is helpful. | 13:28 |
| cjwatson | guruprasad: Different Archives should be disjoint enough, but there might be some buried oddities around things like consistency checks - some of those are cross-archive, think things like publications in PPAs that do some file name conflict checks against the primary archive. Worth some careful thought. And I bet the scripts at the outside of the publication process have assumptions like being | 13:32 |
| cjwatson | able to take a particular lock file name. | 13:32 |
| guruprasad | Hmm, I read through the publisher code a few times recently and did not notice any cross-archive checks. I will check again now. | 13:34 |
| cjwatson | guruprasad: IIRC it's more in the process-upload sort of area | 13:34 |
| guruprasad | We are aware of all the stages in the current publishing process using the same lock file and will have to split them out. | 13:35 |
| cjwatson | Or process-accepted maybe | 13:35 |
| guruprasad | Ah, those I haven't checked. | 13:35 |
| cjwatson | nascentupload can definitely compute cross-archive ancestry, and copy checks can be cross-archive. I don't remember whether copy checks can ever end up being done as part of publication. | 13:38 |
| cjwatson | I don't think any of this should be a fundamental blocker, to be clear, just stuff you'd need to rule out. | 13:38 |
| guruprasad | Sure, thanks. These are helpful things to check. | 13:41 |
| cjwatson | It's likely easier when the Distribution is also different - but Ubuntu accounts for such a high percentage of the data that you likely won't get very much benefit from splitting at that level anyway. | 13:43 |
| guruprasad | Yeah, from what I have seen, everything other than Ubuntu is negligible or a no-op most of the runs. | 13:48 |
| cjwatson | You might save a bit from parallelizing the startup times, I suppose. | 13:52 |
| guruprasad | Can you explain what you meant by 'startup time' here? | 13:56 |
| guruprasad | Is it just the loading of the whole Launchpad stack before a script can run? | 13:56 |
| cjwatson | That | 13:58 |
| guruprasad | There is also a big elephant in the room, "that private distribution", which generates ~50000 OOPSes every day while publishing to Artifactory. :( | 14:01 |
| guruprasad | We will try to fix issues one by one to see how much improvement we can get. | 14:02 |
| guruprasad | Even if we shave of 5 minutes off each PPA publisher run, that will still be helpful because it adds up over time. | 14:02 |
| guruprasad | *shave off | 14:02 |
| cjwatson | Yeah, you'd probably get a fair bit of improvement by doing something about those OOPSes ... | 14:02 |
| guruprasad | Most of those are PoolFileOverwriteErrors, which we (when wgrant was here) investigated and suspected to be due to gaps in mapping the source packages to the wheels like how it is done for the deb packages. | 14:22 |
| guruprasad | So when we fix that one issue, it should solve a large number of those OOPSes. | 14:23 |
| guruprasad | 🤞 | 14:23 |
| cjwatson | Mm, wouldn't be a huge surprise. | 14:24 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!