[00:00] NCommander: They've been implemented for well over a year. [00:00] And permissions use them now. [00:01] wgrant: ah, I thought there were still bits that required implementation in Launchpad (I saw the IPackageset and friends, but not sure what the status was) [00:02] NCommander: There are a few bits and pieces that don't respect those permissions yet (bug nomination appproval and build retries come to mind), but most of it works and is in active use. [00:02] wgrant: they are? [00:02] * NCommander didn't know this [00:03] wgrant: so its possible to generate Packages files with Sets: *blah* in them? [00:03] NCommander: That would not be too difficult. [00:04] Except for the difficulty inherent in the crazy apt-ftparchive integration. [00:05] wgrant: well, APT has to be patched regardless to understand sets [00:06] wgrant: is there a way to query sets currently in LP? [00:07] NCommander: Yes, through the API. Check out edit_acl.py in ubuntu-archive-tools. [00:07] wgrant: ah, I knew about those, just didn't realize those were IPackagesets in the backend [00:10] wgrant: I thought LP was changed to generating the Packages files directly off the backend and obseleting apt-ftparchive [00:10] NCommander: Ha ha ha. [00:11] NCommander: It does for PPAs and partner. [00:11] oops [00:11] But primary and copy archives still use apt-ftparchive. [00:11] wgrant: fun [00:11] wgrant: performance? [00:11] lifeless: probably an issue with Contents-* files and other fun stuff [00:11] * NCommander went down this road with dak [00:11] lifeless: Probably not, but it's unclear. [00:11] The main problem is with overrides. [00:11] NMAF doesn't really support them. [00:13] Contents files are another point, but they're generated somewhat separately. [00:14] (it would be nice if LP could generate them natively for PPAs as well, but a table listing all files would be HUGE. [00:17] Internet doesn't like me at the moment [00:17] wgrant: at the risk of being terrorifed, how do Tasks then end up in the packages file? [00:17] overrides [00:17] Magical magical override files. [00:17] Not really generated by LP. [00:17] But by something that lives in the publisher cronscript. [00:18] elmo: I'm guessing no Soyuz people are around to fix that custom upload bug? [00:19] wgrant: I guess not, sorry. if you have a trivial patch, I'm willing to cowboy it on [00:20] elmo: See the branch on the bug. With tests and everything. [00:20] oh, well, I didn't actually subscribe myself \o/ [00:20] (cjwatson's fix was slightly wrong) [00:21] aaaaa, LP code ui, you make me cry [00:21] What's it doing to you now? [00:22] just going from that bug to your actual diff is requiring me to think [00:22] and I'm not doing well at that right now [00:22] wgrant: I just looked at that crontab and .... [00:22] * NCommander cries [00:23] NCommander: Yes, cron.publish-ftpmaster is scary. [00:23] elmo: It's easy if there's an MP, which there isn't yet. [00:24] wgrant: the changes look pretty straightforward, I just need to amend the extras sweep in cron.germinate (which probably will become to cron.extras) to dump Packagesets in-mass, and then let apt-ftparchive handle it [00:24] http://bazaar.launchpad.net/~wgrant/launchpad/fix-custom-uploads-with-root-dirs/revision/10460 and http://bazaar.launchpad.net/~wgrant/launchpad/fix-custom-uploads-with-root-dirs/revision/10461 [00:24] Although the latter isn't important. [00:24] NCommander: Right. [00:24] wgrant: yeah, I foujnd it eventually [00:24] wgrant: now I'm applying my lack of thinking to the actual diff [00:25] elmo: An issue brought up in https://code.edge.launchpad.net/~julian-edwards/launchpad/symlink-vuln-bug-529710/+merge/20543 is what caused the bug. [00:25] wgrant: do we support a Supported: field ATM? (I see some references to it, but nothing in publisher) [00:26] NCommander: See maintenance-check.py, and whatever calls it. [00:26] wgrant: I'm assuming this is just a context thing (i.e. I don't have enough in the diff), but I don't see how this code actually stops $tmpdir/../../../ other than having to guess $tmpdir? [00:26] oh, duh [00:26] realpath [00:26] nevermind [00:26] Right. [00:26] realpath resolves symlinks and normpaths, eliminating that hole. [00:32] Once you've cowboyed it, process-accepted should successfully process those uploads on its next run. [00:32] I'm happy enough to cowboy this, given it can only break customuploads [00:32] Right. [00:35] wgrant: all files is only about 2GB of data [00:35] wgrant: (for all files + all packages + per-binary file knowledge + relationships between packages) [00:35] lifeless: err, not for PPAs [00:35] wgrant: 'conflictchecker' has this, and can query single package conflicts quickly [00:35] And not for multiple releases of each source package. [00:36] wgrant: conflict checker does precisely that [00:36] Hmm. [00:37] elmo: I don't have the data to estimate for PPAs; if there is interest I could page in conflict checkers schema and figure out what we need to know to estimate it for PPAs. [00:37] by per-binary I mean per-arch [00:37] when you're talking about HUGE tables in launchpad, bear in mind we have TranslationMessage and BranchRevision :-) [00:37] mwhudson: That's true. [00:37] though the last is totally silly [00:37] But BranchRevision is stupidly huge. [00:37] Yes. [00:38] please tell me BranchRevision is not what I think it is [00:38] for added fun, conflictchecker was running on sqlite last I saw. [00:38] elmo: lalalala fairies [00:38] elmo: its what you think it is. [00:38] elmo: Yes, every branch of Launchpad actually has 70000 BranchRevision rows linked to it. [00:38] AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA [00:40] i wonder if we should look at neo4j for this sort of thing [00:40] Hm, maybe. [00:40] however right now i'm going to have lunch and water the garden [00:41] elmo: what machine is conflict-checker running on at the moment? [00:41] lifeless: macaroodle or something [00:41] lifeless: macaroni or macquarie [00:41] probably macaroni [00:42] macaroni indeed. [00:42] yeah, it's the source of constant 1 load on macaroni [00:43] hmm, 3GB now; probably need to ditch some old dev edition packages. [00:45] wgrant: your patch is on production now [00:46] elmo: I guess we'll see if it works in 16 minutes. [00:46] Thanks. [01:00] wgrant: select count(*) from filepath; [01:00] 8319961 [01:01] lifeless: Is that for all SPRs, or just current ones? [01:02] wgrant: thats all filenames seen in binary packages ever. [01:02] wgrant: (but it only scans the official archive) [01:02] Hm, not bad. [01:03] sqlite> select count(*) from binpackagefiles; [01:03] 28807743 [01:03] .schema binpackagefiles [01:03] CREATE TABLE binpackagefiles (package_id INTEGER, file_id INTEGER); [01:03] CREATE INDEX _bpf_fileid on binpackagefiles (file_id); [01:03] CREATE UNIQUE INDEX _unique_packagefile ON binpackagefiles (package_id, file_id); [01:04] I guess we could always purge the lists when we expire PPA sources and binaries, if it becomes a problem. [01:04] elmo: Did it work? [01:04] wgrant: oh, I wasn't looking [01:04] wgrant: didn't the previous upload get rejected? [01:04] elmo: Unlikely. It should have just sat in ACCEPTED. [01:04] wgrant: its really quite modest. You need to be careful to use indices, but thats about it. [01:04] wgrant: ah [01:04] Actually, I should be able to check the queue pages... [01:05] if we had the blacklisting done, for handling hand-verified maintainer-scripts, we would probably keep indefinitely ahead of the archive. [01:09] elmo: They're all DONE now, so it probably worked. [01:10] wgrant: sweet, thanks [01:10] Ah, yes, we have binary publications too. All is good. [01:18] neo4j looks handy, apart from the j bit [01:39] Ah. python2.5 has gone from lucid [01:40] +copy-packages time [01:40] :( [01:43] um. What does it mean when a single source package version has two source publications, one superseding the other?! [01:43] oh, override change? [01:43] Normally, yes. [01:45] oh this could be fun. So I need to pick the right one such that the package ends up in the PPA's main :-) [01:47] ngh [01:47] And I can't retry builds in the LP PPA [01:54] Can someone in ~launchpad hit retry on these, please: https://edge.launchpad.net/~launchpad/+archive/ppa/+build/1553604 https://edge.launchpad.net/~launchpad/+archive/ppa/+build/1553605 [02:00] maxb: done [02:00] thanks [02:29] Hrm. [02:29] launchpad-developer-dependencies Depends spidermonkey-bin [02:29] which has just been removed from lucid [02:31] i wonder what that's for [02:32] * maxb files it under the 'Someone thought it was a good idea' heading :-) [02:38] Added anonymously in 0.35, October 2008 [02:39] sounds like something that could be deleted [03:23] maxb: PPA publications are overridden to main by a deeper hack, so it doesn't matter which you copy. [03:46] :( [03:46] spending lots of time reading code [03:46] and not so much time writing [04:17] * mwhudson is writing some very boring tests [04:18] \\ ? === thekorn_ is now known as thekorn [08:09] stub: I don't suppose fragment caching works within an arbitrary loop? [08:09] It does [08:16] wgrant: hi, shall I land your buildstatus-to-buildmaster branch, or have you already organised it since fixing the test? [08:17] I can't see it in the changelog, so I'll do so. [08:17] stub: how does it know which iteration is which in the cache? [08:18] A counter is stored as a request annotation, and this is used to form part of the key. [08:23] good morning [08:23] Hi adeuring [08:26] hi adeuring, hi noodles775 [08:26] hi noodles775, jtv! [08:26] G'day jtv :) === gmb` is now known as gmb [09:03] Hallo [09:06] noodles775: I haven't organised it. Thanks. [09:07] noodles775: Can you please also land my fix-build-breadcrumbs? It failed a couple of tests overnight which I've fixed. [09:08] wgrant: sure. [09:55] stub: are you around much longer? I'd like to talk to you about how to fix https://bugs.edge.launchpad.net/soyuz/+bug/517101 [09:55] Bug #517101: update-pkgcache.py needs to be database friendly [09:55] I'm free [10:03] Back later [10:03] wgrant: for +copy-packages too? http://ppa.launchpad.net/maxb/ppa/ubuntu/pool/multiverse/s/sun-java5/ says otherwise :-) [10:03] maxb: That was ages ago. [10:03] * wgrant checks the log. [10:05] oh, ok. I admit to not having a more recent example [10:05] bigjools: So, would today be a good day to talk about Copyright stuff for native-source-sync? [10:06] persia: yes maybe in an hour? [10:06] bigjools: That sounds great. Thanks. [10:07] great [10:08] persia: Changelog stuff? [10:08] wgrant: Yes, thanks for the correction. [10:08] henninge: --force-depends works [10:08] Messy :( [10:09] jtv: sure ;-) [10:13] maxb: It was fixed two days after those were published. [10:20] wgrant: can you propose a merge for your custom upload fix [10:23] bigjools: Sure. [10:23] thanks for fixing it, I'll get it landed [10:24] I was out yesterday [10:24] Thanks. [10:25] Unfortunately I last tested the original fix with a real upload before the os.sep change was added :( [10:26] :( [10:32] so I am setting up LP on a new laptop on lucid, and rocketfuel-setup runs aptitide which eventually says launchpad-developer-dependencies depends on spidermonkey-bin ... ?! [10:33] bigjools: Yeah, see the ML. [10:33] It was removed recently. [10:33] ok I'm well behind on email :( [10:34] bigjools: Should I specify you as the reviewer? [10:34] wgrant: yes [10:37] Is there any specific reason rocketfuel-setup runs aptitude? I've heard the aptitude resolver isn't widely tested (and seen any number of bugs for stuff that didn't work with aptitude that did work with other resolvers) [10:46] I suspect whoever wrote that part of the script just happens to also prefer it [10:46] we should change it === matsubara-afk is now known as matsubara === noodles785 is now known as noodles775 [10:57] damn [10:57] call in 6 hours [10:57] best go to be [10:57] d [11:00] Morning [11:09] Do the Canonical webapp engineering teams not talk to each other? [11:09] I have found the same obvious hole in three Canonical webapps in the past two weeks. [11:13] wgrant: not as much as we should, we're setting up an architectural team to address that [11:16] wgrant: not getting my Bob to pick up the jobs... are you sure I don't need to run anything? [11:18] jtv: As long as buildd-manager is working, it should be working. What do buildd-manager's logs say? What does an XML-RPC status() call on the builder say? [11:19] wgrant: hmm... in /var/log/launchpad-buildd/default.log I see it shutting down after a SIGTERM, and nothing after that [11:19] jtv: Heh, you might want to start it. [11:19] yes, I might [11:20] wgrant: time to remove the -e I added to that helper script :) [11:20] A good idea. [11:22] still no joy though :( [11:22] What does buildd-manager say? [11:23] wgrant: nothing in the log, nothing in the shell [11:24] I tell a lie [11:24] jtv: Hmph. The builder is responding with IDLE? [11:24] startup failed because there's "a twistd" already running [11:25] Hmm. [11:25] yup, buildd manager was still running. [11:25] I'll try killing that instance and restarting [11:25] bigjools: Let me know when you have time :) [11:25] How often will bug heat get refreshed do we think? [11:25] persia: right now [11:26] bigjools: Great. So my current understanding is that there are 3-4 things involving changelogs that need to be sorted before we can turn on native-source-sync, and that you know which these are. Is this true? If so, could you expand? [11:26] persia: my memory is very hazy on this, I'll see if I wrote something down [11:27] ok so there's a few bugs [11:28] https://bugs.edge.launchpad.net/soyuz/+bug/55795 [11:28] wgrant: I have a "starting scanning cycle" in the buildd-manager log, but nothing else. :) [11:28] Bug #55795: +changelog includes misleading information related to package versions and authors

[11:28] jtv: The builder is still marked OK, and not manual? [11:28] https://bugs.edge.launchpad.net/soyuz/+bug/139162 [11:28] Bug #139162: Store the pristine debian/changelog for each SourcePackageRelease

[11:28] jtv: Can you see buildd-manager's polls in /var/log/launchpad-buildd/default? [11:28] https://bugs.edge.launchpad.net/soyuz/+bug/247456 [11:28] Bug #247456: changelog listing are incomplete and/or undiscoverable [11:28] persia: basically it's horrible right now [11:29] wgrant: the builder is OK and not manual [11:29] Also we need to be able to create something akin to a changes file. [11:29] wgrant: it's not logging to there, it turns out; it's logging to /var/tmp/development-buildd-manager.log [11:30] All I see there is "Starting scanning cycle." Lots of times. [11:30] jtv: That's buildd-manager. I want launchpad-buildd. [11:30] ahh [11:30] bigjools: There's a few other bugs surrounding that which I've seen, but yeah. So do we actually need the changelog for anything, or could we just store it as requested in 139162 and present it as requested in 55795? [11:31] wgrant: I've been looking at /var/log, but I guess that's on the slave. :) [11:31] persia: 139162 is the first task [11:31] wgrant: 2010-03-10 11:31:18+0000 [HTTPChannel,910,127.0.0.1] 127.0.0.1 - - [10/Mar/2010:11:31:18 +0000] "POST /rpc HTTP/1.0" 200 222 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)" [11:31] wgrant: repeats [11:32] jtv: OK, so it is polling. [11:32] So maybe the candidate selection is not working properly. [11:32] persia: once we've done that the NSS becomes easier [11:32] wgrant: I guess. I was hoping to be able to stay out of that. :) [11:33] But oh well. :) [11:33] jtv: Check your postgres log for the query that buildd-manager is using. [11:34] bigjools: Right. So we have code that does 139162 that we use for the changelog download in apitutude and update-manager, storing stuff on changelogs.ubuntu.com. [11:34] Run that, and work out why it isn't returning your job. [11:34] If it is, there is something more sinister going on. [11:34] persia: we also need to repackage existing records [11:35] bigjools: How do you mean "repackage"? [11:35] wgrant: there's no extra-verbose logging in the script that I can enable? [11:35] bigjools: Can you answer jtv? You know the logging much better now. [11:35] persia: the existing changelogs need to be re-processed [11:36] bigjools: I'm trying to figure out why the buildd isn't picking up my jobs. [11:36] bigjools: Is there any useful information there, or do we just need to extract changelogs from the existing packages? [11:36] jtv: got a b-m log? [11:37] persia: AIUI that's pretty much it [11:37] bigjools: just says "Starting scanning cycle." all the time. [11:38] jtv: then your job is not eligible for dispatch (either a bug or genuinely) [11:38] bigjools: so as wgrant just pointed out, sounds like candidate selection. [11:38] bigjools: And then 55795 becomes easy because we can just expose that blob? [11:38] bigjools: right, so my question is: is there debug logging in the script that'll make this easier to pinpoint? [11:38] jtv: see the query it's issuing in the PG log [11:38] and go from there [11:38] bigjools: that's what he said. OK, I'll try that, thanks. [11:38] jtv: there's no debug logging that will help :( [11:38] it's the candidate selection query [11:39] persia: exactly [11:39] Which I know works generically now, but you may be missing a piece for your translations job. [11:39] persia: the way attribution is done is all messed up as well [11:39] yep [11:39] bigjools: That goes away if we use mvo's code for 139162 [11:40] persia: it would need to work in LP, but that's the general idea yeah [11:40] bigjools: And then 247346 gets resolved trivially as well? [11:40] with any luck :) [11:40] Now, how ought this be implemented in LP? [11:40] jtv: paste the query from the pg log into a psql prompt and experiment [11:41] bigjools: thanks... for now I'm just checking it manually against the data I have. [11:41] persia: depends on what that code looks like, I've not analysed any of this [11:41] bigjools: ahhh, got it: "(buildqueue.virtualized IS NULL OR 'f' IS TRUE)" [11:41] I reviewed that code, so it can't be wrong. :-) [11:41] the classic :) [11:41] bigjools: The code basically just knows the format of a .deb, and extracts the changelog. [11:41] bigjools: That logic can be embedded however is convenient. [11:42] We can do the same for a .dsc, but we need an unpack environment. [11:42] well it already does it for dsc, just not very well [11:42] the upload processor unpacks the source [11:43] We already have code to grab debian/copyright. [11:43] bigjools: it does actually make sense, it's just that it reflects... IIRC whether the buildd manager is looking for arch-independent build candidates. [11:43] Which I believe was touched last week... [11:43] indeed [11:44] So really, that code should just spit debian/copyright into the DB? [11:44] And then we can expose it? [11:44] it does already [11:44] we just don't expose it [11:44] So what's missing in 139162 again? [11:44] https://dev.launchpad.net/Soyuz/Specs/RawSourceChangelog [11:44] ^^ that [11:47] I don't see the missing bit, if we're already stuffing full raw changelogs into the DB. [11:47] we're not, we're extracting bits of it [11:47] Or is it just the exposure and client changes? [11:47] Aha! [11:48] and then trying to re-attribute them [11:48] We store debian/copyright, not debian/changelog yet. [11:48] it all blows quite spectacularly [11:48] right, we store changelog_entry which is a partial changelog [11:49] * persia slaps forehead [11:49] Right. So we just need to pull debian/changelog *when* we pull debian/copyright. [11:50] Now, do we actually need any of the code that attempts to construct a changelog, or can we scrap that entirely? [11:50] I'm not sure yet, I suspect it could be scrapped but bits of it might get re-used in the presentation [11:51] stub: ok can we talk about pkg-cache now? [11:51] yup [11:52] stub: where does the problem lie, in the fact that it's reading too quick? or udpating? [11:52] bigjools: Thanks! [11:52] or both? [11:52] persia: no worries! I hope it's filled a gap [11:52] updating too quickly without regard to the system load [11:52] persia: and sorry it took me so long to get back to you [11:52] stub: so I could put a loop tuner at the top level where it reads package names in [11:52] which would also throttle writes [11:53] Ideally it should block when the system is busy. That logic is centrally encoded in DBLoopTuner at the moment. [11:53] yeah [11:53] I haven't looked at the existing structure of the script [11:53] stub: it's a wrapper around various bits [11:54] stub: it calls IDistribution.updateCompleteSourcePackageCache [11:54] I need to move that out of that model code and into soyuz somewhere [11:54] then I think I can just slap a loop tuner on there and profit [11:54] So updateCompleteSourcePackageCache sounds like a chunk that should be its own DBLoopTuner loop [11:55] exactly [11:55] You could install it into garbo-daily.py if you like - this fits in with its role. [11:56] * stub looks at updateCompleteSourcePackageCache [11:57] Yer - doesn't look like much conversion to turn that loop in updateCompleteSourcePackageCache into a DBLoopTuner loop. [11:59] al-maisan: wgrant, bigjools: looks like my job should have "virtualized" set to true in order to be selected by a non-virtualized builder [11:59] jtv: Didn't I convince you to set bob to virtualised? [12:00] jtv: not setting it (i.e. a NULL) will be eqivalent to virtualized=True [12:00] You should set virtualized to either True or None in order for it to build on a virt builder. [12:00] False will let it build on non-virt. [12:00] wgrant: I may easily have gotten confused between the setting on the builder and on the jobs. [12:02] wgrant: ah, I see it: I skipped a step this time that I had documented [12:02] Ahh. [12:02] sorry for the noise [12:02] now I guess I wait for it to poll... [12:03] It should poll every five seconds. === mrevell is now known as mrevell-lunch [12:41] bigjools: Thanks for landing that. [12:42] noodles775: I see that buildstatus-to-buildmaster should be landing RSN, but I have no EC2 email about the other one yet. [12:43] wgrant: welcome [12:44] is someone already fixing the spidermonkey-bin problem BTW? [12:44] bigjools: Do you have time to talk about PPA download stats? [12:44] wgrant: sure [12:45] so, bug #139855 [12:45] Bug #139855: Display stats about PPA usage [12:45] The only consensus seems to be that we want to store binary download counts like we do LFA downloads. [12:46] Indices probably want to be counted somehow too, but what to do with that is much less obvious. [12:46] I have a working implementation of binary counts. [12:46] I tihnk that doing the binary counts is a very good first step [12:47] do you want me to take a look at it? [12:48] wgrant: checking the ec2 instance... [12:50] wgrant: it seems to be finishing (it's not yet terminated, but I can no longer access it), so I'll wait a bit. If I've not seen anything by my EOD, i'll resend it. [12:50] noodles775: OK, odd... [12:50] At least the other one should be in PQM. [12:51] wgrant: it is (playing now). [12:54] bigjools: I think the backend branches (https://code.edge.launchpad.net/~wgrant/launchpad/parse-ppa-apache-logs is the final one) should be reasonably unobjectionable. [12:55] But https://code.edge.launchpad.net/~wgrant/launchpad/export-basic-binary-download-stats and its successor https://code.edge.launchpad.net/~wgrant/launchpad/export-detailed-binary-download-stats might well be horribly slow. [12:55] wgrant: ok, file an MP and request me as a reviewer and I'll look at that one [12:55] can you paste a diff? I'm lazy :) [12:59] bigjools: The combination is not quite oversized, but it has two logical prereqs which will make it smaller. Shall I propose them separately? [13:00] wgrant: I'm a little confused, you have three branches and the first one is not controversial, right? but the other 2 might be? so I was asking for diffs for those [13:01] bigjools: The first uncontroversial one is really three branches. You just want the two that might be? That's easy. [13:01] * wgrant diffs. [13:02] wgrant: well if it's really big we should split it up and get more reviewers involved [13:03] bigjools: It's not really big. The first branch I mentioned comes it at around 800 lines -- it's just probably a bit easier if it's split into its three logical components. [13:04] ok [13:10] noodles775: Ah, that other run emailed a couple of minutes ago. Great. [13:10] Excellent. === henninge_ is now known as henninge === matsubara is now known as matsubara-afk [13:23] bigjools: A few criss-cross merges and lots of conflicts later: http://pastebin.ubuntu.com/392515/ and http://pastebin.ubuntu.com/392521/ [13:23] heh [13:24] The first just exports a single count per BPPH, the second gives detailed data. [13:30] wgrant: if it's expensive, don't make it a property [13:31] download_count that is [13:31] bigjools: Yeah, I guess not. [13:31] who should i speak to about the design of html pages for launchpad? is it still mpt? [13:31] it will get serialised on every fetch [13:32]