[00:16] maxb: It doesn't depend on -backports? [00:20] Ugh. Soyuz, how I want to refractor thou. [00:20] Which bit in particular? [00:21] ummm... pretty much all of it. [00:22] Heh. [00:22] First I'd refractor publishing [00:22] Just make the archive publisher run faster, kthx. [00:22] The archive publisher isn't really that slow TBH. [00:23] 20 minutes... unacceptable. ;) [00:23] Its 5 minutes [00:23] No. [00:23] Not for the primary archive. [00:23] The primary archive doesn't count. [00:25] What do you see as wrong with the current publisher design? [00:25] Apart from the code being foul and the names sucking and blah blah blurgggh. [00:25] Have you looked at a graph of publishing tables? [00:25] Heh. Yes. [00:26] * wgrant quickly extends http://people.ubuntu.com/~wgrant/launchpad/buildfarm/current-build-model.png with them and cries. [00:27] I'm implementing the ability to delete PPAs. [00:28] The relationship between SourcePackagePublishingHistory and SourcePackageRelease is a mess. [00:28] To what level are you implementing it? [00:28] I presume the DB structures will stay. [00:28] Yes, unfortunately. [00:29] SourcePackageRelease is attached to an archive (ie. upload_archive) when you copy a package to another archive it just creates a new SourcePackagePublishingHistory record that points to the SourcePackageRelease for the source archive. [00:30] Yes. [00:30] But if you're leaving the actual Archive around, i don't see a problem with this. [00:30] upload_archive is only used when saying 'Copied from Some Archive' [00:30] I suggested that bug bigjools doesn't want to keep it around, we really want to delete the archive. [00:31] That's deleting incriminating history, though. [00:31] I don't like deleting history, particularly when it contains records of what arbitrary code people have been executing. [00:31] wgrant, If you copy a copied package then it'll say you copied it from original archive the package was uploaded to, not the archive you actually copied the package from. [00:32] cody-somerville: Yes. That is a stupid bug. [00:32] It will be fixed along with recording who did the copy and when. [00:32] Its not a stupid bug, its a critical design flaw. lol. [00:32] It's not that much of a problem, is it? [00:32] (or more like a lazy hack) [00:32] Lazy hack that is easily removed, right. [00:33] wgrant, It is because we have to keep an archive's SourcePackageRelease around if there are SourcePackagePublishingHistory records for other archives that reference it. [00:33] cody-somerville: You're actually considering deleting the SourcePackageRelease? Why? [00:33] And we already have logic for determining this, in the PPA file expirer. [00:34] Please reuse it, or somebody is again going to get very close to inadvertently deleting a few hundred gigabytes of packages that needed to be kept. [00:34] bigjools said he wants it all deleted. I don't see any reason for keeping it around either myself since it'll have no references to it. [00:35] wgrant, file? [00:36] cody-somerville: The script that expires PPA librarian files has queries to determine whether an SPR is currently in use elsewhere. [00:36] Although I guess you really just care about whether it's been used outside that archive. [00:36] which makes the query trivial [00:39] heh, I wouldn't call the query in expire_archive_files trivial. [00:39] No, that one isn't. [00:39] But your search needs to be more restrictive, so it is. [00:40] wgrant, Maybe you're more familiar with Storm and can recommend a more efficient storm query: http://pastebin.ubuntu.com/398085/ [00:41] Actually I quite like the feature that if you copy a copied package, it still references the original upload archive [00:41] wgrant, oops, extra line in there. [00:42] forget line 3 exists [00:42] cody-somerville: So you /are/ actually removing the Archive? [00:43] wgrant, Thats the current plan, yes. [00:43] Ew. [00:44] * maxb seconds that [00:44] Apart from the impossibility of setting upload_archive = None, and my revulsion at history deletion, that looks OK. [00:44] Why is it so important to delete the history? [00:44] We're patching SourcePackageRelease to drop the NOT NULL constraint. [00:44] As long as you also remove everything else associated with the SPR. [00:44] (SPRFs, that sort of thing) [00:45] Yes, we're deleting everything. [00:45] This is making it very easy to remove a complex hierarchy of important data and accountability. [00:45] If the archive is gone, its gone. [00:46] Sometimes deleting something means really deleting something. [00:46] And why is this desirable !??! [00:46] why is it not? [00:46] We don't want to keep records in the database for a deleted archive. [00:47] We have the ability to disable archives when we want to do that. [00:48] For example, copy archives and rebuild archives would be very nice to be able to delete. [00:48] Why? [00:48] I want to be able to look at the rebuild history. [00:49] What benefit does deleting the records provide, besides allowing a single malicious or mistaken click to delete a vast amount of irreplaceable data, and saving a tiny bit of DB space? [00:49] Do you really care about rebuild history for a rebuild that occurred for a 5 year old release? [00:50] and the launchpad database is like terabytes so I think the latter is valid, yes. [00:51] Also, I don't think this feature will be used primarily to delete archives which actually history but instead ones without. [00:51] for example, if someone wants to rename their account [00:52] an empty PPA unfortunately blocks that [00:52] If they want to rename their account, then we "delete" the PPA. [00:52] We hide it and remove it from disk. [00:52] We probably even remove the librarian files. [00:52] manually [00:52] Incidentally, why don't we just rename the PPA on disk too? [00:53] maxb: The indices need regeneration. [00:53] cody-somerville: The use of present tense there was perhaps a mistake. [00:53] ok, delete and republish from scratch then [00:53] maxb: Right. [00:53] The publisher could just say "oh look, there's no directory there, but there is stuff to publish. Let me take a few more seconds and carefully publish it from scratch" [00:54] I get that discouraging PPA owners from breaking people's sources.lists is good, but an "are you sure?" would be good enough, I would think [00:54] One would think so. [00:55] We allow users to break their OpenIDs after such a warning, for example. [00:57] Heh, icon positioning fail on https://edge.launchpad.net/builders [00:57] Yeah, I reported that a couple of days ago. [00:57] A somewhat intermediate step in the generalisation refactoring landed. [00:58] Bug 540819 [00:58] Bug #540819: BuildFarmBuildJob icons on /builders misplaced [01:03] The PPA publisher seems to take an abnormally long time these days [01:03] maxb: Howso? [01:06] It feels like sometime I'm waiting for multiple 5-minute intervals to happen for a publication [01:06] Hmmm. [01:07] Unless the publisher oversteps its 5 minute window, that should be impossible. [01:08] *right* launchpad-developer-dependencies is installable again [02:55] Where is the vhost config file? [02:55] magcius: The same file as the rest of the config. [02:56] wgrant: which is where? [02:56] wgrant: I'm looking in /configs, nope, /canonical/config, nope [02:56] In the [launchpad] section of your config file. [02:56] Which is probably in configs/. [02:57] http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel/annotate/head%3A/configs/development/launchpad.conf [02:57] nothing useful there [02:57] launchpad-lazr.conf is the important one. [02:57] launchpad.conf is deprecated. [02:57] what is "lazr" [02:58] In that filename it refers to lazr.config, the configuration file parser used by Launchpad. [02:58] Which is part of the LAZR suite of libraries. [02:58] In general, what does it stand for? [02:58] All developed for use in Canonical's web applications. [02:58] It stands for nothing. [02:58] Was LAZR opened when LP was opened? [02:59] Most of it started embedded in various webapps (mostly Launchpad). [02:59] it was progressively split out with the first bits released about a year before LP's source. [02:59] would you mind explaining "traversal"? [02:59] I looked at doc/navigation.txt, but I didn't understand it [03:00] It's how our URLs work. [03:00] Okay. [03:00] /ubuntu/karmic/i386 will traverse to first Ubuntu, then its Karmic release, then its i386 architecture. [03:00] So it's going down a tree of possible endpoints? [03:01] It's taking a path form a URL, applying it to a tree of objects, and ending up with a sequence of objects between the root and the target. [03:01] s/form/from/ [03:01] alright [03:02] It's not too dissimilar from normal Zope traversal. [03:02] do things like "+bug" mean anything special in terms of this navigation? [03:02] Ugh. [03:02] Those are sort of magical and ugly. [03:02] Wonderful. [03:02] Since there's +bugs but also +bug/1234, which doens't really make much sense. [03:02] Say you've got /ubuntu/+bug/1234 [03:02] Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts [03:03] that URL works? (responding to the bot) [03:03] That URL does work, yes. [03:03] +bug doesn't represent an actual object. [03:03] It purely represents a URL namespace. [03:03] I've never used Zope. It lost me at the webapp configuration and object database based on pickle. [03:04] Well, Launchpad is a Zope application, but it doesn't use ZODB at all (the object database of which you speak). [03:04] I understand. [03:04] The thing which really really confuses me about launchpad urls is where special pages like /builders overlap with the namespace of distros/projects [03:05] maxb: That's why we have the name blacklist. [03:05] so does it mean anything to the traversal? [03:05] '+whatever' has historically been used to avoid that namespace issue. [03:06] But that has been abandoned in the root lately, IIRC because it stuffs up breadcrumbs. [03:06] i.e. does "+bug" mean "return this special Bug object for the next lookup" [03:06] And is a tad ugly. [03:06] also, I officially hate Loggerhead. [03:07] 1) the annotate/files distinction for the only purpose of pissing URL editors off [03:07] 2) the %3A in every goddamn URL [03:07] loggerhead isn't really part of LP. [03:07] 3) no left-margin marker, the sans-serif revision marker, especially painful to identify indentation levels with long stretches of code [03:07] magcius: To see how the +bug traversal works, see lp.bugs.browser.bugtask.BugTargetTraversalMixin [03:08] wgrant: does that class go from ubuntu to +bug or +bug to 1234 [03:08] '@stepthrough("+something")' lets the method handle paths of the form '+something/foo', and it will take 'foo' as an argument. [03:08] magcius: It handles ubuntu to +bug/1234 [03:08] There is no intermediate class in this case. [03:08] alright [03:09] how/where are the DNS entries for bugs.launchpad.dev and such on development machines? [03:09] (I never understood DNS) [03:09] /etc/hosts [03:10] I've seen in the "Getting Started" guide that you don't need to modify that, and "launchpad.dev" will work [03:10] Which is what baffled me. [03:10] The installation script (rocketfuel-setup) modifies /etc/hosts itself. [03:10] does it also do that for bugs.launchpad.dev and the rest? [03:10] Yes. [03:11] alright. Does Zope/LP just determine how to dispatch from the request sent, or are there separate locations? [03:12] Separate locations? [03:12] separate servers [03:12] on separate ports [03:12] There is a wildcard Apache vhost which dispatches to a single LP server process. [03:12] That checks the Host header. [03:12] Is that true in the production cluster as well? [03:13] On production I believe there are multiple Squids in front of multiple Apaches in front of multiple appservers each running multiple LP processes. [03:13] Er, there's haproxy in there somewhere too -- probably between Squid and Apache. [03:14] Are some LP servers designated to "Bugs" or "Code Hosting" or something? [03:14] Or maybe Apache and the appservers. [03:14] Wasn't Pound mentioned at one point? [03:14] maxb: It was replaced with haproxy. [03:14] ah [03:14] magcius: The main webapp is all hosted by one set of machines that do not discriminate. [03:15] bazaar.launchpad.net is served by the Codehosting machine, which stores all of the Bazaar branches. [03:15] It might also run Loggerhead, but that could be delegated to another machine. I forget. [03:20] win 16 [03:20] Argh. [03:25] Ah well, that's the first time I've seen you do that :-) [03:25] We have someone at work who does it so often we've written a plugin for our IRC bot to automatically mock him [03:26] Heh === StevenK_ is now known as StevenK === lionel_ is now known as lionel === james_w` is now known as james_w [22:39] If I'm storing changelogs in librarian, should I give it a more descriptive file name beside "changelog"? [22:40] NCommander: I don't think so. [22:40] Thankyou for using the librarian, though! [22:40] Although maybe NAME_VERSION_changelog could be nice. i don't know. [22:49] wgrant: well, I talked it over iwth bigjools once I got a moment to breath [22:50] wgrant: I'll submit it as a second branch proposal, and then let BjornT and stub decide what version they want, and kill the one they don't :-) [22:51] wgrant: I'm open to suggests on this one :-) [22:55] wgrant: also, how do I get stuff OUT of librarian? [22:55] * NCommander isn't so sure on tha tbit [22:55] NCommander: Call .read() on the LFA, IIRC. [22:55] Merge proposals do it to get the diff. [22:55] I can't think of anywhere else in the webapp that does. [22:59] wgrant: as long as its possible, I'm happy. Once changelogs land and are migrated (I have to write the migration utility next), I'd like to do the same for copyright files [22:59] and get those out of the database [23:00] Ran 5 tests with 0 failures and 0 errors in 0.061 seconds. [23:00] woo [23:02] dual screens also woo [23:03] wgrant: is changelog_id a good name for the column? [23:03] or should it be something more decsriptive? [23:03] 'changelog' in the DB. [23:04] wgrant: ah, ok [23:04] * NCommander notes that greatly reduces the number of edits I have to make then [23:04] Why? [23:04] Apart from the comments.sql change. [23:05] wgrant: I was going to call the column changelog_id [23:05] now I don't need to do that [23:05] * NCommander just updates a few comments [23:05] Ah. [23:10] wgrant: how does ForgienKey() work when updating the model [23:10] * NCommander is a bit confused [23:11] wgrant: for instance, the model code to get section in the database is: [23:11] section = ForeignKey(foreignKey='Section', dbName='section') [23:11] So, that's deprecated. [23:11] I'm not sure how to do one for libraryfilealias [23:11] wgrant: oh [23:11] But if the class you're using uses it, I guess you should follow. [23:12] wgrant: is this a zope thing? [23:12] changelog = ForeignKey(foreignKey='LibraryFileAlias', dbName='changelog') [23:12] No. [23:12] * NCommander is still kinda iffy on Zope [23:12] This is SQLObject. [23:12] Which is deprecated by Storm. [23:14] wgrant: ah, I see [23:14] wgrant: so, for the attribute, how's this for a description: changelog = Attribute("LibraryFileAlias for the changelog of this SourcePackageRelease.") [23:15] I think I'd prefer 'LibraryFileAlias containing debian/changelog' [23:16] But I don't really know. There is no consistency. [23:16] wgrant: fair enough [23:17] * NCommander runs the soyuz and archiveuploader tests to make sure nothing else has broke [23:29] * NCommander lets the soyuz and archiveuploader test suites run [23:38] wgrant: ugh, I'm getting a *ton* of test failures on my branch :-/ [23:39] wgrant: http://paste.ubuntu.com/398521/ - like that, which seems odd [23:40] NCommander: There was an error before that that you broke. [23:41] wgrant: no, I'm getting a massive stream of these [23:41] like hundres [23:41] I don't think I could have broken it that badly [23:42] You broke one and it cascaded. [23:42] Wait until the run has finished, then rerun the failing tests with -1. [23:42] -1 will stop on the first failure in each doctest. [23:42] Letting you see what actually caused it all. [23:47] wgrant: k