[00:16] <wgrant> maxb: It doesn't depend on -backports?
[00:20] <cody-somerville> Ugh. Soyuz, how I want to refractor thou.
[00:20] <wgrant> Which bit in particular?
[00:21] <cody-somerville> ummm... pretty much all of it.
[00:22] <wgrant> Heh.
[00:22] <cody-somerville> First I'd refractor publishing
[00:22] <jpds> Just make the archive publisher run faster, kthx.
[00:22] <cody-somerville> The archive publisher isn't really that slow TBH.
[00:23] <jpds> 20 minutes... unacceptable. ;)
[00:23] <cody-somerville> Its 5 minutes
[00:23] <jpds> No.
[00:23] <wgrant> Not for the primary archive.
[00:23] <cody-somerville> The primary archive doesn't count.
[00:25] <wgrant> What do you see as wrong with the current publisher design?
[00:25] <wgrant> Apart from the code being foul and the names sucking and blah blah blurgggh.
[00:25] <cody-somerville> Have you looked at a graph of publishing tables?
[00:25] <wgrant> Heh. Yes.
[00:26]  * wgrant quickly extends http://people.ubuntu.com/~wgrant/launchpad/buildfarm/current-build-model.png with them and cries.
[00:27] <cody-somerville> I'm implementing the ability to delete PPAs.
[00:28] <cody-somerville> The relationship between SourcePackagePublishingHistory and SourcePackageRelease is a mess.
[00:28] <wgrant> To what level are you implementing it?
[00:28] <wgrant> I presume the DB structures will stay.
[00:28] <cody-somerville> Yes, unfortunately.
[00:29] <cody-somerville> SourcePackageRelease is attached to an archive (ie. upload_archive) when you copy a package to another archive it just creates a new SourcePackagePublishingHistory record that points to the SourcePackageRelease for the source archive.
[00:30] <wgrant> Yes.
[00:30] <wgrant> But if you're leaving the actual Archive around, i don't see a problem with this.
[00:30] <wgrant> upload_archive is only used when saying 'Copied from Some Archive'
[00:30] <cody-somerville> I suggested that bug bigjools doesn't want to keep it around, we really want to delete the archive.
[00:31] <wgrant> That's deleting incriminating history, though.
[00:31] <wgrant> I don't like deleting history, particularly when it contains records of what arbitrary code people have been executing.
[00:31] <cody-somerville> wgrant, If you copy a copied package then it'll say you copied it from original archive the package was uploaded to, not the archive you actually copied the package from.
[00:32] <wgrant> cody-somerville: Yes. That is a stupid bug.
[00:32] <wgrant> It will be fixed along with recording who did the copy and when.
[00:32] <cody-somerville> Its not a stupid bug, its a critical design flaw. lol.
[00:32] <wgrant> It's not that much of a problem, is it?
[00:32] <cody-somerville> (or more like a lazy hack)
[00:32] <wgrant> Lazy hack that is easily removed, right.
[00:33] <cody-somerville> wgrant, It is because we have to keep an archive's SourcePackageRelease around if there are SourcePackagePublishingHistory records for other archives that reference it.
[00:33] <wgrant> cody-somerville: You're actually considering deleting the SourcePackageRelease? Why?
[00:33] <wgrant> And we already have logic for determining this, in the PPA file expirer.
[00:34] <wgrant> Please reuse it, or somebody is again going to get very close to inadvertently deleting a few hundred gigabytes of packages that needed to be kept.
[00:34] <cody-somerville> bigjools said he wants it all deleted. I don't see any reason for keeping it around either myself since it'll have no references to it.
[00:35] <cody-somerville> wgrant, file?
[00:36] <wgrant> cody-somerville: The script that expires PPA librarian files has queries to determine whether an SPR is currently in use elsewhere.
[00:36] <wgrant> Although I guess you really just care about whether it's been used outside that archive.
[00:36] <wgrant> which makes the query trivial
[00:39] <cody-somerville> heh, I wouldn't call the query in expire_archive_files trivial.
[00:39] <wgrant> No, that one isn't.
[00:39] <wgrant> But your search needs to be more restrictive, so it is.
[00:40] <cody-somerville> wgrant, Maybe you're more familiar with Storm and can recommend a more efficient storm query: http://pastebin.ubuntu.com/398085/
[00:41] <maxb> Actually I quite like the feature that if you copy a copied package, it still references the original upload archive
[00:41] <cody-somerville> wgrant, oops, extra line in there.
[00:42] <cody-somerville> forget line 3 exists
[00:42] <wgrant> cody-somerville: So you /are/ actually removing the Archive?
[00:43] <cody-somerville> wgrant, Thats the current plan, yes.
[00:43] <wgrant> Ew.
[00:44]  * maxb seconds that
[00:44] <wgrant> Apart from the impossibility of setting upload_archive = None, and my revulsion at history deletion, that looks OK.
[00:44] <maxb> Why is it so important to delete the history?
[00:44] <cody-somerville> We're patching SourcePackageRelease to drop the NOT NULL constraint.
[00:44] <wgrant> As long as you also remove everything else associated with the SPR.
[00:44] <wgrant> (SPRFs, that sort of thing)
[00:45] <cody-somerville> Yes, we're deleting everything.
[00:45] <wgrant> This is making it very easy to remove a complex hierarchy of important data and accountability.
[00:45] <cody-somerville> If the archive is gone, its gone.
[00:46] <cody-somerville> Sometimes deleting something means really deleting something.
[00:46] <maxb> And why is this desirable !??!
[00:46] <cody-somerville> why is it not?
[00:46] <cody-somerville> We don't want to keep records in the database for a deleted archive.
[00:47] <cody-somerville> We have the ability to disable archives when we want to do that.
[00:48] <cody-somerville> For example, copy archives and rebuild archives would be very nice to be able to delete.
[00:48] <wgrant> Why?
[00:48] <wgrant> I want to be able to look at the rebuild history.
[00:49] <wgrant> What benefit does deleting the records provide, besides allowing a single malicious or mistaken click to delete a vast amount of irreplaceable data, and saving a tiny bit of DB space?
[00:49] <cody-somerville> Do you really care about rebuild history for a rebuild that occurred for a 5 year old release?
[00:50] <cody-somerville> and the launchpad database is like terabytes so I think the latter is valid, yes.
[00:51] <cody-somerville> Also, I don't think this feature will be used primarily to delete archives which actually history but instead ones without.
[00:51] <cody-somerville> for example, if someone wants to rename their account
[00:52] <cody-somerville> an empty PPA unfortunately blocks that
[00:52] <wgrant> If they want to rename their account, then we "delete" the PPA.
[00:52] <wgrant> We hide it and remove it from disk.
[00:52] <wgrant> We probably even remove the librarian files.
[00:52] <cody-somerville> manually
[00:52] <maxb> Incidentally, why don't we just rename the PPA on disk too?
[00:53] <wgrant> maxb: The indices need regeneration.
[00:53] <wgrant> cody-somerville: The use of present tense there was perhaps a mistake.
[00:53] <maxb> ok, delete and republish from scratch then
[00:53] <wgrant> maxb: Right.
[00:53] <wgrant> The publisher could just say "oh look, there's no directory there, but there is stuff to publish. Let me take a few more seconds and carefully publish it from scratch"
[00:54] <maxb> I get that discouraging PPA owners from breaking people's sources.lists is good, but an "are you sure?" would be good enough, I would think
[00:54] <wgrant> One would think so.
[00:55] <wgrant> We allow users to break their OpenIDs after such a warning, for example.
[00:57] <maxb> Heh, icon positioning fail on https://edge.launchpad.net/builders
[00:57] <wgrant> Yeah, I reported that a couple of days ago.
[00:57] <wgrant> A somewhat intermediate step in the generalisation refactoring landed.
[00:58] <wgrant> Bug 540819
[00:58] <mup> Bug #540819: BuildFarmBuildJob icons on /builders misplaced <trivial> <ui> <Soyuz:Triaged by jtv> <https://launchpad.net/bugs/540819>
[01:03] <maxb> The PPA publisher seems to take an abnormally long time these days
[01:03] <wgrant> maxb: Howso?
[01:06] <maxb> It feels like sometime I'm waiting for multiple 5-minute intervals to happen for a publication
[01:06] <wgrant> Hmmm.
[01:07] <wgrant> Unless the publisher oversteps its 5 minute window, that should be impossible.
[01:08] <maxb> *right* launchpad-developer-dependencies is installable again
[02:55] <magcius> Where is the vhost config file?
[02:55] <wgrant> magcius: The same file as the rest of the config.
[02:56] <magcius> wgrant: which is where?
[02:56] <magcius> wgrant: I'm looking in /configs, nope, /canonical/config, nope
[02:56] <wgrant> In the [launchpad] section of your config file.
[02:56] <wgrant> Which is probably in configs/.
[02:57] <magcius> http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel/annotate/head%3A/configs/development/launchpad.conf
[02:57] <magcius> nothing useful there
[02:57] <wgrant> launchpad-lazr.conf is the important one.
[02:57] <wgrant> launchpad.conf is deprecated.
[02:57] <magcius> what is "lazr"
[02:58] <wgrant> In that filename it refers to lazr.config, the configuration file parser used by Launchpad.
[02:58] <wgrant> Which is part of the LAZR suite of libraries.
[02:58] <magcius> In general, what does it stand for?
[02:58] <wgrant> All developed for use in Canonical's web applications.
[02:58] <wgrant> It stands for nothing.
[02:58] <magcius> Was LAZR opened when LP was opened?
[02:59] <wgrant> Most of it started embedded in various webapps (mostly Launchpad).
[02:59] <wgrant> it was progressively split out with the first bits released about a year before LP's source.
[02:59] <magcius> would you mind explaining "traversal"?
[02:59] <magcius> I looked at doc/navigation.txt, but I didn't understand it
[03:00] <wgrant> It's how our URLs work.
[03:00] <magcius> Okay.
[03:00] <wgrant> /ubuntu/karmic/i386 will traverse to first Ubuntu, then its Karmic release, then its i386 architecture.
[03:00] <magcius> So it's going down a tree of possible endpoints?
[03:01] <wgrant> It's taking a path form a URL, applying it to a tree of objects, and ending up with a sequence of objects between the root and the target.
[03:01] <wgrant> s/form/from/
[03:01] <magcius> alright
[03:02] <wgrant> It's not too dissimilar from normal Zope traversal.
[03:02] <magcius> do things like "+bug" mean anything special in terms of this navigation?
[03:02] <wgrant> Ugh.
[03:02] <wgrant> Those are sort of magical and ugly.
[03:02] <magcius> Wonderful.
[03:02] <wgrant> Since there's +bugs but also +bug/1234, which doens't really make much sense.
[03:02] <wgrant> Say you've got /ubuntu/+bug/1234
[03:02] <mup> Bug #1234: Gina is an unmaintainable mess of command line options, environment variables and shell scripts <Launchpad Foundations:Fix Released by debonzi> <https://launchpad.net/bugs/1234>
[03:03] <magcius> that URL works? (responding to the bot)
[03:03] <wgrant> That URL does work, yes.
[03:03] <wgrant> +bug doesn't represent an actual object.
[03:03] <wgrant> It purely represents a URL namespace.
[03:03] <magcius> I've never used Zope. It lost me at the webapp configuration and object database based on pickle.
[03:04] <wgrant> Well, Launchpad is a Zope application, but it doesn't use ZODB at all (the object database of which you speak).
[03:04] <magcius> I understand.
[03:04] <maxb> The thing which really really confuses me about launchpad urls is where special pages like /builders overlap with the namespace of distros/projects
[03:05] <wgrant> maxb: That's why we have the name blacklist.
[03:05] <magcius> so does it mean anything to the traversal?
[03:05] <wgrant> '+whatever' has historically been used to avoid that namespace issue.
[03:06] <wgrant> But that has been abandoned in the root lately, IIRC because it stuffs up breadcrumbs.
[03:06] <magcius> i.e. does "+bug" mean "return this special Bug object for the next lookup"
[03:06] <wgrant> And is a tad ugly.
[03:06] <magcius> also, I officially hate Loggerhead.
[03:07] <magcius> 1) the annotate/files distinction for the only purpose of pissing URL editors off
[03:07] <magcius> 2) the %3A in every goddamn URL
[03:07] <wgrant> loggerhead isn't really part of LP.
[03:07] <magcius> 3) no left-margin marker, the sans-serif revision marker, especially painful to identify indentation levels with long stretches of code
[03:07] <wgrant> magcius: To see how the +bug traversal works, see lp.bugs.browser.bugtask.BugTargetTraversalMixin
[03:08] <magcius> wgrant: does that class go from ubuntu to +bug or +bug to 1234
[03:08] <wgrant> '@stepthrough("+something")' lets the method handle paths of the form '+something/foo', and it will take 'foo' as an argument.
[03:08] <wgrant> magcius: It handles ubuntu to +bug/1234
[03:08] <wgrant> There is no intermediate class in this case.
[03:08] <magcius> alright
[03:09] <magcius> how/where are the DNS entries for bugs.launchpad.dev and such on development machines?
[03:09] <magcius> (I never understood DNS)
[03:09] <wgrant> /etc/hosts
[03:10] <magcius> I've seen in the "Getting Started" guide that you don't need to modify that, and "launchpad.dev" will work
[03:10] <magcius> Which is what baffled me.
[03:10] <wgrant> The installation script (rocketfuel-setup) modifies /etc/hosts itself.
[03:10] <magcius> does it also do that for bugs.launchpad.dev and the rest?
[03:10] <wgrant> Yes.
[03:11] <magcius> alright. Does Zope/LP just determine how to dispatch from the request sent, or are there separate locations?
[03:12] <wgrant> Separate locations?
[03:12] <magcius> separate servers
[03:12] <magcius> on separate ports
[03:12] <wgrant> There is a wildcard Apache vhost which dispatches to a single LP server process.
[03:12] <wgrant> That checks the Host header.
[03:12] <magcius> Is that true in the production cluster as well?
[03:13] <wgrant> On production I believe there are multiple Squids in front of multiple Apaches in front of multiple appservers each running multiple LP processes.
[03:13] <wgrant> Er, there's haproxy in there somewhere too -- probably between Squid and Apache.
[03:14] <magcius> Are some LP servers designated to "Bugs" or "Code Hosting" or something?
[03:14] <wgrant> Or maybe Apache and the appservers.
[03:14] <maxb> Wasn't Pound mentioned at one point?
[03:14] <wgrant> maxb: It was replaced with haproxy.
[03:14] <maxb> ah
[03:14] <wgrant> magcius: The main webapp is all hosted by one set of machines that do not discriminate.
[03:15] <wgrant> bazaar.launchpad.net is served by the Codehosting machine, which stores all of the Bazaar branches.
[03:15] <wgrant> It might also run Loggerhead, but that could be delegated to another machine. I forget.
[03:20] <wgrant> win 16
[03:20] <wgrant> Argh.
[03:25] <maxb> Ah well, that's the first time I've seen you do that :-)
[03:25] <maxb> We have someone at work who does it so often we've written a plugin for our IRC bot to automatically mock him
[03:26] <wgrant> Heh
[22:39] <NCommander> If I'm storing changelogs in librarian, should I give it a more descriptive file name beside "changelog"?
[22:40] <wgrant> NCommander: I don't think so.
[22:40] <wgrant> Thankyou for using the librarian, though!
[22:40] <wgrant> Although maybe NAME_VERSION_changelog could be nice. i don't know.
[22:49] <NCommander> wgrant: well, I talked it over iwth bigjools once I got a moment to breath
[22:50] <NCommander> wgrant: I'll submit it as a second branch proposal, and then let BjornT and stub decide what version they want, and kill the one they don't :-)
[22:51] <NCommander> wgrant: I'm open to suggests on this one :-)
[22:55] <NCommander> wgrant: also, how do I get stuff OUT of librarian?
[22:55]  * NCommander isn't so sure on tha tbit
[22:55] <wgrant> NCommander: Call .read() on the LFA, IIRC.
[22:55] <wgrant> Merge proposals do it to get the diff.
[22:55] <wgrant> I can't think of anywhere else in the webapp that does.
[22:59] <NCommander> wgrant: as long as its possible, I'm happy. Once changelogs land and are migrated (I have to write the migration utility next), I'd like to do the same for copyright files
[22:59] <NCommander> and get those out of the database
[23:00] <NCommander>   Ran 5 tests with 0 failures and 0 errors in 0.061 seconds.
[23:00] <NCommander> woo
[23:02] <NCommander> dual screens also woo
[23:03] <NCommander> wgrant: is changelog_id a good name for the column?
[23:03] <NCommander> or should it be something more decsriptive?
[23:03] <wgrant> 'changelog' in the DB.
[23:04] <NCommander> wgrant: ah, ok
[23:04]  * NCommander notes that greatly reduces the number of edits I have to make then <g>
[23:04] <wgrant> Why?
[23:04] <wgrant> Apart from the comments.sql change.
[23:05] <NCommander> wgrant: I was going to call the column changelog_id
[23:05] <NCommander> now I don't need to do that
[23:05]  * NCommander just updates a few comments
[23:05] <wgrant> Ah.
[23:10] <NCommander> wgrant: how does ForgienKey() work when updating the model
[23:10]  * NCommander is a bit confused
[23:11] <NCommander> wgrant: for instance, the model code to get section in the database is:
[23:11] <NCommander>     section = ForeignKey(foreignKey='Section', dbName='section')
[23:11] <wgrant> So, that's deprecated.
[23:11] <NCommander> I'm not sure how to do one for libraryfilealias
[23:11] <NCommander> wgrant: oh
[23:11] <wgrant> But if the class you're using uses it, I guess you should follow.
[23:12] <NCommander> wgrant: is this a zope thing?
[23:12] <wgrant> changelog = ForeignKey(foreignKey='LibraryFileAlias', dbName='changelog')
[23:12] <wgrant> No.
[23:12]  * NCommander is still kinda iffy on Zope
[23:12] <wgrant> This is SQLObject.
[23:12] <wgrant> Which is deprecated by Storm.
[23:14] <NCommander> wgrant: ah, I see
[23:14] <NCommander> wgrant: so, for the attribute, how's this for a description:     changelog = Attribute("LibraryFileAlias for the changelog of this SourcePackageRelease.")
[23:15] <wgrant> I think I'd prefer 'LibraryFileAlias containing debian/changelog'
[23:16] <wgrant> But I don't really know. There is no consistency.
[23:16] <NCommander> wgrant: fair enough
[23:17]  * NCommander runs the soyuz and archiveuploader tests to make sure nothing else has broke
[23:29]  * NCommander lets the soyuz and archiveuploader test suites run
[23:38] <NCommander> wgrant: ugh, I'm getting a *ton* of test failures on my branch :-/
[23:39] <NCommander> wgrant: http://paste.ubuntu.com/398521/ - like that, which seems odd
[23:40] <wgrant> NCommander: There was an error before that that you broke.
[23:41] <NCommander> wgrant: no, I'm getting a massive stream of these
[23:41] <NCommander> like hundres
[23:41] <NCommander> I don't think I could have broken it that badly
[23:42] <wgrant> You broke one and it cascaded.
[23:42] <wgrant> Wait until the run has finished, then rerun the failing tests with -1.
[23:42] <wgrant> -1 will stop on the first failure in each doctest.
[23:42] <wgrant> Letting you see what actually caused it all.
[23:47] <NCommander> wgrant: k