[03:37]  * thumper EODs (for now), back tonight
[03:59] <NCommander> Is there a good way to iterate thorugh SourceReleasePackage records? I need to iterate through ALL of them, download the source package, pop out the changelog, upload changelog to librarian, etc.
[03:59] <NCommander> (or at least, all published ones)
[04:04] <wgrant> You'll probably have to use query using Storm directly.
[04:04] <wgrant> There's no way to iterate over them all, AFAIK.
[04:12] <NCommander> wgrant: ugh .... *groan*
[04:13] <NCommander> wgrant: do I have to use Storm in writing migrationtools?
[04:13]  * NCommander is not even sure how to attack this problem in a sane manner
[04:13] <StevenK> NCommander: Given the number of packages, is there a sane manner?
[04:15] <NCommander> StevenK: I'm honestly not sure, since we're also including PPA packages in this
[04:16] <StevenK> NCommander: Do the math -- 9,000 * Dapper, Hardy, Jaunty, Karmic and Lucid times about 40% is still 18,000
[04:17] <NCommander> StevenK: ugh ....
[04:17] <NCommander> migration == PAIN
[04:18] <NCommander> StevenK: (necessary pain, but still pain)
[04:18] <persia> why 9000?
[04:18] <NCommander> StevenK: actually, we may want to include every source package so the changelog can be individually queried on a given SRP
[04:18] <NCommander> That requires discussion with BjornT, and bigjools
[04:19] <wgrant> We don't have the files for lots of SPRs.
[04:19] <NCommander> wgrant: we don't?
[04:19] <NCommander> I thought ever SPR had at least a few files
[04:19] <NCommander> for the source packages anyway
[04:19] <wgrant> All PPA sources that have been unpublished for more than a week, and all non-final primary archive versions from obsolete series.
[04:20] <NCommander> wgrant: oh, that greatly reduces the numbers I was planning
[04:21] <NCommander> wgrant: is there a special flag on SPR that states if there are files?
[04:21] <wgrant> NCommander: No.
[04:21] <NCommander> i.e., published, superseeded, obsolete, etc.
[04:21] <NCommander> ugh
[04:22]  * NCommander thought we had a removed from librarian status
[04:22] <wgrant> You need to scan the LFCs of the LFAs of the SPRFs.
[04:22] <wgrant> Sorry, the LFCs of the LFAs of the SPRFs of the SPRs.
[04:22] <NCommander> wgrant: *blink*
[04:22] <NCommander> ow
[04:22] <ajmitch> needs more acronyms
[04:22]  * wgrant gives ajmitch a DASBPR
[04:22] <mwhudson> i know let's switch launchpad to a nosql database that doesn't support joins!
[04:23] <ajmitch> wgrant: it's worrying that I know what that is
[04:23]  * NCommander whacks mwhudson :-P
[04:23] <NCommander> wgrant: https://edge.launchpad.net/ubuntu/jaunty/+source/hello/2.2-2 - they seem ot be there
[04:23] <wgrant> mwhudson: My NMAF optimisation does a 9 table join.
[04:23] <wgrant> NCommander: Jaunty isn't obsolete.
[04:23] <NCommander> oh, d'oh
[04:23] <NCommander> fail
[04:24] <wgrant> It's possible that we only removed obsolete series' binaries. I reviewed the query, but really do not remember.
[04:24]  * wgrant checks.
[04:24] <NCommander> wgrant: https://edge.launchpad.net/ubuntu/feisty/+source/hello/2.2-1build1 - seems thats the case
[04:24] <NCommander> yeah, the binaries were deleted
[04:25] <wgrant> Oh, right, we kept the old primary sources just in case.
[04:25] <NCommander> wgrant: that's a LOT of sources :-/
[04:25] <wgrant> Although they were very very close to being accidentally nuked.
[04:25] <ajmitch> perhaps to satisfy some legalities?
[04:25] <wgrant> ajmitch: And to satisfy the bzr importer.
[04:25] <NCommander> ajmitch: we have to provide sources as long as we provide debs, so as long as old-releases exist ...
[04:25] <wgrant> NCommander: But old-releases still holds sources, doesn't it?
[04:26] <NCommander> wgrant: oooh, bzr importer does pretty close to what I'm going to need to do, where's the source
[04:26] <NCommander> wgrant: eh, I'm notsure
[04:26] <ajmitch> NCommander: I never knew if you were also required to provide sources for every package version published as well :)
[04:26] <wgrant> NCommander: It uses the API and stuff, though. You have DB access.
[04:27] <NCommander> ajmitch: legally speaking, we probably do as long as we distributed it on a release CD
[04:27] <NCommander> there's only a time limit to how long we provide sources if we use a written offer to get the source
[04:27] <NCommander> (at least on GPL'ed sources)
[04:27] <NCommander> Other licenses, YMMV
[04:29] <NCommander> wgrant: re: bzr importer: but it had to iterate on all SPRs, and if it uses the API, won't be cleaner?
[06:42] <wgrant> NCommander: But you need to get the SPR from the DB to set the changelog.
[06:43] <NCommander> wgrant: wait, what?
[06:44] <wgrant> NCommander: The bzr importer doesn't have DB access.
[06:44] <wgrant> It uses launchpadlib.
[06:44] <NCommander> wgrant: ugh
[06:44] <NCommander> crap
[06:44] <wgrant> You can't set the changelog with launchpadlib, so you need DB access.
[06:44] <NCommander> *sigh*
[09:07] <mrevell> Morning all
[09:33] <mwhudson> gmb, allenap: have you seen ParallelLimitedTaskConsumer (in the context of bugwatch updating)?
[09:34] <allenap> mwhudson: Yes, I looked at it briefly yesterday after I saw your message on IRC. That works with the job runner doesn't it?
[09:34] <mwhudson> allenap: no, it's not at that level
[09:34] <allenap> mwhudson: Okay, I'll look again.
[09:35] <mwhudson> (the branch puller uses it, and that doesn't even connect to the database)
[09:35] <NCommander> mwhudson: your up late :-/
[09:35] <mwhudson> NCommander: a bit yeah
[09:35] <mwhudson> NCommander: why don't i look at that branch of yours
[09:35] <mwhudson> NCommander: it must be a considerably sillier time of night for you
[09:36] <NCommander> mwhudson: can't sleep unfortunately
[09:36] <mwhudson> NCommander: :(
[09:36] <NCommander> mwhudson: insommia == suck
[09:41] <allenap> mwhudson: Would this be a replacement for the ThreadPool stuff I've done in TwistedThreadScheduler?
[09:41] <mwhudson> allenap: maybe
[09:41]  * mwhudson has to go away sorry
[09:41] <allenap> mwhudson: Thanks for the pointer, bye :)
[09:41] <mwhudson> allenap: talk to jml maybe
[09:41] <allenap> Okay.
[10:06] <deryck> Morning, all.
[10:07] <wgrant> bigjools: Did you inspect any of the data collected by the script yesterday?
[10:08] <NCommander> mwhudson: thanks for taking a look at the branch again
[10:17] <bigjools> wgrant: no
[10:18] <bigjools> well, I looked at a few rows in the table but didn't closely check
[10:34] <NCommander> When are data migrations done usually? (i.e., importing changelogs into librarian and updating the SPR record)
[10:39] <wgrant> NCommander: That's a special one -- it can be done at any time after the release, and will take a long while.
[10:40] <NCommander> wgrant: I wonder if we're looking at hours, days, or weeks :-/
[10:43] <wgrant> NCommander: We're probably looking at around 200000 sources.
[10:44] <NCommander> bigjools, wgrant: so it sounds like to me, to migrate the changelog data, I'm going to need a script that uses Storm to access the DB, access each SRP where changelog = NULL, download the source package, pop out the debian/changelog, upload to librarian, update the SPR and then go on to the next record
[10:44] <NCommander> wgrant: that seems low ...
[10:45] <jml> allenap, what?
[10:45] <bigjools> NCommander: sounds about right to me
[10:46] <NCommander> bigjools: I assume at some point we may want to do this for copyright data as well?
[10:46] <wgrant> NCommander: There are only ~18000 primary sources per series, multiplied by a couple for the non-final versions, multiplied by say 10 for the number of releases, excluding most of those before Dapper (they weren't in LP).
[10:46] <bigjools> NCommander: you can make it faster by using store.execute()
[10:46] <wgrant> Then 30000 published PPA sources.
[10:46] <NCommander> (I think stub would be very happy to make the database loose several GiB of data)
[10:46] <bigjools> I don't care about PPA sources
[10:46] <NCommander> bigjools: hrm ...
[10:47] <NCommander> bigjools: we don't? I thought part of native sync sources was to let us go PPA -> main archive with a button click
[10:47] <allenap> jml: mwhudson mentioned ParallelLimitedTaskConsumer as (maybe) another/better way to solve the problem in checkwatches, of half-way Twistifying it.
[10:47] <bigjools> we also need to be very, very wary of the increase in librarian disk space required
[10:47] <wgrant> bigjools: The construction time for the few million Storm objects is going to be negligible compared to extraction time.
[10:47] <bigjools> I think the losas would quite like an estimate
[10:47] <jml> allenap, oh yes
[10:47] <stub> bigjools: Its easier to scale the Librarian than scale PG.
[10:47] <bigjools> NCommander: well I meant historic PPA sources
[10:47] <jml> allenap, do you want to run a bunch of tasks in parallel but with limitations until there's nothing left to do?
[10:47] <allenap> jml: We don't have time for it now, but yes, it does look interesting. It's working for now :)
[10:47] <NCommander> bigjools: oh, of course.
[10:47] <NCommander> bigjools: if we did the entire PPA history, I think the datacentre would probably implode
[10:48] <bigjools> wgrant: I am more concerned about exhausting memory
[10:48] <wgrant> bigjools: Historic PPA sources don't exist any more, so there's no question of that.
[10:48] <allenap> jml: Yes, basically. At the moment, because each task must always run in a thread, I'm just using a ThreadPool.
[10:48] <jml> allenap, :(
[10:48] <NCommander> stub: oh, so to answer your quesiton, yess, we will have NULLs for changelogs
[10:48] <bigjools> wgrant: quite true
[10:48] <allenap> jml: But when we get time to switch to non-blocking IO then something PLTC looks more interesting.
[10:48] <NCommander> bigjools: it sounds like where I'm sitting we should just attempt a changelog import wherever we have sources available, and make sure the code can handle changelog=NULL
[10:49] <allenap> jml: I've not looked at how it's used in LP, only at its implementation, so I'm not 100% sure how it's meant to be used.
[10:49] <wgrant> NCommander: I think that's all it's possible to do.
[10:49] <bigjools> NCommander: +1
[10:49] <jml> allenap, the implementation is surprisingly hairy for a simple concept.
[10:49] <NCommander> bigjools: great, now I need to figure out how to write it :-/
[10:49] <bigjools> NCommander: good luck :)
[10:50] <allenap> jml: PLTC or the one in checkwatches? If PLTC, I agree! ;)
[10:50] <jml> allenap, concurrency is hard. luckily you're using threads so you can pretend it isn't.
[10:50] <NCommander> bigjools: (I think raw_source_changelog will probably land this week in db-devel, I got a stub +1, just waiting on BjornT and the second code review)
[10:50] <wgrant> NCommander: Stealing code from archiveuploader will make your job very easy.
[10:50] <NCommander> wgrant: which code?
[10:50] <allenap> jml: Yeah, that's the approach we're taking for now. Ear defenders and blinkers at the ready.
[10:50]  * NCommander notes ripping packages apart isn't that hard
[10:50] <jml> allenap, actually, PLTC is perhaps an ideal candidate for an informative doctest
[10:51] <wgrant> NCommander: The code that takes a source package, grabs files into a temp dir (some from the DB), extracts them, and inserts the changelog into the DB.
[10:51] <NCommander> wgrant: oh, the code we ripped apart last week :-)
[10:51] <wgrant> Most of it, yes.
[10:51] <NCommander> wgrant: that wasn't the part I was worried about, I was more worried about iterating through the databsae, but I think thats just due to not knowing much of storm.
[10:51] <allenap> jml: You're right. I'm so used to doctests being muddled that I didn't even think to look for one.
[10:51] <jml> allenap, it doesn't have one
[10:52]  * NCommander isn't even sure how we can test the importer script sanely
[10:52] <bigjools> NCommander: juse use store.execute() and you can write raw SQL.  This is better in this case otherwise it will exhaust memory when caching objects.
[10:52] <bigjools> this is a real problem in some scripts that iterate a lot of objects
[10:53] <bigjools> NCommander: we can test this initally locally on your laptop, then on dogfood
[10:53] <NCommander> bigjools: fair enough, but I'm going to need quite a few joins to get all the files.
[10:53] <jml> allenap, http://paste.ubuntu.com/399839/ is probably the best example
[10:53] <NCommander> bigjools: my laptop has a small SSD. Your going to blow it up
[10:53] <bigjools> NCommander: welcome to Soyuz SQL
[10:53] <bigjools> NCommander: I have an SSD too.  Aren't they great? :)
[10:53] <NCommander> bigjools: is that SQL or SOL :-P
[10:53] <jml> allenap, the consumer just has an upper limit of parallel tasks and a logger. You tell it to consume from a "source" and it just runs whatever the source gives it until the source is exhausted
[10:53] <NCommander> bigjools: I dunno, I didn't want to burn my out within a week
[10:54] <NCommander> bigjools: this laptop takes 510 minutes to run the entire LP test suite (although it didn't swap doing it, which was nice)
[10:54] <allenap> jml: That's nice.
[10:54] <bigjools> that's what guarantees are for :)
[10:54] <bigjools> NCommander: 510?!  jeez, mine takes about 180
[10:54] <NCommander> bigjools: I dunno why it took so long.
[10:54] <bigjools> how much free memory and what cpu?
[10:54] <NCommander> bigjools: 8 GiB of RAM + 2.2Ghz dual core processor
[10:55] <NCommander> I'm thinking something must have hung for awhile
[10:55] <bigjools> yes
[10:55] <NCommander> I got more failures than ec2test
[10:55] <jml> allenap, the source just has to implement start(task_consumer) and stop(), and to call appropriate methods on the consumer when it has tasks. we've already implemented a polling source.
[10:55]  * NCommander will probably setup a karmic instance or UEC on my old laptop
[10:55]  * bigjools needs to go back to working now
[10:55] <NCommander> bigjools: sorry to distract you :-)
[10:55] <bigjools> that's my life
[10:56] <jml> allenap, the nice thing for us is that other than the source, the rest of the code would be identical even if we had some kind of event-driven infrastructure (e.g. message queues)
[10:56] <bigjools> I have a dead desktop today, I'm upset
[10:56] <jml> allenap, anyway, I'll let you get to it.
[10:56] <NCommander> bigjools: ugh, what happened to it?
[10:56]  * NCommander notes he's run LP onhis ia64 desktop before :-)
[10:56] <bigjools> NCommander: it powers up but the monitor doesn't wake up, no bios messages etc.
[10:56] <allenap> jml: I'm can't immediately imagine how that'll fit into checkwatches, but it probably will. Thanks for explaining it.
[10:56] <jml> allenap, np.
[10:56] <NCommander> bigjools: sounds like your mainboard died
[10:57] <bigjools> NCommander: I am thinking gfx card actually
[10:57] <NCommander> bigjools: try the intergrated graphics?
[10:57] <bigjools> it was working last night until I tried to suspend it
[10:57] <bigjools> and hung on the way down
[11:15] <wgrant> bigjools: lp.services.buildfarm?
[11:15] <bigjools> wgrant: possibly
[11:15] <bigjools> increases finger strain though :)
[11:16] <bigjools> naming is the hardest part of software engineering
[11:16] <wgrant> Yes :(
[11:18] <jml> funding is the hardest part of software engineering
[11:18] <bigjools> fun is the hardest part of software engineering
[11:18] <NCommander> bigjools: not with python; just import fun
[11:18] <jml> (or maybe I meant concurrency, or dealing with massive applications)
[11:19] <bigjools> no, naming stuff is still the hardest :)
[11:20] <jml> bigjools, I'll respectfully disagree.
[11:21] <bigjools> jml: it's tongue-in-cheek
[11:21] <jml> bigjools, :)
[11:23] <wgrant> What became of the generalised builder history discussion?
[11:24] <bigjools> ongoing AFAIK
[11:25] <bigjools> it got a bit stuck on the fact that the translations jobs don't have a build record
[11:25] <wgrant> Right.
[11:26] <wgrant> And adding one is going to make everything even more horribly duplicated.
[11:30]  * wgrant likes http://people.ubuntu.com/~wgrant/launchpad/buildfarm/new-build-model.png as a solution to that.
[11:43] <bigjools> wgrant: yes, we had something similar in mind
[11:43] <bigjools> noodles775 and I discussed this
[11:43] <wgrant> Ah, good.
[11:44] <bigjools> I want to ditch the use of Job
[11:44] <bigjools> it's been a royal PITA
[11:44] <wgrant> Yep.
[11:49] <bigjools> running intel gfx at 1920x1200 really shows up how awful the performance is :/
[11:49] <wgrant> Lucid or otherwise?
[11:49] <bigjools> anything
[11:50] <bigjools> lucid right now
[11:50] <wgrant> Hmm.
[11:50] <NCommander> bigjools: I've nevcer had an issue with intel gfx running two screens at about that resolution
[11:50] <bigjools> it's fine at laptop resolutions, but on external monitors it's quite sluggish
[11:51] <wgrant> My old i915 drives a 1920x1080 TV fine.
[11:55] <bigjools> it could of course be kwin...
[11:56]  * wgrant is reminded of the oddities in Wellington.
[11:56] <bigjools> but what about the laptop?

[12:03] <gmb> Does anyone have any idea why I'd be getting psycopg2 import errors on db-devel branches? AFAICT it's installed correctly but LP seems not to see it. Could it be connected with the buildbot failures we saw yesterday?
[12:06] <wgrant> jml: Hm, lp/__init__.py isn't exactly an obvious place for that documentation.
[12:06] <jml> wgrant, it is if you are browsing API docs, I think.
[12:06] <jml> wgrant, such as at http://people.canonical.com/~mwh/canonicalapi/index.html
[12:07] <wgrant> Hm, those are nice.
[12:08] <jml> wgrant, I reckon we could do a lot worse than emulate Bazaar, and have a doc/hacking/ directory
[12:08] <jml> or however they spell it -- it's easy to find when you need it.
[12:11] <jml> I really really need to end this zope testing branch and figure out what the hell to do with these images
[12:11] <bigjools> gnargh, testfix mode
[12:12] <wgrant> jml: Any idea why http://people.canonical.com/~mwh/canonicalapi/lp.soyuz.interfaces.build.IBuild.html has only IBuildFarmJob linked, and none of the others?
[12:12] <wgrant> bigjools: That db_lp failure from four hours ago with me in the blamelist?
[12:12] <jml> wgrant, backticks, I think.
[12:12]  * bigjools is checking
[12:13] <jml> wgrant, alternatively, it's because the others are backticked but lack sufficient context for pydoctor to find them
[12:13] <wgrant> jml: getFileByName' ILibraryFileAlias is backticked.
[12:13] <bigjools> wgrant: test_sigusr2 failure?
[12:13] <wgrant> And IBuildFarmJob isn't imported in the other file.
[12:13] <wgrant> bigjools: No idea. I can't see buildbot failures.
[12:13] <bigjools> oh
[12:14] <bigjools> looks spurious
[12:14] <jml> wgrant, in that case, I don't know. I *do* know that it's a bit of a black art, and that mwhudson is the person to ask.
[12:15] <jml> heck, I don't even know if we have a convenience thing in tree to build those docs
[12:16]  * wgrant didn't find one.
[12:18] <bigjools> jml: +1 for separate branch but it will be hard until we can split up LP better than it is
[12:19] <jml> bigjools, it will be hard.
[12:19] <wgrant> That sounds really really hard.
[12:19] <bigjools> but totally necessary for a sane development experience
[12:20] <jml> bigjools, it will always be hard. so will collaborating on a 350k line tightly-coupled Python app.
[12:21] <bigjools> there are some strong arguments to make it more loose
[12:21] <bigjools> it also begs the question of why we made lib/lp/xxx in the first place
[12:21] <jml> bigjools, to make the structure clearer.
[12:22] <noodles775> step-by-step
[12:22] <bigjools> what structure?
[12:22] <jml> bigjools, it then turned out that we had less structure than we'd hoped
[12:22] <bigjools> exactly :)
[12:22] <jml> bigjools, but we had even less of an idea of that or the scope of that when everything was in one big directory
[12:23] <jml> bigjools, also, I think that it's almost always a good idea to change your code to match the way you think about the problem space
[12:24] <bigjools> no arguments from me there
[12:24] <jml> bigjools, in our case, we were talking always about "code" this and "bugs" that and "registry" whatever. the lp apocalypse has actually made our code clearer imo
[12:25] <wgrant> But the API constraints make a complete split difficult.
[12:25] <wgrant> And the Soyuz<->Registry split is insane.
[12:25] <bigjools> I agree, but it's also pointed out the nightmare of tight coupling
[12:26] <wgrant> (although I guess most of the Soyuz-related methods in Registry have belonged on IArchive since archive-rework three years ago)
[12:26] <bigjools> wgrant: it's a good idea to split it, the problems come because of other reasons
[12:26] <bigjools> yes
[12:27] <jml> wgrant, API constraints are the biggest technical blocker I can see.
[12:27] <jml> wgrant, bigjools: does the buildfarm code actually depend on stuff outside of the buildfarm?
[12:27] <wgrant> jml: Only a little.
[12:27] <wgrant> I've removed most of them.
[12:27] <jml> so it wouldn't be _that_ hard to move to a separate branch
[12:28] <wgrant> eg. BuildBase stuff depends on Archive.
[12:28] <wgrant> But BuildBase could reasonably move to Soyuz.
[12:28] <wgrant> And the builder UI code is still in Soyuz.
[12:28] <bigjools> well
[12:28] <jml> I mean, apart from the base level of hard involved in doing anything that changes our build & deployment processes.
[12:28] <bigjools> there's probably room for another module between the buildfarm and soyuz
[12:28] <wgrant> jml: But splitting something out when it depends on the DB?
[12:29] <bigjools> whatever "soyuz" is nowadays anyway
[12:29] <jml> wgrant, I don't see why that's an issue
[12:29] <wgrant> jml: Oh, also, buildmaster is currently tested with Soyuz objects, since it has no concrete job implementations of its own.
[12:32] <wgrant> What is the purpose of splitting it into a separate branch? It can't be run independently.
[12:33] <bigjools> ideally it should be
[12:33] <jml> wgrant, can't it? perhaps I misunderstand what it is.
[12:33] <jml> I'd put the bloody buildd slave code in a separate branch first, tbh.
[12:33] <wgrant> Ideally it should be.
[12:33] <wgrant> But it probably depends on DB details and blah blah blah.
[12:33] <wgrant> Although I guess it really needn't.
[12:34] <bigjools> the only thing I know about is tachandler.py
[12:34] <jml> wgrant, ideally it wouldn't, but that's not _actually_ a blocker
[12:34] <bigjools> otherwise it really should be ripped out
[12:34] <wgrant> Why is tachandler not somewhere like Twisted?
[12:34] <bigjools> nfi
[12:34] <jml> wgrant, a few reasons
[12:35] <jml> wgrant, in fact, Twisted might have grown better APIs since we wrote tachandler
[12:35] <jml> wgrant, it's not code that a lot of Twisted apps feel the need for.
[12:36] <wgrant> jml: Well, a few unrelated bits of LP do. What's so special about them?
[12:36] <jml> wgrant, tbh, I don't know why we use it.
[12:36] <jml> wgrant, other than for integration tests.
[12:37]  * jml greps
[12:38] <jml> ReadyService, afaict could be in Twisted and is only really there to make testing easier (icbw of course)
[12:40] <jml> and TacTestSetup is essentially an API for manipulating out-of-process Twisted applications
[12:44] <jml> wgrant, I guess someone could spend a few hours and end up by getting rid of tachandler
[12:45] <jml> maybe I'll take on splitting out the buildd code as my hacking task after I land the zope.testing upgrade and my poppy work.
[12:48] <jml> wgrant, there you go, I asked on #twisted.
[12:48] <jml> that is progress.
[12:51] <wgrant> I should probably get my buildmaster cleanup branches into potentially mergable condition.
[12:54] <jml> I should probably do some strategy today :)
[12:57] <NCommander> wgrant: jml: any chance you can make the ABORT button work?:-)
[12:57] <bigjools> so, removing my gfx card and booting works fine - I just don't get a desktop :)
[12:57] <wgrant> bigjools: Ah, easy fix, good.
[12:57] <NCommander> bigjools: hope that card is under warrenty?
[12:58] <bigjools> well easy, but expensive :()
[12:58] <bigjools> no, it's about 18 months old
[12:58] <wgrant> NCommander: The slave changes aren't hard. The main difficulty is working out how to transfer the request through the DB.
[12:58] <wgrant> Do we have new Aborting and Aborted DB statuses?
[12:59] <bigjools> we also need a CANCELLED
[12:59] <bigjools> we're abusing SUPERSEDED right now
[12:59] <wgrant> Right.
[12:59] <bigjools> lamont will love whoever fixes that
[13:00] <wgrant> This should probably be thought about in the schema redesign, if that ends up happening.
[13:00] <wgrant> Since it applies to all job types.
[13:00] <bigjools> yes
[13:01] <bigjools> but as much as I hate it, the job status is a separate status to the build status
[13:02] <wgrant> +ALTER TABLE ONLY build ADD CONSTRAINT build__archive__fk FOREIGN KEY (archive) REFERENCES archive(id) on delete cascade;
[13:02]  * wgrant cries.
[13:02] <bigjools> ha, if I had a hdmi cable my desktop would be working, it has an onboard nforce 750a
[13:02] <wgrant> bigjools: It has onboard HDMI but not VGA?
[13:02] <bigjools> yep :/
[13:03] <wgrant> WTF
[13:03] <bigjools> yep :/
[13:03] <jml> NCommander, abort button? what do I look like? a programmer?!
[13:03] <bigjools> wgrant: gar, that one is wrong isn't it. Crapola.
[13:03] <wgrant> That is a seriously fucking dangerous DB patch.
[13:04] <bigjools> oO
[13:05] <wgrant> Oh, no, it does leave a constraint or two that will stop it cascading too far into other archives. Good.
[13:06] <wgrant> (If BPPH.binarypackagerelease had been altered to cascade... oh dear)
[13:10] <bigjools> wgrant: so, I could make the Build.archive on delete NULL, but that's gonna break lots of shit
[13:10]  * bigjools thinks
[13:10] <wgrant> That's what I said last night :)
[13:11] <bigjools> so there's a few options
[13:11] <wgrant> There is no precedent for doing this sort of thing.
[13:11] <bigjools> 1. don't delete the archive
[13:11] <bigjools> 2. delete the binaries
[13:11] <bigjools> 3, ??
[13:12] <bigjools> 4. profit
[13:12]  * bigjools is fed up with people insisting that ppa deletion is easy
[13:12] <wgrant> Deleting the archive means deleting all sorts of stuff. Future copy logging will be difficult, PackageUploads (and thus all upload accountability information and changes files) will have to be removed...
[13:13] <wgrant> It's not like a branch, which is one object, pontentially with a non-critical tendril attached to another branch.
[13:14] <bigjools> yes
[13:14] <bigjools> however I feel very strongly that we should delete them in their entirety if at all possible
[13:15] <jml> wgrant, stacking is a critical tendril
[13:15] <wgrant> I am in two minds about it. It's difficult and wrong, but also much cleaner.
[13:15] <wgrant> jml: But you deny in that case, don't you?
[13:15] <jml> wgrant, maybe.
[13:15] <bigjools> we could delay the full deletion until the last PPA that uses another's packages disappears
[13:15]  * jml really likes the infrastructure abentley set up for deleting branches
[13:16] <bigjools> I'm starving and I really like the fact that my fridge is full of food, bbiab
[13:17] <jml> heh heh
[13:48] <stub> You can delete the things its safe to manually, and leave the constraints there to raise exceptions when necessary.
[14:00] <kfogel> intellectronica: you saw https://code.edge.launchpad.net/~kfogel/launchpad/78565-bug-comment-link-to-bug/+merge/21896 ?
[14:04] <dobey> hi all. i'm having some trouble filtering bugs for rhythmbox-ubuntuone-music-store (Ubuntu). they seem to be getting past my procmailrc filter (though all other Ubuntu bugs are being filtered just fine)
[14:05] <maxb> Hi dobey. Perhaps you mean to be in #launchpad ?
[14:05] <dobey> but for rhythmbox-ubuntuone-music-store bugs, there seems to be a newline inside the header... perhaps causing procmail to not interpret the header properly?
[14:07] <dobey> maxb: no, i don't think so. this seems to be a bug in the launchpad bug mail creator
[14:07] <maxb> Please paste the mail headers to a pastebin.
[14:10] <bigjools> wgrant: still around
[14:10] <bigjools> ?
[14:14] <dobey> maxb: http://pastebin.ubuntu.com/399993/ <- this is from one mail that's failing
[14:15] <dobey> maxb: and http://pastebin.ubuntu.com/399994/ is from a mail that is successfully filtered
[14:16] <maxb> dobey: I believe that's header folding, as specified by rfc2822.
[14:16] <maxb> I.e. it's your filter's fault
[14:16] <maxb> sorry
[14:17] <dobey> * ^X-Launchpad-Bug: distribution=ubuntu; sourcepackage=\/[^;]+
[14:18] <dobey> that's the filter... and it works for the 59 other Ubuntu packages that i've gotten bug mails for
[14:18] <dobey> i'm just trying to understand what the difference is
[14:18] <intellectronica> kfogel: i did, and for some reason i thought i approved it. it was early in the morning so i was probably confused. r=me and will tick the mp now
[14:18] <dobey> and others seem to have 'folded headers' too
[14:19] <thekorn> dobey: the "\n " between distribution=... and sourcepackage=
[14:20] <dobey> oh right
[14:20] <dobey> blah
[14:22] <maxb> dobey: A fully compliant filtering solution would apply rfc822 unfolding before applying regexps to header values.
[14:34] <maxb> OOI, do db patches ever get renumbered, or is the idea that the DBA only assigns them a real version number when they will be nearly guaranteed to land in the upcoming dev cycle?
[14:41] <BjornT> maxb: the db patch numbers aren't version numbers. they are more like identifiers. it doesn't matter of a db patch gets a number and never lands.
[14:41] <maxb> But, they do control the relative ordering of patch application
[14:41] <maxb> (right?)
[14:48] <BjornT> maxb: yes. so in theory it could cause problems not landing them in order, but not really in practice. i can't ever recall that causing a problem.
[15:15] <bac> hi leonardr
[15:16] <leonardr> hi bac
[15:16] <leonardr> i'm on the phone but type away
[15:17] <bac> leonardr: when using utilities/ec2 i'm getting the "File name too long" error from httplib2.  you did a fix for that in lazr.restfulclient.  should it not be in 0.9.12?
[15:18] <leonardr> bac, i'd think so
[15:20] <bac> leonardr: your fix was on 2010-02-09 so it should've been in 0.9.11 and definitely 0.9.12
[15:20] <leonardr> bac, give me any information about the failure you can
[15:21] <bac> leonardr: http://paste.ubuntu.com/400029/
[15:28] <james_w> leonardr: is there a way to test the wadl that is generated for a particular entry from the LP test suite?
[15:29] <james_w> leonardr: I have to make a change, but the current tests pass and they use the webservice calls in the story tests, and show that the representation is correct. The bug is that the wadl is incorrect.
[15:33] <leonardr> james_w: unfortunately you are #4 in line
[15:44] <mars> sinzui, quick question: if I have a bug about a page's HTTP cache headers, should that bug go against launchpad-web or launchpad-foundations?
[15:44] <sinzui> foudations
[15:45] <mars> sinzui, so launchpad-web is for UI only?  Not browser mechanics?
[15:45] <sinzui> mars: web is for css/markup issues that are common to all applications. web is thus about the site design
[15:45] <mars> right
[15:45] <sinzui> mars: no mechanics. these bugs should be fixable by designers
[15:46] <mars> hmm, good category
[15:46] <mars> I was going through the list of bugs tagged CSS, and a number fell into that category.
[15:47] <mars> There were three categories IIRC: inconsistent application of the LP style, stuff where the current UI design can't cope with the data, and browser-specific display issues
[15:47] <sinzui> yes, I came to that understanding when I triaged the bugs gary_poster was uncertain of.
[15:48] <sinzui> I note that users started putting application specific bugs in launchpad-web. I moved them to the specific app
[15:49] <mars> hmm. orthogonal concerns: responsibility, and category
[15:50] <bac> leonardr: i see in your safename replacement you limit to 117 (152-32-1) so it is less than 150.  but that is just limiting the key portion.  httplib2 then adds the cachedir name to it, which in my case is 52 characters
[15:50] <bac> leonardr:         cacheFullPath = os.path.join(self.cache, self.safe(key))
[16:05] <leonardr> bac: i see. you have a long cachedir name
[16:06] <bac> leonardr: not really.  it is the default.  yours would be longer.  (leonardr > bac)
[16:06] <bac> leonardr: i'll open a bug
[16:06] <leonardr> is it the path name that is limited or the filename?
[16:07] <bac> currently on the key is limited
[16:07] <leonardr> i meant in the underlying ecryptfs filesystem
[16:10] <leonardr> james_w: i'm not sure what you mean by the wadl generated for a particular entry. do you mean the wadl representation of /~leonardr? i suspect you mean something else but i'm not sure what, nor which wadl is incorrect
[16:14] <james_w> leonardr: the wadl declaration for ICodeImport is incorrect
[16:15] <james_w> it asserts there will be a "branch" parameter, when there is in fact a "branch_link" parameter
[16:15] <james_w> I know the code fix, but I don't know how to test it from that angle
[16:15] <james_w> I could do a test that ICodeImport.branch is a ReferenceChoice and not a Choice, but that doesn't seem ideal
[16:17] <james_w> the issue is that lazr.restful changes its behaviour between the declarations and the marshalling when the attribute is an entry, but not declared with the extra information of a ReferenceChoice
[16:19]  * james_w heads for some lunch, any suggestions welcome
[16:19] <leonardr> james_w: ok, i need to think about this
[16:37] <bdmurray> How can I know when my branch is rolled out on edge?
[16:38] <bigjools> bdmurray: if you know the revision number that it landed with, you can compare against the rNNNNN at the bottom of each edge page
[16:38] <bac> leonardr: https://bugs.edge.launchpad.net/lazr.restfulclient/+bug/545197
[16:38] <mup> Bug #545197: Cache file paths are too long for encryptfs <lazr.restfulclient:New> <https://launchpad.net/bugs/545197>
[16:38] <bdmurray> bigjools: I was hoping for a notification not something I'd have to check ;-)
[16:39] <bigjools> bdmurray: ah :)  edge is normally gets rolled out each day
[16:39] <bigjools> and please correct my English
[16:40] <bigjools> bdmurray: if there was a bug linked to the branch, you'll see its status change when it sees the branch on edge or staging
[16:41] <bdmurray> bigjools: okay, awesome that'll work
[16:41] <leonardr> bigjools: the problem is a misconfiguration on dogfood
[16:41] <leonardr> the service root for the 'api' virtual host needs to have its /beta/ removed
[16:42] <bigjools> leonardr: ah ok, where do I edit that?
[16:43] <leonardr> bigjools: launchpad-lazr.conf
[16:43] <leonardr> [vhost.api].rooturl
[16:44] <leonardr> you _also_ need to remove /beta/ from your dogfood root url, or launchpadlib's first request will fail
[16:44] <leonardr> if you remove /beta/ from lputils.py but don't change dogfoot, the first request succeeds and the second one fails
[16:46] <leonardr> if you fix all this and dogfood credentials still aren't written to disk, let me know
[16:48]  * bigjools is trying now
[16:53] <leonardr> james_w: ok, i understand what you're asking for
[16:58] <leonardr> if you look at lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt you'll see a test that retrieves the wadl description of launchpad and does minimal tests to it
[16:58] <leonardr> if you look at lazr.restful/src/lazr/restful/example/base/tests/wadl.txt you'll see a more detailed test that dissects a wadl file
[16:59] <leonardr> you could use an xpath selector to grab the part of the wadl file you want to examine
[17:00] <leonardr> but you might like this idea better
[17:00] <leonardr> the launchpad wadl description is transformed into an html document
[17:01] <leonardr> lib/canonical/launchpad/pagetests/webservice/apidoc.txt grabs that html document and prints it out
[17:01] <leonardr> you could do something similar to verify that branch_link shows up under code_import
[17:01] <leonardr> so you have several options
[17:02] <bigjools> leonardr: ok, success on dogfood!  Thanks very much.
[17:02] <leonardr> yay
[18:01] <james_w> leonardr: what about ObjectLookupFieldMarshaller  checking for field.schema? Would that be too intrusive?
[18:02] <james_w> leonardr: the idea being that it would refuse to unmarshall something that wasn't declared as a ReferenceChoice, and so not cause the skew between the wadl and the representation?
[18:06] <mrevell> night
[18:08] <leonardr> james_w: that might be a good way to stop errors from begin committed, since the normal wadl test would fail
[18:08] <leonardr> try it and see if it interferes with existing tests
[18:09] <james_w> ok, but I'm not keen on having to deal with eggs and buildout :-)
[18:14] <leonardr> james_w: if you just want to try it, change the code in your existing lazr.restful egg
[18:14] <leonardr> ~/.buildout/eggs/lazr.restful.*/...
[18:14] <leonardr> it'lll take effect immediately
[18:16] <james_w> ah, cool, thanks
[18:31] <bac> Ursinha: i approved your MP
[18:32] <bac> thanks
[18:33] <Ursinha> bac: oh, thanks :)
[18:49] <cody-somerville> wgrant, ping
[19:01] <mwhudson> good morning
[19:17] <jml> mwhudson, hi
[19:17] <jml> mwhudson, I need an answer for you re https://code.edge.launchpad.net/~jml/launchpad/update-ec2-image/+merge/21804
[19:19]  * mwhudson hadn't noticed it was back in question land
[19:19] <mwhudson> oh, "do we need to support karmic"
[19:19] <mwhudson> i guess not
[19:19] <mwhudson> jml: replied
[19:20] <jml> mwhudson, thanks. :)
[19:23] <bac> leonardr: so the 144 limit you discussed with dustin is just for the name portion, not including the path?
[19:31] <mwhudson> jml: "someone" should write a twisted success story for launchpad
[19:44] <leonardr> bac: right, the path limit is more like 4096 characters
[19:44] <bac> leonardr: ok.  sorry then for the red herring in the bug report
[19:44] <leonardr> np, it worked out
[20:01] <didrocks> gmb: flacoste: hey o/ will it be possible to finish the second merge review on exposing ssh key this week (https://code.edge.launchpad.net/~didrocks/launchpad/expose-sshkeys-bug-357235/+merge/20995)? it prevents me from uploading last Quickly into ubuntu
[20:24] <james_w> is anyone else seeing a failure in lib/canonical/launchpad/ftests/../doc/webservice-marshallers.txt ?
[20:26] <james_w> I'm getting a TypeError on devel
[21:48] <james_w> make clean && make hasn't fixed it, and neither has anything else I have tried
[21:49] <james_w> would someone try running "./bin/test -cvvt lib/canonical/launchpad/ftests/../doc/webservice-marshallers.txt" in a (fairly) up to date copy of devel?
[22:03] <gary_poster> sinzui: would you mind helping me categorize the following as registry,  foundations,  or something else? https://bugs.edge.launchpad.net/launchpad-foundations/+bug/231797  It doesn't feel terribly foundations-y to me, but I'll take it if you think it's appropriate.
[22:03] <mup> Bug #231797: no sensible way to use debian/watch files with launchpad hosted tarballs <Launchpad Foundations:New> <devscripts (Ubuntu):Invalid> <https://launchpad.net/bugs/231797>
[22:11] <wgrant> cody-somerville: Hi.
[22:12] <wgrant> james_w: What's the failure?
[22:14] <james_w> TypeError: a float is required
[22:14] <wgrant> More traceback?
[22:15] <james_w> canonical.launchpad.webapp.adapter.set_request_started() isn't being called, or some variation on that
[22:15] <wgrant> Hm. You've updated sourcecode and download-cache and everything lately?
[22:16] <james_w> and I can't see what is supposed to call LaunchpadBrowserPublication.beforeTraversal
[22:16] <james_w> hmm, I haven't done sourcecode
[22:16] <wgrant> It'll be somewhere in Zope, probably.
[22:17] <james_w> yeah, that's my problem :-)
[22:22] <mwhudson> james_w: i get the same
[22:22] <james_w> hooray!
[22:23] <james_w> well, not really, but I'm glad I'm not alone in this ;-)
[22:23] <mwhudson> http://pastebin.ubuntu.com/400246/
[22:25] <james_w> yep
[22:25]  * mwhudson tries db-devel too
[22:28] <james_w> updated sourcecode and download-cache, did make clean && make and still see it
[22:29] <james_w> also, why isn't buildbot complaining?
[22:30] <mwhudson> lucid vs hardy maybe?
[22:32] <james_w> that's possible
[22:33] <james_w> I haven't upgraded any packages since this test was working, so I think there is a code change associated too
[22:34] <mwhudson> fails in db-devel too
[22:51] <mwhudson> james_w: i guess email the list and see what is said there
[23:04] <mwhudson> james_w: it fails for me too in production-devel which suggests some environmental change to me
[23:05] <mwhudson> and test failures from ec2
[23:05] <james_w> right
[23:05]  * mwhudson grumps off to lunch
[23:20]  * maxb runs 'lpnochange python-apt', and is very happy to have scripted these rebuilds
[23:27] <wgrant> Wasn't staging meant to be moved to Lucid reasonably soon?
[23:30] <mwhudson> "reasonably soon after release" i think?
[23:31] <wgrant> I thought it was meant to be before release.
[23:31] <wgrant> But I heard that a few months ago, so things might have changed.
[23:32] <maxb> It would seem a bit odd to run a beta distro in the datacentre?
[23:44] <lifeless> maxb: why ?
[23:44] <maxb> What's the rush?
[23:44] <wgrant> Test.
[23:44] <wgrant> Make it less disastrous than Hardy.
[23:44] <wgrant> That sort of thing.
[23:44] <lifeless> can't make changes to lucid after it releases
[23:44] <lifeless> can't tell if its suitable for the dc if we don't deploy it
[23:44] <wgrant> s/can't/shouldn't/
[23:45] <wgrant> Much better to get the applications running on it semi-production now.
[23:45] <lifeless> s/cant'/won't/ then
[23:45] <wgrant> lifeless: Hardy proved otherwise.
[23:46] <maxb> ?!
[23:47] <wgrant> Hardy had far too many SRUs right after release. That showed that there wasn't enough testing, as usual.
[23:47] <wgrant> The more testing Lucid gets in real environments before release, the better.