=== jamalta-afk is now known as jamalta
mwhudsonthumper: woo, new code import mail with actual information in it \o/01:04
jelmermwhudson: it looks like bzr-git will switch to a new caching format as of the next version01:12
mwhudsonjelmer: does that mean we'll need to do something to update the old caches?01:12
jelmermwhudson: better performant, but I guess it'll also mean there's some changes that have to happen in lp-code to cope with new filenames, etc01:12
mwhudsonor will regenerating them work?01:13
mwhudsonyeah, guess so01:13
mwhudsonshould be easy though01:13
mwhudsonjelmer: is the new format going to be pack based?01:13
jelmerregenerating them should work, if it doesn't that's a bug in bzr-git :-)01:13
jelmermwhudson: not pack based, but based on bzr indexes (there's no content so no need for packs)01:13
mwhudsonah, ok01:14
mwhudsonwill it be a single file still?01:14
jelmerno, it's a directory with multiple files01:15
jelmerthough if you wanted to you could trigger a repack so it ends up being just a single file (and a format indicator) in that directory01:15
mwhudsonmmf, that sounds boring01:16
mwhudsonprobably tarring it up is easier01:16
mwhudsonjelmer: will it be .bzr/repository/git-cache or something like that?01:17
jelmermwhudson: yeah, .bzr/repository/git01:17
jelmermwhudson: of course, Bazaar should really have hooks for fetch that allow us to copy this data along with the branch..01:17
mwhudsonjelmer: have you seen how AMAZINGLY SLOW the kernel import is going? https://code.edge.launchpad.net/~kiko/linux/2.6.3101:18
jelmermwhudson: yeah01:18
lifelessjelmer: yes, we should.01:18
jelmerlifeless: is this the moment where you suggest I submit a patch ? ;-)01:18
lifelessjelmer: I wouldn't insult your intelligence that way :)01:19
lifelessactually this is not trivial to do well01:19
lifelessand doing it badly will make push and pull suck arse for everyone01:19
jelmermwhudson: it'll be a bit faster with the new cache01:19
mwhudsonjelmer: i *think* we have tests that will fail when bzr-git changes here01:19
jelmermwhudson: also, there's some Google folks working on improving the performance of Dulwich, that'll help a bit01:20
mwhudsonjelmer: it will also be a bit faster when we get that new machine online...01:20
mwhudsonjelmer: heh, interesting01:20
jelmerlifeless: what are you thinking might make it problematic ? Figuring out what revisions to fetch?01:21
mwhudsondo you know why it's so slow for the kernel?01:21
mwhudsonjust because it's a big tree?01:21
lifelessjelmer: not triggering VFS on every pull01:21
lifelessjelmer: and integrating with streaming01:21
lifelesshow much to fetch and so on is also interesting01:22
lifelessbut I was figuring plugins will need their own protocol for negotiating that shit01:22
jelmerlifeless: yeah - how hard is it to add custom commands to the smart server protocol?01:22
jelmerhopefully using the bzr btree indexes will also make it a bit easier to just fetch those keys that we need - it'd be trickier than tdb01:23
wgrantmwhudson: It seemed to go from 3-5 hours per run to > 12 a couple of days ago.01:24
jelmerwgrant: it depends on the machine01:24
jelmergalapagos in particular is very slow01:24
mwhudsonwgrant: galapagos seems particularly slow01:24
mwhudsoni think elmo said one of the machines was one of canonical's very first machines :)01:25
lifelessjelmer: custom commands are easy; making it all hook in well not so much - we don't want plugin-count extra round trips either01:25
wgrantmwhudson: Ah, yeah, but neumayer isn't fast either by the look of things.01:25
wgrantYeah, galapagos is from the original naming scheme, so it makes sense that it's old...01:25
mwhudsonoh right, yeah01:26
jelmerlifeless: so even a single extra roundtrip if there's no data would be a problem? I guess in that case we really need something like what's specified in the 'branch baggage' spec to make it perform well01:26
mwhudsonneumayer and russkaya are both antartic bases01:26
mwhudsonthere's a new machine ready and waiting apparently01:26
lifelessjelmer: something-mumble-mumble01:27
wgrantPenguins, Antarctic bases, elements, fruits... is that it?01:27
lifelessjelmer: imagine 10 plugins doing this01:27
mwhudsonwaiting for our sysadmin to recover from dental surgery :/01:27
wgrantmwhudson: Yeah :(01:27
lifelessjelmer: if that was a rtt per just to probe, we'd spend 3 more seconds on an empty pull01:27
lifelessfor me <-> London01:27
mwhudsonwgrant: yeah, still on friut afaik01:27
mwhudsonjelmer: in other news01:28
mwhudsonjelmer: bzr-hg's tests are appalling01:28
jelmermwhudson: don't say I didn't warn you :-P01:29
mwhudsonthis is true01:29
jelmermwhudson, thumper: oh, btw, is it possible to upgrade lp:django ? www.isgitbetterthanx.com uses it to show that bzr is slow, and since it uses 0.92 it's indeed very slow01:36
mwhudsonjelmer: with a losa to help, yes :/01:37
mwhudsonjelmer: ask a question?01:38
jelmeragainst launchpad-code and assign to losas?01:38
mwhudsoni'll post some more details01:38
lifelesscan't a branch expert upgrade it on the web ui?01:39
mwhudsonnot for an import branch, no01:39
EdwinGrubbswgrant, jelmer: Can you look at this proposal if you have time? https://dev.launchpad.net/Registry/InvolvementPortletRefactor01:39
wgrantEdwinGrubbs: Looking.01:40
EdwinGrubbssorry, I have to leave now01:41
jelmerEdwinGrubbs: Sure, I'll have a look (might be tomorrow morning CET though)01:42
mwhudsoncalling mercurial.commands.commit() in a test is asking me to choose an editor01:42
jelmermwhudson: I think you can specify a custom ui object to prevent that01:43
jelmermwhudson: we have one that wraps a bzr ui factory01:43
mwhudsonyeah, seems likely01:44
mwhudsonoh right01:44
mwhudsonwoo, i think my tests are ok now01:48
mwhudsonnow why doesn't stop_revision to InterBranch.fetch() work...01:48
thumperand now to attempt to focus on work...02:01
mwhudsonjelmer: still there?02:04
jelmermwhudson: yeah02:04
mwhudsonjelmer: i'm not sure where to do the limit of importing in bzr-hg02:04
mwhudsonit doesn't seem to produce a nice toposorted list of the hg revisions to import anywhere02:05
mwhudsonunless a mercurial.changegroup.chunkiter() iterates in toposorted order?02:05
jelmeryeah, that's guaranteed to be toposorted02:06
jelmeralthough it would be nice to not fetch those revisions in the first place02:06
thumperjelmer: any movement on the bzr-hg bug that is causing 80% of them to fail?02:07
mwhudsonjelmer: is that going to be possible?02:07
mwhudsongiven a head, you'll have to walk back to null to find which ones to import first, won't you?02:07
pooliethumper: is a 'revision feed' a feed of all revisions for a particular project02:09
mwhudsonpoolie: or person02:10
thumperor team02:10
thumperor project group02:10
thumperpoolie: but yes02:10
lifelessnot quite02:11
thumperlatest revisions02:11
lifelessthere is a feed of new braches02:11
thumperin the last 30 days02:11
lifelessand per branch a feed of revisions02:12
lifelessI filed a bug asking for project feeds the day they went into beta02:12
thumperlifeless: and a revision feed for a project for over a year02:12
thumperlifeless: and I did it02:12
jelmermwhudson: I think that might be possible by doing the right thing in findmissing()02:12
lifelessthumper: ah! For some reason I thought it hadn't been done.02:12
lifelessthumper: sorry!02:12
thumperlifeless: it has been so long I've forgotten when I did it02:12
jelmerthumper: not yet, I need to spend some quality time with a particular branch that's failing to figure out what's going wrong02:12
jelmerthumper: and preferably write some tests that demonstrate the problem, that bit is probably the hardest02:13
thumperjelmer: ok02:24
* mwhudson is completely confused02:29
mwhudsonmaking a method which is called in one place which doesn't look at the return value return something causes tests to fail02:30
* mwhudson slaps himself02:31
mwhudsonjelmer: findmissing() is fairly opaque to me :/02:36
* mwhudson stares at it some more02:36
thumpermwhudson: what are you doing?02:37
mwhudsonthumper: incremental imports for bzr-hg02:38
* thumper is still doing code review comment emails...02:45
jtvwgrant: bob seems to think it's building for me02:46
wgrantjtv: I saw your bug reports that implied that, yeah.02:48
wgrantIs it actually working?02:48
wgrantWhat does the slave think it's doing?02:48
jtvthat's what I wanted to ask you: is there anything more to look at slave-side than the launchpad-buildd.log?02:49
wgrantjtv: No. That's all there is.02:50
jtvand I just zapped that for my next attempt :)02:50
wgrantBut it should give you lots of information.02:50
jtvI can go through the setup pretty quickly now though02:51
jtvThere wasn't much in it last time02:51
wgrantWhy are you reinstalling each time?02:51
jtvAm I?02:53
jtvI mean, I'm reinstalling every time I reinstall, obviously, but...02:54
wgrantWhy do you have to go through the setup again?02:56
* NCommander wonders if anyone can explain why we keep copyright file contents in the database and not in librarian02:58
wgrantNCommander: I asked that myself.02:58
NCommanderwgrant: was there an answer?02:58
wgrantNCommander: No.02:58
jtvwgrant: various reasons.  So far I haven't had to re-do the slave setup at all.02:59
jtvI have had to reboot a few times though.02:59
jtvwgrant: in many cases I had changes I _could_ just patch through without re-doing anything, but didn't want to because I'm trying to write up something reproducible.  It's easy to mess that up without knowing it.03:01
mwhudsoni don't like the mercurial code very much :/03:01
wgrantjtv: Ah, right.03:02
jtvwgrant: for example, between henning's work and mine we had a case where for him "dpkg -i <package> ; apt-get -f install" worked, and for me the other way around worked, because the --force-depends was missing from the dpkg command line.03:03
jtvWe'd like one order that works for everyone, but differences in our setup hid the problem in different ways03:03
wgrantThe dpkg, apt-get order should always work.03:04
wgrantIf it doesn't, give me examples because I don't believe you :P03:04
jtvNote the bit where I said --force-depends was missing...03:06
jtvif you do dpkg -i first, well, launchpad-buildd won't install because the dependencies aren't there03:06
jtvif you do apt-get -f first, I don't think there's necessarily anything there that'll make it install all dependencies03:07
jtvfor the package you have yet to install03:07
jtv...or maybe other stuff was broken in my case, which is the sort of thing you get when you don't re-do from scratch from time to time, which was my original point :-)03:08
wgrantdpkg -i should keep the package installed but unconfigured.03:08
wgrantThen apt-get -f install will satisfy the dependencies and finish installation.03:08
jtvdpkg -i leaves it unconfigured?03:09
wgrantIf the dependencies are not satisfied, yeah.03:09
jtvAnyway, when you do dpkg -i first, it refuses service because the dependencies aren't there, doesn't it?03:09
jtvThe --force-depends forces it to break things, and then the apt-get -f can fix the deliberate breakage03:09
wgrantIt probably looks like it refuses, but it actually lets it go half way.03:10
jtvah, like so03:11
jtvwgrant: meanwhile, it seems as if my job has stopped cold somewhere below the radar :(03:21
jtvThe buildd-manager log says it's dispatching.  The slave log says nothing.03:21
jtvMaybe the previous instance of the slave, which I killed a little late, grabbed it.03:21
wgrantjtv: Is buildd-manager repeatedly dispatching it?03:22
jtvno, just once03:22
wgrantAnd can you see buildd-manager currently polling in the slave log?03:22
jtvThe launchpad-buildd log on the slave ended with "daemon ready."03:24
jtvSo I think it went to the wrong buildd instance... I'm retrying.03:24
jtvwgrant: marking Bob as not-OK is not stopping its ongoing job this time...03:43
jtvDoes changing the OK setting POST to the slave on xmlrpc?03:44
wgrantjtv: Only buildd-manager can talk XML-RPC to the slave.03:44
wgrantMarking it not-OK will stop scanning.03:44
wgrantIt will not touch the slave, but it should deassign the job in the UI.03:44
wgrantAnd when you mark the builder OK again, buildd-manager should declare the builder lost (since it's building an unassigned job), and abort the builder.03:45
wgrantThe lost builder detection may not work in this case, however.03:45
jtvThat sounds plausible03:46
jtvwhat would break it?03:46
wgrantIt being crap.03:46
* wgrant finds the bug.03:46
jtvwgrant: ahhhh, _that_ explains _everything_!  :-)03:46
wgrantBug #46304103:47
mupBug #463041: Lost builder detection is insufficiently aggressive <buildd-manager> <Soyuz:Triaged> <https://launchpad.net/bugs/463041>03:47
wgrantIs there a way to make a @stepthrough consume multiple segments?03:47
wgrantI want a @stepthrough method to take multiple arguments, without having to create intermediate objects.03:48
* thumper stabs his tests03:48
mwhudsonwgrant: i think you need to write your own navigation (?) to do that03:50
mwhudsonthere's something more general than @stepthrough, anyway03:51
wgrantThe closest thing I can think of is that BranchNamespace thingy.03:51
* jtv has to be off for a bit03:53
wgrantmwhudson: Ah, thanks.03:54
thumpertesting jobs is a PITA03:56
wgrantIt seems sort of evil, but it might work.03:56
jtvamen to that03:59
* mwhudson EODs04:22
* thumper waves at mwhudson04:23
* thumper sprinkes some removeSecurityProxy around04:28
mwhudsonheh heh04:30
pooliethumper: did robert ask you yesterday what happens if you set a bmp to state 'queued'?04:43
pooliedoes the world end? does the ui cope?04:43
thumperpoolie: he did04:43
thumperpoolie: I suggested that he shouldn't04:43
thumperpoolie: weird shit may happen04:44
thumperit might work04:44
thumperit might not04:44
lifelesspoolie: thumper also said that staging would probably be a good enough indication04:44
thumperit should04:44
lifelessof what weird shit would happen04:44
thumperit'll either work or not04:44
thumperstaging should be able to tell you04:44
* persia wonders of !ohmy is supposed to apply in launchpad channels04:46
wgrantIt is often violated here, but it seems unproblematic.04:48
wgrantNo bot :(04:49
lifelesswhat doth it mean?04:55
wgrant15:55:19 <ubottu> Please remember that all Ubuntu IRC channels share the same attitude of providing friendly and polite interaction with all users of all ages and cultures. Basically, this means no  foul language and no abuse towards others.04:55
thumperpersia: NZ has a slightly more relaxed attitude to swearing04:56
thumperpersia: there are several prime time adds that use "bugger" and "bloody idiot"04:57
thumpersorry if I offended anyone04:57
persiathumper: I'm sure.  I don't claim that ohmy applies here, just asking.  Mostly because of e.g. http://irclogs.ubuntu.com/2010/03/11/#launchpad-dev.txt04:57
thumper'twas not my intention04:57
* persia certainly isn't offended, but is so used to typing !ohmy in other channels, that the message had to be erased before the query04:58
thumperwhich bit?04:58
lifelesspersia: I don't see the thing that triggered you to say that05:00
poolielifeless: i think you meant to say "wtf does that mean?" :)05:00
pooliethe s-word05:00
* thumper EODs05:00
thumperlater people05:00
pooliethump em and thump em again05:01
lifelessoh, wow.05:01
pooliepersia: i think we share that attitude but i don't think tim's expression trips it05:04
pooliethis may just show i need sensitivity training05:04
wgrantIn some channels it does.05:04
lifelessas colloquial for 'things' 'my shit', 'weird shit' etc are pretty ingrained. 'your shit' would be an aggressive use and would trip my bad-censor05:05
lifelessI blame the US for popular culture and tv shots.05:05
persiapoolie: Personally speaking, I'm open to lots of expressions.  That said, I've seen long-winded arguments about usage of terms, and self-admitted 8-year-old people in some development channels.05:05
* persia agrees with lifeless05:05
=== jamalta is now known as jamalta-afk
lifelessok, EODing05:07
NCommanderwhere can I get a log of the local keyserver? I'm getting a 500 when I try to create a soyuz user06:03
* persia has a sense that the query is dropping into the nebulous dark span that lies between NZ and EU timezones06:06
NCommandernm, got it by killed twistd and restating everything06:08
* wgrant appears.06:58
wgrantNCommander: It all ended up working?06:58
NCommanderwgrant: when applied a heavy enough wrench, although the soyuz script didn't add my GPG key to the ppa-user, and I'm getting issues with rejection based on "Can not build for architecture: any"06:59
wgrantNCommander: You specified the right email address?07:01
wgrantThe "Cannot build for architecture" thing normally means that your DASes either aren't enabled for virtualisation, or do not have chroots.07:01
NCommanderwgrant: yes, and the key posted to the internal keyserver07:02
NCommanderwgrant: it might have gotten confused though, I have my keys on a GPG smartcard07:02
NCommanderwhich makes gpg --list-keys look a bit different07:02
NCommanderwgrant: second BTW, for pygpgme, there's a patch in the archive that fixes it07:02
wgrantjtv: Did you have any more luck with the slave?07:03
wgrantNCommander: We don't use the archive version :(07:03
jtvwgrant: no...07:03
NCommanderwgrant: right, but that patch can just be applied to the version in the tree07:03
wgrantjtv: You didn't try, or it didn't work?07:03
NCommanderwgrant: I would do it, but I know diddly about pushing to launchpad-pqm or doing a review on an external component07:03
jtvwgrant: I didn't try, and waiting didn't help either :)07:04
jtv(helping out with something urgent here)07:05
NCommanderwgrant: can you help me get ready to push my first changes to LP? I have a database change that needs to be made, and I'm kinda confused on how I edit the schema07:05
wgrantNCommander: Sure. What's the change?07:05
NCommanderwgrant: first pat of implemented raw source changelogs07:06
NCommanderwgrant: the part I wrote causes new packages to have the changelog stripped out and put in the database07:06
NCommanderon upload07:06
wgrantNCommander: ie. new column SPR.changelog -> LFA.id?07:06
NCommanderwgrant: no. just SPR.changelog07:07
NCommanderper bigjools and sadbfl's comments07:07
wgrantBut changelogs get unsmall.07:07
persiaTHey don't "get" unsmall, they are by default.  Still : https://bugs.edge.launchpad.net/soyuz/+bug/139028/comments/107:09
mupBug #139028: SPR.changelog should be renamed to 'changelog_entry' <soyuz-core> <trivial> <Soyuz:Fix Released by julian-edwards> <https://launchpad.net/bugs/139028>07:09
wgrantHas the DBA been consulted?07:11
lifelesspep8 names are  preferred07:12
lifelessif thats your concern07:12
wgrantlifeless: I'm more wondering about the storing large strings in the DB thing.07:13
NCommanderwgrant: well, worse case, I rewrite a bit of code, and some comments :-)07:13
wgrantstub: Do you have an opinion on this?07:13
NCommanderwgrant: while we wait, maybe you can explain how do I integrate a schema change? I've already gotten the bit htat I need db-devel07:26
wgrantNCommander: Ah, right. I tried to find the wiki page, but then U1 ate my bandwidth.07:28
wgrantI've killed it, so maybe it will work better this time.07:28
wgrantI believe that https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess is the latest.07:29
NCommanderwgrant: cool, so now I need to learn how to interact better with bzr07:31
* NCommander reads on bzr loom07:32
wgrantNCommander: I prefer pipeline these days.07:33
NCommanderwgrant: pipeline?07:33
* NCommander admits he perfers git when dealing with non-trivial archives07:33
wgrantNCommander: https://launchpad.net/bzr-pipeline07:33
wgrantNCommander: Why?07:33
NCommanderwgrant: git rebase is my best friend with dealing with local trees before submisison07:35
wgrantNCommander: Ew.07:36
wgrantHistory destruction is revolting.07:36
wgrant(but bzr can rebase too)07:36
NCommanderwgrant: having all of ones changes ontop of the history is very handy07:36
NCommanderBut we digress07:36
* NCommander decides not to bother with looms or pipeline with this change07:37
adeuringgood morning08:21
* thumper prefers pipelines too08:25
lifelessthumper: different use cases08:29
lifelessthumper: hey, how is your bzr using project going08:29
thumperlifeless: slowly08:29
thumperlifeless: how to I create a preview tree for a particular revision?08:29
thumperlifeless: I want to commit to it08:29
thumperlifeless: and either pull or merge into the main branch08:30
stubwgrant, NCommander: (from 1 hour 15 mins ago) It depends :-) changelogs are not that big. If we want to search on them, they certainly can go in the database. If we don't, we are juggling the convenience of stuffing them in the database vs. sticking them in the librarian. I want to keep binary blobs out of the database, so if the text is in an unknown encoding it should definitely get stuffed in the Librarian.08:31
lifelessthumper: a preview tree is the result of a merge done into a revision tree AIUI08:32
lifelessthumper: so I'd look at the pipeline pump code08:32
thumperlifeless: ok08:32
thumperlifeless:  a revision tree is tree based on a revision?08:32
lifelessyou can commit from the transform I believe08:32
thumperlifeless: can you commit to a revision tree?08:33
lifelessthumper: a revision tree - b.basis_tree(), or b.repository.revision_tree(arbitrary_revid)08:33
lifelessthumper: no08:33
lifelessrevision trees are immutable08:33
lifeless'MutableTree.commit()' commits a new tree to a branch.08:33
NCommanderstub: we already have copyright in the database and the encoding is guessed (although I realize that kinda is hit or miss). sabdfl made a comment that we should stuff it in the DB already as well08:33
lifelessthumper: give my phone a ring; just going for a walk and voice will sort this quicker08:33
NCommanderstub: plus I want to retroactively add all the past changelogs, which is going to be about 5 GiB when done08:33
stubNCommander: What do we do with them once they are stuffed in the database? Are we going to display them inline in the web UI or similar?08:34
NCommanderstub: ideally08:35
stubIf we are guessing encoding, is there a use case for keeping an unadulterated copy in the Librarian (eg. are they GPG signed)08:35
NCommanderstub: the file ripped out of the source package, the original can always be retrieved there08:35
NCommanderstub: right now, the code just exists to put them in the database. I need to talk to bigjools and persia on how we want to change +changelogs and the SRP page specifically.08:35
stubYup. Also, do we want to store the entire changelog (a potential DOS perhaps), or truncate it to some sane limit.08:36
wgrantWe pretty much need the whole thing.08:36
* NCommander thinks wgrant can explain better than I can08:37
wgrantSince we need to be able to produce an Ubuntu->Debian changelog delta.08:37
stubBasic procedure for altering the schema is to create a database/schema/patch-2207-99-0.sql file containing a sequence of CREATE TABLE, ALTER TABLE etc. DDL commands. The first line and last line are dead chickens you can steal from one of the other patches. When requesting a review, request 'db' reviews from me (stub) and bjorn (bjornt).08:38
stubSo if I upload a sourcepackage to my PPA that when uncompressed contains a 60GB changelog.... ?08:38
wgrantRight, we're screwed.08:39
thumperlifeless: sorry, called away08:39
thumperlifeless: I'll talk to you tomorrow about it08:39
stubI suspect we do want some limit. Just pick an arbitrary number we won't hit except in case of a) major screwup or b) attack.08:39
jtvwgrant: meanwhile, on my simulated buildd, I've deleted the job that was in the way but my own is just not getting picked up.  :(  Still sits there w status 008:39
* NCommander coughs08:39
wgrantjtv: What does the query do?08:39
jtvwgrant: the candidate selection query?08:40
noodles775jtv: put a break point in the candidate selection and follow it through.08:40
jtvLast time I tried that, it worked.  Maybe it's failing for some new reason.08:40
stubNCommander: Of course, the 2GB limit on a text column is the arbitrary limit if you don't set one, so it probably doesn't matter.08:41
NCommanderstub: ugh :-/.08:41
noodles775jtv: related to bug 53679708:42
mupBug #536797: Builder pages break for job without build <wellington> <Soyuz:New> <https://launchpad.net/bugs/536797>08:42
jtvnoodles775: no, it's before it gets to that08:42
noodles775jtv: I think it's straight forward to ensure that the builder index works for your jobs, but the harder part is how we present the history.08:42
NCommanderstub: thats really nasty :-/08:42
noodles775jtv: (no, I have a question about it ;)).08:42
jtvah :)08:43
jtv(al-maisan, what's supposed to happen right after this query?  SELECT SUM(BuildQueue.estimated_duration) FROM Archive, Build, buildpackagejob, BuildQueue, DistroArchSeries, Processor WHERE buildpackagejob.job = BuildQueue.job AND buildpackagejob.build = Build.id AND Build.distroarchseries = DistroArchSeries.id AND Build.archive = Archive.id AND DistroArchSeries.processorfamily = Processor.family AND Build.buildstate = 0 AND Archive.ena08:43
jtv )08:43
noodles775jtv: Currently the builder histor relies on the builds (as they are the only permanent record that something happened on the builder), but I'm guessing we'll need to modify it to instead rely on BuildFarmJob records, and not delete those immediately. Let me know what you think (you too wgrant) and I'll update the bug.08:44
jtvnoodles775: I think I see what you mean.  Yes, getting this right is going to be an interesting question; it also came up in the sprint.  My immediate worry was getting it not oopsing though.  :-)08:44
al-maisanjtv: where's that query from?08:44
wgrantnoodles775: I think we'll have to present each Job, with adapters for each type.08:44
jtval-maisan: the buildd manager08:44
jtvI agree with wgrant08:44
wgrantnoodles775: Hmm, I guess TTBJs don't actually have a builder once the BuildQueue dies, do they?08:44
jtv(I have it not oopsing for now)08:44
noodles775wgrant: nope, we'd need to defer the deletion of those, perhaps truncating the history to 1000 BuildFarmJobs or something.08:45
wgrantI think we have too many tables.08:45
wgrantnoodles775: Why truncate them?08:46
noodles775wgrant: at the moment we delete the BFJ, Job, QueueItem etc. once the (soyuz) build completes.08:46
wgrantnoodles775: I think just the BuildQueue is deleted.08:46
wgrantThe other three (Build, BPJ, Job) should survive.08:46
wgrantI hope.08:46
* wgrant checks.08:47
noodles775Yes, the Build is permanent of course, but I'd thought the others were deleted, but haven't checked. Doing so now too.08:47
jtvwgrant: there's some propagation of deletions in the python though08:48
wgrantnoodles775: BuildBase appears to kill the buildqueue.08:48
wgrantBut not the others.08:48
lifelessthumper: de nada08:48
wgrantBut the BuildQueue kills the rest.08:48
wgrantI think we should just keep them all.08:49
al-maisanFWIW: we also discussed briefly *not* deleting the BuildQueue records upon job completion and putting in place a purge mechanism that deletes them after some configurable period of time (e.g. a month)08:49
wgrantWe are going to have at least three five rows in other tables from each build for eternity anyway.08:49
wgrantSo three extra very tiny rows for each really shouldn't make much difference...08:50
NCommanderstub: so no issue with changelog in database aside from a sanity check on size?08:50
wgrantEr, s/three // in that first sentence.08:50
wgrantal-maisan: IIRC we argued this during the sprint, and nobody brought up a compelling reason to purge them.08:51
al-maisanwgrant: data volume?08:51
wgrantal-maisan: It's trivial compared to the other stuff we store for each build.08:52
noodles775jtv: so you've got a branch that ensures the builder-index doesn't oops for jobs without builds?08:52
wgrant(the build, build log, changes file, upload entries...)08:52
jtvnoodles775: yes08:52
jtvnot pretty, but...08:53
wgrantjtv: Does it just not link them if they have no build attribute?08:53
stubNCommander: Should be fine. (Assuming this is SourcepackageRelease) We probably want to split out the bulk data from SourcepackageRelease such as copyright, dsc_* etc. into a seperate table as it is getting pretty fat. Fine in theory, but because we use an ORM we generally load all columns even when we don't need them.08:53
wgrantal-maisan: Plus, if we keep some of the others than we can trim Build.08:54
al-maisanwgrant: I am inclined to agree .. however: we currently operate on a BuildQueue table with a very *small* number of rows and if that were to change we may see some performance issues08:54
NCommanderstub: ugh :-/. So maybe make a sourcepackagerelease_metadata in the nearish future?08:54
jtvwgrant: close.  It names the job type and says there's no build for it; and it shows the log regardless of who you are.08:54
stubNCommander: Yes, but with a name that doesn't suck :)08:54
bigjoolsmorning guys08:54
NCommanderstub: I'm from the platform team, we have a bad history of sucky names. (i.e., bundles, petals, leafs, etc. :-))08:54
wgrantMorning bigjools.08:54
noodles775Morning bigjools08:55
al-maisanGood morning bigjools08:55
wgrantNCommander: Heh, I remember that discussion.08:55
stubNCommander: It might not be a problem in reality - it depends if we have pages that load a large number of SourcePackageRelease records08:55
NCommanderstub: suggestions welcome08:55
NCommanderstub: no, its usually just loaded one at a time except for +copy-packages (I think)08:55
stubSourcePackageReleaseData sucks less in my opinion08:55
wgrantNCommander: It's a lot more than that, sadly.08:55
wgrantNCommander: Anything that displays a version number is loading an SPR to get it.08:56
NCommanderwgrant: ugh. Maybe I'll make mychangeset grow so we only need one or two hits against db-devel08:56
NCommanderwgrant: ow.08:56
stubBut that can be future anyway - mention it in the review request and open a bug if you can be bothered.08:56
* NCommander remember he could make the database timeout by access the archive copy-package page 08:56
lifelessis there a 'create a project' API script yet ?08:56
NCommanderDoing the near-equivelent ofSELECT * FROM SourcereleasePackages didn't make the database happy :-)08:56
wgrantbigjools: Do you have an opinion on preserving the buildqueue, job and specificjob for completed builds?08:57
bigjoolsNCommander: that happens because it does a lot of checks08:57
bigjoolswgrant: strongly against preserving08:57
wgrantkfogel: Hm, that single "Merging db-stable" is interesting.09:00
wgrantNot sure why that is.09:00
bigjoolswgrant: why has this come up?09:03
wgrantbigjools: There was a little bit of a discussion about how to present a generalised history.09:04
bigjoolsthose table rows are cruft once the job is done.  We should find a better way of keeping history.09:06
bigjoolsand I really, really like having a single table that contains all the waiting jobs09:06
wgrantbigjools: OK, so maybe BuildQueue could be purged. But the Job seems useful.09:07
wgrant(in the current model, all record of a translations job is destroyed once the build completes)09:08
bigjoolsdoesn't it have build records?09:08
wgrantI don't believe so.09:09
wgrantjtv: ^^?09:09
* jtv reads09:09
jtvNo builds.09:10
bigjoolsit should do!09:10
jtvWould they be the same kind of build records?  Do build records need to go into buildfarm?09:10
* jtv wonders if we should have a buildfarm team09:10
bigjoolsplease no more teams :)09:11
bigjoolsso noodles775 and I discussed last night that we would rename Build -> BinaryBuild and then a new Build table would have common fields from all build types09:11
wgrantkfogel: Ah! They were merged in db-devel r8326, which is the rev before the one c-c.py listed as being the first merge.09:12
lifelesswgrant: pop quiz, connecting lplib to staging09:12
bigjoolshaving persistent data hanging around in a set of tables that represent a queue is crack IMO09:12
jtvbigjools: and those will stay around "forever"?09:12
bigjoolsjtv: yes09:12
noodles775bigjools: but that's only really suitable for SPRecipeBuilds and BinaryBuilds... not for the TranslationTemplatesBuildJob tyep.09:13
noodles775I thought.09:13
wgrantlifeless: Launchpadlib.login_anonymously('your app', 'staging')09:13
wgrantbigjools: Does Job represent a queue?09:13
* noodles775 reads the back-scroll.09:13
bigjoolsnoodles775: if that's the case then we should take a look09:13
lifelesswgrant: is there a constant like EDGE_SERVICE_ROOT ?09:13
bigjoolswgrant: no, but buildqueue does :)09:13
wgrantbigjools: right.09:13
jtvwell, all 1:1 anyway in the cases we care about...09:14
wgrantlifeless: I think they're deprecated by the simple strings. But yes, STAGING_SERVICE_ROOT should exist.09:14
jtvbigjools, noodles775: anyway the new Build could be just an id to hold things together and still be useful.09:15
bigjoolsjtv: id, builder are the 2 main things09:15
noodles775OK, so if there was a build for TranslationTemplatesBuildJob's then that would enable us to generalise for all three build farm job types.09:15
lifelesswgrant: is there any convention for environment variables to control this?09:15
jtvbigjools: I'm sure there's stuff like dates that would otherwise perish with the more transient job/buildqueue rows09:15
noodles775But I think we need to think through this carefully.09:15
bigjoolsyes very carefully09:16
jtvwhy?  hack the thing up & roll it out I say!09:16
lifelesswgrant: like LAUNCHPAD_API=staging09:16
wgrantlifeless: Not sure.09:16
* lifeless makes one up09:16
=== jtv is now known as jtv-otp
bigjoolsjtv: date is also another good one :)09:16
* noodles775 updates the relevant bugs trying to summarise.09:16
wgrantnoodles775, bigjools: Why Build rather than Job?09:16
jtv-otpwgrant: lets us denormalize to our little hearts' content09:17
wgrantAll build farm jobs might not always be Builds.09:17
al-maisanthe actual build duration is another piece of data  for the generic `Build` table09:17
bigjoolsbuild farm jobs might not always be Jobs09:17
bigjoolsno wait :)09:17
bigjoolsJobs might not always be build farm jobs, is what I meant to say09:17
wgrantAnd indeed they are not. True.09:18
wgrantSo.. we need BuildFarmJob?09:18
bigjoolsor... Build :)09:18
bigjoolswhich is the same thing09:18
bigjoolsbut it could be called BuildFarmJob09:18
wgrantI thought you were averse to the idea of calling something Build ever again.09:18
bigjoolsI'm not precious about it09:18
wgrantI think we need a good list of all the columns on all the tables, and what we need to keep.09:19
wgrantIs there such a thing around?09:19
bigjoolsI don't even remember what I did yesterday so if I said that then great :)09:19
bigjoolssuch a thing is the db schema. no?09:20
wgrantThe second point is a subjectively processed version of the DB schema.09:20
bigjoolsif we could factor stuff out, that'd be great09:20
bigjoolsyeah it needs analysis09:20
lifelesswgrant: thanks09:21
noodles785Not that I know of, I'll try to create something like http://people.canonical.com/~michaeln/bfb/sourcepackagerecipe.png for the build/queue related tables today, if it's helpful.09:22
=== noodles785 is now known as noodles775
bigjoolsnoodles775: if it's easy, yes please09:23
lifelessnow thats fairly cool09:23
lifelessoauth tokens for edge work on staging09:24
bigjoolsnoodles775, wgrant, jtv-otp: Job on its own is too generic for a build farm job.  I'd even be happy to ditch our use of Job and migrate to BuildFarmJob completely.09:25
wgrantbigjools: Yeah, quite possibly.09:25
lifelesscall it task09:25
lifelessthat way you can really confuse people09:25
bigjoolsI've never been really happy with using Job09:26
jtv-otpbigjools: that whole subsystem needs a cleanup... not sure I'd enjoy yet more duplication09:26
bigjoolsI felt press-ganged into using it09:26
wgrantIt never seemed like a really good idea to use it.09:26
wgrantWe do not use much of it.09:27
bigjoolsI continually fail to see the duplication between Job and build farm jobs09:27
bigjoolssomeone wanted the buildd manager to become more like a job runner09:27
bigjoolsit's not going to happen09:27
=== jtv-otp is now known as jtv
jtvbigjools, wgrant: I'm not sure the Job runner uses most of what you see in the schema there either.09:40
jtvMy impression is, Job failed to become all it was planned to be, with failover mechanisms that never materialized.  Okay, I'm not trying to force anyone to stick with Job, but it'd be nice to have something that everyone could use and be happier for it.  Message queue plus log table maybe?09:49
jtvbigjools: unrelated note... what happens to a build slave when its chroot falls out of date?09:50
jtvWould it be noticed at all?  Would jobs still be dispatched to it?09:51
* persia has seen the results of such dispatches09:53
persiaDunno if it still can happen09:53
jtvwgrant, could this be related to my buildd stuckage?  lib/canonical/twistedsupport/processmonitor.py:254: twisted.internet.error.PotentialZombieWarning: spawnProcess called, but the SIGCHLD handler is not installed. This probably means you have not yet called reactor.run, or called reactor.run(installSignalHandler=0). You will probably never see this process finish, and it may become a zombie process.09:57
jtvpersia: what did those results look like?  I'm talking about my local system, so weeeird things can happen :)09:58
persiajtv: Generally FTBFS because the build-deps couldn't be installed.10:03
jtvpersia: hmm... so sounds like it wouldn't affect me.10:04
persiajtv: chroot-out-of-date shouldn't do anything particularly odd, other than make the build take longer.10:05
jtvpersia: because of the apt-get update?  I'm skipping those in this case, re-using the same slave in the same state, so I guess for my case even that wouldn't apply.10:06
bigjoolsjtv: sorry was otp10:06
jtvit happens10:06
jtvto all of us with hones10:06
persiajtv: There's two bits.  The update, and the build-dep install.  But yeah, if neither of those breaks because of old-chroot, you shouldn't care.10:06
bigjoolsjtv: the job runner is being improved all the time, but I see it as orthogonal to the build farm10:06
bigjoolsjtv: as for chroots, what do you mean by out of date?10:07
jtvpersia: ok, thanks; that means I'd better look in other directions for the explanation of my troubles.  Thanks!10:07
jtvbigjools: I've got a test setup as per https://dev.launchpad.net/Translations/BuildTemplatesOnLocalBuildFarm (which I'm writing), but this time around my jobs fail to dispatch.10:07
jtvWhen I restarted everything just now, I got a timeout from an attempt to talk to the Librarian though.10:08
jtvMaybe it's just the order in which I did things.10:08
bigjoolsjtv: quite possibly10:08
bigjoolsjtv: the builder does an apt-get dist-upgrade as part of setting up the chroot, BTW10:09
jtvbigjools: as a prelude to dispatch, I guess?10:09
jtvOr in the actual slave-side FSM?10:10
bigjoolswell the slave does it as part of kicking off the build10:10
jtvok, so that's not working I guess until I get my dispatch working10:10
bigjoolsthe chroot comes from the librarian so I guess not :)10:11
jtvI'll try registering my chroot earlier on in the process10:13
jtvokay, my librarian is down.  that'll do it.10:14
wgrantjtv: The chroot stuff is all done as part of the build process.10:14
wgrantAs long as you've run ensurepresent with that SHA1 before, the librarian doesn't matter.10:15
wgrant(as long as you're not calling ensurepresent again -- are you dispatching manually now, or with buildd-manager?)10:15
lifelesswgrant: you might like  merge_sorted in bzrlib.tsort10:15
jtvwgrant: I'm using manage-chroot; where would ensurepresent be called from?10:15
wgrantjtv: ensurepresent is a buildd XML-RPC. It's called from buildd-manager.10:16
wgrantlifeless: For what?10:17
jtvwgrant: so would that fail right there if the chroot on the slave were out of date _and_ the librarian is down?10:17
lifelessdetermining the revision thats shallowest10:17
lifelessand/or which ones should show up in a log-like situation10:17
wgrantjtv: "out of date" meaning you've uploaded a new one with manage-chroot.py?10:18
wgrantlifeless: Hmm. Possibly, yes10:18
jtvwgrant: I wouldn't swear to it never having happened since I created this slave image10:18
jtvSo I could just create a new slave image, but it takes a while10:18
wgrantOr start the librarian.10:19
jtvFixing the librarian seems like a better idea.10:19
jtvExactly.  :-)10:19
jtvI've started poppy... what else?10:19
wgrantmake run10:20
wgrantOr that.10:20
wgrantpoppy isn't important for you.10:21
wgrantUnless you're uploading source packages with dput.10:21
jtvNo, no package uploads for this10:21
jtvI tried make run_codehosting10:21
jtv...which should have started the librarian.  ook.10:25
* jtv looks for pidfiles10:25
lifelesswgrant: how do you say 'me' in LP apis ?10:37
wgrantlifeless: lp.me10:38
lifelessso team.owner = self.launchpad.me?10:39
wgrantlifeless: Yes.10:39
* lifeless tries10:39
lifelesshmm, its not .owner10:40
wgrantAh, yeah, team_owner10:41
lifelessoh well10:42
lifelessIPerson claims owner, in the apidoc10:42
lifelessand team claims to extend person10:42
wgrantteam has team_owner_link.10:42
wgrantWhere do you see IPerson's owner on +apidoc?10:43
lifelessperson ... owner 'Link to a person'10:44
lifelessoh, method declaration10:44
* lifeless whines about apidoc again10:44
lifelessI don't think I've seen a more confusing api documentation layout, ever.10:44
lifelesswgrant: any idea what 'team_owner_link: Constraint not satisfied.' actually means?10:46
wgrantlifeless: You're probably trying to unset it on a team, set it on a person, or set it to a Private Membership Team.10:47
lifelessah, no10:48
lifelessI was trying to something I was sure worked, but apparently not.10:48
lifelesswhich is to set it's owner to itself.10:48
wgrantWhat was it?10:48
lifelessWhich means you can't have a trivial anarchy.10:48
wgrantThere has to be someone at the root.10:48
lifelesscan you do A->B->A ?10:49
wgrantMemberships can't work like that, so ownerships probably can't. But you could try.10:49
lifelessmeh, its ok10:49
lifelessI just want to automate all the manual 'good practice' stuff I do on new projects.10:49
lifelessif I was imagining that I did something, thats ok :)10:50
deryckMorning, all.11:00
lifelesshow do you say 'project uses launchpad for bug tracking' via API's ?11:05
lifelesslikewise 'code is published in bzr branches' etc11:09
wgrantHow do I identify a binarypackagerelease's archtag?11:09
wgrantDo I have to go through the build's DAS?11:09
derycklifeless, you want to set this via the API or just get at the info?11:17
lifelessset it11:17
derycklifeless, there is official_malone on product, distribution, etc. but it's not exported in the API.11:20
lifelessderyck: exactly :)11:20
lifelessderyck: in case you're wondering, I hit my DRY threshold for making projects on launchpad.11:24
lifelessI already copy n paste 90% of the docs and settings do that.11:24
deryckah, ok.11:25
lifelesswhile I can think of interesting things to do by reading these settings11:26
lifelessI really do want to set them11:26
lifelessdo we have staging codehosting?11:27
wgrantlifeless: Yes.11:27
wgrantlifeless: The branch content isn't copied from prod, but you can push branches up.11:27
lifelesslp: doesn't seem to support it yet11:27
wgrantlifeless: lp://staging/~blah/blah/blah11:28
lifelessoh, handled in the xmlrpc server side?11:28
wgrantThat is all magic to me.11:28
wgrantI haven't bothered to look.11:28
wgrantbigjools: What should the BPRDC URL? The easiest appears to be something like +binaryhits/foo_1.0_i386.deb/2010-03-11/AU11:29
lifelesswgrant: you know far too much about our systems :)11:30
lifelesscheck out staging.launchpad.net/duh911:31
wgrantlifeless: Maybe.11:33
lifelesshmm, maintainer isn't set yet11:34
wgrantThat was set up automatically?11:34
lifelessis there a better url than self_link ?11:35
wgrantlifeless: web_link, but that doesn't exist yet.11:36
lifelessis there a bug for that yet ?11:36
wgrantBug #31669411:37
mupBug #316694: Add web_link property to resources <launchpadlib :Triaged> <https://launchpad.net/bugs/316694>11:37
lifelessok, duh11 has the right maintainer11:38
* lifeless commits and proposes for merge11:38
lifelesswgrant: lp:~lifeless/lptools/lp-project/ if you are interested.11:41
* lifeless now uses it to make his project11:43
lifelessderyck: you might be interested in that script too11:44
* deryck looks11:44
lifelessdid we end up turning on auto-list-approval?11:49
lifeless(the answer is yes)11:52
bigjoolswgrant: +binaryhits/foo/1.0/20100311/AU would be nicer11:55
lifelessderyck: what do you think ?11:56
wgrantbigjools: Yes, that's sort of what I have now, but the arch is a bit awkward.11:57
wgrantbigjools: Should I just use the archtag of the build DAS? That will be a bit odd for arch-indep?11:58
bigjoolsit has precedent though11:59
derycklifeless, its nice, definitely.  I'll use it myself. :-)12:00
=== matsubara-afk is now known as matsubara
lifelessderyck: :) high complement - thanks12:05
didrocksmars: if you are interested to track the bug we discussed yesterday: bug #53729812:06
mupBug #537298: Launchpad make firefox rendering pages very slow with lucid version <Launchpad itself:New> <https://launchpad.net/bugs/537298>12:06
=== Ursinha-lunch is now known as Ursinha
bigjoolswgrant: sorry, phone again. Meh, I thought it was used for DASSP[R] but those are linked to publications of course12:30
wgrantbigjools: Yes :(12:30
bigjoolswgrant: I can't see any great harm just using the arch-indep though?12:31
wgrantbigjools: ie. i386 instead?12:31
wgrantI think it is probably ambiguous, although it shouldn't be a problem in practice.12:32
wgrant(think foo_1.0_i386.deb, then foo_1.0_all.deb appears in another upload.12:32
* jtv got his job to dispatch again, but has no clue why it fails12:34
wgrantjtv: What does the slave log say this time?12:34
jtvwgrant: it gets stuck...  I snipped copies at the bottom of the wiki page: https://dev.launchpad.net/Translations/BuildTemplatesOnLocalBuildFarm12:35
jtvOn the bright side, the librarian was working and it looks as if the slave tried to get the tarball12:35
wgrantjtv: Does the fetch complete?12:35
jtvI don't know.12:35
wgrantsha1sum /home/buildd/filecache-default/f1f10b8402ed686aaf0307676c76f07b45af2a0912:36
jtvf1f10b8402ed686aaf0307676c76f07b45af2a09  /home/buildd/filecache-default/f1f10b8402ed686aaf0307676c76f07b45af2a0912:36
jtv(hey, we're a remote terminal session)12:36
wgrantSo it worked.12:37
jtv...yup, that's the sha of my tarball.12:37
wgrantjtv: Can you flip buildd-manager into debug mode?12:37
wgrantI'm not sure if it's any more useful now, but it's worth a try.12:38
wgrantNight lifeless.12:38
jtvg'night lifeless12:38
jtvwgrant: how do I do that?12:38
wgrantjtv: s/logging.INFO/logging.DEBUG/ in lib/lp/buildmaster/manager.py, IIRC.12:39
* jtv edits12:39
jtvnow that would be a nice thing for the config...12:39
jtvand now I guess I restart the daemon to make it take effect?12:40
jtvseems to be running again, and talking to bob12:43
jtv010-03-11 19:42:52+0700 [-] Dispatching: <bob:http://localhost:8221/>12:43
jtv2010-03-11 19:42:52+0700 [-] <bob:http://localhost:8221/> -> ensurepresent((u'f1f10b8402ed686aaf0307676c76f07b45af2a09', 'http://launchpad.dev:58080/93/chroot-ubuntu-lucid-i386.tar.bz2', '', ''))12:43
jtv2010-03-11 19:42:52+0700 [QueryWithTimeoutProtocol,client] <bob:http://localhost:8221/> response for "ensurepresent": [True, 'Cache']12:43
jtvAnd the slave got another Twisted/XMLRPClib post12:44
jtvbut now once again everything seems to sit still12:44
jtvBob is listed as doing "Job of type TranslationTemplatesBuildJob with no associated Build."  Which is what I taught it to say.12:45
jtvMy job has status 3, so that explains why there's no progress; but why no more scanning?12:48
wgrantWhat is status 3?12:48
jtvfailure iirc12:49
jtvit had that before I restarted the buildd, so why did it re-dispatch?12:50
jtvmaybe the best thing is to delete the job, restart the daemons, and retry12:50
wgrantIt does what it wants.12:50
wgrantProbably, yeah.12:50
jtvI can confidently confirm that debug output has been enabled...12:52
jtvit's trying to add a bunch of distroarchseries every 5 seconds12:53
jtvwell, here goes nothing!12:53
jtvxmlrpclib post on slave12:55
wgrantjtv: Yeah, the DAS-driven approach is pretty stupid.12:55
jtvjob is marked as started.12:55
jtvdamage assessment strike?12:55
jtvoh, that would work too12:56
wgrantFortunately I trialled a branch to remove it, and it worked.12:56
wgrantSo... what's the final POST?12:56
jtv2010-03-11 12:54:33+0000 [HTTPChannel,104,] - - [11/Mar/2010:12:54:32 +0000] "POST /rpc HTTP/1.0" 200 212 "-" "Twisted/XMLRPClib"12:56
jtvis what the slave says12:56
wgrantWell, how useful.12:56
wgrantWhat if you call status() on it manually?12:56
stubDistroSeries.getQueueItems tries and fails to return rows in a particular order. This is failing under PG 8.4. Should I just work around the bug by fixing the test? The method is flagged as depricated.12:57
jtvwgrant: what, "make harness," retrieve the slave, and invoke it on that?12:57
wgrantjtv: import xmlrpclib; xmlrpclib.ServerProxy('http://localhost:8221/rpc').status()12:58
jtvah like that... IDLE12:58
wgrantAnd it's marked OK in the UI?12:59
jtvbut the job still has status 112:59
jtvI think that's the part where Build came in...  :/12:59
wgrantjtv: Is the slave code reasonably similar to what's in trunk now?13:00
jtvwgrant: no, I merged henninge's branch; see the top of the page under "patches"13:01
wgrantjtv: I don't see a henning branch there13:01
=== henninge_ is now known as henninge
henningewgrant: it's henningE ;-)13:02
henningeAlthough my name *is* Henning13:02
jtvwgrant: hang on, I'll add that right away13:02
wgranthenninge: Right, I just didn't want to ping you unnecessarily.13:03
henningewgrant: thanks ;-)13:03
* henninge has an alarm on henning, too13:03
* henninge lunches13:03
jtvwgrant: lp:~launchpad/bug-509557-invoke-pottery13:05
wgrantjtv: s@~@~henninge/@?13:05
jtvwgrant: yes13:05
* wgrant looks.13:05
wgrantjtv: I'd be trying to work out what the XML-RPC call was.13:08
wgrantBut I should also be sleeping.13:08
jtvwgrant: both of those make sense13:09
jtvI guess I could just try the state machine.13:09
jtvoh! the slave just showed some activity again13:09
wgrantSo it was just hanging for a while?13:09
wgrantMaybe you're not using the state machine properly.13:09
jtvthis is with henning's branch that just re-uses the debian builder13:10
wgrantHm, OK.13:10
=== mrevell is now known as mrevell-lunch
wgrantAnyway, ping me when you're giving up for the day if you can't get it working, and I'll have a poke around myself in a few hours.13:10
=== salgado is now known as salgado-afk
jtvwgrant: I'm being kicked out of here right now, in fact13:11
jtvwgrant: but you mentioned sleep.  :)13:12
wgrantYeah, I should. Uni is forcing me to sleep at reasonable hours for now.13:12
jtvgood night!13:13
jtvdamn those fascists13:13
NCommanderWhen building a test case, how do I properly catch something raised so the test passes?13:29
* NCommander kills his laptop13:40
allenapNCommander: Have a look at TestCase.assertRaises()13:41
NCommanderallenap: thanks, although I don't seem to be having much luck with it13:46
allenapNCommander: How are you invoking it?13:47
NCommanderallenap: the wrong way it seems :-)13:47
NCommander        self.FailUnlessRaises(UploadError, self.findCopyright, self.dsc_file,
                              self.tmpdir, mock_logger_quiet)13:47
NCommander(the raise error comes up in a function that findCopyright calls, but changing the arguments to that also fails)13:48
allenapNCommander: Can you paste against LP (I assume this is LP) so I can replicate?13:49
allenaps/paste/paste a diff/13:49
NCommanderallenap: its a set of changes with a database mod, not a small set :-/13:49
=== mrevell-lunch is now known as mrevell
allenapNCommander: I'm kinda here to help, so it's fine :) If you'd rather plug away at it, I don't mind either :)13:52
NCommanderallenap: let me post to launchpad13:53
NCommanderallenap: its based off db-devel versus devel, but given my internet connection ATM, this will take awhile13:56
NCommanderallenap: lp:~mcasadevall/launchpad/raw_source_changelog13:57
NCommanderI think I pushed everything properly w.r.t. to database changes13:57
allenapNCommander: Okay, I'm getting that now.13:58
=== jamalta-afk is now known as jamalta
allenapNCommander: Okay, I have a fix for you I think.14:11
NCommanderallenap: I assume you know this, but you can run the test code that breaks with ./bin/test -t lp.archiveuploader.tests.test_dscfile14:13
allenapNCommander: The new function findFile() that you've added raises UploadError, whereas errors are yielded elsewhere.14:13
allenapNCommander: I can get the test to pass with http://paste.ubuntu.com/393289/, but I think the callers of findFile need to change.14:14
=== Chex_ is now known as Chex
NCommanderallenap: well, I refactored the code this way so I don't have to duplicate code between findChangelog/findCopyright14:15
allenapNCommander: Yes. The way errors are handled - by yielding them - is interesting but a bit weird, so the refactoring in your branch doesn't quite work. I've nearly got a suggestion for you...14:17
allenapNCommander: http://paste.ubuntu.com/393291/14:19
allenapNCommander: Callers of findFile() must catch UploadError and yield it to their callers.14:20
NCommanderallenap: that's ugly, but :-/14:21
allenapNCommander: Yes it is :)14:22
NCommanderallenap: I'm kinda concerned that won't make it through a review :-/14:22
NCommanderallenap: *sigh* NameError: global name 'errors' is not defined14:25
NCommanderallenap: (on a second note, how can I land a change for one of the sourcedeps; I have a fix that gets LP going again on lucid)14:25
NCommanderallenap: wait, that a fail to save error :-)14:26
allenapNCommander: It's been a while since I did that... First, get an LP dev to add your sourcedep to lp:~launchpad/lp-source-dependencies/trunk, then propose merging an LP branch to update versions.cfg.14:27
allenapNCommander: In findFile(), why do you do os.path.exists()? Does globbing not demonstrate that the name exists?14:30
NCommanderallenap: that may just be a case of code being copied; I refactored old code into findFile, I didn't write it14:30
allenapNCommander: Ah, okay, it might be safest to leave it then.14:31
bigjoolsallenap, NCommander: errors are yielded instead of raised so that we can collect as many as possible to return to the uploader14:32
allenapbigjools: Yeah, the problem NCommander's facing is that it's difficult for helper functions to raise errors; the caller must add boilerplate to catch the error and yield it. It works, it's just ugly.14:33
NCommanderbigjools: ah, that explains it14:33
bigjoolsallenap: what's the context here14:34
NCommanderbigjools: I just refactored findChangelog to call a new findFiles14:35
NCommanderwhich is also called by findCopyright :-)14:35
NCommanderas I didn't want to duplicate code logic14:35
bigjoolswhere is findChangelog?14:36
allenapNCommander: Here's another approach: http://paste.ubuntu.com/393298/14:36
NCommanderbigjools: in code not yet merged ;-)14:36
bigjoolsah :)14:36
NCommanderallenap: I need to add another sanity check to findFile to prevent a possible DOS, so I like the former approach. Thanks for your help14:37
allenapNCommander: Okay, you're welcome :)14:38
NCommanderbigjools: this is the code that implement raw_source_changelog in soyuz (I'm not sure I grok zope UI yet, but at least the mind numbling part is in)14:38
bigjoolsNCommander: ah interesting14:39
* bigjools will need to review that ;)14:39
NCommanderbigjools: branch up on LP already14:39
NCommander lp:~mcasadevall/launchpad/raw_source_changelog14:39
bigjoolsNCommander: so are you fixing changesfile.py?14:39
NCommanderbigjools: not yet. That requires a proper changelog parser unfortunately14:40
bigjoolsNCommander: so you're just storing the raw changelog?14:40
NCommanderbigjools: ATM, yes. We ran it by stub this morning14:41
* bigjools looks at the branch instead of postulating14:41
* NCommander hears bigjools screaming for some reason14:43
bigjoolsNCommander: be careful with those yield error texts14:43
bigjoolsthey are copied verbatim into emails14:43
bigjoolsso "Failed to copy changelog file" isn't that useful14:43
NCommanderbigjools: that came from Failed to copy copyright file :-)14:43
bigjoolsthat sucks too :)14:44
bigjoolsopen(fullpath).read().strip() should probably be broken down into different stages and a separate error for each14:44
* NCommander is getting the feeling that since I'm now the most recent to touch this file, I get to fix and make it publish14:47
bigjoolsNCommander: if we didn't do drive-by fixes they'd never get done :/14:47
bigjoolsso we like to encourage it :)14:47
NCommanderbigjools: this doesn't seem like the right way to maintain critical infrastructure ...14:48
bigjoolswhy's that?14:48
persiaShhhh !14:48
* NCommander wonders who persia is shhhhing14:48
bigjoolsit seems a great way to me, but perhaps we're thinking about different things14:49
bigjoolsNCommander: anyway your change looks great from what I can see so far14:51
NCommanderbigjools: I'm probably going to run into issues when it gets to the point of bending the UI to my will14:52
bigjoolsNCommander: yeah probably, but we can help you!  Get this one done first though.14:52
NCommanderbigjools: I also have no clue how to write the migration tool when we get there14:54
bigjoolsNCommander: yeah that will be tough14:54
NCommanderbigjools: there's code in dak that can be recycled :-)14:55
bigjoolswe need something to parse the changelogs and poke the database14:55
bigjoolsit will be an interesting script14:55
* NCommander rewrote a good chunk of dak pre-Canonical days, including support to generate dists/* from the database14:55
bigjoolsI don't think it's too hard though14:55
bigjoolstough but not impossible :)14:56
NCommanderbigjools: its more just going one by one through sourcereleasepackage, finding its files, downloading the package from librarian, sucking out a changelog, and updating the database14:56
NCommanderIts going to be bloody time consuming14:56
bigjoolswell we only need to scan published sources14:56
NCommanderbigjools: we don't want everything?14:57
bigjoolsNCommander: what's the point?14:57
bigjoolsif there's a good reason it's fine of course14:57
NCommander^- persia - please provide a point if there is one14:58
* persia reads backscroll14:59
* bigjools going otp14:59
persiaHrm.  I think I have to defer to mvo on that.15:00
persiaThe point would depend on where/how the comparisons are happening for the update-manager display.15:00
persiaIf it's happening between two pulls from the server, we need everything.  If it's happening between the server and the client, we only need urrently published stuff.15:01
deryckadeuring, , is bug 505999 really a dupe?  It was marked fixed so I assume you passed the bug number to the commit message?15:01
mupBug #505999: Unsubscribe OOPS <oops> <qa-needstesting> <Launchpad Bugs:Fix Committed by adeuring> <https://launchpad.net/bugs/505999>15:01
adeuringderyck: yes, this really looks like a dupe.15:02
adeuringthe OOPS data is quite similar15:02
deryckadeuring, ok, thanks.15:02
deryckadeuring, did you QA the doNotSnapshot-related OOPS bugs?15:04
adeuringderyck: yes, I tried for example to (un)subscripbe from bug one15:04
deryckadeuring, can you update all the related bugs to change to the qa-ok tag then?15:05
adeuringderyck: sure15:05
deryckadeuring, sorry, many questions about this. :-)  Bug #518254 was fixed, too?15:06
adeuringderyck: probably.... but I need to check this15:08
deryckadeuring, ok, just claim it and mark it fixed if so.  if not, ping me to follow up about it.15:08
=== matsubara is now known as matsubara-lunch
leonardrflacoste, i'm having trouble iterating over a shell glob in the launchpad makefile. can you take a look?15:24
leonardrthat same code works in a shell script15:24
mcasadevallallenap: I have no luck with exceptions :-/15:25
allenapmcasadevall: Hehe :) Can I help?15:25
mcasadevall        errors = findFile(self.tmpdir, self.changelog_file, mock_logger_quiet)
        self.failUnlessRaises(UploadError, list, errors)                      
mcasadevallallenap: I'm trying to trigger an upload error in findFiles that I added for files being too big15:26
=== mcasadevall is now known as NCommander
allenapmcasadevall: Sorry, I have to go out for a minute; I'll be back in <10m.15:26
leonardrflacoste: i changed $(WADL_FILE_GLOB) to $(wildcard WADL_FILE_GLOB), but that's expanding to the empty string for some reason15:38
jelmerleonardr: you'd want $(wildcard $(WADL_FILE_GLOB)) probably?15:39
leonardrjelmer: madness!15:39
persiaNo, just make15:39
leonardrjelmer, thanks, that works15:39
allenapNCommander: Do you have a diff of your latest changes?15:49
NCommanderallenap: see PM15:53
=== matsubara-lunch is now known as matsubara
NCommanderbigjools: so, I think I'm at the point where I need to run the local test suite to make sure this all is working sanely16:12
NCommanderbigjools: how do I do that? :-)16:12
bigjoolsNCommander: best thing is to type:16:12
bigjoolsbin/test -cvv -m soyuz -m archiveuploader -m archivepublisher16:12
bigjoolsand go get coffee for 30m16:12
* NCommander lets it run16:13
bigjoolsoh -m buildmaster too16:13
NCommanderTest-module import failures:

Module: lp.soyuz.browser.tests.test_archive_admin_view
NCommanderuh oh16:13
NCommanderWhat did I break16:13
bigjoolshappens here too16:13
bigjoolsdon't worry16:13
bigjoolsjelmer was going to look into that16:14
jelmerI'll be mailing the list about it before EOD16:14
NCommanderbigjools: w.r.t. to findFile(), allenap is concerned thats not going to pass review. If thats the case, I rather rewrite it now. Care to weigh in? (or any other soyuz person)16:16
bigjoolswhat is the concern, exactly?16:17
NCommanderbigjools: it kinda handles errors funny16:17
NCommanderbigjools: it uses raise to raise an error, then catches them one level up and yeilds them so I don't have a lot of code duplication across changelog and copyright file findings stuff16:17
bigjoolsNCommander: I can't think of a better way of doing it16:18
allenapNCommander: I don't see the exception catching code in the branch...?16:19
allenapNCommander: If you catch UploadError in findCopyright() and findChangelog() and yield it then it's fine. The test we just worked on would not work though, and would need to be changed; that's why I assumed you weren't doing that.16:22
        copyright_file = findFile(source_dir, 'debian/copyright', logger);
    except UploadError, error:
        yield error
NCommanderthat came out a bit flatter than I expected16:23
NCommanderunless I'm misunderstanding the code patch you wrote16:23
allenapNCommander: I only got "try:" then "hrm" there; maybe you hit some flood protection.16:24
=== deryck is now known as deryck[lunch]
allenapNCommander: Cool :)16:34
leonardrgary: i'm going to create a .pt file for the apidoc index file. should that go into lib/canonical/launchpad/templates?16:37
gary_posterleonardr: mm.  I'd be tempted to put it closer to the machinery that you are building.  It is a not a template that the application should use itself, except in this rare case, yes?16:39
leonardrgary: right16:39
leonardrshould it go in utilities or utilities/templates?16:39
gary_posterleonardr: in favor of ./lib/canonical/launchpad/utilities16:40
leonardrok, cool16:40
leonardrbut, that's not really anywhere near create-lp-wadl.py, which is in /utilities16:41
gary_posterleonardr: oh16:41
gary_posterleonardr: I thought it would be over there.16:42
leonardrgary: i think it's in /utilities because it's only called by the makefile?16:42
gary_posterleonardr: right.  since this template needs to run within lp, it should be in the lp tree.  if there's not an obvious place for it, but it in the standard template location you mentioned first16:43
NCommanderzeca failed to start in the test suite :-/17:14
bigjoolsNCommander: yeah it's very problematic sometimes17:29
NCommanderbigjools: so five tests failed, I'm looking over the failures now. Is the full log automatically saved somewhere?17:31
bigjoolsNCommander: no :/17:32
* NCommander accidently cleared his screen17:32
bigjoolsNCommander: it sounds spurious to me17:32
NCommandernow I need to wait an hour for the other failures to repop up17:32
bigjoolsdo you remember the test names?>17:32
* bigjools keeps meaning to make bin/test and make check write a log17:32
=== deryck[lunch] is now known as deryck
NCommanderbigjools: nops :-/17:33
NCommanderI have one as a doc test, although I'm unsure how to rerun it individually17:33
=== gary_poster is now known as gary-lunch
* kfogel is away: switching consoles for a bit, back soon18:19
=== Ursinha is now known as Ursinha-lunch
=== leonardr is now known as leonardr-lunch
=== salgado-afk is now known as salgado
=== leonardr-lunch is now known as leonardr
kfogelwgrant: I was thinking, now that the community-contributions.py script is counting Canonical devs (who aren't on the launchpad team) too, maybe it no longer makes sense to have these hardcode start revs in the file?19:03
NCommanderStupid question but are there any know broken tests in soyuz, I have a test thats failing that doesn't seem be related to my changes, and it fails in devel and in my branch19:11
=== gary-lunch is now known as gary_poster
thumperNCommander: which tests are breaking for you?19:43
NCommanderthumper: FTPconfig19:43
NCommander   test_generateConfig (lp.archivepublisher.tests.test_ftparchive.TestFTPArchive)
thumperNCommander: I'll check mine on devel19:44
NCommanderthumper: it fails consistantly on my branch, and on stock devel19:44
NCommanderthumper: I'd also like to start the review process for my changeset into db-devel19:44
thumperNCommander: propose your branch for merging19:48
thumperNCommander: that's all you have to do to start the process19:48
thumperTestFTPArchive tests pass for me locally with up to date devel19:48
NCommanderthumper: are you onlucid?19:49
thumperNCommander: no, karmic still19:49
* NCommander is suspecting he's having some unique failures because of it19:49
NCommanderthumper: thats probably why19:49
NCommanderthumper: who should I put down for reviewer19:52
thumperNCommander: just leave it as launchpad19:52
thumperNCommander: it should have that already19:52
thumperNCommander: although are you wanting to land on devel or db-devel?19:53
NCommanderthumper: I'm just reading on the wiki about who to set for reviewer and cover letter19:53
NCommanderthumper: db-devel19:53
thumperNCommander: ok, lp:launchpad is the right branch then19:53
NCommanderthumper: I'm running make lint as recommended by the cover page instructiosn, and I'm getting a bunch of can't import messages20:18
NCommander    29: [F0401] Unable to import 'zope.component'20:18
thumperNCommander: have you linked the external sourcecode in your branch?20:18
NCommanderthumper: yes20:19
* thumper confesses to not running make lint in a while20:19
* NCommander has run make run and make schema without issues20:19
thumperNCommander: I'd say ignore those errors/warnings20:19
wgrantNCommander: generateConfig is the only test broken on lucid.20:22
wgrantkfogel: Hm, possibly.20:23
NCommanderwgrant: make lint seems broke too :-/20:23
wgrantNCommander: What does it do?20:23
NCommanderwgrant: runs pylint, the submission directions say your supposed to run it and attach the output to any merge cover letter20:24
wgrantNCommander: I mean, why is it broken?20:24
wgrantIt works for me.20:24
NCommanderwgrant:     17: [F0401] Unable to import 'zope.schema'
NCommanderwgrant: and other such messages20:25
wgrantThat's normal.20:25
wgrantIgnore external missing imports like that; they're in eggs.20:25
wgrantInternal imports should work fine, though.20:25
NCommandermerge proposed20:26
wgrantkfogel: I suspect we'll probably need to extend the lists, then, but let's just see what happens.20:26
* wgrant runs.20:26
=== fjlacoste is now known as flacoste
NCommanderthumper: so I take it now I just wait?20:30
thumperNCommander: yep or go into #lauchpad-reviews and bug the on call reviewer20:30
NCommanderthumper: don't DB changes need some special reviewing as well?20:31
wgrantRequest 'db' reviews from stub and BjornT.20:32
NCommanderwgrant: before or after the technical review?20:34
wgrantNCommander: I don't believe it matters.20:34
thumperNCommander: same time20:36
lifelessjml: morning20:44
jmllifeless, hi20:46
lifelessI'd love it if I could understand the progress-to-hard issue. I feel like I just don't get the crux of it20:49
jmlI don't want to write a thing that displays a progress bar while dumping subunit output20:50
kfogeljml: for changes to the community-contributions script (by wgrant) it's okay if I just review and land them, right?20:50
jmlkfogel, yes.20:50
lifelessjml: ok. So you're saying that it is possible, but you don't want to do it. I was saying that its pointless (you can get the bar from subunit in real time anyway).20:52
jmllifeless, it's also pointless20:52
lifelessjml: ok, so I think we're agreed at this level. The reason I suggested saying pointless in the docstring, was so that other people examining it won't think you're discarding an important feature.20:53
jmllifeless, ok.20:53
jml    # subunit output is designed for computers, so displaying a progress bar20:54
jml    # isn't helpful.20:54
jmlalso, we're on the wrong channel :)20:54
lifelessjml: well, its a launchpad patch.20:55
lifelessjml: so I think this is the right channel :)20:55
jelmerYou should crosspost.20:55
lifelesshah. Now perhaps finally thats a reason to install bip20:56
jmllifeless, uhh, no, it's a patch to zope.testing.20:57
lifelessoh, my bad20:57
lifelessI thought it was going to go into the lp tree for some reason.20:57
lifelessjml: should I join #zope? I think we're essentially done on the review, right?20:59
kfogelwgrant: one teeny little thing left for the community-contributions-fixes branch:20:59
wgrantkfogel: Sure.21:00
kfogelwgrant: the line "lr.merge_depth < ec.seen_revs[lr.rev.revision_id][0].merge_depth):" and the one below it go longer than 78 columns.  https://dev.launchpad.net/PythonStyleGuide#Whitespace%20and%20Wrapping21:00
jmllifeless, I think we're done, yeah21:00
jmllifeless, #zope3-dev is the appropriate channel AIUI. Also, you're CCd on the email to the zope mailing list.21:00
wgrantkfogel: Really? Argh.21:00
* wgrant fixes.21:00
kfogelwgrant: if you fix up, then we have a clean set of commits by you.21:00
kfogelwgrant: (I could fix it, but I hate to taint the history with my shameful name...)21:01
lifelessjml: sweet.21:01
wgrantkfogel: Pushing.21:02
kfogelwgrant: thanks.  I will land.21:02
wgrantkfogel: Thanks.21:02
jmlnow all I need to do is trick gary_poster or barry or another person w/ commit privs to land it for me :)21:03
lifelessjml: I'm a little nervous about the embedded protocol, because I kindof expect the chunked stuff to change - martin is working on a patch to use a ascii coding rather than 8bit21:03
lifelesswhen I say working on, I mean he wants it and I keep saying patches accepted.21:03
lifelessI think barry makes a good victim^Wassistance21:05
=== matsubara is now known as matsubara-afk
wgrantkfogel: The current version of /Draft was generated from r1 of both branches, with an extended name map.21:23
kfogelwgrant: oh, you filled in the name map with those earlier devs?  you rock again21:24
* kfogel visits21:24
kfogelwgrant: looks good to me21:25
kfogelwgrant: are all your changes pushed to the lp branch now?21:25
wgrantkfogel: No, I haven't committed these yet.21:25
wgrantShould I remove the start_revno support?21:25
kfogelwgrant: I think it's kind of pointless now.  Do you concur?21:26
wgrantIndeed. If we end up needing it, we have a VCS...21:26
* wgrant removes.21:26
wgrantkfogel: r10469 pushing.21:32
kfogelwgrant: thank you21:32
lifelesssidnei: you are da man21:33
lifelessbah echannel21:33
kfogelwgrant: got your changes, looking now; expect to land today.22:27
wgrantkfogel: Great.22:28
lifelessoh jml - https://code.edge.launchpad.net/~lifeless/lptools/lp-project/+merge/21124 - may interest you; I'm going to do a further variation for bzr plugins.22:29
* kfogel is away: machine fan cleanup; back later22:29
=== salgado is now known as salgado-afk
=== jamalta is now known as jamalta-afk
wgrantI have twisted.web.xmlrpc.Proxy.callRemote hanging when one of its arguments is a dict containing a None value.23:20
shen-longok ... wow ...23:50
shen-longtrying to https://dev.launchpad.net/Running/RemoteAccess23:50
shen-longgetting a 503 error23:50
shen-longfollowed directions to a T, not using two ip's though, just one public23:51
shen-longthis is on a fresh ubuntu 9.1023:51
shen-longImportError: No module named _pythonpath in apache error log is the only thing I can see23:56
shen-longfrom scripts/branch-rewrite.py23:57
shen-longeverything else loops fine23:57
shen-longwrong version of python23:58
shen-longthanks for talking me through it #launchpad-dev ;p23:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!