[00:47] is anyone looking at Person:+commentedbugs ? [00:54] hi wallyworld_, sorry for the lag, do you still want to talk? [00:54] poolie: ok [01:27] * thumper stabs TAL in the face [01:42] * thumper is getting confused by PackageBuildDerived [01:42] wgrant: ping [01:42] or StevenK ping [01:51] thumper: Hi. [01:51] wgrant: nm [01:51] I'm walking through build farm job derived [01:52] gee, could it be more convoluted? [01:53] It is a little awful, yes. [01:53] thanks for the QA thumper, i was looking for it to land [01:53] np [01:54] thumper: I've been meaning to excise the obsolete bits, but I should probably study instead. [01:57] thumper: In related news, trying to fit inheritance into an RDBMS sucks. [01:57] wgrant: that it does [01:59] thumper, what are you looking at in the build farm jobs? [02:00] Pain and suffering, I imagine. [02:03] wgrant, that's not what you're looking for usually, but it's what you get. [02:13] rockstar: yes [02:19] * thumper stares with wonder at the steps he needs to build a test page [02:22] so... a source package release is one of: a recipe build; or an upload? [02:23] wgrant: ? [02:23] well, "is the result of" surely? [02:23] maybe not [02:23] * mwhudson lets wgrant answer [02:31] mwhudson: yeah, that's what I ment [02:31] the result of a successful recipe build [02:31] I was a little terse [02:32] Oh man, I can totally do the overlay reflowing if we trash our overlay and use Y.Overlay. [02:32] gah [02:34] stupid factory [02:34] rockstar: how about morphing dialogs? [02:34] thumper, that's what I meant. [02:34] rockstar: DO IT! [02:34] thumper, it's freakin' easy, assuming that the [beta] tag doesn't mean it's going to change much. [02:35] thumper, doing it now. :) [02:37] I'm also trying to listen in on a panel entitled "The Future of Frontend Engineering." [02:37] stub: hi ? [02:39] lifeless: yo [02:40] I've been meaning to grab some time to talk about db evolution [02:40] are the technologies around that can do modest online DDL ? [02:40] things like add a table, drop a column ? [02:41] I'm wondering about an approach where we do small steps and use things like appserver code / triggers to update an old-format table and new-format table insync, [02:41] thumper: Right, a SourcePackageRelease is the result of an upload, whether it be by FTP or recipe. [02:41] use the garbo to migrate untouched rows across [02:41] yep [02:42] and once we're fully synced stop writing to the old table, break all its references and discard it [02:42] lifeless: Surely we are the not the first people to encounter this problem. [02:43] wgrant: yes, but we're using slony [02:43] wgrant: single db server is trivial; sharded is trivial. [02:43] lifeless: There is a commercial product that claims to do modest online DDL, but I doubt it would work with replication. [02:44] wgrant: the drizzle folk claim nbd can do what we need, but I haven't dug in in detail - and thats not replication either. [02:44] lifeless: Ah, fair point. [02:45] lifeless: For simple DDL (add a table, add a new NULLable column, drop a column), we are much better off when we are running Slony 2.x or maybe some other replication. The stuff in PG 9.0 may well be suitable for us soon too (once we ween off using replication to ship data to ISD) [02:46] lifeless: So we are stuck where we are until we upgrade or switch replication. [02:46] ok [02:46] stub: with RFWTAD roughly done, this is now next up as TA work in kanban [02:46] TA? [02:47] technical architecture [02:47] I've given myself two slots in kanban; currently performance + continuous deployment [02:49] Does test suite runs fall under that, or is that seperate? [02:49] StevenK: I don't think it does [02:49] StevenK: the kanban limits don't stop me doing spikes and helping folk [02:50] StevenK: but they are where I'm focusing the vast bulk of my effort [02:50] lifeless: Well, sure, but you were doing some work with parallel tests, and then you just ... stopped :-) [02:50] StevenK: I did a spike [02:50] Ah [02:50] StevenK: so did Ian [02:51] stub: could you perhaps have a look at why https://bugs.edge.launchpad.net/malone/+bug/668138 is slow? I think its a 8.4 thing, and it would be nice to be able to tell what the issue is [02:51] <_mup_> Bug #668138: Person:+commentedbugs timeouts [02:56] * thumper stabs soyuz repeatedly [02:57] probably not entirely fair [02:57] but it is the closest thing [02:57] where is the code that schedules recipe builds? [02:57] thumper: 'schedules' meaning 'creates'? [02:57] wgrant: yeah [02:57] I'm messing around with the factory [02:58] so I can test my page [02:58] I created a source package recipe build [02:58] LOF, or the SPRB factory? [02:58] but it has no buildqueue_record [02:58] so no eta [02:58] what creates the buildqueue_record? [02:58] Try calling sprb.queueBuild() [02:59] (this whole thing is scheduled for demolition) [03:07] * thumper sighs [03:07] Hmmmmm? [03:09] I can't seem to get it to say anything other than "No suitable builders" [03:09] ahh LBYL, gotta love it [03:09] you'll need an i386 capable builder [03:11] i386 virt, no less. [03:11] (this is nothing to do with Soyuz; that message is purely a Code invention) [03:12] thumper, the tests for source package recipes should be a good template for how to get all this set up (and there's a lot of shit to set up) [03:12] rockstar: yeah, I'm going through one, but it isn't helping [03:12] I'm calling factory.makeBuilder() [03:18] thumper, do the distros/distroserieses match? [03:18] rockstar: eh? [03:19] thumper, I'm pulling from memory here, but I remember some issues where the builder wasn't set up to build against the distro and/or distroseries the recipe build was for. [03:19] rockstar: right now it is the processor doesn't match [03:20] thumper, ah yes. [03:20] the buildqueue processor it is 10 [03:20] if I call makeBuilder I get a processor id 1 [03:22] the buildqueue that was created has a factory generated processor family [03:22] that's why [03:22] :( [03:23] thumper, yeah, was just about to say that. [03:23] * thumper wants to smack this around [03:23] thumper, I have some model tests that set up what you want. [03:23] now the question is what do I need to change? [03:23] rockstar: can you point me at one plz? [03:23] thumper, one sec. [03:24] thumper: Create a builder for the processor in that family. [03:24] Or unfuck the factory. [03:24] wgrant: I'd rather make the buildqueue work with the default builder [03:25] I want to know what is making factory processors [03:26] thumper: Probably whatever is constructing a distroseries. [03:26] * thumper thinks he has it [03:27] thumper, lib/lp/code/model/tests/test_sourcepackagerecipebuild.py line 62 [03:27] thumper: Recipes should use the distroseries' nominatedarchindep distroarchseries' processorfamily's processor. [03:28] * thumper need to unfuck the factory [03:29] thumper, that's a hell of a spike. Fair warning. [04:08] Project devel build (202): STILL FAILING in 3 hr 43 min: https://hudson.wedontsleep.org/job/devel/202/ [04:09] \o/ [04:09] finally have a page showing pending build [05:05] * thumper EODs === _mup__ is now known as _mup_ === almaisan-away is now known as al-maisan [07:27] wgrant: around ? [07:28] my improvement to https://bugs.launchpad.net/soyuz/+bug/662523 is on qastaging; I'm trying to find the page to use to reproduce ;) [07:28] <_mup_> Bug #662523: Archive:EntryResource:getBuildSummariesForSourceIds times out [07:28] lifeless: Archive:+packages might use something like that method. [07:28] Otherwise there's the API. [07:29] yeah [07:29] Yippie, build fixed! [07:29] Project devel build (203): FIXED in 3 hr 20 min: https://hudson.wedontsleep.org/job/devel/203/ [07:29] Launchpad Patch Queue Manager: [r=lifeless][ui=none][bug=661441] Do checkwatches custom logging to [07:29] info instead of error to avoid duplicate OOPS reports. [07:47] wgrant: do you have an api script thats vaguely relevant here, handy? [07:47] lifeless: No. fta has one which uses it a lot. [08:36] 'morning [09:18] Well hello [11:03] Why can't we "make" in db-stable? It complains about a missing module, difftacular. [11:03] Seems to be a bzr plugin that I don't have. === matsubara-afk is now known as matsubara [11:04] Same for db-devel but not for devel. [11:05] http://paste.ubuntu.com/528623/ [11:05] jtv, henninge: utilities/update-sourcecode ? [11:06] bigjools: dunno about that but I did rocketfuel-get [11:06] but let me try [11:06] What did I miss? [11:06] I sometimes need to do it separately [11:07] jtv: bigjools suggested utilities/update-sourcecode [11:07] But rocketfuel-get does that too, doesn't it? [11:07] (Which naturally I ran) [11:07] yes, what I said [11:07] bigjools: lots of "no change" [11:07] :( [11:08] I suspect that db-devel has got a dependency change that is not in devel [11:08] I believe we were once supposed to have special sources.list entries for a bzr PPA, but those became unnecessary. [11:09] I'll see if I can re-create here [11:10] bigjools: this is no time to play around with the missus, we need your help! [11:11] it. takes. too. damn. long. to. run. make. [11:12] Hey, I just took a minute off "make schema." If everyone tried to do the same… [11:14] really? [11:15] jtv: make build now worked for me in db-devel [11:15] It may really be related to udpate-sourcode ... strange [11:15] trying recife now [11:16] ok re-created, now I am trying something [11:18] henninge: I got an update to dulwich, but that must have come in minutes ago because I've been running rocketfuel-get [11:18] so [11:18] the update-sourcecode in db-devel has a change that is not in devel [11:18] "Getting difftacular from lp:difftacular at revision 6" [11:18] we've had this issue before [11:18] Grr [11:20] But why don't I see anything about it in the output of update-source code. [11:21] Hmm? [11:21] difftacular is a new dep in db-devel's sourcedeps.conf. [11:22] and I guess rocketfuel-get uses the one from devel, right? [11:22] Yes, I guess on the assumption that dependencies would appear there first. [11:22] yes [11:23] Ah great: no module named _gpgme [11:23] that's progress, right? :) [11:23] jtv: recife now builds for me. [11:23] bigjools, wgrant: thanks [11:24] I think I probably tried to do too much before all the downloads were finished. [11:24] thanks bigjools! [11:25] :) [11:29] matsubara: Do you think you could document that recipe build of bzr-pqm into the launchpad PPA you've got on dev.lp.net/LaunchpadPpa, fix it to build for maverick too, and maybe set its ownership to ~launchpad-committers? [11:37] maxb, what kind of documentation are you looking for? [11:38] I guess a link to the recipe would be fine? Just enough for someone to figure out where the package came from and why it is there [11:38] Ideally enough to ascertain when, at some point in the future, it is no longer necessary to maintain a separate package [11:44] maxb, will do [12:06] Morning, all. === Ursinha-afk is now known as Ursinha === mrevell is now known as mrevell-lunch === mrevell-lunch is now known as mrevell [14:34] mars, ping [14:36] Hi deryck, in the team standup, will ping back in a few minutes [14:37] mars, ok, thanks [14:45] jml: Mumble? [14:52] difftacular? Did we grow a new dependency without buildout or lp packages love? [14:53] ahh... sorted. Needed to relink sourcecode/ [14:53] stub: you need to use update-sourcecode in db-devel [14:55] Hi deryck, what's up? [15:02] mars, hi, now I'm on call. Will ping shortly. [15:02] deryck, heh, ok :) [15:11] mars, off now [15:11] mars, so windmill is driving me insane ;) [15:11] ah, I'm sorry to hear that [15:12] I'm trying to get rockstar's branch to update to latest lazr-js landed.... [15:12] this is also the yui 3.2 upgrade [15:12] but it's blocking our subscription story work and has been for a few weeks now [15:12] flacoste: ping [15:12] so I need to get it done [15:12] ok [15:13] deryck, does the lazr-js work depend on the yui 3.2 work? [15:13] as in, they need to be upgraded at the same time? [15:13] flacoste: sudo apt-get install gobby-0.5 [15:14] mars, yes. the branch updates lp to lazr-js rev 188, which is yui 3.2 plus other fixes. [15:14] mars, I have a high degree of confidence in the upgrade it self -- windmill tests pass individually and playing with the dev site locally looks good.... [15:14] mars, but the tests will not pass in ec2 [15:18] deryck, ok, do you have some sample failures [15:18] ? [15:19] mars, yes, let me forward you the two recent logs in email [15:19] deryck, I'd like to see if the failures are new, or similar to existing ones [15:19] timing, or DOM [15:25] mars, they appear to be similar to issues before where the tests collectively suddenly start to fail. I'm hesistant to call that timing, but it smells the same. [15:25] mars, email sent with logs and a bit more info. [15:25] jml: https://bugs.edge.launchpad.net/launchpad-code/+bug/634466 [15:25] <_mup_> Bug #634466: bzr branch lp:subvertpy fails as it gets a private branch url. [15:25] deryck, got it, give me a bit to browse them === matsubara is now known as matsubara-lunch [15:27] mars, sure. thanks for the help! [15:30] deryck, ec2 lag between branches - weird. I fixed the limit to 8 hours during UDS, so that should have been fixed (I hope!) [15:30] mars, ah, well that would explain it then. The js branch is just taking longer to complete. Not sure why. [15:32] deryck, ugh, well, the js-refresh run took 7.6 hours, eh? [15:33] "Total duration of test run 27166.2 seconds" [15:37] yeah [15:37] and the older- run, someone killed it? [15:37] KeyboardInterrupt [15:37] mars, I didn't. It seems to have died on it's own. [15:37] hmmm [15:37] I didn't notice that [15:37] how would I have keyboard interrupted it? [15:37] * mars shrugs [15:37] well, looking at the test run time [15:38] 12357.3 seconds. [15:38] 3.4 hours [15:38] deryck, so the older run, and the newer run - what is the difference between them exactly? [15:38] because there is a big difference in the suite run times [15:38] thanks to subunit, I can probably narrow that down [15:39] mars, nothing that I know of. I made no changes. Same revno of my branch. the first failed mysteriously, so I immediately fired off the second run. [15:39] mars, up until this, I was getting long runs but pretty clear errors that I could progressively chip away at. real failures. After fixing the last of the real failures, then this mystery. [15:40] deryck, ah, so the keyboard interrupt - that was a headless ec2 run you fired off? And I presume that you didn't ssh into the server and kill it... [15:41] mars, right. Just a normal ec2 land run. I did nothing differently that I normally do. Did not ssh in. [15:41] weird [15:42] deryck, did you receive summary mails for these runs? (sorry if you already told me, and I'm forgetting) [15:43] mars, yes [15:43] mars, for both. I just sent you the logs from the mails rather than forwarding the mails. Would you like the mails? [15:44] deryck, please [15:45] mars, sent. and I just noticed something.... [15:45] ? [15:46] mars, in the older KeyboardInterupt one, the error up further is Firefox IO error. I get this locally everytime when running ./bin/test -cvv --layer=WindmillLayer [15:46] I just quit running the tests this way, thinking I was doing it wrong. [15:46] deryck, which line, which file? [15:46] ah, the older one [15:47] ack [15:47] ok [15:47] just read backwards in the older one [15:47] WARNING: A test appears to be hung. There has been no output for 600 seconds. [15:51] deryck, ok, in the older file, which line is the Firefox IO error? I am looking at line 39084, which is the last output before the hang [15:52] mars, line 39216 [15:54] ah, ok, I get it. So line 39084 is the last line of normal output. Everything from there between there to line 39270 was caught in the output buffer. Now I get it. [15:59] deryck, ok, so I am going to focus on the second run, since it at least completed [15:59] it may be worth comparing the execution times of both [15:59] mars, ok [15:59] to reach the same point [16:00] bigjools, ping [16:01] sinzui: yo [16:01] bigjools, do we have a function that can compare to debversions and state if one version is newer than the other? [16:02] sinzui: of course [16:02] just digging up what we use [16:03] sinzui: apt_pkg.VersionCompare() [16:03] bigjools, I think I neglected to mention that I am actually comparing a spr version to a product release version. I expect to munge something to make this work [16:03] thanks! [16:03] see lib/lp/archiveuploader/nascentupload.py:_checkVersion() [16:03] sinzui: /o\ [16:03] good luck [16:04] I think I will change scope if this is more than two hours of work [16:08] sinzui, it should be simple with python-debian [16:08] from debian.changelog import Version [16:09] product_version = Version( + "-1"); package_version = Version() [16:10] print Version(product_version.upstream_version) > Version(package_version.upstream_version) [16:10] james_w, thank you very much. [16:10] it looks odd, but I think that's the behaviour that you want [16:11] I want to tell packagers looking at a source package page that upstream has a new version, or if Ubuntu is ahead of upstream (presumably Lp is missing the a registered upstream release) [16:13] bigjools: mumble? [16:13] flacoste: yep === beuno is now known as beuno-lunch === Ursinha is now known as Ursinha-doctor [16:36] * jelmer_ wonders why jml is naming branches after Greek islands === matsubara-lunch is now known as matsubara [16:37] jelmer_: The Book of Revelation (aka "The Apocalypse") was written on the island of Patmos. [16:40] deryck, ping, I might have a test you can try [16:41] mars, ok, cool. What's up? === al-maisan is now known as almaisan-away [16:41] deryck, hmm - I noticed the YUI unittest runs were failing [16:41] deryck, lib/lp/testing/__init__.py:741] [16:42] deryck, the waits.forElement on that line has no time limit - is it an infinite wait? [16:42] mars, which test? [16:43] deryck, three of them I think - lp.registry.windmill.tests.test_yuitests.RegistryYUIUnitTestCase.runTest [16:43] lp.registry.windmill.tests.test_yuitests.RegistryYUIUnitTestCase.runTest [16:43] lp.code.windmill.tests.test_yuitests.CodeYUIUnitTestCase.runTest [16:43] lp.bugs.windmill.tests.test_yuitests.BugsYUIUnitTestCase.runTest [16:44] mars, ok, I see what you mean now. I have no idea why there's isn't a time there. [16:44] I didn't know that was possible actually. [16:45] so that is the immediate cause of three of the failures. deryck, those YUIUnitTestCase classes pass locally? [16:46] mars, yes, they do. with ./bin/test -cvv -m bugs -t test_yuitests (for example) [16:46] deryck, and have you considered running just these testing in ec2? "ec2 test -o '-vv --layer=FooWindmillLayer'" [16:46] ok [16:46] mars, no, hadn't thought of that. I can try that. [16:47] mars, you can have an indefinite wait in forElement. I did not know that and I certainly hope I did not write those lines of code [16:47] deryck, run it with --post-mortem. If they fail, you can ssh in and run them with 'bin/test -D' [16:47] sinzui, well, deryck's tests are failing, so the wait does not appear to be inifinite [16:48] (inifinite?) [16:48] mars, ok, I'll try some additional runs with these suggestions and see what I turn up. [16:49] mars, thanks for your help! [16:49] that would tease out the problem with the YUI parts [16:49] looking at the others [16:49] deryck, np [16:51] deryck, also, both tests take roughly the same time to execute: up to 15 minutes [16:51] errm [16:52] deryck, rather, one took 2hours 55 minutes, the other took 3 hours 11 minutes. That implies that they would have both run up to a full seven hour run. Too long. [16:52] mars, by "one" and "other" do you mean just the yuitests took that long? [16:52] deryck, sorry, the two subunit streams you gave me [16:53] ah, right [16:53] here, I have a run from Nov. 3rd, I'll compare to that [16:54] 12591 seconds - so 3.5 hours for the full run [16:56] 2 hours and 15 minutes to finish the test_me_too bugs windmill test [16:57] deryck, so your branch has taken an average of 48 minutes more than mine to execute up to the test_me_too windmill test (assuming execution order is the same) [16:57] ok, that's interesting [16:58] I wonder why? [17:01] deryck, well, your and my test runs took 2:41 and 2:47 respectively to reach the first windmill test [17:02] deryck, could be networking - if your branch is choking on network timeouts or waits.forWhatever(), then that would take the time right out [17:02] 'could not resolve host' takes a while [17:03] but that should not be happening... hmm [17:04] deryck, your test run also has a lot of thread trash - I fixed that exact bug before, where the threads were from the testrunner trying to erroneously access an external web service provider [17:05] sorry, the windmill test proxy server was trying to do the access on behalf of the page [17:05] those connections timed out, and left thread trash [17:08] deryck, hi, back? [17:09] hi mars, yes sorry. Connection issues. [17:09] deryck, how much of the last bit did you see? [17:09] mars, I saw mars> deryck, well, your and my test runs took 2:41 and 2:47 respectively to reach the first windmill test [17:09] and that's it [17:10] deryck, ok, so I say your test run had a lot of thread trash - I fixed that exact error with the tests in the past [17:10] ok [17:10] deryck, IIRC it means the page is trying to access web resources outside of the test server [17:10] hmmm === beuno-lunch is now known as beuno === benji is now known as benji-lunch [17:17] mars, ok, I'm poking at this a bit more now, trying to work it out. [17:19] deryck, I am going to run a test with ec2, see if I can track down the off-site connections. [17:19] (if that is what is happening) [17:19] mars, ok, thanks! [17:22] mars, would the hard coded ports cause any issue here? (thinking of wallyworld's email recently about not having this anymore to get ready for parallel runs) [17:23] well, couple weeks ago recently :-) [17:23] deryck, when was this branch last updated? [17:23] the ports should be transparent [17:23] mars, merged devel daily, even yesterday. but the ports are still there in the tests. [17:24] oh? [17:24] mars, yeah. but they're also there in devel, too. [17:24] was wallyworld's work in preparation for removing ports, or did it actually start to remove them? [17:25] mars, I don't see the ports in db-devel. So maybe this is completely inrequired for devel at this point. [17:26] deryck, ok, so you are saying the windmill test harness has drifted between devel and db-devel? [17:26] jml: would you have a moment to help me diagnose that latest problem in the buildd-manager please? [17:27] mars, I don't know what I'm saying :-) I'm asking. :-) I am saying devel and db-devel are not the same for me. db-devel does not have hard coded ports, devel does. [17:27] mars, I don't know where wallyworld did his work. [17:27] checking the list for that [17:28] bigjools: I have half an hour. [17:28] jml: oddly, so do I [17:28] jml: thanks. The traceback is in here: https://bugs.launchpad.net/soyuz/+bug/671242 [17:28] <_mup_> Bug #671242: New buildd-manager disabling everything in sight [17:28] f'n wrapping [17:28] yes [17:29] someone should fix that :) [17:29] mars, it was merged to db-devel. so shouldn't be merged to devel at this point. [17:30] deryck, cool, so that should not factor into this [17:30] mars, right. Sorry, that was just noise. Just trying to cover all bases. [17:30] bigjools: http://paste.ubuntu.com/528801/ [17:30] jml: it's failing inside rescueBuilderIfLost() which is bascically the first thing that scan() does [17:31] deryck, no problem, you just reduced the problem space. That's a good thing :) [17:31] cool :-) [17:31] jml: at some point the builder object goes wonky as wgrant seems to suggest [17:32] the behavioural code that it's using is not consistent with the fact that it's not got a current job [17:32] deryck, I need to take lunch, should be back in about an hour [17:32] mars, np. thanks, again! [17:32] bigjools: hmm. [17:33] jml: given that this is the very first piece of code that's run after it re-fetches the builder object in scan(), I am stumped. [17:33] bigjools: is that a given? [17:33] bigjools: I'm still paging in the cdoe === Ursinha-doctor is now known as Ursinha [17:44] bigjools: buildqueue = getUtility(IBuildQueueSet).getByJob(self.job) [17:48] bigjools, hi :) do you know what are all those PoolFileOverwriteError oopses about? [17:48] Ursinha: not really :/ I need to investigate but there's about 100 other critical things to fix first :( [17:48] bigjools, :/ [17:48] I see [17:48] do you want me to file a bug about it? [17:51] Ursinha: there might be one already, but yes please [17:53] bigjools, I couldn't find one, I'll try a bit more and file one if not succeed [17:53] thank you [17:56] good night everyone [18:06] I should be testing bug 666496 on qastaging right? [18:06] <_mup_> Bug #666496: findSimilarBugs() returns bug task you are using