[03:37] <micahg> wgrant: ajax seems broke on bug 741528, is there anything I can do (file a bug, cry)?
[03:39] <wgrant> micahg: Hmm, it all seems to work except for the task table.
[03:40] <nigelb> Has LP gotten past the sacrifice goat stage? :p
[03:40] <micahg> oh, yeah, sorry, should've been more specific :)
[03:41] <spm> nigelb: so to speak. tho we're using black cockerels. does that count?
[03:42] <nigelb> spm: definite improvement :D
[03:43] <micahg> wgrant: it's the task table I need to work tonight :)
[03:44] <wgrant> micahg: I'm trying to work out what's up. Can you use the non-AJAX instead?
[03:44] <micahg> wgrant: yeah, I am, just taking a lot longer :)
[03:44] <wgrant> Yeah.
[03:47] <wgrant> Oh.
[03:47] <wgrant>           tal:condition="not:view/many_bugtasks"
[03:48] <wgrant> lib/lp/bugs/browser/bugtask.py:        self.many_bugtasks = len(self.bugtasks) >= 10
[03:48] <wgrant> That is slightly suboptimal.
[03:50] <micahg> yeah, especially with 5 releases, you're limited to 2 source packages or less across all, and that's when you need it most
[03:51] <wgrant> Yup.
[03:51] <wgrant> I guess that means someone found a performance issue with lots of tasks.
[03:52] <micahg> my guess is originally the ajax code was per widget
[03:52] <wgrant> It still sort of is.
[03:52] <lifeless> what does annotate say is up ?
[03:52] <micahg> ah, that would explain it, the actions should be factored out and just be a function call w/parameters if they're not already
[03:54] <wgrant> I think we should probably migrate to the new inline widget stuff that thumper did.
[03:55] <micahg> wgrant: should I file a bug?
[03:56] <wgrant> Bug #430288
[03:56] <wgrant> That's why it was disabled.
[03:56] <mwhudson> wgrant: i think it was something even worse, like n**2 behaviour in number of tasks or something
[03:56] <mwhudson> it was definitely a cop-out
[03:57] <wgrant> Let's see if it's still bad.
[03:58] <lifeless> wgrant: btw there is a bug for the massive javascript redundancy on that page
[03:58] <wgrant> lifeless: Everything has massive JavaScript redundancy.
[03:59] <lifeless> wgrant: yes, just saying ;)
[04:05] <wgrant> Hmm.
[04:06] <wgrant> It's not quite as bad as it used to be.
[04:06] <wgrant> Chromium does 100 bugtasks with full AJAX in 3s.
[04:06] <wgrant> Still bad.
[04:06] <wgrant> But not terrible.
[04:09] <micahg> wgrant: what's FF4 time onIt was discovered that several invalid HTTPS certificates were issued and
[04:09] <micahg> revoked. These were placed on the certificate blacklist to prevent their
[04:09] <micahg> misuse.? that
[04:09] <micahg> oops
[04:09]  * micahg shakes fist at xchat
[04:10] <wgrant> Heh.
[04:10] <wgrant> Let's see.
[04:11] <micahg> wgrant: also, which version of chromium?
[04:11] <wgrant> micahg: 11.something
[04:11] <micahg> ok, so that'll be out next month :)
[04:11] <wgrant> So the dev build.
[04:11] <wgrant> Yeah.
[04:12] <wgrant> Ah, Firebug is out now.
[04:12]  * wgrant grabs.
[04:12] <wgrant> Ah, now we see the problem.
[04:12] <wgrant> Firefox just hangs for a few seconds.
[04:13] <micahg> wgrant: AJAX related?
[04:13] <lifeless> javascript
[04:13] <wgrant> Yeah.
[04:13] <wgrant> Well, yeah, JS.
[04:13] <wgrant> Our JS is crap.
[04:13] <lifeless> micahg: the way that the javascript is created and linked to the rows is extremely expensive
[04:14] <micahg> wgrant: if you have a good test case, we can try to get that fixed, performance is a big focus for them
[04:14] <wgrant> micahg: It's really bad JS.
[04:14] <micahg> lifeless: yeah, but if chromium can do it, firefox should be able to as well :)
[04:14] <lifeless> micahg: when ff gets v8
[04:15] <lifeless> micahg: anyhow, its bad for chromium too
[04:15] <micahg> lifeless: part of v8 is in FF4
[04:15] <lifeless> micahg: this should be something lazy like: bind all the buttons to a single callback; when someone mouses over a button that callback can hook up the real code, and we only have one copy of the real code.
[04:16] <wgrant> 2.8/3s in addPickerPatcher
[04:16] <micahg> lifeless: exactly :)
[04:16] <lifeless> micahg: but its not
[04:16]  * micahg was trying to suggest something like that above
[04:16] <lifeless> micahg: its js emitted as literals inline with each task
[04:16] <nigelb> lifeless: that is the recommended way to do JS
[04:16] <lifeless> nigelb: of course it is
[04:17] <nigelb> lifeless: (but everyone ignores it for code that works)
[04:17] <lifeless> nigelb: well, I filed a bug on it
[04:17] <wgrant> lifeless: Mmmm, not exactly.
[04:17] <wgrant> lifeless: The slow code is not reproduced.
[04:17] <lifeless> nigelb: but we have a queue of problems to fix :> - patches appreciated
[04:17] <wgrant> lifeless: Each task calls setup_bugtask_row
[04:17] <lifeless> wgrant: well, thats a small mercy
[04:17] <wgrant> Which is slow.
[04:17] <nigelb> If I get lp on my system working like ever, I'll help.
[04:17] <nigelb> Right now, I fail at getting it working :(
[04:18] <lifeless> wgrant: nigelb thats pretty straight forward if you're interested - I run it in a VM
[04:18] <lifeless> bah
[04:18] <lifeless> nigelb: ^
[04:18] <micahg> lifeless: I'd like to discuss this with you in launchpad-dev in an hour or so if you're still around
[04:18] <lifeless> micahg: sure
[04:18] <nigelb> lifeless: I might grab you for help sometime then
[04:18]  * nigelb back2work
[04:38] <ScottK> lifeless: The package accept page feels snappier now too.  Thanks.
[04:39] <wgrant> ScottK: And not just because it times out more quickly? :)
[04:40] <ScottK> No.  It's actually working so far (we just hit Beta 1 freeze and I did my first pass through the queue)
[04:40] <lifeless> ScottK: \o/
[04:40]  * lifeless pokes tongue out at the doubters
[04:40] <wgrant> Excellent.
[04:42] <ScottK> Let's not get too excited too fast.  If I get through Beta 1 with no timeouts, then I'll start to feel good.
[04:43] <lifeless> no promises :)
[06:04] <micahg> wgrant: OOPS-1910F577, delete packages timeout, I'm updating bug 590523
[06:04] <wgrant> micahg: ppa:ubuntu-security/ppa?
[06:04] <micahg> wgrant: yeah :)
[06:04] <wgrant> I guess that has a bit of history.
[06:04] <wgrant> And lots of builds.
[06:05] <micahg> yep, should I file a new bug since it's a P3A?
[06:05] <wgrant> No.
[06:05] <micahg> wgrant: update existing?
[06:07] <micahg> updated
[06:07] <wgrant> micahg: Yeah, the OOPS ID is useful. Thanks.
[06:08] <micahg> and another pg83 bug is made relevant :)
[07:52] <geser> lifeless: re the FTBFS report: do you have any ideas which parts is causing the trouble and how to improve it? (I guess gettig bug 558907 fixed would be helpful)
[07:53] <lifeless> geser: its timing out because the collection iteration in APIs depends on cheap indexing
[07:53] <lifeless> e.g. foo[1000] depends on determining item 1000 cheaply
[07:53] <lifeless> but its not cheap
[07:53] <lifeless> if you process incrementally rather than reading all FTBFS for a series, there will be no problem
[07:54] <lifeless> and probably be faster too as you'll be doing less cross-network work
[07:56] <geser> what do you mean with "process incrementally"?
[08:07] <geser> lifeless: what do you mean with "process incrementally"?
[08:08] <wgrant> geser: Keep a local cache of builds.
[08:14] <geser> wgrant: I still don't understand how to improve the script. Would doing a list() on the results of getBuildRecords and then iterate over that list help? Or am I looking at the wrong part of the script?
[08:14] <geser> the script tries to fetch everything only once to avoid the delays from each http request
[08:15] <wgrant> geser: So, the idea is to avoid retrieving all the data every time, keeping a persistent local store of the failed builds so we don't have to request the whole lot every time the script runs.
[08:15] <geser> ah
[08:18] <geser> I thought about that but could think of a way that it would work. The main problem I see is how to get the new failed builds on the next run?
[08:18] <geser> s/could/couldn't/
[08:18] <lifeless> geser: ask each package that was a fail, if it has a newer publication
[08:19] <lifeless> geser: if it does, its been fixed.
[08:19] <geser> and new uploads that didn't fail previously?
[08:19] <lifeless> walk the failure list - its in order - until you see a date that you saw on the previous run
[08:20] <lifeless> unless your script is shutdown for a month, that will be fast
[08:20] <wgrant> (I already have code to detect superseded packages)
[08:20] <wgrant> It's still reasonably slow, but not that slow :)
[08:24] <geser> lifeless: ah, so the list is in "fail" order and not order when the build records got created? e.g. armel fails a day later that the amd64 build (for the same package) because of a large build queue
[08:25] <geser> would that be the "datebuilt" attribute from the "build" object?
[08:25] <lifeless> geser: erg
[08:25] <lifeless> geser: uhm, ok, walk back with 3 days overlap or something:)
[08:25] <lifeless> we really need an api that matches your needs
[08:25] <wgrant> The sort order changes depending on the status you request.
[08:25] <lifeless> generic programming over high latency links is fail
[08:26] <wgrant> And the object on which you request it.
[08:26] <lifeless> wgrant: failed is in date_created in as far as the ftbfs scripts requests are concerned
[08:26] <geser> lifeless: why not get the script (or more precisely the report) directly into LP? with more direct DB access it would probably cause less trouble
[08:27] <wgrant> lifeless: Sure?
[08:27] <wgrant> getBuildsByArchIds seems to sort on date_finished.
[08:27] <lifeless> wgrant: check the timeout
[08:28] <wgrant> lifeless: Is that for Failed to Build?
[08:28] <lifeless> yes
[08:28] <lifeless> huh, it is finished
[08:28] <lifeless> mea culpa
[08:33] <geser> to sum it up: I store the data from a previous run in some file, and on the next run skip over the builds which have a datebuilt before the first run, and only fetch those after that? (of course check the old stored builds if it's superseded or fixed (given-back))
[08:34] <geser> and I don't miss new records that way?
[08:34] <lifeless> failures are by date_finished
[08:34] <lifeless> so you shouldn't miss anything
[08:34] <geser> is that in the api?
[08:34] <lifeless> is it documented? probably not
[08:35] <lifeless> geser: I'd be happy to do this in-launchpad
[08:35] <wgrant> DistroSeries.getBuildRecords orders by date_finished except for NEEDSBUILD, BUILDING, UPLOADED and SUPERSEDED.
[08:35] <lifeless> geser: but as a working [mostly] thing already, working on that is kindof lower priority than all the stuff that isn't working
[08:35] <lifeless> wgrant: so what does it do when you get both FTB and NB ?
[08:36] <wgrant> lifeless: Huh?
[08:36] <lifeless> wgrant: if you ask for both a failed to build and a needsbuild status
[08:36] <wgrant> You can't ask for both at once.
[08:36] <wgrant> You can ask for one or all.
[08:36] <lifeless> whack
[08:36] <wgrant> Yes, our API is stupid :)
[08:36] <lifeless> we could just sort by date finished desc null first, date_created
[08:37] <lifeless> or something like that
[08:37] <wgrant> That breaks on retries, but retries are broken anyway.
[08:38] <wgrant> COEALESCE(date_finished, date_created) almost makes sense.
[08:38] <geser> lifeless: on https://launchpad.net/+apidoc/1.0.html#build I see only datebuilt (Date finished; The timestamp when the build farm job was finished). Is that the date_finished or a different attribute?
[08:39] <wgrant> geser: datebuild == date_finished. The internal model has changed, while the API remains compatible to avoid breaking scripts.
[08:39] <wgrant> *datebuilt
[08:44] <geser> wgrant: UPLOADED == UPLOADFAIL in the API?
[08:45] <wgrant> geser: UPLOADING, not UPLOADED, sorry.
[08:46] <geser> but is it the same or a different state?
[08:46] <wgrant> UPLOADING == "Uploading build"
[08:48] <geser> ok, so the sorting is ok for the cases the script uses
[08:49] <geser> will try to work on the script over the weekend
[08:49] <wgrant> Thanks. If you need any help give me a yell.
[08:50] <lifeless> geser: thanks!
[08:50] <lifeless> geser: we will need to make the collection faster in LP eventually, but thats likely to be feature level work.
[08:50] <lifeless> and we have /////many////// collections to fix
[08:56] <geser> how hard would it be (and how long would it take) to add filtering based on date_finished to distro_series.getBuildRecords? so the script doesn't have to fetch old records it knows already about (or are even superseded by now) just to skip them. would that help the timeout issue?
[08:57] <wgrant> geser: That's not too difficult, but you can do the same thing by just iterating through the collection until you see one that is older than you want.
[08:58] <geser> ok, so this doesn't cause problems? just the refetching of the data we're still interested in on each run?
[08:58] <wgrant> geser: Retrieving the early parts of the collections is no problem.
[08:59] <wgrant> It's when you're doing stuff like OFFSET 3000 LIMIT 75 that it gets terribly slow.
[08:59] <wgrant> Because it basically has to work out the first 3075, then return the last 75.
[08:59] <wgrant> And the queries involved are not terribly fast.
[09:01] <geser> at which part of the script do such queries get issued? so I understand the problem better
[09:01] <wgrant> geser: Iterating through getBuildRecords.
[09:01] <wgrant> The API limits it to batches of something like 75
[09:02] <wgrant> So it grabs 0-74 in one request, 75-149 in another, and so on.
[09:03] <geser> so it's the "for build in series.getBuildRecords(build_state = state):" causing such queries?
[09:03] <wgrant> Yes.
[09:04] <geser> how can I avoid that even when I save the data from the previous run?
[09:04] <wgrant> You'll get them in date_built order. When you see one which is older than your cut-off date, break out of the loop.
[09:05] <wgrant> The getBuildRecords call doesn't grab everything at once.
[09:05] <geser> ah, so fresh ones are at the bedinning of the list?
[09:05] <wgrant> Right.
[09:05] <geser> beginning
[09:05] <wgrant> It's date_finished DESC
[09:05] <geser> now everything gets more clear
[09:08] <geser> would it be enough to save the date of the last still published build record (for each state)? and skip later ones
[20:44] <pedro> hi
[20:46] <KB1JWQ> Hello.
[21:03] <pedro> hi
[21:32] <bjf> sinzui, i've switched over to using qastaging as opposed to staging, it looks like the database hasn't been sync'd with production for a while
[21:33] <bjf> sinzui, is there any kind of schedule when the database gets reloaded ?
[21:36] <lifeless> bjf: in theory every week
[21:36] <lifeless> in practice possibly 2-3 week
[21:36] <lifeless> s
[21:37] <bjf> lifeless, hmmm, ok could be, i added a project to production about 3 to 4 weeks ago and qastaging doesn't know about it
[21:38] <lifeless> it was last synced 14 days ago
[21:38] <lifeless> from memory
[21:39] <bjf> lifeless, is this handled by some kind of cron job or is it something someone has to remember to do and do it "manually" ?
[21:40] <hyperair> c
[21:40] <hyperair> oops
[21:40] <sinzui> bjf staging was restored on the 13th with data from the 12.
[21:41] <sinzui> a cronscript does start the restore, every Saturday, but we are luck when we get to restores a month
[21:41] <lifeless> actually
[21:41] <lifeless> qastaging isn't in cron yet
[21:42] <lifeless> staging is restored weekly
[21:42] <sinzui> I do not think qastaging has ever gotten data since it was setup
[21:42] <lifeless> sinzui: yes, it was done a couple of weeks back
[21:42] <sinzui> I use staging's db exclusively for research
[21:43] <bjf> sinzui, i've tried using staging but been told qastaging is the correct place because there is no priority on fixing staging when it busts and it gets updated "often"
[21:44] <sinzui> bjf: qastaging has always has current code. Most users do not care about data. I do because I work with spam and I research user issues
[22:08] <pedro> hi
[22:08] <pedro> hi
[22:09] <pedro> its nice that people form canonical are here