[00:00] <wgrant> NCommander: They've been implemented for well over a year.
[00:00] <wgrant> And permissions use them now.
[00:01] <NCommander> wgrant: ah, I thought there were still bits that required implementation in Launchpad (I saw the IPackageset and friends, but not sure what the status was)
[00:02] <wgrant> NCommander: There are a few bits and pieces that don't respect those permissions yet (bug nomination appproval and build retries come to mind), but most of it works and is in active use.
[00:02] <NCommander> wgrant: they are?
[00:02]  * NCommander didn't know this
[00:03] <NCommander> wgrant: so its possible to generate Packages files with Sets: *blah* in them?
[00:03] <wgrant> NCommander: That would not be too difficult.
[00:04] <wgrant> Except for the difficulty inherent in the crazy apt-ftparchive integration.
[00:05] <NCommander> wgrant: well, APT has to be patched regardless to understand sets
[00:06] <NCommander> wgrant: is there a way to query sets currently in LP?
[00:07] <wgrant> NCommander: Yes, through the API. Check out edit_acl.py in ubuntu-archive-tools.
[00:07] <NCommander> wgrant: ah, I knew about those, just didn't realize those were IPackagesets in the backend
[00:10] <NCommander> wgrant: I thought LP was changed to generating the Packages files directly off the  backend and obseleting apt-ftparchive
[00:10] <wgrant> NCommander: Ha ha ha.
[00:11] <wgrant> NCommander: It does for PPAs and partner.
[00:11] <NCommander> oops
[00:11] <wgrant> But primary and copy archives still use apt-ftparchive.
[00:11] <NCommander> wgrant: fun
[00:11] <lifeless> wgrant: performance?
[00:11] <NCommander> lifeless: probably an issue with Contents-* files and other fun stuff
[00:11]  * NCommander went down this road with dak
[00:11] <wgrant> lifeless: Probably not, but it's unclear.
[00:11] <wgrant> The main problem is with overrides.
[00:11] <wgrant> NMAF doesn't really support them.
[00:13] <wgrant> Contents files are another point, but they're generated somewhat separately.
[00:14] <wgrant> (it would be nice if LP could generate them natively for PPAs as well, but a table listing all files would be HUGE.
[00:17] <NCommander> Internet doesn't like me at the moment
 wgrant: at the risk of being terrorifed, how do Tasks then end up in the packages file?
[00:17] <elmo> overrides
[00:17] <wgrant> Magical magical override files.
[00:17] <wgrant> Not really generated by LP.
[00:17] <wgrant> But by something that lives in the publisher cronscript.
[00:18] <wgrant> elmo: I'm guessing no Soyuz people are around to fix that custom upload bug?
[00:19] <elmo> wgrant: I guess not, sorry.  if you have a trivial patch, I'm willing to cowboy it on
[00:20] <wgrant> elmo: See the branch on the bug. With tests and everything.
[00:20] <elmo> oh, well, I didn't actually subscribe myself \o/
[00:20] <wgrant> (cjwatson's fix was slightly wrong)
[00:21] <elmo> aaaaa, LP code ui, you make me cry
[00:21] <wgrant> What's it doing to you now?
[00:22] <elmo> just going from that bug to your actual diff is requiring me to think
[00:22] <elmo> and I'm not doing well at that right now
[00:22] <NCommander> wgrant: I just looked at that crontab and ....
[00:22]  * NCommander cries
[00:23] <wgrant> NCommander: Yes, cron.publish-ftpmaster is scary.
[00:23] <wgrant> elmo: It's easy if there's an MP, which there isn't yet.
[00:24] <NCommander> wgrant: the changes look pretty straightforward, I just need to amend the extras sweep in cron.germinate (which probably will become to cron.extras) to dump Packagesets in-mass, and then let apt-ftparchive handle it
[00:24] <wgrant> http://bazaar.launchpad.net/~wgrant/launchpad/fix-custom-uploads-with-root-dirs/revision/10460 and http://bazaar.launchpad.net/~wgrant/launchpad/fix-custom-uploads-with-root-dirs/revision/10461
[00:24] <wgrant> Although the latter isn't important.
[00:24] <wgrant> NCommander: Right.
[00:24] <elmo> wgrant: yeah, I foujnd it eventually
[00:24] <elmo> wgrant: now I'm applying my lack of thinking to the actual diff
[00:25] <wgrant> elmo: An issue brought up in https://code.edge.launchpad.net/~julian-edwards/launchpad/symlink-vuln-bug-529710/+merge/20543 is what caused the bug.
[00:25] <NCommander> wgrant: do we support a Supported: field ATM? (I see some references to it, but nothing in publisher)
[00:26] <wgrant> NCommander: See maintenance-check.py, and whatever calls it.
[00:26] <elmo> wgrant: I'm assuming this is just a context thing (i.e. I don't have enough in the diff), but I don't see how this code actually stops $tmpdir/../../../ other than having to guess $tmpdir?
[00:26] <elmo> oh, duh
[00:26] <elmo> realpath
[00:26] <elmo> nevermind
[00:26] <wgrant> Right.
[00:26] <wgrant> realpath resolves symlinks and normpaths, eliminating that hole.
[00:32] <wgrant> Once you've cowboyed it, process-accepted should successfully process those uploads on its next run.
[00:32] <elmo> I'm happy enough to cowboy this, given it can only break customuploads
[00:32] <wgrant> Right.
[00:35] <lifeless> wgrant: all files is only about 2GB of data
[00:35] <lifeless> wgrant: (for all files + all packages + per-binary file knowledge + relationships between packages)
[00:35] <elmo> lifeless: err, not for PPAs
[00:35] <lifeless> wgrant: 'conflictchecker' has this, and can query single package conflicts quickly
[00:35] <wgrant> And not for multiple releases of each source package.
[00:36] <lifeless> wgrant: conflict checker does precisely that
[00:36] <wgrant> Hmm.
[00:37] <lifeless> elmo: I don't have the data to estimate for PPAs; if there is interest I could page in conflict checkers schema and figure out what we need to know to estimate it for PPAs.
[00:37] <lifeless> by per-binary I mean per-arch
[00:37] <mwhudson> when you're talking about HUGE tables in launchpad, bear in mind we have TranslationMessage and BranchRevision :-)
[00:37] <wgrant> mwhudson: That's true.
[00:37] <mwhudson> though the last is totally silly
[00:37] <wgrant> But BranchRevision is stupidly huge.
[00:37] <wgrant> Yes.
[00:38] <elmo> please tell me BranchRevision is not what I think it is
[00:38] <lifeless> for added fun, conflictchecker was running on sqlite last I saw.
[00:38] <mwhudson> elmo: lalalala fairies
[00:38] <lifeless> elmo: its what you think it is.
[00:38] <wgrant> elmo: Yes, every branch of Launchpad actually has 70000 BranchRevision rows linked to it.
[00:38] <elmo> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
[00:40] <mwhudson> i wonder if we should look at neo4j for this sort of thing
[00:40] <wgrant> Hm, maybe.
[00:40] <mwhudson> however right now i'm going to have lunch and water the garden
[00:41] <lifeless> elmo: what machine is conflict-checker running on at the moment?
[00:41] <elmo> lifeless: macaroodle or something
[00:41] <elmo> lifeless: macaroni or macquarie
[00:41] <elmo> probably macaroni
[00:42] <wgrant> macaroni indeed.
[00:42] <elmo> yeah, it's the source of constant 1 load on macaroni
[00:43] <lifeless> hmm, 3GB now; probably need to ditch some old dev edition packages.
[00:45] <elmo> wgrant: your patch is on production now
[00:46] <wgrant> elmo: I guess we'll see if it works in 16 minutes.
[00:46] <wgrant> Thanks.
[01:00] <lifeless> wgrant: select count(*) from filepath;
[01:00] <lifeless> 8319961
[01:01] <wgrant> lifeless: Is that for all SPRs, or just current ones?
[01:02] <lifeless> wgrant: thats all filenames seen in binary packages ever.
[01:02] <lifeless> wgrant: (but it only scans the official archive)
[01:02] <wgrant> Hm, not bad.
[01:03] <lifeless> sqlite> select count(*) from binpackagefiles;
[01:03] <lifeless> 28807743
[01:03] <lifeless>  .schema binpackagefiles
[01:03] <lifeless> CREATE TABLE binpackagefiles (package_id INTEGER, file_id INTEGER);
[01:03] <lifeless> CREATE INDEX _bpf_fileid on binpackagefiles (file_id);
[01:03] <lifeless> CREATE UNIQUE INDEX _unique_packagefile ON binpackagefiles (package_id, file_id);
[01:04] <wgrant> I guess we could always purge the lists when we expire PPA sources and binaries, if it becomes a problem.
[01:04] <wgrant> elmo: Did it work?
[01:04] <elmo> wgrant: oh, I wasn't looking
[01:04] <elmo> wgrant: didn't the previous upload get rejected?
[01:04] <wgrant> elmo: Unlikely. It should have just sat in ACCEPTED.
[01:04] <lifeless> wgrant: its really quite modest. You need to be careful to use indices, but thats about it.
[01:04] <elmo> wgrant: ah
[01:04] <wgrant> Actually, I should be able to check the queue pages...
[01:05] <lifeless> if we had the blacklisting done, for handling hand-verified maintainer-scripts, we would probably keep indefinitely ahead of the archive.
[01:09] <wgrant> elmo: They're all DONE now, so it probably worked.
[01:10] <elmo> wgrant: sweet, thanks
[01:10] <wgrant> Ah, yes, we have binary publications too. All is good.
[01:18] <mwhudson> neo4j looks handy, apart from the j bit
[01:39] <maxb> Ah. python2.5 has gone from lucid
[01:40] <maxb> +copy-packages time
[01:40] <wgrant> :(
[01:43] <maxb> um. What does it mean when a single source package version has two source publications, one superseding the other?!
[01:43] <maxb> oh, override change?
[01:43] <wgrant> Normally, yes.
[01:45] <maxb> oh this could be fun. So I need to pick the right one such that the package ends up in the PPA's main :-)
[01:47] <maxb> ngh
[01:47] <maxb> And I can't retry builds in the LP PPA
[01:54] <maxb> Can someone in ~launchpad hit retry on these, please: https://edge.launchpad.net/~launchpad/+archive/ppa/+build/1553604 https://edge.launchpad.net/~launchpad/+archive/ppa/+build/1553605
[02:00] <jelmer> maxb: done
[02:00] <maxb> thanks
[02:29] <maxb> Hrm.
[02:29] <maxb> launchpad-developer-dependencies Depends spidermonkey-bin
[02:29] <maxb> which has just been removed from lucid
[02:31] <mwhudson> i wonder what that's for
[02:32]  * maxb files it under the 'Someone thought it was a good idea' heading :-)
[02:38] <maxb> Added anonymously in 0.35, October 2008
[02:39] <mwhudson> sounds like something that could be deleted
[03:23] <wgrant> maxb: PPA publications are overridden to main by a deeper hack, so it doesn't matter which you copy.
[03:46] <thumper> :(
[03:46] <thumper> spending lots of time reading code
[03:46] <thumper> and not so much time writing
[04:17]  * mwhudson is writing some very boring tests
[04:18] <lifeless> \\ ?
[08:09] <jtv> stub: I don't suppose fragment caching works within an arbitrary loop?
[08:09] <stub> It does
[08:16] <noodles775> wgrant: hi, shall I land your buildstatus-to-buildmaster branch, or have you already organised it since fixing the test?
[08:17] <noodles775> I can't see it in the changelog, so I'll do so.
[08:17] <jtv> stub: how does it know which iteration is which in the cache?
[08:18] <stub> A counter is stored as a request annotation, and this is used to form part of the key.
[08:23] <adeuring> good morning
[08:23] <noodles775> Hi adeuring
[08:26] <jtv> hi adeuring, hi noodles775
[08:26] <adeuring> hi noodles775, jtv!
[08:26] <noodles775> G'day jtv :)
[09:03] <mrevell> Hallo
[09:06] <wgrant> noodles775: I haven't organised it. Thanks.
[09:07] <wgrant> noodles775: Can you please also land my fix-build-breadcrumbs? It failed a couple of tests overnight which I've fixed.
[09:08] <noodles775> wgrant: sure.
[09:55] <bigjools> stub: are you around much longer?  I'd like to talk to you about how to fix https://bugs.edge.launchpad.net/soyuz/+bug/517101
[09:55] <mup> Bug #517101: update-pkgcache.py needs to be database friendly <Soyuz:Triaged> <https://launchpad.net/bugs/517101>
[09:55] <stub> I'm free
[10:03] <stub> Back later
[10:03] <maxb> wgrant: for +copy-packages too? http://ppa.launchpad.net/maxb/ppa/ubuntu/pool/multiverse/s/sun-java5/ says otherwise :-)
[10:03] <wgrant> maxb: That was ages ago.
[10:03]  * wgrant checks the log.
[10:05] <maxb> oh, ok. I admit to not having a more recent example
[10:05] <persia> bigjools: So, would today be a good day to talk about Copyright stuff for native-source-sync?
[10:06] <bigjools> persia: yes maybe in an hour?
[10:06] <persia> bigjools: That sounds great.  Thanks.
[10:07] <bigjools> great
[10:08] <wgrant> persia: Changelog stuff?
[10:08] <persia> wgrant: Yes, thanks for the correction.
[10:08] <jtv> henninge: --force-depends works
[10:08] <wgrant> Messy :(
[10:09] <henninge> jtv: sure ;-)
[10:13] <wgrant> maxb: It was fixed two days after those were published.
[10:20] <bigjools> wgrant: can you propose a merge for your custom upload fix
[10:23] <wgrant> bigjools: Sure.
[10:23] <bigjools> thanks for fixing it, I'll get it landed
[10:24] <bigjools> I was out yesterday
[10:24] <wgrant> Thanks.
[10:25] <wgrant> Unfortunately I last tested the original fix with a real upload before the os.sep change was added :(
[10:26] <bigjools> :(
[10:32] <bigjools> so I am setting up LP on a new laptop on lucid, and rocketfuel-setup runs aptitide which eventually says launchpad-developer-dependencies depends on spidermonkey-bin ... ?!
[10:33] <wgrant> bigjools: Yeah, see the ML.
[10:33] <wgrant> It was removed recently.
[10:33] <bigjools> ok I'm well behind on email :(
[10:34] <wgrant> bigjools: Should I specify you as the reviewer?
[10:34] <bigjools> wgrant: yes
[10:37] <persia> Is there any specific reason rocketfuel-setup runs aptitude?  I've heard the aptitude resolver isn't widely tested (and seen any number of bugs for stuff that didn't work with aptitude that did work with other resolvers)
[10:46] <bigjools> I suspect whoever wrote that part of the script just happens to also prefer it
[10:46] <bigjools> we should change it
[10:57] <thumper> damn
[10:57] <thumper> call in 6 hours
[10:57] <thumper> best go to be
[10:57] <thumper> d
[11:00] <deryck> Morning
[11:09] <wgrant> Do the Canonical webapp engineering teams not talk to each other?
[11:09] <wgrant> I have found the same obvious hole in three Canonical webapps in the past two weeks.
[11:13] <bigjools> wgrant: not as much as we should, we're setting up an architectural team to address that
[11:16] <jtv> wgrant: not getting my Bob to pick up the jobs... are you sure I don't need to run anything?
[11:18] <wgrant> jtv: As long as buildd-manager is working, it should be working. What do buildd-manager's logs say? What does an XML-RPC status() call on the builder say?
[11:19] <jtv> wgrant: hmm... in /var/log/launchpad-buildd/default.log I see it shutting down after a SIGTERM, and nothing after that
[11:19] <wgrant> jtv: Heh, you might want to start it.
[11:19] <jtv> yes, I might
[11:20] <jtv> wgrant: time to remove the -e I added to that helper script  :)
[11:20] <wgrant> A good idea.
[11:22] <jtv> still no joy though :(
[11:22] <wgrant> What does buildd-manager say?
[11:23] <jtv> wgrant: nothing in the log, nothing in the shell
[11:24] <jtv> I tell a lie
[11:24] <wgrant> jtv: Hmph. The builder is responding with IDLE?
[11:24] <jtv> startup failed because there's "a twistd" already running
[11:25] <wgrant> Hmm.
[11:25] <jtv> yup, buildd manager was still running.
[11:25] <jtv> I'll try killing that instance and restarting
[11:25] <persia> bigjools: Let me know when you have time :)
[11:25] <stub> How often will bug heat get refreshed do we think?
[11:25] <bigjools> persia: right now
[11:26] <persia> bigjools: Great.  So my current understanding is that there are 3-4 things involving changelogs that need to be sorted before we can turn on native-source-sync, and that you know which these are.  Is this true?  If so, could you expand?
[11:26] <bigjools> persia: my memory is very hazy on this, I'll see if I wrote something down
[11:27] <bigjools> ok so there's a few bugs
[11:28] <bigjools> https://bugs.edge.launchpad.net/soyuz/+bug/55795
[11:28] <jtv> wgrant: I have a "starting scanning cycle" in the buildd-manager log, but nothing else.  :)
[11:28] <mup> Bug #55795: +changelog includes misleading information related to package versions and authors <soyuz-core> <soyuz-upload> <ui> <Soyuz:Triaged> <https://launchpad.net/bugs/55795>
[11:28] <wgrant> jtv: The builder is still marked OK, and not manual?
[11:28] <bigjools> https://bugs.edge.launchpad.net/soyuz/+bug/139162
[11:28] <mup> Bug #139162: Store the pristine debian/changelog for each SourcePackageRelease <soyuz-core> <soyuz-upload> <Soyuz:Triaged> <https://launchpad.net/bugs/139162>
[11:28] <wgrant> jtv: Can you see buildd-manager's polls in /var/log/launchpad-buildd/default?
[11:28] <bigjools> https://bugs.edge.launchpad.net/soyuz/+bug/247456
[11:28] <mup> Bug #247456: changelog listing are incomplete and/or undiscoverable <soyuz-core> <ui> <Soyuz:New> <https://launchpad.net/bugs/247456>
[11:28] <bigjools> persia: basically it's horrible right now
[11:29] <jtv> wgrant: the builder is OK and not manual
[11:29] <wgrant> Also we need to be able to create something akin to a changes file.
[11:29] <jtv> wgrant: it's not logging to there, it turns out; it's logging to /var/tmp/development-buildd-manager.log
[11:30] <jtv> All I see there is "Starting scanning cycle."  Lots of times.
[11:30] <wgrant> jtv: That's buildd-manager. I want launchpad-buildd.
[11:30] <jtv> ahh
[11:30] <persia> bigjools: There's a few other bugs surrounding that which I've seen, but yeah.  So do we actually need the changelog for anything, or could we just store it as requested in 139162 and present it as requested in 55795?
[11:31] <jtv> wgrant: I've been looking at /var/log, but I guess that's on the slave.  :)
[11:31] <bigjools> persia: 139162 is the first task
[11:31] <jtv> wgrant: 2010-03-10 11:31:18+0000 [HTTPChannel,910,127.0.0.1] 127.0.0.1 - - [10/Mar/2010:11:31:18 +0000] "POST /rpc HTTP/1.0" 200 222 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)"
[11:31] <jtv> wgrant: repeats
[11:32] <wgrant> jtv: OK, so it is polling.
[11:32] <wgrant> So maybe the candidate selection is not working properly.
[11:32] <bigjools> persia: once we've done that the NSS becomes easier
[11:32] <jtv> wgrant: I guess.  I was hoping to be able to stay out of that.  :)
[11:33] <jtv> But oh well. :)
[11:33] <wgrant> jtv: Check your postgres log for the query that buildd-manager is using.
[11:34] <persia> bigjools: Right.  So we have code that does 139162 that we use for the changelog download in apitutude and update-manager, storing stuff on changelogs.ubuntu.com.
[11:34] <wgrant> Run that, and work out why it isn't returning your job.
[11:34] <wgrant> If it is, there is something more sinister going on.
[11:34] <bigjools> persia: we also need to repackage existing records
[11:35] <persia> bigjools: How do you mean "repackage"?
[11:35] <jtv> wgrant: there's no extra-verbose logging in the script that I can enable?
[11:35] <wgrant> bigjools: Can you answer jtv? You know the logging much better now.
[11:35] <bigjools> persia: the existing changelogs need to be re-processed
[11:36] <jtv> bigjools: I'm trying to figure out why the buildd isn't picking up my jobs.
[11:36] <persia> bigjools: Is there any useful information there, or do we just need to extract changelogs from the existing packages?
[11:36] <bigjools> jtv: got a b-m log?
[11:37] <bigjools> persia: AIUI that's pretty much it
[11:37] <jtv> bigjools: just says "Starting scanning cycle." all the time.
[11:38] <bigjools> jtv: then your job is not eligible for dispatch (either a bug or genuinely)
[11:38] <jtv> bigjools: so as wgrant just pointed out, sounds like candidate selection.
[11:38] <persia> bigjools: And then 55795 becomes easy because we can just expose that blob?
[11:38] <jtv> bigjools: right, so my question is: is there debug logging in the script that'll make this easier to pinpoint?
[11:38] <bigjools> jtv: see the query it's issuing in the PG log
[11:38] <bigjools> and go from there
[11:38] <jtv> bigjools: that's what he said.  OK, I'll try that, thanks.
[11:38] <bigjools> jtv: there's no debug logging that will help :(
[11:38] <bigjools> it's the candidate selection query
[11:39] <bigjools> persia: exactly
[11:39] <wgrant> Which I know works generically now, but you may be missing a piece for your translations job.
[11:39] <bigjools> persia: the way attribution is done is all messed up as well
[11:39] <bigjools> yep
[11:39] <persia> bigjools: That goes away if we use mvo's code for 139162
[11:40] <bigjools> persia: it would need to work in LP, but that's the general idea yeah
[11:40] <persia> bigjools: And then 247346 gets resolved trivially as well?
[11:40] <bigjools> with any luck :)
[11:40] <persia> Now, how ought this be implemented in LP?
[11:40] <bigjools> jtv: paste the query from the pg log into a psql prompt and experiment
[11:41] <jtv> bigjools: thanks...  for now I'm just checking it manually against the data I have.
[11:41] <bigjools> persia: depends on what that code looks like, I've not analysed any of this
[11:41] <jtv> bigjools: ahhh, got it: "(buildqueue.virtualized IS NULL OR 'f' IS TRUE)"
[11:41] <jtv> I reviewed that code, so it can't be wrong.  :-)
[11:41] <bigjools> the classic :)
[11:41] <persia> bigjools: The code basically just knows the format of a .deb, and extracts the changelog.
[11:41] <persia> bigjools: That logic can be embedded however is convenient.
[11:42] <persia> We can do the same for a .dsc, but we need an unpack environment.
[11:42] <bigjools> well it already does it for dsc, just not very well
[11:42] <bigjools> the upload processor unpacks the source
[11:43] <wgrant> We already have code to grab debian/copyright.
[11:43] <jtv> bigjools: it does actually make sense, it's just that it reflects... IIRC whether the buildd manager is looking for arch-independent build candidates.
[11:43] <wgrant> Which I believe was touched last week...
[11:43] <bigjools> indeed
[11:44] <persia> So really, that code should just spit debian/copyright into the DB?
[11:44] <persia> And then we can expose it?
[11:44] <bigjools> it does already
[11:44] <bigjools> we just don't expose it
[11:44] <persia> So what's missing in 139162 again?
[11:44] <bigjools> https://dev.launchpad.net/Soyuz/Specs/RawSourceChangelog
[11:44] <bigjools> ^^ that
[11:47] <persia> I don't see the missing bit, if we're already stuffing full raw changelogs into the DB.
[11:47] <bigjools> we're not, we're extracting bits of it
[11:47] <persia> Or is it just the exposure and client changes?
[11:47] <persia> Aha!
[11:48] <bigjools> and then trying to re-attribute them
[11:48] <wgrant> We store debian/copyright, not debian/changelog yet.
[11:48] <bigjools> it all blows quite spectacularly
[11:48] <bigjools> right, we store changelog_entry which is a partial changelog
[11:49]  * persia slaps forehead
[11:49] <persia> Right.  So we just need to pull debian/changelog *when* we pull debian/copyright.
[11:50] <persia> Now, do we actually need any of the code that attempts to construct a changelog, or can we scrap that entirely?
[11:50] <bigjools> I'm not sure yet, I suspect it could be scrapped but bits of it might get re-used in the presentation
[11:51] <bigjools> stub: ok can we talk about pkg-cache now?
[11:51] <stub> yup
[11:52] <bigjools> stub: where does the problem lie, in the fact that it's reading too quick?  or udpating?
[11:52] <persia> bigjools: Thanks!
[11:52] <bigjools> or both?
[11:52] <bigjools> persia: no worries!  I hope it's filled a gap
[11:52] <stub> updating too quickly without regard to the system load
[11:52] <bigjools> persia: and sorry it took me so long to get back to you
[11:52] <bigjools> stub: so I could put a loop tuner at the top level where it reads package names in
[11:52] <bigjools> which would also throttle writes
[11:53] <stub> Ideally it should block when the system is busy. That logic is centrally encoded in DBLoopTuner at the moment.
[11:53] <bigjools> yeah
[11:53] <stub> I haven't looked at the existing structure of the script
[11:53] <bigjools> stub: it's a wrapper around various bits
[11:54] <bigjools> stub: it calls IDistribution.updateCompleteSourcePackageCache
[11:54] <bigjools> I need to move that out of that model code and into soyuz somewhere
[11:54] <bigjools> then I think I can just slap a loop tuner on there and profit
[11:54] <stub> So updateCompleteSourcePackageCache sounds like a chunk that should be its own DBLoopTuner loop
[11:55] <bigjools> exactly
[11:55] <stub> You could install it into garbo-daily.py if you like - this fits in with its role.
[11:56]  * stub looks at updateCompleteSourcePackageCache
[11:57] <stub> Yer - doesn't look like much conversion to turn that loop in updateCompleteSourcePackageCache into a DBLoopTuner loop.
[11:59] <jtv> al-maisan: wgrant, bigjools: looks like my job should have "virtualized" set to true in order to be selected by a non-virtualized builder
[11:59] <wgrant> jtv: Didn't I convince you to set bob to virtualised?
[12:00] <al-maisan> jtv: not setting it (i.e. a NULL) will be eqivalent to virtualized=True
[12:00] <wgrant> You should set virtualized to either True or None in order for it to build on a virt builder.
[12:00] <wgrant> False will let it build on non-virt.
[12:00] <jtv> wgrant: I may easily have gotten confused between the setting on the builder and on the jobs.
[12:02] <jtv> wgrant: ah, I see it: I skipped a step this time that I had documented
[12:02] <wgrant> Ahh.
[12:02] <jtv> sorry for the noise
[12:02] <jtv> now I guess I wait for it to poll...
[12:03] <wgrant> It should poll every five seconds.
[12:41] <wgrant> bigjools: Thanks for landing that.
[12:42] <wgrant> noodles775: I see that buildstatus-to-buildmaster should be landing RSN, but I have no EC2 email about the other one yet.
[12:43] <bigjools> wgrant: welcome
[12:44] <bigjools> is someone already fixing the spidermonkey-bin problem BTW?
[12:44] <wgrant> bigjools: Do you have time to talk about PPA download stats?
[12:44] <bigjools> wgrant: sure
[12:45] <wgrant> so, bug #139855
[12:45] <mup> Bug #139855: Display stats about PPA usage <feature> <Soyuz:Triaged> <https://launchpad.net/bugs/139855>
[12:45] <wgrant> The only consensus seems to be that we want to store binary download counts like we do LFA downloads.
[12:46] <wgrant> Indices probably want to be counted somehow too, but what to do with that is much less obvious.
[12:46] <wgrant> I have a working implementation of binary counts.
[12:46] <bigjools> I tihnk that doing the binary counts is a very good first step
[12:47] <bigjools> do you want me to take a look at it?
[12:48] <noodles775> wgrant: checking the ec2 instance...
[12:50] <noodles775> wgrant: it seems to be finishing (it's not yet terminated, but I can no longer access it), so I'll wait a bit. If I've not seen anything by my EOD, i'll resend it.
[12:50] <wgrant> noodles775: OK, odd...
[12:50] <wgrant> At least the other one should be in PQM.
[12:51] <noodles775> wgrant: it is (playing now).
[12:54] <wgrant> bigjools: I think the backend branches (https://code.edge.launchpad.net/~wgrant/launchpad/parse-ppa-apache-logs is the final one) should be reasonably unobjectionable.
[12:55] <wgrant> But https://code.edge.launchpad.net/~wgrant/launchpad/export-basic-binary-download-stats and its successor https://code.edge.launchpad.net/~wgrant/launchpad/export-detailed-binary-download-stats might well be horribly slow.
[12:55] <bigjools> wgrant: ok, file an MP and request me as a reviewer and I'll look at that one
[12:55] <bigjools> can you paste a diff?  I'm lazy :)
[12:59] <wgrant> bigjools: The combination is not quite oversized, but it has two logical prereqs which will make it smaller. Shall I propose them separately?
[13:00] <bigjools> wgrant: I'm a little confused, you have three branches and the first one is not controversial, right?  but the other 2 might be?  so I was asking for diffs for those
[13:01] <wgrant> bigjools: The first uncontroversial one is really three branches. You just want the two that might be? That's easy.
[13:01]  * wgrant diffs.
[13:02] <bigjools> wgrant: well if it's really big we should split it up and get more reviewers involved
[13:03] <wgrant> bigjools: It's not really big. The first branch I mentioned comes it at around 800 lines -- it's just probably a bit easier if it's split into its three logical components.
[13:04] <bigjools> ok
[13:10] <wgrant> noodles775: Ah, that other run emailed a couple of minutes ago. Great.
[13:10] <noodles775> Excellent.
[13:23] <wgrant> bigjools: A few criss-cross merges and lots of conflicts later: http://pastebin.ubuntu.com/392515/ and http://pastebin.ubuntu.com/392521/
[13:23] <bigjools> heh
[13:24] <wgrant> The first just exports a single count per BPPH, the second gives detailed data.
[13:30] <bigjools> wgrant: if it's expensive, don't make it a property
[13:31] <bigjools> download_count that is
[13:31] <wgrant> bigjools: Yeah, I guess not.
[13:31] <leonardr> who should i speak to about the design of html pages for launchpad? is it still mpt?
[13:31] <bigjools> it will get serialised on every fetch
[13:32] <bigjools> leonardr: good point - it was beuno but he's moved to U1
[13:32] <bigjools> he might still want to help :)
[13:32] <wgrant> I'm not sure how expensive it will be. It could be a SUM across several hundred rows of a single-table join.
[13:33] <bigjools> I want to avoid making properties that hide complex queries anyway
[13:33] <bigjools> it's not really a property :)
[13:33] <wgrant> This is true.
[13:36] <asabil> hi all
[13:37] <bigjools> wgrant: I think sum is pretty quick anyway
[13:38] <bigjools> wgrant: but I wonder if we shouldn't just store that total to avoid doing it?
[13:40]  * bigjools thinks about splitting up archive.txt
[13:41] <wgrant> bigjools: Is archive.txt the 3000 line one?
[13:41] <bigjools> yarp
[13:41] <bigjools> it's useless as documentation
[13:41] <bigjools> and lots of it can be a unit test
[13:41] <wgrant> bigjools: We could, but there's no convenient table representing (archive, bpr)
[13:42] <wgrant> Right.
[13:43] <bigjools> presumably you've got a schema change hiding somewhere?
[13:44] <wgrant> It's in the previous branch.
[13:44]  * wgrant diffs.
[13:44] <bigjools> ok
[13:44] <bigjools> what is the .count counting then?
[13:44] <bigjools> Sum(BinaryPackageReleaseDownloadCount.count)
[13:44] <beuno> bigjools, I always want to help
[13:45] <bigjools> if we're counting total downloads per archive we can add a column to Archive
[13:45] <bigjools> beuno: yo da man
[13:46] <wgrant> bigjools: Like LFDC, it maps (target (in this case (archive, bpr)), date, country) -> count
[13:46] <jml> I'm sorry, I wasn't paying attention. Did I hear someone say they were adding download counts for PPAs to Launchpad?
[13:46] <bigjools> heh
[13:48] <wgrant> bigjools: http://pastebin.ubuntu.com/392534/ is the work before the two diffs I gave above.
[13:48] <bigjools> wgrant: so we could use another column on IArchive for total download count and maintain it in the log scraper?
[13:48] <jml> leonardr, there's no one person with responsibility for that any more. You consult the launchpad-dev list, basically.
[13:48] <leonardr> jml: ok
[13:48] <wgrant> jml: Has that been shown to work at all yet?
[13:48] <jml> wgrant, asking launchpad-dev for advice?
[13:48] <wgrant> bigjools: But counts per archive, per (archive, bpr) and per (archive, bpr, day) are useful.
[13:48] <wgrant> We need a generic cache mechanism :(
[13:48] <leonardr> well, i'm going to write something, and if launchpad folks don't like it they can lump^H^H^H^Hfix it
[13:48] <wgrant> jml: For UI advice.
[13:49] <jml> wgrant, how would you tell if it worked?
[13:49] <bigjools> wgrant: yes I'm saying to do it instead of that new property
[13:49] <bigjools> so you can avoid the Sum
[13:50] <wgrant> jml: Well, given that LP now has no UI designers, and nobody from UX has replied to the RFC-UI emails on the list up to now, it seems like it probably won't.
[13:50] <bigjools> wgrant: the other thing is that exposing database IDs on URL traversals makes me a bit sad
[13:50] <wgrant> bigjools: But that's on BPPH, so it needs a download count cache on (archive, bpr), which is a tuple that the DB doesn't know about.
[13:51] <wgrant> bigjools: Yes :(
[13:51] <jml> wgrant, although I would highly value contributions from Canonical UX, I think that the broader launchpad developer community almost always has useful things to say about UI design.
[13:51] <bigjools> can you traverse with the package name/version instead?
[13:52] <wgrant> bigjools: Possibly. I suppose they should generally be unique.
[13:52] <bigjools> wgrant: IArchive.getPackageDownloadTotal() - we can replace that with a new column on Archive
[13:52] <wgrant> And any cases in which they are not are bugs.
[13:52] <bigjools> unless I am misreading something?
[13:52] <wgrant> bigjools: It takes a BPR as an argument.
[13:53] <bigjools> yes I am misreading :)
[13:53] <bigjools> so that's the total of all versions of that package in this archive
[13:53] <wgrant> All versions? A BPR is one version.
[13:54] <wgrant> All publications, yes.
[13:54] <bigjools> all published versions is what I meant
[13:54] <wgrant> Right.
[13:54] <bigjools> looks great
[13:55] <bigjools> just needs to be a method instead of a property on publishing
[13:57] <wgrant> bigjools: Great, thanks. It is getting late, so I will fix that tomorrow. Shall I then request reviews from you for the three, or should I use OCR now you're obviously not too unhappy with the implementation?
[13:58] <bigjools> wgrant: the traversal should not use the ID, other than that the approach looks great, thanks!
[13:58] <wgrant> Oh, right, yeah.
[14:19] <kfogel> adeuring: re bug #532022, I'm just starting to figure out where to add the criterion to suppress bugs whose status is "Fix Released".  Is this something that would go into those special DB queries you made for +patches?
[14:19] <mup> Bug #532022: bugs with patches shows fix released bugs as well <Launchpad Bugs:In Progress by kfogel> <https://launchpad.net/bugs/532022>
[14:21] <adeuring> kfogel: we call bugtarget.searchTasks() with a set of status values. I think it is enough to change this set of values, let me look...
[14:23] <kfogel> adeuring: ah, heh -- that'd be easy.  But let me check: it *might* be the case that there's some big efficiency to be had by doing the filtering at a lower level, not sure.
[14:23] <adeuring> kfogel: right, we call searchTasks(status=UNRESOLVED_PLUS_FIXRELEASED_BUGTASK_STATUSES, ...), so this value should be changed.
[14:23] <kfogel> adeuring: okay, good.  In fact, I think I may have created that combination status explicitly for this, so if this is the only use of it, it can go away too.  I'll check.
[14:23] <adeuring> kfogel: no, I don't think so. If the current query works, another one with a smaller set of status values should be fine
[14:24] <adeuring> kfogel: yes, it can probably go again, if it's used only here..
[14:24] <kfogel> adeuring: right, because the raw query that goes into the DB starts here (i.e., it's not like we're getting a bigger set of results back and then filtering down).
[14:24] <adeuring> kfogel: right
[14:24] <kfogel> adeuring: just talking out loud partly to make sure my understandings are right.  Thanks for reading :-).
[14:24] <adeuring> kfogel: there are one more changes:
[14:25] <adeuring> the number shown in the link to the +patches view
[14:25] <kfogel> adeuring: oh, that's not connected to the searchTasks query?
[14:25] <kfogel> adeuring: (I'd already forgotten how we implemented that number.)
[14:25] <adeuring> and the test that ensures that this number euals the number of patches shown in the +patches view
[14:26] <adeuring> s/euals/equals/
[14:27] <kfogel> adeuring: thanks for the reminder -- will take a look.
[14:27] <kfogel> adeuring: but anyway I can base my branch for this off of devel, not db-devel.
[14:27] <adeuring> kfogel: sure, using devel is fine
[14:28] <adeuring> kfogel:  lib/lp/bugs/browser/bugtask.py: BugsStatsMixin.bugs_with_patches_count
[14:29] <adeuring> kfogel: and the test is: lib/lp/bugs/browser/tests/test_bugtarget_patches_view.py
[14:29] <kfogel> adeuring: thank you.  It's starting to come back to me now...
[14:29] <kfogel> adeuring: I have a tremendous ability to wipe my mind clean after finishing something.  I've often thought I could give lessons to trauma victims.
[14:29] <adeuring> (see also bzr diff -r10453..10454
[14:30] <adeuring> kfogel: sounds interesting ;) I could help there perhaps too... I had to look through "bzr log" to find the right revision, and then looked at the diff ;)
[14:31] <kfogel> adeuring: yeah, but I actually wrote this code, like, a week ago IIRC :-]
[14:32] <kfogel> wgrant: hey, I will review your branch today
[14:32] <kfogel> wgrant: "today" for me means in the next eight hours (I'm in New York)
[15:09] <mars> gary_poster, ping, trying to work around a buildout dependency.
[15:09] <gary_poster> mars, what's up
[15:09] <mars> gary_poster, I'm seeing 'mercurial' listed in the branch root's setup.py, and I do not see mercurial in versions.cfg.  Is that to be expected?
[15:10] <mars> gary_poster, and bin/buildout is spitting errors at me because of mercurial for some odd reason.
[15:10] <gary_poster> mars, no, doesn't sound right.
[15:10] <mars> weird for a 3 line change...
[15:10] <gary_poster> mars, investigating.  this is devel?
[15:11] <mars> gary_poster, yes, fresh branch yesterday
[15:11] <gary_poster> k
[15:11] <mars> gary_poster, line 46, setup.py
[15:12] <mars> looks like it is needed by code hosting.  Maybe a system package is supposed to install it?
[15:12] <gary_poster> mars, typically still should have a version
[15:12] <mars> ah
[15:14] <gary_poster> mars, jumping around a bit, but trying to pull devel now.  In regards to your hypothesis, have you checked to see if launchpad-developer-dependencies are up to date for you?
[15:17] <mars> gary_poster, chekcing
[15:18] <mars> yes, they are up to date
[15:18] <gary_poster> ok
[15:18] <gary_poster> I'm now building Twisted 10...
[15:21] <mars> gary_poster, here is what I am getting on the CLI when I run bin/buildout:
[15:21] <mars> http://paste.ubuntu.com/392596/
[15:22] <gary_poster> mars, ok, I just built fine...investigating setup.py and versions.cfg next, and will look at your pastebin
[15:22] <gary_poster> mars, mercurial is in my devel's versions.cfg
[15:23] <mars> line?
[15:23] <gary_poster> mars, the error you are getting is exactly because it is not in your versions.cfg
[15:23] <mars> hehe
[15:23] <gary_poster> we have a setting to cause that
[15:23] <gary_poster> mars, line 39
[15:23] <gary_poster> mechanize = 0.1.11
[15:23] <gary_poster> mercurial = 1.3.1
[15:23] <gary_poster> mocker = 0.10.1
[15:24] <mars> gary_poster, ok, thanks, I am missing that line.  More detective work then
[15:24] <gary_poster> mars, ok cool
[15:25] <deryck> Ursinha, ping
[15:25] <Ursinha> deryck: hi
[15:26] <deryck> Ursinha, hi.  Could the script that does qa tagging, which links the milestone if needed, also link the branch if it's not linked yet?
[15:26] <deryck> Ursinha, not to give you more work even ;)
[15:27] <Ursinha> deryck: hmmmmmmm I'll see what I can do :) I have to finish polishing the burndown chart, and have other task after that, but I put that in the queue, sure :)
[15:27] <Ursinha> *I'll put
[15:28] <deryck> Ursinha, np.  Thanks for considering it! :-)
[15:28] <Ursinha> deryck: no problem :)
[15:28] <mars> gary_poster, found the problem, thanks for the help.
[15:29] <gary_poster> mars, great
[15:35] <adiroiban> hi, any idea why I get this error on launchpad.dev when I try to login ? http://paste.ubuntu.com/392604/ ?
[15:37] <mars> adiroiban, something to do with the new OpenID changes perhaps.  Maybe check the config variable listed on line 15 of your post?
[15:38] <salgado> adiroiban, need to add testopenid.dev to /etc/hosts
[15:39] <adiroiban> salgado: true. thanks!
[15:39] <mars> salgado, if he re-runs rocketfuel-setup, will that happen automatically?
[15:39] <salgado> mars, yes
[15:39] <salgado> actually, maybe nt
[15:39] <mars> salgado, so now we all have to do that to use launchpad.dev?
[15:40] <salgado> not
[15:40] <salgado> mars, yes
[15:41] <salgado> adiroiban, mars, yes, running rocketfuel-setup should add it there
[15:41] <mars> salgado, have we sent a mail to launchpad-dev about having to run rf-setup to add that host?
[15:42] <mars> "Please re-run rocketfuel-setup" or something
[15:43] <salgado> no, I didn't.  forgot about that
[15:44] <salgado> mars, do you think it's still worth to do so?  I suspect most people have already found that out or they run rocketfuel-setup frequently and didn't notice it at all
[15:45] <mars> salgado, depends on how long ago the change landed.  It could be that most people don't log out and in again.  And sending it can't hurt :)
[15:47] <salgado> indeed.  although it's landed 2 weeks ago, it won't hurt to send an email
[15:49] <mars> the more people that know about the problem and the answer, the fewer times you will have to answer it yourself :)
[18:14] <didrocks> Launchpad screen refresh being very slow is a known issue? For instance, switching tabs in firefox, 2 to 4 seconds to refresh the page, without hitting F5; opening option bug in edit -> options takes 3 seconds to have it refreshed too
[18:14] <didrocks> and this, only if I am on a LP page current tab
[18:14] <didrocks> no issue with other sites in other tabs
[18:14] <didrocks> apparently seb128 has the same issue too
[18:15] <didrocks> I tried in edge/production server
[18:15] <didrocks> and disabling greasemonkey too
[18:17] <didrocks> (just to be sure, I'm not speaking about loading a page, just having something on a already loaded page)
[18:19] <deryck> didrocks, I'm not familiar with this as a known issue... maybe mars knows something?
[18:19] <didrocks> I would think the page content is not supported by either xulrunner or if there is an ajax things "on activate"
[18:22] <mars> ?
[18:22] <mars> didrocks, do you have a specific page you were trying this on?
[18:23] <didrocks> mars: it's an all LP page for both seb128 and I (particularly bugs page). I can make a short screencast if you want to see the difference
[18:24] <mars> didrocks, ok, if that is easiest :)
[18:24] <mars> didrocks, do you mean, all LP pages on edge have this problem?
[18:25] <didrocks> mars: no, LP edge and production server (non edge)
[18:25] <didrocks> with firefox on lucid
[18:29] <rockstar> leonardr, ping
[18:30] <leonardr> rockstar, hi
[18:30] <rockstar> leonardr, abentley and I were talking about bug 520446 this morning.
[18:30] <mup> Bug #520446: OOPS creating duplicate merge proposal using API <api> <oops> <trivial> <Launchpad Bazaar Integration:In Progress by rockstar> <https://launchpad.net/bugs/520446>
[18:30] <leonardr> ok
[18:31] <rockstar> leonardr, basically, the fix is to catch the (more meaningful) exception and raise a ValueError.
[18:31] <rockstar> leonardr, have you seen this kind of problem before?
[18:31] <didrocks> mars: http://people.canonical.com/~didrocks/slowLP-1.ogv that should explain the issue :)
[18:31] <mars> didrocks, looking
[18:32] <leonardr> rockstar: so the problem is that a specific exception results in an oops when it should result in a 400 status code?
[18:32] <leonardr> yes, that happens all the time
[18:33] <leonardr> you can annotate the exception with @webservice_error(400) and it will be handled correctly if the error happens during field validation
[18:33] <leonardr> if it happens in named operation invocation i don't think that will work
[18:34] <mars> didrocks, so Chrome is fine
[18:34] <didrocks> mars: right
[18:34] <didrocks> (it's chromium, but the same base ^^)
[18:34] <rockstar> leonardr, what do you mean annotate the exception?
[18:35] <leonardr> rockstar, let me try to find an example
[18:35] <mars> didrocks, could you try that tab-switching test with the gnome system monitor open?  To the Resources page, with the graphs on it
[18:35] <didrocks> mars: sure
[18:35] <leonardr> i have a feeling that this is used all over launchpad but maybe no one is using it
[18:35] <mars> didrocks, should show a CPU spike, if there is one
[18:35] <rockstar> leonardr, it seems like it should be used all over launchpad at least.  :)
[18:36] <mars> didrocks, no need to record the CPU, just let me know if it spikes while the browser is hung
[18:36] <didrocks> mars: right, one CPU is going to 100%
[18:36] <didrocks> for the 3-4 seconds
[18:36] <mars> hmm
[18:36] <rockstar> leonardr, I found some examples.
[18:36] <rockstar> lib/lp/hardwaredb/interfaces/hwdb.py
[18:37] <mars> didrocks, thinking, who's the culprit?  The FF JS engine?  Or the FF rendering engine?
[18:37] <leonardr> rockstar: yeah, try that and see if it works
[18:37] <mars> didrocks, I don't think it is any JS on our page, but can't rule that out entirely either.
[18:38] <didrocks> mars: I don't know enough Firefox to have a strong opinion on that. I can ask chriscoulson who is our new maintainer
[18:38] <didrocks> mars: can I tell him to see that with you (tomorrow probably)
[18:38] <didrocks> ?
[18:38] <mars> didrocks, sure.  You say you have confirmed the behaviour with someone else using FF+Lucid
[18:38] <didrocks> that's just pretty obvious in lucid and only affecting LP pages. So, there is something that FF doesn't like :)
[18:38] <mars> ?
[18:39] <didrocks> mars: right, I was thinking about a local profile issue first, but seb128 begins to have the same behavior
[18:39] <mars> didrocks, ok, I can think of a test: I just need to find a relatively JS-free page first
[18:40] <didrocks> mars: sure, going to get my dinner, you can ping me, bbiab
[18:40] <mars> didrocks, ok
[18:41] <rockstar> leonardr, oh man, MUCH better solution.  Thank you!
[18:42] <leonardr> awesome
[19:05] <didrocks> mars: so, chrisccoulson is our new firefox rockstar :)
[19:05] <chrisccoulson> heh
[19:05] <didrocks> mars: do you have any idea of tests I can do?
[19:05] <didrocks> chrisccoulson: did you see the video?
[19:05] <chrisccoulson> i did
[19:06] <didrocks> chrisccoulson: you can see how my life is hard nowdays :-)
[19:19] <mars> hi chrisccoulson
[19:19] <chrisccoulson> hi
[19:19] <mars> didrocks, looking around
[19:20] <mars> chrisccoulson, did didrocks show you his screencast with the strange behaviour?
[19:20] <chrisccoulson> mars - he did
[19:20] <chrisccoulson> i've not noticed behaviour like that before though
[19:21] <mars> chrisccoulson, I was thinking it may be the Firefox JS engine, the rendering/paint engine, or maybe something in our JavaScript itself that doesn't play nice with the new FF JS engine.
[19:21] <mars> chrisccoulson, but I don't have the FF profiling tools at hand to narrow it down.  I'm trying to find a non-js page to test on.
[19:21] <mars> would rule out the paint operation
[19:22] <didrocks> mars: I mentionned also to chrisccoulson that I don't have accelerated GPU driver. That should emphasize, but as we can see, chromium plays a better role there
[19:24] <mars> didrocks, try it with this page: https://code.launchpad.net/oops
[19:24] <mars> that is the 404 page: quite minimal, and no javascript
[19:24] <mars> bit of a dumb test, but better than nothing to start I guess
[19:24] <didrocks> mars: no issue with this 404 page
[19:25] <mars> didrocks, and https://code.launchpad.net/  ?
[19:25] <didrocks> its fast to say "you screw something" :)
[19:25] <didrocks> mars: I have a delay
[19:25] <didrocks> not as long as on a bug page
[19:25] <mars> interesting
[19:25] <didrocks> but still, 1.5s I would say
[19:26] <mars> didrocks, are you on a 64bit system?
[19:26] <didrocks> mars: no
[19:26] <didrocks> mars: google reader and gmail are quite fast
[19:26] <didrocks> (for the js part)
[19:27] <mars> chrisccoulson, ^ fun FF bug for you - our automated integration testing tool using FF goes nuts and eats 100% CPU if our uncompressed JS exceeds 512Kb, but only on  64 bit systems.  Not a problem here though :)
[19:29] <chrisccoulson> sorry, i just tried using venkman to profile the javascript
[19:29] <chrisccoulson> i need to figure out how to use it really though ;)
[19:29] <mars> didrocks, ok, so it is *not* the invisible iframe we use for the help system.  That is evidence against a DOM issue.
[19:30] <mars> hmm
[19:30] <mars> Google PageSpeed could tease out the issue.  It hooks right in the guts of firefox
[19:31] <mars> chrisccoulson, I take it you see the same behaviour on your system?
[19:31] <chrisccoulson> mars - i don't
[19:31] <chrisccoulson> which is a bit of a pain ;)
[19:31] <mars> ...
[19:32] <didrocks> chrisccoulson: can you try to switch to a 2D driver?
[19:32] <didrocks> maybe that'll be more obvious
[19:32] <mars> didrocks, do you see this behaviour when you run in -safemode (I think that's the switch?)
[19:32] <didrocks> on seb128's laptop is something like 2s
[19:32] <chrisccoulson> didrocks - i could do, but that might have to wait until later
[19:32] <didrocks> trying
[19:32] <mwhudson> morning
[19:32] <mars> morning mwhudson
[19:33] <mars> didrocks, ah: "firefox -safe-mode"  http://support.mozilla.com/en-US/kb/Safe+Mode
[19:33] <mars> forgot the dash
[19:35] <didrocks> mars: a little bit better, only 1.5 - 2s now. What's strange is that the only extension I have is the ubuntu one (ubufox)
[19:36] <mars> chrisccoulson, didrocks, since I don't have access to a proper FF profiler (though I have read about them), it maybe be possible to use the PageSpeed extension to narrow the problem down to JS or the browser: http://code.google.com/speed/page-speed/docs/using.html#activities
[19:36] <chrisccoulson> yeah, i just installed that
[19:36] <chrisccoulson> it looks pretty neat
[19:36] <mars> didrocks, do you see the delay/hang when you first open a page?
[19:37] <mars> didrocks, or just when visiting a tab?
[19:37] <didrocks> mars: as it's loading at the same time, it's hard to tell
[19:37] <didrocks> mars: but in the video you can see that when firefox has something to show (like the option dialog box), there is the same hang
[19:38] <mars> didrocks, the same hang?
[19:38] <kfogel> I have never seen this happen before: I did 'bzr branch devel 532022-patches-view-omit-fixreleased' to create a working branch, and cd'd into the new branch and did 'utilities/link-external-sourcecode ../devel'.  But for whatever reason, my new branch does not have a bin/ directory at the top level!
[19:38]  * mars re-reads the last word
[19:38] <didrocks> mars: the same delay before showing the dialog box
[19:38] <mars> didrocks, ok.
[19:39] <mars> didrocks, chrisccoulson, if you see the same delay when showing a dialog, then I suspect two things
[19:39] <salgado> kfogel, need to 'make build' -- that's automatically done by rocketfuel-branch
[19:39] <kfogel> salgado: aaaah.  thanks
[19:39] <mars> didrocks, chrisccoulson: 1) The browser paint and render code is messing up in a bad way
[19:40] <mars> didrocks, chrisccoulson: 2) There is a focus/blur event that our JavaScript is processing and choking on
[19:40] <didrocks> chrisccoulson, mars: showing a dialog only over a LP page, I must precise
[19:42] <mars> I don't know enough about the Firefox 3.6 architecture to say if the JS execution thread has been decoupled from the application threads: to say if it is possible for JS to block page paints
[19:42] <mars> didrocks, it is only on tab focus/blur, not window focus/blur?
[19:43] <didrocks> mars: you mean, other windows?
[19:44] <didrocks> mars: if I, for instance, move a gnome-terminal over it, I get the same issue
[19:44] <mars> didrocks, yes.  If you have the tab open, click on another window, click back on Firefox
[19:44] <mars> chrisccoulson, ^ !
[19:44] <didrocks> seems that FF has some issue to just redraw the LP page. It's just strange that it only happens with LP and not on other "big page"
[19:45] <mars> didrocks, so it chugs if you move another window of the Firefox painted area
[19:45] <didrocks> mars: exactly
[19:45] <mars> ok, so that means it is probably not the JavaScript events
[19:46] <mars> It is the Firefox rendering engine trying to paint the screen
[19:46] <mars> didrocks, there is a way to test that hypothesis
[19:46] <mars> didrocks, save the page to a local temp folder, then open the local page and try the tests again
[19:47] <didrocks> mars: ok, doing this
[19:47] <mars> didrocks, if it is the page structure, or something in the page, then it will probably show up when the page is local
[19:48] <didrocks> mars: yes! same issue with the local page :)
[19:48] <mars> didrocks, to be extra sure, we can break the javascript after you have saved it to disk.  No JS + chugging means it is the page structure or content
[19:48] <didrocks> oh, good idea
[19:48] <mars> didrocks, you can shut of JS in Firefox itself, can't you?
[19:49] <mars> (wish I'd thought of that a few minutes ago)
[19:51] <mars> didrocks, after shutting off JS, try the test.  If it still chugs, then shut off images, and please try it again.
[19:52] <mars> should narrow the bug down to the content, or the page structure itself.
[19:52] <didrocks> mwhudson: yeah, still the bug with JS unactivated
[19:52] <didrocks> now images :)
[19:56] <didrocks> mars: not showing images fix it
[19:56] <mars> ok, so it might be painting images from Launchpad that is messed up
[19:56] <thumper> morning
[19:56] <didrocks> morning thumper
[19:56] <didrocks> mars: right
[19:57] <mars> morning thumper
[19:57] <didrocks> (I still have the LP logo even when selecting "not showing images"), but the others are gone
[19:57] <mars> So, so so so....
[19:58] <mars> we could try cherry-picking just the images from the page, put them on an empty HTML page, see if the problem persists
[19:58] <mars> shouldn't take long
[19:58] <chrisccoulson> sorry, i disappeared for a bit there
[19:58] <mars> didrocks, what was the URL you have been testing with?
[19:59] <didrocks> mars: I cached locally https://bugs.edge.launchpad.net/launchpad-registry/+bug/357235
[19:59] <mup> Bug #357235: A user's ssh keys are not currently available throug the APIs <api> <feature> <quickly> <Launchpad Registry:Triaged by didrocks> <https://launchpad.net/bugs/357235>
[19:59] <mars> chrisccoulson, np.  We narrowed it down to the Launchpad page images.  The bug is present irregardless of JS being on or off.  But deaactivating images made the bug go away.
[19:59] <chrisccoulson> ah, ok
[19:59] <didrocks> I would never thought it was image related :)
[20:00] <chrisccoulson> there's not many images is there?
[20:00] <mars> well, lets see
[20:00] <chrisccoulson> oh, i notice all the images when i disable them ;)
[20:00] <mars> Firebug Net tab has that nice "images only" filter
[20:00] <didrocks> chrisccoulson: that's weird, there is one that obviously FF doesn't like :)
[20:01] <mars> chrisccoulson, after finding that 64bit 512Kb JS limit bug.... And some of the DOM bugs I've seen...
[20:01] <thumper> sinzui, bac: ping
[20:01] <thumper> re bug 246594
[20:01] <mup> Bug #246594: Registering a code import from a series is clunky. <confusing-ui> <Launchpad Bazaar Integration:In Progress by bac> <https://launchpad.net/bugs/246594>
[20:01] <mars> oh.  hmmm
[20:02] <mars> thumper, they are sprinting this week, may not be around
[20:02] <mars> IIRC
[20:02] <thumper> mars: ah
[20:02] <thumper> I'll comment on the bug
[20:02] <leonardr> sinzui, can you help me with xslt-fu?
[20:03] <mars> leonardr, see previous comment to thumper :)
[20:03] <leonardr> ah, thanks mars
[20:04] <mars> didrocks, chrisccoulson, 26 images on that bugs page according to Firebug.
[20:04] <mars> didrocks, chrisccoulson, give me a bit, I can make a temporary page with just those images.  *might* have the paint bug.
[20:05] <didrocks> mars: sure, thanks. I saw too that there is not that much image (just reinstalled firebug, that was a while… ;))
[20:06] <mars> didrocks, you need to use Control-F5 in case the browser cache was being used, to get the complete image count.
[20:06] <mars> ok, this test is shaky, but better than nothing.  Off to code
[20:06] <didrocks> mars: right, that's what I did when I test :)
[20:07] <didrocks> thanks mars :)
[20:07] <mars> didrocks, is seb128 online?
[20:07] <didrocks> mars: I guess he is away for a couple of horus
[20:07] <didrocks> hours*
[20:07] <mars> yes, and he is PST, so he'll be around :)
[20:08] <didrocks> mars: hum, he is in France too, so, even he's going to bed late, I don't think so :)
[20:09] <didrocks> mars: in any case, you can still ping me and I will do the test tomorrow morning
[20:09] <mars> didrocks, ?  His whois info says otherwise: seb128, right?
[20:09] <didrocks> (my irc is always opened)
[20:10] <didrocks> mars: he maybe connects to an PST located IRC server, but I can ensure you he is French and live in France :)
[20:10] <mars> must like using US/west servers
[20:10] <didrocks> https://launchpad.net/~seb128
[20:10] <mars> ah
[20:11] <didrocks> I just guess I made the issue more noticeable in not having 3D driver currently installed and so making FF software rendered painting suffers :)
[20:12] <mars> yeah, that was quite the hiccup though.  2-3 seconds.  And why oh why did it have to be Launchpad, of all pages? :D
[20:15] <mars> hmmmm]
[20:15] <mars> didrocks, I wonder if it is the transparent PNGs doing it?
[20:15] <mars> we'll see
[20:16] <mars> back to code
[20:16] <didrocks> mars: hum, maybe  :)
[20:16] <didrocks> ok
[20:27] <mars> didrocks, please try the test using the following page: http://people.canonical.com/~mars/lp-images-test.html
[20:27] <didrocks> mars: no latency
[20:28] <mars> phooey.
[20:28] <didrocks> mars: I guess you expected some, right?
[20:28] <mars> didrocks, I'm not surprised.  It was the only test I could think of at the moment :)
[20:29] <mars> so we have a launchpad page that, when shown with images and HTML, causes the browser paint to mess up
[20:29] <mars> shut off the images, everything is fine
[20:30] <mars> shut off the content, everything is fine
[20:30] <mars> so images+content produced the bug
[20:30] <mars> brutal.  That is hard to debug.
[20:31] <didrocks> right, I tried again and even with the online version, that's definitely reproduceable
[20:31] <didrocks> definitively
[20:32] <mars> didrocks, ok, we could try to rule out alpha using the offline version.  Could you please upload your offline version to p.c.c?  I'll hack on it.
[20:32] <thumper> abentley: AssertionError: Wrong min time to next available builder (47099 > 300)
[20:32] <thumper> abentley: that's what I get
[20:32] <mwhudson> 47099 is quite a lot bigger than 300
[20:32] <abentley> That's not the same one I get.
[20:32] <didrocks> mars: sure, one sec
[20:32] <abentley> I get 120 != 30
[20:33]  * thumper makes schema
[20:33] <didrocks> mars: FYI, if I keep the images on and I unactivate the css, no bug
[20:33] <didrocks> mars: so, definitively images + layout
[20:33] <mars> ok
[20:33] <mars> that is good to know
[20:34] <didrocks> mars: http://people.canonical.com/~didrocks/357235.html
[20:37] <mars> didrocks, perfect, thanks.  So I can try cutting out the transparent pngs.  That is something at least.
[20:37] <mars> Unless there is some other way to look at the problem.
[20:37] <didrocks> that can be a good idea at least to ensure it's not layout + transparent pngs
[20:38] <mars> maybe not the transparent PNGs.  They aren't visible on the default page.
[20:38] <mars> oh, wait, at least one is, in the footer.
[20:38] <didrocks> oh, the little logos have white bg? I hadn't a look TBH
[20:48] <mwhudson> thumper, abentley, rockstar: http://pastebin.ubuntu.com/392769/
[20:50] <thumper> http://pastebin.ubuntu.com/392770/
[20:51] <mwhudson> https://bugs.edge.launchpad.net/soyuz/+bug/525329
[20:51] <mup> Bug #525329: spurious failure in test_min_time_to_next_builder <qa-needstesting> <spurious-test-failure> <Soyuz:Fix Committed by al-maisan> <https://launchpad.net/bugs/525329>
[20:55] <mwhudson> thumper, abentley, rockstar: do you all want to approve this mp then? https://code.edge.launchpad.net/~mwhudson/launchpad/fix-test_min_time_to_next_builder-non-utc-bug-525329/+merge/21082
[21:01] <mars> didrocks, this file has heavily mangled CSS, does it show the bug?  http://people.canonical.com/~mars/lp-images-test/1/357235.html
[21:01] <mars> didrocks, this is a copy of what you sent me.  It absolutely should have the bug: http://people.canonical.com/~mars/lp-images-test/0/357235.html
[21:03] <thumper> mwhudson: we could ask mars to approve it too and go for the patch with the most reviews :)
[21:04] <mars> thumper, I must have missed where the 'utc' variable came from
[21:04] <didrocks> mars: yeah! the first one has no bug. The second one has it
[21:04] <mars> didrocks, ok.  Doesn't help much, except that almost every image is broken in /1/
[21:05] <mars> didrocks, so again, it is an image+layout problem.  Narrowing it down is going to be a really big pain though.  It will take time.
[21:05] <thumper> mars: the utc comes from pytz at the top of the file: from pytz import utc
[21:06] <didrocks> mars: right, it's really not obvious, but at least, the issue is narrowed much more than an hour ago :)
[21:09] <mars> mwhudson, nice change.  Easier to read than before.
[21:11]  * mwhudson pqm-submits
[21:11] <thumper> mwhudson: don't forget to add mars to the r= list :)
[21:11] <mars> :D
[21:12] <mwhudson> thumper: i used ec2 land --dry-run to make the list, never fear
[21:12] <thumper> :)
[21:22] <rockstar> I assume there's no reviewers meeting today...
[21:22] <mwhudson> rockstar: bac cancelled it
[21:23] <rockstar> mwhudson, yeah, I figured.  I actually had something bring up too.
[21:31] <mwhudson> rockstar: gosh
[21:33] <mars> didrocks, I think I am going to call it a night.  Do you think you could file a bug about this problem?  Be sure to attach the screencast.
[21:34] <didrocks> mars: sure, just a bug against LP itself so?
[21:34] <mars> didrocks, for now, until we narrow it down.
[21:34] <didrocks> mars: I'll link to your file too
[21:34] <mars> didrocks, htanks
[21:34] <didrocks> mars: doing that tomorrow morning as well, it's getting late there :)
[21:37] <didrocks> thanks a lot for your debugging too :)
[21:50] <mwhudson> wgrant: congrats on getting your name into a debian-security mail :-)
[21:55] <wgrant> Ah, so that's out now.
[21:56] <wgrant> I assume that production is patched?
[22:09] <NCommander> does anyone know who would be the person to talk to about extending packagesets to be able to work with binary packages in addition to source packages?
[22:10] <wgrant> NCommander: Distro people.
[22:11] <NCommander> wgrant: care to be more specific?
[22:11] <wgrant> NCommander: Core relevant members of the distro team.
[22:12] <lifeless> NCommander: start with cjwatson or persia
[22:12] <lifeless> as a hint
[22:12] <wgrant> Something like that, yes.
[22:13] <NCommander> lifeless: I've been talking with persia, but I need to poke someones brain about the current implementation in launchpad because I'm not quite sure I have a full grasp of its current status and/or limitations.
[22:13] <wgrant> NCommander: Extending it to binary packages is relatively easy.
[22:13] <wgrant> If such functionality is desired.
[22:14] <wgrant> Which it may be, to perhaps obsolete cron.germinate's Task extra-overrides.
[22:14] <NCommander> wgrant: well, talking with persia, to fully replace the current OGRE mode, we need per-binary package set selections
[22:14] <NCommander> wgrant: since we do have source packages who binaries are split between main/universe
[22:14] <wgrant> Ah, yes, true.
[22:15] <wgrant> So, get cjwatson/TB to work out what needs doing, and ask for it.
[22:15] <wgrant> Extending the current implementation to binary packages is not difficult.
[22:15] <NCommander> wgrant: right, but to make a proposal, I need to know where we current stand w.r.t. to backend capability :-)
[22:15]  * NCommander has had some interest in getting involved in soyuz development as a hobby for some time now and has quite a lot of interest in assisting with archive reorg
[22:19] <NCommander> wgrant: correct me if I'm wrong, but the basic way to getting packagesets implemented into LP would be extending IPackageset to have binary methods in addition to the current source methods. The other issue I see is packagesets don't seem to work against series; its an all or nothing things looking at IPackageset
[22:21] <EdwinGrubbs> wgrant: ping
[22:21] <wgrant> EdwinGrubbs: Hi.
[22:21] <wgrant> NCommander: I thought packagesets were series-specific now
[22:21]  * wgrant checks.
[22:22] <wgrant> NCommander: packageset.distroseries
[22:22] <NCommander> wgrant: oops
[22:23] <NCommander> wgrant: so that removes that concern. Then all thats left is a way to assiocate packagesets with a BinaryPackageRelease record so sets can be added like sections/component overrides can
[22:26] <wgrant> NCommander: History is not currently stored for source<->packageset links. This needs to be discussed
[22:26] <NCommander> wgrant: history?
[22:27] <wgrant> NCommander: At the moment, you can see the full component/section/priority history for a source or binary package.
[22:27] <wgrant> You can do no such thing with package set memberships.
[22:27] <wgrant> This could be confusing when it starts being used for ogreing.
[22:28] <wgrant> And package set membership should probably be between a Packageset and a BinaryPackageName, not a BinaryPackageRelease.
[22:29] <NCommander> wgrant: how is history recorded for those?
[22:30]  * thumper fires off an ec2 land before closing laptop
[22:30] <wgrant> NCommander: By keeping multiple (Binary|Source)PackagePublishingHistories.
[22:30] <wgrant> NCommander: They are the things that link an archive-agnostic BPR to an archive, in a particular component, section and (for binaries) priority.
[22:31] <wgrant> When an override is changed, a new [SB]PPH is created, and the old one is marked Superseded.
[22:31] <wgrant> So you can see the full history.
[22:31] <NCommander> wgrant: so we need a SourcePackagesetPublishingHistory(Public), and records to be created for each change
[22:32] <wgrant> NCommander: I don't know. I don't think this has been discussed before.
[22:32] <NCommander> and when binary packagesets materialize, BinaryPagesetPublishingHistory
[22:33] <wgrant> This also makes it just about impossible to change package sets in post-Release pockts.
[22:33] <EdwinGrubbs> wgrant: hi. you were mentioned as somebody that might have insight into the benefits of crowdsourcing the linking of projects to upstream bugtrackers and linking the development focus series to an imported upstream branch.
[22:34] <NCommander> wgrant:?
[22:35] <wgrant> EdwinGrubbs: Yes. At the moment Ubuntu bug triagers can't do any of that, so are crippled and have to go and talk to ~registry or $ARBITRARY_PROJECT_REGISTRANT in order to streamline the upstream integration workflow with projects that do not maintain an LP-hosted presence.
[22:35] <NCommander> wgrant: oh, because packagesets have no concepts of components, right?
[22:35] <wgrant> NCommander: No concepts of *pockets*.
[22:35] <wgrant> They don't need to know about components.
[22:35] <NCommander> s/components/pockets/g :-)
[22:35] <wgrant> But at the moment I can technically promote something in -updates, for example.
[22:35] <wgrant> This is potentially useful, although it needs discussion.
[22:36] <NCommander> wgrant: its been done before
[22:36] <wgrant> It would be unfortunate if this capability was lost.
[22:36] <NCommander> (promotion in a pocket)
[22:36] <wgrant> Right.
[22:36]  * NCommander remembers we had a KDE bug that needed that done, and we had to test on dogfood to make sure Soyuz won't blow up on publishing
[22:36] <wgrant> Then again, somebody has also done a promotion in the Release pocket after a release, which ends up a little odd...
[22:36] <wgrant> Yep.
[22:36] <NCommander> wgrant: LP prevents promotion in Release after release
[22:37] <wgrant> NCommander: Heh, well, it should.
[22:37] <wgrant> But it doesn't.
[22:37] <wgrant> The new publication will just sit there forever.
[22:37] <wgrant> (because the publisher won't touch new records in Release)
[22:37] <NCommander> wgrant: it doesn't?
[22:37] <NCommander> ugh
[22:37] <wgrant> I need to fix that.
[22:39]  * wgrant -> uni
[22:39] <NCommander> wgrant: cya later
[23:42] <mwhudson> gosh lunchtime already
[23:42] <mwhudson> where is the day going