/srv/irclogs.ubuntu.com/2010/03/10/#launchpad-dev.txt

wgrantNCommander: They've been implemented for well over a year.00:00
wgrantAnd permissions use them now.00:00
NCommanderwgrant: ah, I thought there were still bits that required implementation in Launchpad (I saw the IPackageset and friends, but not sure what the status was)00:01
wgrantNCommander: There are a few bits and pieces that don't respect those permissions yet (bug nomination appproval and build retries come to mind), but most of it works and is in active use.00:02
NCommanderwgrant: they are?00:02
* NCommander didn't know this00:02
NCommanderwgrant: so its possible to generate Packages files with Sets: *blah* in them?00:03
wgrantNCommander: That would not be too difficult.00:03
wgrantExcept for the difficulty inherent in the crazy apt-ftparchive integration.00:04
NCommanderwgrant: well, APT has to be patched regardless to understand sets00:05
NCommanderwgrant: is there a way to query sets currently in LP?00:06
wgrantNCommander: Yes, through the API. Check out edit_acl.py in ubuntu-archive-tools.00:07
NCommanderwgrant: ah, I knew about those, just didn't realize those were IPackagesets in the backend00:07
NCommanderwgrant: I thought LP was changed to generating the Packages files directly off the  backend and obseleting apt-ftparchive00:10
wgrantNCommander: Ha ha ha.00:10
wgrantNCommander: It does for PPAs and partner.00:11
NCommanderoops00:11
wgrantBut primary and copy archives still use apt-ftparchive.00:11
NCommanderwgrant: fun00:11
lifelesswgrant: performance?00:11
NCommanderlifeless: probably an issue with Contents-* files and other fun stuff00:11
* NCommander went down this road with dak00:11
wgrantlifeless: Probably not, but it's unclear.00:11
wgrantThe main problem is with overrides.00:11
wgrantNMAF doesn't really support them.00:11
wgrantContents files are another point, but they're generated somewhat separately.00:13
wgrant(it would be nice if LP could generate them natively for PPAs as well, but a table listing all files would be HUGE.00:14
NCommanderInternet doesn't like me at the moment00:17
NCommander<NCommander> wgrant: at the risk of being terrorifed, how do Tasks then end up in the packages file?00:17
elmooverrides00:17
wgrantMagical magical override files.00:17
wgrantNot really generated by LP.00:17
wgrantBut by something that lives in the publisher cronscript.00:17
wgrantelmo: I'm guessing no Soyuz people are around to fix that custom upload bug?00:18
elmowgrant: I guess not, sorry.  if you have a trivial patch, I'm willing to cowboy it on00:19
wgrantelmo: See the branch on the bug. With tests and everything.00:20
elmooh, well, I didn't actually subscribe myself \o/00:20
wgrant(cjwatson's fix was slightly wrong)00:20
elmoaaaaa, LP code ui, you make me cry00:21
wgrantWhat's it doing to you now?00:21
elmojust going from that bug to your actual diff is requiring me to think00:22
elmoand I'm not doing well at that right now00:22
NCommanderwgrant: I just looked at that crontab and ....00:22
* NCommander cries00:22
wgrantNCommander: Yes, cron.publish-ftpmaster is scary.00:23
wgrantelmo: It's easy if there's an MP, which there isn't yet.00:23
NCommanderwgrant: the changes look pretty straightforward, I just need to amend the extras sweep in cron.germinate (which probably will become to cron.extras) to dump Packagesets in-mass, and then let apt-ftparchive handle it00:24
wgranthttp://bazaar.launchpad.net/~wgrant/launchpad/fix-custom-uploads-with-root-dirs/revision/10460 and http://bazaar.launchpad.net/~wgrant/launchpad/fix-custom-uploads-with-root-dirs/revision/1046100:24
wgrantAlthough the latter isn't important.00:24
wgrantNCommander: Right.00:24
elmowgrant: yeah, I foujnd it eventually00:24
elmowgrant: now I'm applying my lack of thinking to the actual diff00:24
wgrantelmo: An issue brought up in https://code.edge.launchpad.net/~julian-edwards/launchpad/symlink-vuln-bug-529710/+merge/20543 is what caused the bug.00:25
NCommanderwgrant: do we support a Supported: field ATM? (I see some references to it, but nothing in publisher)00:25
wgrantNCommander: See maintenance-check.py, and whatever calls it.00:26
elmowgrant: I'm assuming this is just a context thing (i.e. I don't have enough in the diff), but I don't see how this code actually stops $tmpdir/../../../ other than having to guess $tmpdir?00:26
elmooh, duh00:26
elmorealpath00:26
elmonevermind00:26
wgrantRight.00:26
wgrantrealpath resolves symlinks and normpaths, eliminating that hole.00:26
wgrantOnce you've cowboyed it, process-accepted should successfully process those uploads on its next run.00:32
elmoI'm happy enough to cowboy this, given it can only break customuploads00:32
wgrantRight.00:32
lifelesswgrant: all files is only about 2GB of data00:35
lifelesswgrant: (for all files + all packages + per-binary file knowledge + relationships between packages)00:35
elmolifeless: err, not for PPAs00:35
lifelesswgrant: 'conflictchecker' has this, and can query single package conflicts quickly00:35
wgrantAnd not for multiple releases of each source package.00:35
lifelesswgrant: conflict checker does precisely that00:36
wgrantHmm.00:36
lifelesselmo: I don't have the data to estimate for PPAs; if there is interest I could page in conflict checkers schema and figure out what we need to know to estimate it for PPAs.00:37
lifelessby per-binary I mean per-arch00:37
mwhudsonwhen you're talking about HUGE tables in launchpad, bear in mind we have TranslationMessage and BranchRevision :-)00:37
wgrantmwhudson: That's true.00:37
mwhudsonthough the last is totally silly00:37
wgrantBut BranchRevision is stupidly huge.00:37
wgrantYes.00:37
elmoplease tell me BranchRevision is not what I think it is00:38
lifelessfor added fun, conflictchecker was running on sqlite last I saw.00:38
mwhudsonelmo: lalalala fairies00:38
lifelesselmo: its what you think it is.00:38
wgrantelmo: Yes, every branch of Launchpad actually has 70000 BranchRevision rows linked to it.00:38
elmoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA00:38
mwhudsoni wonder if we should look at neo4j for this sort of thing00:40
wgrantHm, maybe.00:40
mwhudsonhowever right now i'm going to have lunch and water the garden00:40
lifelesselmo: what machine is conflict-checker running on at the moment?00:41
elmolifeless: macaroodle or something00:41
elmolifeless: macaroni or macquarie00:41
elmoprobably macaroni00:41
wgrantmacaroni indeed.00:42
elmoyeah, it's the source of constant 1 load on macaroni00:42
lifelesshmm, 3GB now; probably need to ditch some old dev edition packages.00:43
elmowgrant: your patch is on production now00:45
wgrantelmo: I guess we'll see if it works in 16 minutes.00:46
wgrantThanks.00:46
lifelesswgrant: select count(*) from filepath;01:00
lifeless831996101:00
wgrantlifeless: Is that for all SPRs, or just current ones?01:01
lifelesswgrant: thats all filenames seen in binary packages ever.01:02
lifelesswgrant: (but it only scans the official archive)01:02
wgrantHm, not bad.01:02
lifelesssqlite> select count(*) from binpackagefiles;01:03
lifeless2880774301:03
lifeless .schema binpackagefiles01:03
lifelessCREATE TABLE binpackagefiles (package_id INTEGER, file_id INTEGER);01:03
lifelessCREATE INDEX _bpf_fileid on binpackagefiles (file_id);01:03
lifelessCREATE UNIQUE INDEX _unique_packagefile ON binpackagefiles (package_id, file_id);01:03
wgrantI guess we could always purge the lists when we expire PPA sources and binaries, if it becomes a problem.01:04
wgrantelmo: Did it work?01:04
elmowgrant: oh, I wasn't looking01:04
elmowgrant: didn't the previous upload get rejected?01:04
wgrantelmo: Unlikely. It should have just sat in ACCEPTED.01:04
lifelesswgrant: its really quite modest. You need to be careful to use indices, but thats about it.01:04
elmowgrant: ah01:04
wgrantActually, I should be able to check the queue pages...01:04
lifelessif we had the blacklisting done, for handling hand-verified maintainer-scripts, we would probably keep indefinitely ahead of the archive.01:05
wgrantelmo: They're all DONE now, so it probably worked.01:09
elmowgrant: sweet, thanks01:10
wgrantAh, yes, we have binary publications too. All is good.01:10
mwhudsonneo4j looks handy, apart from the j bit01:18
maxbAh. python2.5 has gone from lucid01:39
maxb+copy-packages time01:40
wgrant:(01:40
maxbum. What does it mean when a single source package version has two source publications, one superseding the other?!01:43
maxboh, override change?01:43
wgrantNormally, yes.01:43
maxboh this could be fun. So I need to pick the right one such that the package ends up in the PPA's main :-)01:45
maxbngh01:47
maxbAnd I can't retry builds in the LP PPA01:47
maxbCan someone in ~launchpad hit retry on these, please: https://edge.launchpad.net/~launchpad/+archive/ppa/+build/1553604 https://edge.launchpad.net/~launchpad/+archive/ppa/+build/155360501:54
jelmermaxb: done02:00
maxbthanks02:00
maxbHrm.02:29
maxblaunchpad-developer-dependencies Depends spidermonkey-bin02:29
maxbwhich has just been removed from lucid02:29
mwhudsoni wonder what that's for02:31
* maxb files it under the 'Someone thought it was a good idea' heading :-)02:32
maxbAdded anonymously in 0.35, October 200802:38
mwhudsonsounds like something that could be deleted02:39
wgrantmaxb: PPA publications are overridden to main by a deeper hack, so it doesn't matter which you copy.03:23
thumper:(03:46
thumperspending lots of time reading code03:46
thumperand not so much time writing03:46
* mwhudson is writing some very boring tests04:17
lifeless\\ ?04:18
=== thekorn_ is now known as thekorn
jtvstub: I don't suppose fragment caching works within an arbitrary loop?08:09
stubIt does08:09
noodles775wgrant: hi, shall I land your buildstatus-to-buildmaster branch, or have you already organised it since fixing the test?08:16
noodles775I can't see it in the changelog, so I'll do so.08:17
jtvstub: how does it know which iteration is which in the cache?08:17
stubA counter is stored as a request annotation, and this is used to form part of the key.08:18
adeuringgood morning08:23
noodles775Hi adeuring08:23
jtvhi adeuring, hi noodles77508:26
adeuringhi noodles775, jtv!08:26
noodles775G'day jtv :)08:26
=== gmb` is now known as gmb
mrevellHallo09:03
wgrantnoodles775: I haven't organised it. Thanks.09:06
wgrantnoodles775: Can you please also land my fix-build-breadcrumbs? It failed a couple of tests overnight which I've fixed.09:07
noodles775wgrant: sure.09:08
bigjoolsstub: are you around much longer?  I'd like to talk to you about how to fix https://bugs.edge.launchpad.net/soyuz/+bug/51710109:55
mupBug #517101: update-pkgcache.py needs to be database friendly <Soyuz:Triaged> <https://launchpad.net/bugs/517101>09:55
stubI'm free09:55
stubBack later10:03
maxbwgrant: for +copy-packages too? http://ppa.launchpad.net/maxb/ppa/ubuntu/pool/multiverse/s/sun-java5/ says otherwise :-)10:03
wgrantmaxb: That was ages ago.10:03
* wgrant checks the log.10:03
maxboh, ok. I admit to not having a more recent example10:05
persiabigjools: So, would today be a good day to talk about Copyright stuff for native-source-sync?10:05
bigjoolspersia: yes maybe in an hour?10:06
persiabigjools: That sounds great.  Thanks.10:06
bigjoolsgreat10:07
wgrantpersia: Changelog stuff?10:08
persiawgrant: Yes, thanks for the correction.10:08
jtvhenninge: --force-depends works10:08
wgrantMessy :(10:08
henningejtv: sure ;-)10:09
wgrantmaxb: It was fixed two days after those were published.10:13
bigjoolswgrant: can you propose a merge for your custom upload fix10:20
wgrantbigjools: Sure.10:23
bigjoolsthanks for fixing it, I'll get it landed10:23
bigjoolsI was out yesterday10:24
wgrantThanks.10:24
wgrantUnfortunately I last tested the original fix with a real upload before the os.sep change was added :(10:25
bigjools:(10:26
bigjoolsso I am setting up LP on a new laptop on lucid, and rocketfuel-setup runs aptitide which eventually says launchpad-developer-dependencies depends on spidermonkey-bin ... ?!10:32
wgrantbigjools: Yeah, see the ML.10:33
wgrantIt was removed recently.10:33
bigjoolsok I'm well behind on email :(10:33
wgrantbigjools: Should I specify you as the reviewer?10:34
bigjoolswgrant: yes10:34
persiaIs there any specific reason rocketfuel-setup runs aptitude?  I've heard the aptitude resolver isn't widely tested (and seen any number of bugs for stuff that didn't work with aptitude that did work with other resolvers)10:37
bigjoolsI suspect whoever wrote that part of the script just happens to also prefer it10:46
bigjoolswe should change it10:46
=== matsubara-afk is now known as matsubara
=== noodles785 is now known as noodles775
thumperdamn10:57
thumpercall in 6 hours10:57
thumperbest go to be10:57
thumperd10:57
deryckMorning11:00
wgrantDo the Canonical webapp engineering teams not talk to each other?11:09
wgrantI have found the same obvious hole in three Canonical webapps in the past two weeks.11:09
bigjoolswgrant: not as much as we should, we're setting up an architectural team to address that11:13
jtvwgrant: not getting my Bob to pick up the jobs... are you sure I don't need to run anything?11:16
wgrantjtv: As long as buildd-manager is working, it should be working. What do buildd-manager's logs say? What does an XML-RPC status() call on the builder say?11:18
jtvwgrant: hmm... in /var/log/launchpad-buildd/default.log I see it shutting down after a SIGTERM, and nothing after that11:19
wgrantjtv: Heh, you might want to start it.11:19
jtvyes, I might11:19
jtvwgrant: time to remove the -e I added to that helper script  :)11:20
wgrantA good idea.11:20
jtvstill no joy though :(11:22
wgrantWhat does buildd-manager say?11:22
jtvwgrant: nothing in the log, nothing in the shell11:23
jtvI tell a lie11:24
wgrantjtv: Hmph. The builder is responding with IDLE?11:24
jtvstartup failed because there's "a twistd" already running11:24
wgrantHmm.11:25
jtvyup, buildd manager was still running.11:25
jtvI'll try killing that instance and restarting11:25
persiabigjools: Let me know when you have time :)11:25
stubHow often will bug heat get refreshed do we think?11:25
bigjoolspersia: right now11:25
persiabigjools: Great.  So my current understanding is that there are 3-4 things involving changelogs that need to be sorted before we can turn on native-source-sync, and that you know which these are.  Is this true?  If so, could you expand?11:26
bigjoolspersia: my memory is very hazy on this, I'll see if I wrote something down11:26
bigjoolsok so there's a few bugs11:27
bigjoolshttps://bugs.edge.launchpad.net/soyuz/+bug/5579511:28
jtvwgrant: I have a "starting scanning cycle" in the buildd-manager log, but nothing else.  :)11:28
mupBug #55795: +changelog includes misleading information related to package versions and authors <soyuz-core> <soyuz-upload> <ui> <Soyuz:Triaged> <https://launchpad.net/bugs/55795>11:28
wgrantjtv: The builder is still marked OK, and not manual?11:28
bigjoolshttps://bugs.edge.launchpad.net/soyuz/+bug/13916211:28
mupBug #139162: Store the pristine debian/changelog for each SourcePackageRelease <soyuz-core> <soyuz-upload> <Soyuz:Triaged> <https://launchpad.net/bugs/139162>11:28
wgrantjtv: Can you see buildd-manager's polls in /var/log/launchpad-buildd/default?11:28
bigjoolshttps://bugs.edge.launchpad.net/soyuz/+bug/24745611:28
mupBug #247456: changelog listing are incomplete and/or undiscoverable <soyuz-core> <ui> <Soyuz:New> <https://launchpad.net/bugs/247456>11:28
bigjoolspersia: basically it's horrible right now11:28
jtvwgrant: the builder is OK and not manual11:29
wgrantAlso we need to be able to create something akin to a changes file.11:29
jtvwgrant: it's not logging to there, it turns out; it's logging to /var/tmp/development-buildd-manager.log11:29
jtvAll I see there is "Starting scanning cycle."  Lots of times.11:30
wgrantjtv: That's buildd-manager. I want launchpad-buildd.11:30
jtvahh11:30
persiabigjools: There's a few other bugs surrounding that which I've seen, but yeah.  So do we actually need the changelog for anything, or could we just store it as requested in 139162 and present it as requested in 55795?11:30
jtvwgrant: I've been looking at /var/log, but I guess that's on the slave.  :)11:31
bigjoolspersia: 139162 is the first task11:31
jtvwgrant: 2010-03-10 11:31:18+0000 [HTTPChannel,910,127.0.0.1] 127.0.0.1 - - [10/Mar/2010:11:31:18 +0000] "POST /rpc HTTP/1.0" 200 222 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)"11:31
jtvwgrant: repeats11:31
wgrantjtv: OK, so it is polling.11:32
wgrantSo maybe the candidate selection is not working properly.11:32
bigjoolspersia: once we've done that the NSS becomes easier11:32
jtvwgrant: I guess.  I was hoping to be able to stay out of that.  :)11:32
jtvBut oh well. :)11:33
wgrantjtv: Check your postgres log for the query that buildd-manager is using.11:33
persiabigjools: Right.  So we have code that does 139162 that we use for the changelog download in apitutude and update-manager, storing stuff on changelogs.ubuntu.com.11:34
wgrantRun that, and work out why it isn't returning your job.11:34
wgrantIf it is, there is something more sinister going on.11:34
bigjoolspersia: we also need to repackage existing records11:34
persiabigjools: How do you mean "repackage"?11:35
jtvwgrant: there's no extra-verbose logging in the script that I can enable?11:35
wgrantbigjools: Can you answer jtv? You know the logging much better now.11:35
bigjoolspersia: the existing changelogs need to be re-processed11:35
jtvbigjools: I'm trying to figure out why the buildd isn't picking up my jobs.11:36
persiabigjools: Is there any useful information there, or do we just need to extract changelogs from the existing packages?11:36
bigjoolsjtv: got a b-m log?11:36
bigjoolspersia: AIUI that's pretty much it11:37
jtvbigjools: just says "Starting scanning cycle." all the time.11:37
bigjoolsjtv: then your job is not eligible for dispatch (either a bug or genuinely)11:38
jtvbigjools: so as wgrant just pointed out, sounds like candidate selection.11:38
persiabigjools: And then 55795 becomes easy because we can just expose that blob?11:38
jtvbigjools: right, so my question is: is there debug logging in the script that'll make this easier to pinpoint?11:38
bigjoolsjtv: see the query it's issuing in the PG log11:38
bigjoolsand go from there11:38
jtvbigjools: that's what he said.  OK, I'll try that, thanks.11:38
bigjoolsjtv: there's no debug logging that will help :(11:38
bigjoolsit's the candidate selection query11:38
bigjoolspersia: exactly11:39
wgrantWhich I know works generically now, but you may be missing a piece for your translations job.11:39
bigjoolspersia: the way attribution is done is all messed up as well11:39
bigjoolsyep11:39
persiabigjools: That goes away if we use mvo's code for 13916211:39
bigjoolspersia: it would need to work in LP, but that's the general idea yeah11:40
persiabigjools: And then 247346 gets resolved trivially as well?11:40
bigjoolswith any luck :)11:40
persiaNow, how ought this be implemented in LP?11:40
bigjoolsjtv: paste the query from the pg log into a psql prompt and experiment11:40
jtvbigjools: thanks...  for now I'm just checking it manually against the data I have.11:41
bigjoolspersia: depends on what that code looks like, I've not analysed any of this11:41
jtvbigjools: ahhh, got it: "(buildqueue.virtualized IS NULL OR 'f' IS TRUE)"11:41
jtvI reviewed that code, so it can't be wrong.  :-)11:41
bigjoolsthe classic :)11:41
persiabigjools: The code basically just knows the format of a .deb, and extracts the changelog.11:41
persiabigjools: That logic can be embedded however is convenient.11:41
persiaWe can do the same for a .dsc, but we need an unpack environment.11:42
bigjoolswell it already does it for dsc, just not very well11:42
bigjoolsthe upload processor unpacks the source11:42
wgrantWe already have code to grab debian/copyright.11:43
jtvbigjools: it does actually make sense, it's just that it reflects... IIRC whether the buildd manager is looking for arch-independent build candidates.11:43
wgrantWhich I believe was touched last week...11:43
bigjoolsindeed11:43
persiaSo really, that code should just spit debian/copyright into the DB?11:44
persiaAnd then we can expose it?11:44
bigjoolsit does already11:44
bigjoolswe just don't expose it11:44
persiaSo what's missing in 139162 again?11:44
bigjoolshttps://dev.launchpad.net/Soyuz/Specs/RawSourceChangelog11:44
bigjools^^ that11:44
persiaI don't see the missing bit, if we're already stuffing full raw changelogs into the DB.11:47
bigjoolswe're not, we're extracting bits of it11:47
persiaOr is it just the exposure and client changes?11:47
persiaAha!11:47
bigjoolsand then trying to re-attribute them11:48
wgrantWe store debian/copyright, not debian/changelog yet.11:48
bigjoolsit all blows quite spectacularly11:48
bigjoolsright, we store changelog_entry which is a partial changelog11:48
* persia slaps forehead11:49
persiaRight.  So we just need to pull debian/changelog *when* we pull debian/copyright.11:49
persiaNow, do we actually need any of the code that attempts to construct a changelog, or can we scrap that entirely?11:50
bigjoolsI'm not sure yet, I suspect it could be scrapped but bits of it might get re-used in the presentation11:50
bigjoolsstub: ok can we talk about pkg-cache now?11:51
stubyup11:51
bigjoolsstub: where does the problem lie, in the fact that it's reading too quick?  or udpating?11:52
persiabigjools: Thanks!11:52
bigjoolsor both?11:52
bigjoolspersia: no worries!  I hope it's filled a gap11:52
stubupdating too quickly without regard to the system load11:52
bigjoolspersia: and sorry it took me so long to get back to you11:52
bigjoolsstub: so I could put a loop tuner at the top level where it reads package names in11:52
bigjoolswhich would also throttle writes11:52
stubIdeally it should block when the system is busy. That logic is centrally encoded in DBLoopTuner at the moment.11:53
bigjoolsyeah11:53
stubI haven't looked at the existing structure of the script11:53
bigjoolsstub: it's a wrapper around various bits11:53
bigjoolsstub: it calls IDistribution.updateCompleteSourcePackageCache11:54
bigjoolsI need to move that out of that model code and into soyuz somewhere11:54
bigjoolsthen I think I can just slap a loop tuner on there and profit11:54
stubSo updateCompleteSourcePackageCache sounds like a chunk that should be its own DBLoopTuner loop11:54
bigjoolsexactly11:55
stubYou could install it into garbo-daily.py if you like - this fits in with its role.11:55
* stub looks at updateCompleteSourcePackageCache11:56
stubYer - doesn't look like much conversion to turn that loop in updateCompleteSourcePackageCache into a DBLoopTuner loop.11:57
jtval-maisan: wgrant, bigjools: looks like my job should have "virtualized" set to true in order to be selected by a non-virtualized builder11:59
wgrantjtv: Didn't I convince you to set bob to virtualised?11:59
al-maisanjtv: not setting it (i.e. a NULL) will be eqivalent to virtualized=True12:00
wgrantYou should set virtualized to either True or None in order for it to build on a virt builder.12:00
wgrantFalse will let it build on non-virt.12:00
jtvwgrant: I may easily have gotten confused between the setting on the builder and on the jobs.12:00
jtvwgrant: ah, I see it: I skipped a step this time that I had documented12:02
wgrantAhh.12:02
jtvsorry for the noise12:02
jtvnow I guess I wait for it to poll...12:02
wgrantIt should poll every five seconds.12:03
=== mrevell is now known as mrevell-lunch
wgrantbigjools: Thanks for landing that.12:41
wgrantnoodles775: I see that buildstatus-to-buildmaster should be landing RSN, but I have no EC2 email about the other one yet.12:42
bigjoolswgrant: welcome12:43
bigjoolsis someone already fixing the spidermonkey-bin problem BTW?12:44
wgrantbigjools: Do you have time to talk about PPA download stats?12:44
bigjoolswgrant: sure12:44
wgrantso, bug #13985512:45
mupBug #139855: Display stats about PPA usage <feature> <Soyuz:Triaged> <https://launchpad.net/bugs/139855>12:45
wgrantThe only consensus seems to be that we want to store binary download counts like we do LFA downloads.12:45
wgrantIndices probably want to be counted somehow too, but what to do with that is much less obvious.12:46
wgrantI have a working implementation of binary counts.12:46
bigjoolsI tihnk that doing the binary counts is a very good first step12:46
bigjoolsdo you want me to take a look at it?12:47
noodles775wgrant: checking the ec2 instance...12:48
noodles775wgrant: it seems to be finishing (it's not yet terminated, but I can no longer access it), so I'll wait a bit. If I've not seen anything by my EOD, i'll resend it.12:50
wgrantnoodles775: OK, odd...12:50
wgrantAt least the other one should be in PQM.12:50
noodles775wgrant: it is (playing now).12:51
wgrantbigjools: I think the backend branches (https://code.edge.launchpad.net/~wgrant/launchpad/parse-ppa-apache-logs is the final one) should be reasonably unobjectionable.12:54
wgrantBut https://code.edge.launchpad.net/~wgrant/launchpad/export-basic-binary-download-stats and its successor https://code.edge.launchpad.net/~wgrant/launchpad/export-detailed-binary-download-stats might well be horribly slow.12:55
bigjoolswgrant: ok, file an MP and request me as a reviewer and I'll look at that one12:55
bigjoolscan you paste a diff?  I'm lazy :)12:55
wgrantbigjools: The combination is not quite oversized, but it has two logical prereqs which will make it smaller. Shall I propose them separately?12:59
bigjoolswgrant: I'm a little confused, you have three branches and the first one is not controversial, right?  but the other 2 might be?  so I was asking for diffs for those13:00
wgrantbigjools: The first uncontroversial one is really three branches. You just want the two that might be? That's easy.13:01
* wgrant diffs.13:01
bigjoolswgrant: well if it's really big we should split it up and get more reviewers involved13:02
wgrantbigjools: It's not really big. The first branch I mentioned comes it at around 800 lines -- it's just probably a bit easier if it's split into its three logical components.13:03
bigjoolsok13:04
wgrantnoodles775: Ah, that other run emailed a couple of minutes ago. Great.13:10
noodles775Excellent.13:10
=== henninge_ is now known as henninge
=== matsubara is now known as matsubara-afk
wgrantbigjools: A few criss-cross merges and lots of conflicts later: http://pastebin.ubuntu.com/392515/ and http://pastebin.ubuntu.com/392521/13:23
bigjoolsheh13:23
wgrantThe first just exports a single count per BPPH, the second gives detailed data.13:24
bigjoolswgrant: if it's expensive, don't make it a property13:30
bigjoolsdownload_count that is13:31
wgrantbigjools: Yeah, I guess not.13:31
leonardrwho should i speak to about the design of html pages for launchpad? is it still mpt?13:31
bigjoolsit will get serialised on every fetch13:31
bigjoolsleonardr: good point - it was beuno but he's moved to U113:32
bigjoolshe might still want to help :)13:32
wgrantI'm not sure how expensive it will be. It could be a SUM across several hundred rows of a single-table join.13:32
bigjoolsI want to avoid making properties that hide complex queries anyway13:33
bigjoolsit's not really a property :)13:33
wgrantThis is true.13:33
asabilhi all13:36
bigjoolswgrant: I think sum is pretty quick anyway13:37
bigjoolswgrant: but I wonder if we shouldn't just store that total to avoid doing it?13:38
* bigjools thinks about splitting up archive.txt13:40
wgrantbigjools: Is archive.txt the 3000 line one?13:41
bigjoolsyarp13:41
bigjoolsit's useless as documentation13:41
bigjoolsand lots of it can be a unit test13:41
wgrantbigjools: We could, but there's no convenient table representing (archive, bpr)13:41
wgrantRight.13:42
bigjoolspresumably you've got a schema change hiding somewhere?13:43
wgrantIt's in the previous branch.13:44
* wgrant diffs.13:44
bigjoolsok13:44
bigjoolswhat is the .count counting then?13:44
bigjoolsSum(BinaryPackageReleaseDownloadCount.count)13:44
beunobigjools, I always want to help13:44
bigjoolsif we're counting total downloads per archive we can add a column to Archive13:45
bigjoolsbeuno: yo da man13:45
wgrantbigjools: Like LFDC, it maps (target (in this case (archive, bpr)), date, country) -> count13:46
jmlI'm sorry, I wasn't paying attention. Did I hear someone say they were adding download counts for PPAs to Launchpad?13:46
bigjoolsheh13:46
wgrantbigjools: http://pastebin.ubuntu.com/392534/ is the work before the two diffs I gave above.13:48
bigjoolswgrant: so we could use another column on IArchive for total download count and maintain it in the log scraper?13:48
jmlleonardr, there's no one person with responsibility for that any more. You consult the launchpad-dev list, basically.13:48
leonardrjml: ok13:48
wgrantjml: Has that been shown to work at all yet?13:48
jmlwgrant, asking launchpad-dev for advice?13:48
wgrantbigjools: But counts per archive, per (archive, bpr) and per (archive, bpr, day) are useful.13:48
wgrantWe need a generic cache mechanism :(13:48
leonardrwell, i'm going to write something, and if launchpad folks don't like it they can lump^H^H^H^Hfix it13:48
wgrantjml: For UI advice.13:48
jmlwgrant, how would you tell if it worked?13:49
bigjoolswgrant: yes I'm saying to do it instead of that new property13:49
bigjoolsso you can avoid the Sum13:49
wgrantjml: Well, given that LP now has no UI designers, and nobody from UX has replied to the RFC-UI emails on the list up to now, it seems like it probably won't.13:50
bigjoolswgrant: the other thing is that exposing database IDs on URL traversals makes me a bit sad13:50
wgrantbigjools: But that's on BPPH, so it needs a download count cache on (archive, bpr), which is a tuple that the DB doesn't know about.13:50
wgrantbigjools: Yes :(13:51
jmlwgrant, although I would highly value contributions from Canonical UX, I think that the broader launchpad developer community almost always has useful things to say about UI design.13:51
bigjoolscan you traverse with the package name/version instead?13:51
wgrantbigjools: Possibly. I suppose they should generally be unique.13:52
bigjoolswgrant: IArchive.getPackageDownloadTotal() - we can replace that with a new column on Archive13:52
wgrantAnd any cases in which they are not are bugs.13:52
bigjoolsunless I am misreading something?13:52
wgrantbigjools: It takes a BPR as an argument.13:52
bigjoolsyes I am misreading :)13:53
bigjoolsso that's the total of all versions of that package in this archive13:53
wgrantAll versions? A BPR is one version.13:53
wgrantAll publications, yes.13:54
bigjoolsall published versions is what I meant13:54
wgrantRight.13:54
bigjoolslooks great13:54
bigjoolsjust needs to be a method instead of a property on publishing13:55
wgrantbigjools: Great, thanks. It is getting late, so I will fix that tomorrow. Shall I then request reviews from you for the three, or should I use OCR now you're obviously not too unhappy with the implementation?13:57
bigjoolswgrant: the traversal should not use the ID, other than that the approach looks great, thanks!13:58
wgrantOh, right, yeah.13:58
=== mrevell-lunch is now known as mrevell
=== jamalta-afk is now known as jamalta
kfogeladeuring: re bug #532022, I'm just starting to figure out where to add the criterion to suppress bugs whose status is "Fix Released".  Is this something that would go into those special DB queries you made for +patches?14:19
mupBug #532022: bugs with patches shows fix released bugs as well <Launchpad Bugs:In Progress by kfogel> <https://launchpad.net/bugs/532022>14:19
adeuringkfogel: we call bugtarget.searchTasks() with a set of status values. I think it is enough to change this set of values, let me look...14:21
kfogeladeuring: ah, heh -- that'd be easy.  But let me check: it *might* be the case that there's some big efficiency to be had by doing the filtering at a lower level, not sure.14:23
adeuringkfogel: right, we call searchTasks(status=UNRESOLVED_PLUS_FIXRELEASED_BUGTASK_STATUSES, ...), so this value should be changed.14:23
kfogeladeuring: okay, good.  In fact, I think I may have created that combination status explicitly for this, so if this is the only use of it, it can go away too.  I'll check.14:23
adeuringkfogel: no, I don't think so. If the current query works, another one with a smaller set of status values should be fine14:23
adeuringkfogel: yes, it can probably go again, if it's used only here..14:24
kfogeladeuring: right, because the raw query that goes into the DB starts here (i.e., it's not like we're getting a bigger set of results back and then filtering down).14:24
adeuringkfogel: right14:24
kfogeladeuring: just talking out loud partly to make sure my understandings are right.  Thanks for reading :-).14:24
adeuringkfogel: there are one more changes:14:24
adeuringthe number shown in the link to the +patches view14:25
kfogeladeuring: oh, that's not connected to the searchTasks query?14:25
kfogeladeuring: (I'd already forgotten how we implemented that number.)14:25
adeuringand the test that ensures that this number euals the number of patches shown in the +patches view14:25
adeurings/euals/equals/14:26
kfogeladeuring: thanks for the reminder -- will take a look.14:27
kfogeladeuring: but anyway I can base my branch for this off of devel, not db-devel.14:27
adeuringkfogel: sure, using devel is fine14:27
adeuringkfogel:  lib/lp/bugs/browser/bugtask.py: BugsStatsMixin.bugs_with_patches_count14:28
adeuringkfogel: and the test is: lib/lp/bugs/browser/tests/test_bugtarget_patches_view.py14:29
kfogeladeuring: thank you.  It's starting to come back to me now...14:29
kfogeladeuring: I have a tremendous ability to wipe my mind clean after finishing something.  I've often thought I could give lessons to trauma victims.14:29
adeuring(see also bzr diff -r10453..1045414:29
adeuringkfogel: sounds interesting ;) I could help there perhaps too... I had to look through "bzr log" to find the right revision, and then looked at the diff ;)14:30
kfogeladeuring: yeah, but I actually wrote this code, like, a week ago IIRC :-]14:31
kfogelwgrant: hey, I will review your branch today14:32
kfogelwgrant: "today" for me means in the next eight hours (I'm in New York)14:32
=== matsubara-afk is now known as matsubara
marsgary_poster, ping, trying to work around a buildout dependency.15:09
gary_postermars, what's up15:09
marsgary_poster, I'm seeing 'mercurial' listed in the branch root's setup.py, and I do not see mercurial in versions.cfg.  Is that to be expected?15:09
marsgary_poster, and bin/buildout is spitting errors at me because of mercurial for some odd reason.15:10
gary_postermars, no, doesn't sound right.15:10
marsweird for a 3 line change...15:10
gary_postermars, investigating.  this is devel?15:10
marsgary_poster, yes, fresh branch yesterday15:11
gary_posterk15:11
marsgary_poster, line 46, setup.py15:11
marslooks like it is needed by code hosting.  Maybe a system package is supposed to install it?15:12
gary_postermars, typically still should have a version15:12
marsah15:12
gary_postermars, jumping around a bit, but trying to pull devel now.  In regards to your hypothesis, have you checked to see if launchpad-developer-dependencies are up to date for you?15:14
marsgary_poster, chekcing15:17
=== jamalta is now known as jamalta-afk
marsyes, they are up to date15:18
gary_posterok15:18
gary_posterI'm now building Twisted 10...15:18
marsgary_poster, here is what I am getting on the CLI when I run bin/buildout:15:21
marshttp://paste.ubuntu.com/392596/15:21
gary_postermars, ok, I just built fine...investigating setup.py and versions.cfg next, and will look at your pastebin15:22
gary_postermars, mercurial is in my devel's versions.cfg15:22
marsline?15:23
gary_postermars, the error you are getting is exactly because it is not in your versions.cfg15:23
marshehe15:23
gary_posterwe have a setting to cause that15:23
gary_postermars, line 3915:23
gary_postermechanize = 0.1.1115:23
gary_postermercurial = 1.3.115:23
gary_postermocker = 0.10.115:23
marsgary_poster, ok, thanks, I am missing that line.  More detective work then15:24
gary_postermars, ok cool15:24
deryckUrsinha, ping15:25
Ursinhaderyck: hi15:25
=== jamalta-afk is now known as jamalta
deryckUrsinha, hi.  Could the script that does qa tagging, which links the milestone if needed, also link the branch if it's not linked yet?15:26
deryckUrsinha, not to give you more work even ;)15:26
Ursinhaderyck: hmmmmmmm I'll see what I can do :) I have to finish polishing the burndown chart, and have other task after that, but I put that in the queue, sure :)15:27
Ursinha*I'll put15:27
deryckUrsinha, np.  Thanks for considering it! :-)15:28
Ursinhaderyck: no problem :)15:28
marsgary_poster, found the problem, thanks for the help.15:28
gary_postermars, great15:29
adiroibanhi, any idea why I get this error on launchpad.dev when I try to login ? http://paste.ubuntu.com/392604/ ?15:35
marsadiroiban, something to do with the new OpenID changes perhaps.  Maybe check the config variable listed on line 15 of your post?15:37
salgadoadiroiban, need to add testopenid.dev to /etc/hosts15:38
adiroibansalgado: true. thanks!15:39
marssalgado, if he re-runs rocketfuel-setup, will that happen automatically?15:39
salgadomars, yes15:39
salgadoactually, maybe nt15:39
marssalgado, so now we all have to do that to use launchpad.dev?15:39
salgadonot15:40
salgadomars, yes15:40
salgadoadiroiban, mars, yes, running rocketfuel-setup should add it there15:41
marssalgado, have we sent a mail to launchpad-dev about having to run rf-setup to add that host?15:41
mars"Please re-run rocketfuel-setup" or something15:42
salgadono, I didn't.  forgot about that15:43
salgadomars, do you think it's still worth to do so?  I suspect most people have already found that out or they run rocketfuel-setup frequently and didn't notice it at all15:44
marssalgado, depends on how long ago the change landed.  It could be that most people don't log out and in again.  And sending it can't hurt :)15:45
salgadoindeed.  although it's landed 2 weeks ago, it won't hurt to send an email15:47
marsthe more people that know about the problem and the answer, the fewer times you will have to answer it yourself :)15:49
=== jamalta is now known as jamalta-afk
=== salgado is now known as salgado-lunch
=== salgado-lunch is now known as salgado
=== abentley_ is now known as abentley
=== beuno is now known as beuno-lunch
=== Ursinha is now known as Ursinha-lunch
didrocksLaunchpad screen refresh being very slow is a known issue? For instance, switching tabs in firefox, 2 to 4 seconds to refresh the page, without hitting F5; opening option bug in edit -> options takes 3 seconds to have it refreshed too18:14
didrocksand this, only if I am on a LP page current tab18:14
didrocksno issue with other sites in other tabs18:14
didrocksapparently seb128 has the same issue too18:14
didrocksI tried in edge/production server18:15
didrocksand disabling greasemonkey too18:15
didrocks(just to be sure, I'm not speaking about loading a page, just having something on a already loaded page)18:17
deryckdidrocks, I'm not familiar with this as a known issue... maybe mars knows something?18:19
didrocksI would think the page content is not supported by either xulrunner or if there is an ajax things "on activate"18:19
mars?18:22
marsdidrocks, do you have a specific page you were trying this on?18:22
=== beuno-lunch is now known as beuno
didrocksmars: it's an all LP page for both seb128 and I (particularly bugs page). I can make a short screencast if you want to see the difference18:23
marsdidrocks, ok, if that is easiest :)18:24
marsdidrocks, do you mean, all LP pages on edge have this problem?18:24
didrocksmars: no, LP edge and production server (non edge)18:25
didrockswith firefox on lucid18:25
rockstarleonardr, ping18:29
leonardrrockstar, hi18:30
rockstarleonardr, abentley and I were talking about bug 520446 this morning.18:30
mupBug #520446: OOPS creating duplicate merge proposal using API <api> <oops> <trivial> <Launchpad Bazaar Integration:In Progress by rockstar> <https://launchpad.net/bugs/520446>18:30
leonardrok18:30
rockstarleonardr, basically, the fix is to catch the (more meaningful) exception and raise a ValueError.18:31
rockstarleonardr, have you seen this kind of problem before?18:31
didrocksmars: http://people.canonical.com/~didrocks/slowLP-1.ogv that should explain the issue :)18:31
marsdidrocks, looking18:31
leonardrrockstar: so the problem is that a specific exception results in an oops when it should result in a 400 status code?18:32
leonardryes, that happens all the time18:32
leonardryou can annotate the exception with @webservice_error(400) and it will be handled correctly if the error happens during field validation18:33
=== gary_poster is now known as gary-lunch
leonardrif it happens in named operation invocation i don't think that will work18:33
marsdidrocks, so Chrome is fine18:34
didrocksmars: right18:34
didrocks(it's chromium, but the same base ^^)18:34
rockstarleonardr, what do you mean annotate the exception?18:34
leonardrrockstar, let me try to find an example18:35
marsdidrocks, could you try that tab-switching test with the gnome system monitor open?  To the Resources page, with the graphs on it18:35
=== bjf is now known as bjf-afk
didrocksmars: sure18:35
leonardri have a feeling that this is used all over launchpad but maybe no one is using it18:35
marsdidrocks, should show a CPU spike, if there is one18:35
rockstarleonardr, it seems like it should be used all over launchpad at least.  :)18:35
marsdidrocks, no need to record the CPU, just let me know if it spikes while the browser is hung18:36
didrocksmars: right, one CPU is going to 100%18:36
didrocksfor the 3-4 seconds18:36
marshmm18:36
rockstarleonardr, I found some examples.18:36
rockstarlib/lp/hardwaredb/interfaces/hwdb.py18:36
marsdidrocks, thinking, who's the culprit?  The FF JS engine?  Or the FF rendering engine?18:37
leonardrrockstar: yeah, try that and see if it works18:37
marsdidrocks, I don't think it is any JS on our page, but can't rule that out entirely either.18:37
didrocksmars: I don't know enough Firefox to have a strong opinion on that. I can ask chriscoulson who is our new maintainer18:38
didrocksmars: can I tell him to see that with you (tomorrow probably)18:38
didrocks?18:38
marsdidrocks, sure.  You say you have confirmed the behaviour with someone else using FF+Lucid18:38
didrocksthat's just pretty obvious in lucid and only affecting LP pages. So, there is something that FF doesn't like :)18:38
mars?18:38
didrocksmars: right, I was thinking about a local profile issue first, but seb128 begins to have the same behavior18:39
marsdidrocks, ok, I can think of a test: I just need to find a relatively JS-free page first18:39
didrocksmars: sure, going to get my dinner, you can ping me, bbiab18:40
marsdidrocks, ok18:40
rockstarleonardr, oh man, MUCH better solution.  Thank you!18:41
leonardrawesome18:42
didrocksmars: so, chrisccoulson is our new firefox rockstar :)19:05
chrisccoulsonheh19:05
didrocksmars: do you have any idea of tests I can do?19:05
didrockschrisccoulson: did you see the video?19:05
chrisccoulsoni did19:05
didrockschrisccoulson: you can see how my life is hard nowdays :-)19:06
marshi chrisccoulson19:19
chrisccoulsonhi19:19
marsdidrocks, looking around19:19
marschrisccoulson, did didrocks show you his screencast with the strange behaviour?19:20
chrisccoulsonmars - he did19:20
chrisccoulsoni've not noticed behaviour like that before though19:20
marschrisccoulson, I was thinking it may be the Firefox JS engine, the rendering/paint engine, or maybe something in our JavaScript itself that doesn't play nice with the new FF JS engine.19:21
marschrisccoulson, but I don't have the FF profiling tools at hand to narrow it down.  I'm trying to find a non-js page to test on.19:21
marswould rule out the paint operation19:21
didrocksmars: I mentionned also to chrisccoulson that I don't have accelerated GPU driver. That should emphasize, but as we can see, chromium plays a better role there19:22
marsdidrocks, try it with this page: https://code.launchpad.net/oops19:24
marsthat is the 404 page: quite minimal, and no javascript19:24
marsbit of a dumb test, but better than nothing to start I guess19:24
didrocksmars: no issue with this 404 page19:24
marsdidrocks, and https://code.launchpad.net/  ?19:25
didrocksits fast to say "you screw something" :)19:25
didrocksmars: I have a delay19:25
didrocksnot as long as on a bug page19:25
marsinteresting19:25
didrocksbut still, 1.5s I would say19:25
marsdidrocks, are you on a 64bit system?19:26
didrocksmars: no19:26
didrocksmars: google reader and gmail are quite fast19:26
didrocks(for the js part)19:26
marschrisccoulson, ^ fun FF bug for you - our automated integration testing tool using FF goes nuts and eats 100% CPU if our uncompressed JS exceeds 512Kb, but only on  64 bit systems.  Not a problem here though :)19:27
chrisccoulsonsorry, i just tried using venkman to profile the javascript19:29
chrisccoulsoni need to figure out how to use it really though ;)19:29
marsdidrocks, ok, so it is *not* the invisible iframe we use for the help system.  That is evidence against a DOM issue.19:29
marshmm19:30
marsGoogle PageSpeed could tease out the issue.  It hooks right in the guts of firefox19:30
marschrisccoulson, I take it you see the same behaviour on your system?19:31
chrisccoulsonmars - i don't19:31
chrisccoulsonwhich is a bit of a pain ;)19:31
mars...19:31
didrockschrisccoulson: can you try to switch to a 2D driver?19:32
didrocksmaybe that'll be more obvious19:32
marsdidrocks, do you see this behaviour when you run in -safemode (I think that's the switch?)19:32
didrockson seb128's laptop is something like 2s19:32
chrisccoulsondidrocks - i could do, but that might have to wait until later19:32
didrockstrying19:32
mwhudsonmorning19:32
marsmorning mwhudson19:32
marsdidrocks, ah: "firefox -safe-mode"  http://support.mozilla.com/en-US/kb/Safe+Mode19:33
marsforgot the dash19:33
didrocksmars: a little bit better, only 1.5 - 2s now. What's strange is that the only extension I have is the ubuntu one (ubufox)19:35
marschrisccoulson, didrocks, since I don't have access to a proper FF profiler (though I have read about them), it maybe be possible to use the PageSpeed extension to narrow the problem down to JS or the browser: http://code.google.com/speed/page-speed/docs/using.html#activities19:36
chrisccoulsonyeah, i just installed that19:36
chrisccoulsonit looks pretty neat19:36
marsdidrocks, do you see the delay/hang when you first open a page?19:36
marsdidrocks, or just when visiting a tab?19:37
didrocksmars: as it's loading at the same time, it's hard to tell19:37
didrocksmars: but in the video you can see that when firefox has something to show (like the option dialog box), there is the same hang19:37
marsdidrocks, the same hang?19:38
kfogelI have never seen this happen before: I did 'bzr branch devel 532022-patches-view-omit-fixreleased' to create a working branch, and cd'd into the new branch and did 'utilities/link-external-sourcecode ../devel'.  But for whatever reason, my new branch does not have a bin/ directory at the top level!19:38
* mars re-reads the last word19:38
didrocksmars: the same delay before showing the dialog box19:38
marsdidrocks, ok.19:38
marsdidrocks, chrisccoulson, if you see the same delay when showing a dialog, then I suspect two things19:39
salgadokfogel, need to 'make build' -- that's automatically done by rocketfuel-branch19:39
kfogelsalgado: aaaah.  thanks19:39
marsdidrocks, chrisccoulson: 1) The browser paint and render code is messing up in a bad way19:39
marsdidrocks, chrisccoulson: 2) There is a focus/blur event that our JavaScript is processing and choking on19:40
didrockschrisccoulson, mars: showing a dialog only over a LP page, I must precise19:40
marsI don't know enough about the Firefox 3.6 architecture to say if the JS execution thread has been decoupled from the application threads: to say if it is possible for JS to block page paints19:42
=== gary-lunch is now known as gary_poster
marsdidrocks, it is only on tab focus/blur, not window focus/blur?19:42
didrocksmars: you mean, other windows?19:43
didrocksmars: if I, for instance, move a gnome-terminal over it, I get the same issue19:44
marsdidrocks, yes.  If you have the tab open, click on another window, click back on Firefox19:44
marschrisccoulson, ^ !19:44
didrocksseems that FF has some issue to just redraw the LP page. It's just strange that it only happens with LP and not on other "big page"19:44
marsdidrocks, so it chugs if you move another window of the Firefox painted area19:45
didrocksmars: exactly19:45
marsok, so that means it is probably not the JavaScript events19:45
marsIt is the Firefox rendering engine trying to paint the screen19:46
marsdidrocks, there is a way to test that hypothesis19:46
marsdidrocks, save the page to a local temp folder, then open the local page and try the tests again19:46
didrocksmars: ok, doing this19:47
marsdidrocks, if it is the page structure, or something in the page, then it will probably show up when the page is local19:47
didrocksmars: yes! same issue with the local page :)19:48
marsdidrocks, to be extra sure, we can break the javascript after you have saved it to disk.  No JS + chugging means it is the page structure or content19:48
didrocksoh, good idea19:48
marsdidrocks, you can shut of JS in Firefox itself, can't you?19:48
mars(wish I'd thought of that a few minutes ago)19:49
marsdidrocks, after shutting off JS, try the test.  If it still chugs, then shut off images, and please try it again.19:51
marsshould narrow the bug down to the content, or the page structure itself.19:52
didrocksmwhudson: yeah, still the bug with JS unactivated19:52
didrocksnow images :)19:52
didrocksmars: not showing images fix it19:56
marsok, so it might be painting images from Launchpad that is messed up19:56
thumpermorning19:56
didrocksmorning thumper19:56
didrocksmars: right19:56
marsmorning thumper19:57
didrocks(I still have the LP logo even when selecting "not showing images"), but the others are gone19:57
marsSo, so so so....19:57
marswe could try cherry-picking just the images from the page, put them on an empty HTML page, see if the problem persists19:58
marsshouldn't take long19:58
chrisccoulsonsorry, i disappeared for a bit there19:58
marsdidrocks, what was the URL you have been testing with?19:58
didrocksmars: I cached locally https://bugs.edge.launchpad.net/launchpad-registry/+bug/35723519:59
mupBug #357235: A user's ssh keys are not currently available throug the APIs <api> <feature> <quickly> <Launchpad Registry:Triaged by didrocks> <https://launchpad.net/bugs/357235>19:59
marschrisccoulson, np.  We narrowed it down to the Launchpad page images.  The bug is present irregardless of JS being on or off.  But deaactivating images made the bug go away.19:59
chrisccoulsonah, ok19:59
didrocksI would never thought it was image related :)19:59
chrisccoulsonthere's not many images is there?20:00
marswell, lets see20:00
chrisccoulsonoh, i notice all the images when i disable them ;)20:00
marsFirebug Net tab has that nice "images only" filter20:00
didrockschrisccoulson: that's weird, there is one that obviously FF doesn't like :)20:00
marschrisccoulson, after finding that 64bit 512Kb JS limit bug.... And some of the DOM bugs I've seen...20:01
thumpersinzui, bac: ping20:01
thumperre bug 24659420:01
mupBug #246594: Registering a code import from a series is clunky. <confusing-ui> <Launchpad Bazaar Integration:In Progress by bac> <https://launchpad.net/bugs/246594>20:01
marsoh.  hmmm20:01
marsthumper, they are sprinting this week, may not be around20:02
marsIIRC20:02
thumpermars: ah20:02
thumperI'll comment on the bug20:02
leonardrsinzui, can you help me with xslt-fu?20:02
marsleonardr, see previous comment to thumper :)20:03
leonardrah, thanks mars20:03
marsdidrocks, chrisccoulson, 26 images on that bugs page according to Firebug.20:04
marsdidrocks, chrisccoulson, give me a bit, I can make a temporary page with just those images.  *might* have the paint bug.20:04
didrocksmars: sure, thanks. I saw too that there is not that much image (just reinstalled firebug, that was a while… ;))20:05
marsdidrocks, you need to use Control-F5 in case the browser cache was being used, to get the complete image count.20:06
marsok, this test is shaky, but better than nothing.  Off to code20:06
didrocksmars: right, that's what I did when I test :)20:06
didrocksthanks mars :)20:07
marsdidrocks, is seb128 online?20:07
didrocksmars: I guess he is away for a couple of horus20:07
didrockshours*20:07
marsyes, and he is PST, so he'll be around :)20:07
didrocksmars: hum, he is in France too, so, even he's going to bed late, I don't think so :)20:08
didrocksmars: in any case, you can still ping me and I will do the test tomorrow morning20:09
marsdidrocks, ?  His whois info says otherwise: seb128, right?20:09
didrocks(my irc is always opened)20:09
didrocksmars: he maybe connects to an PST located IRC server, but I can ensure you he is French and live in France :)20:10
marsmust like using US/west servers20:10
didrockshttps://launchpad.net/~seb12820:10
marsah20:10
didrocksI just guess I made the issue more noticeable in not having 3D driver currently installed and so making FF software rendered painting suffers :)20:11
marsyeah, that was quite the hiccup though.  2-3 seconds.  And why oh why did it have to be Launchpad, of all pages? :D20:12
marshmmmm]20:15
marsdidrocks, I wonder if it is the transparent PNGs doing it?20:15
marswe'll see20:15
marsback to code20:16
didrocksmars: hum, maybe  :)20:16
didrocksok20:16
marsdidrocks, please try the test using the following page: http://people.canonical.com/~mars/lp-images-test.html20:27
didrocksmars: no latency20:27
marsphooey.20:28
didrocksmars: I guess you expected some, right?20:28
=== bjf-afk is now known as bjf
marsdidrocks, I'm not surprised.  It was the only test I could think of at the moment :)20:28
marsso we have a launchpad page that, when shown with images and HTML, causes the browser paint to mess up20:29
marsshut off the images, everything is fine20:29
marsshut off the content, everything is fine20:30
marsso images+content produced the bug20:30
marsbrutal.  That is hard to debug.20:30
didrocksright, I tried again and even with the online version, that's definitely reproduceable20:31
didrocksdefinitively20:31
marsdidrocks, ok, we could try to rule out alpha using the offline version.  Could you please upload your offline version to p.c.c?  I'll hack on it.20:32
thumperabentley: AssertionError: Wrong min time to next available builder (47099 > 300)20:32
thumperabentley: that's what I get20:32
mwhudson47099 is quite a lot bigger than 30020:32
abentleyThat's not the same one I get.20:32
didrocksmars: sure, one sec20:32
abentleyI get 120 != 3020:32
* thumper makes schema20:33
didrocksmars: FYI, if I keep the images on and I unactivate the css, no bug20:33
didrocksmars: so, definitively images + layout20:33
marsok20:33
marsthat is good to know20:33
didrocksmars: http://people.canonical.com/~didrocks/357235.html20:34
marsdidrocks, perfect, thanks.  So I can try cutting out the transparent pngs.  That is something at least.20:37
marsUnless there is some other way to look at the problem.20:37
didrocksthat can be a good idea at least to ensure it's not layout + transparent pngs20:37
marsmaybe not the transparent PNGs.  They aren't visible on the default page.20:38
marsoh, wait, at least one is, in the footer.20:38
didrocksoh, the little logos have white bg? I hadn't a look TBH20:38
mwhudsonthumper, abentley, rockstar: http://pastebin.ubuntu.com/392769/20:48
thumperhttp://pastebin.ubuntu.com/392770/20:50
mwhudsonhttps://bugs.edge.launchpad.net/soyuz/+bug/52532920:51
mupBug #525329: spurious failure in test_min_time_to_next_builder <qa-needstesting> <spurious-test-failure> <Soyuz:Fix Committed by al-maisan> <https://launchpad.net/bugs/525329>20:51
mwhudsonthumper, abentley, rockstar: do you all want to approve this mp then? https://code.edge.launchpad.net/~mwhudson/launchpad/fix-test_min_time_to_next_builder-non-utc-bug-525329/+merge/2108220:55
marsdidrocks, this file has heavily mangled CSS, does it show the bug?  http://people.canonical.com/~mars/lp-images-test/1/357235.html21:01
marsdidrocks, this is a copy of what you sent me.  It absolutely should have the bug: http://people.canonical.com/~mars/lp-images-test/0/357235.html21:01
thumpermwhudson: we could ask mars to approve it too and go for the patch with the most reviews :)21:03
marsthumper, I must have missed where the 'utc' variable came from21:04
didrocksmars: yeah! the first one has no bug. The second one has it21:04
marsdidrocks, ok.  Doesn't help much, except that almost every image is broken in /1/21:04
marsdidrocks, so again, it is an image+layout problem.  Narrowing it down is going to be a really big pain though.  It will take time.21:05
thumpermars: the utc comes from pytz at the top of the file: from pytz import utc21:05
didrocksmars: right, it's really not obvious, but at least, the issue is narrowed much more than an hour ago :)21:06
=== matsubara is now known as matsubara-afk
marsmwhudson, nice change.  Easier to read than before.21:09
=== salgado is now known as salgado-afk
* mwhudson pqm-submits21:11
thumpermwhudson: don't forget to add mars to the r= list :)21:11
mars:D21:11
mwhudsonthumper: i used ec2 land --dry-run to make the list, never fear21:12
thumper:)21:12
rockstarI assume there's no reviewers meeting today...21:22
mwhudsonrockstar: bac cancelled it21:22
rockstarmwhudson, yeah, I figured.  I actually had something bring up too.21:23
mwhudsonrockstar: gosh21:31
marsdidrocks, I think I am going to call it a night.  Do you think you could file a bug about this problem?  Be sure to attach the screencast.21:33
didrocksmars: sure, just a bug against LP itself so?21:34
marsdidrocks, for now, until we narrow it down.21:34
didrocksmars: I'll link to your file too21:34
marsdidrocks, htanks21:34
didrocksmars: doing that tomorrow morning as well, it's getting late there :)21:34
didrocksthanks a lot for your debugging too :)21:37
mwhudsonwgrant: congrats on getting your name into a debian-security mail :-)21:50
wgrantAh, so that's out now.21:55
wgrantI assume that production is patched?21:56
NCommanderdoes anyone know who would be the person to talk to about extending packagesets to be able to work with binary packages in addition to source packages?22:09
wgrantNCommander: Distro people.22:10
NCommanderwgrant: care to be more specific?22:11
=== jamalta-afk is now known as jamalta
wgrantNCommander: Core relevant members of the distro team.22:11
lifelessNCommander: start with cjwatson or persia22:12
lifelessas a hint22:12
wgrantSomething like that, yes.22:12
NCommanderlifeless: I've been talking with persia, but I need to poke someones brain about the current implementation in launchpad because I'm not quite sure I have a full grasp of its current status and/or limitations.22:13
wgrantNCommander: Extending it to binary packages is relatively easy.22:13
wgrantIf such functionality is desired.22:13
wgrantWhich it may be, to perhaps obsolete cron.germinate's Task extra-overrides.22:14
NCommanderwgrant: well, talking with persia, to fully replace the current OGRE mode, we need per-binary package set selections22:14
NCommanderwgrant: since we do have source packages who binaries are split between main/universe22:14
wgrantAh, yes, true.22:14
wgrantSo, get cjwatson/TB to work out what needs doing, and ask for it.22:15
wgrantExtending the current implementation to binary packages is not difficult.22:15
NCommanderwgrant: right, but to make a proposal, I need to know where we current stand w.r.t. to backend capability :-)22:15
* NCommander has had some interest in getting involved in soyuz development as a hobby for some time now and has quite a lot of interest in assisting with archive reorg22:15
NCommanderwgrant: correct me if I'm wrong, but the basic way to getting packagesets implemented into LP would be extending IPackageset to have binary methods in addition to the current source methods. The other issue I see is packagesets don't seem to work against series; its an all or nothing things looking at IPackageset22:19
EdwinGrubbswgrant: ping22:21
wgrantEdwinGrubbs: Hi.22:21
wgrantNCommander: I thought packagesets were series-specific now22:21
* wgrant checks.22:21
wgrantNCommander: packageset.distroseries22:22
NCommanderwgrant: oops22:22
NCommanderwgrant: so that removes that concern. Then all thats left is a way to assiocate packagesets with a BinaryPackageRelease record so sets can be added like sections/component overrides can22:23
wgrantNCommander: History is not currently stored for source<->packageset links. This needs to be discussed22:26
NCommanderwgrant: history?22:26
wgrantNCommander: At the moment, you can see the full component/section/priority history for a source or binary package.22:27
wgrantYou can do no such thing with package set memberships.22:27
wgrantThis could be confusing when it starts being used for ogreing.22:27
wgrantAnd package set membership should probably be between a Packageset and a BinaryPackageName, not a BinaryPackageRelease.22:28
NCommanderwgrant: how is history recorded for those?22:29
* thumper fires off an ec2 land before closing laptop22:30
wgrantNCommander: By keeping multiple (Binary|Source)PackagePublishingHistories.22:30
wgrantNCommander: They are the things that link an archive-agnostic BPR to an archive, in a particular component, section and (for binaries) priority.22:30
wgrantWhen an override is changed, a new [SB]PPH is created, and the old one is marked Superseded.22:31
wgrantSo you can see the full history.22:31
NCommanderwgrant: so we need a SourcePackagesetPublishingHistory(Public), and records to be created for each change22:31
wgrantNCommander: I don't know. I don't think this has been discussed before.22:32
NCommanderand when binary packagesets materialize, BinaryPagesetPublishingHistory22:32
wgrantThis also makes it just about impossible to change package sets in post-Release pockts.22:33
EdwinGrubbswgrant: hi. you were mentioned as somebody that might have insight into the benefits of crowdsourcing the linking of projects to upstream bugtrackers and linking the development focus series to an imported upstream branch.22:33
NCommanderwgrant:?22:34
wgrantEdwinGrubbs: Yes. At the moment Ubuntu bug triagers can't do any of that, so are crippled and have to go and talk to ~registry or $ARBITRARY_PROJECT_REGISTRANT in order to streamline the upstream integration workflow with projects that do not maintain an LP-hosted presence.22:35
NCommanderwgrant: oh, because packagesets have no concepts of components, right?22:35
wgrantNCommander: No concepts of *pockets*.22:35
wgrantThey don't need to know about components.22:35
NCommanders/components/pockets/g :-)22:35
wgrantBut at the moment I can technically promote something in -updates, for example.22:35
wgrantThis is potentially useful, although it needs discussion.22:35
NCommanderwgrant: its been done before22:36
wgrantIt would be unfortunate if this capability was lost.22:36
NCommander(promotion in a pocket)22:36
wgrantRight.22:36
* NCommander remembers we had a KDE bug that needed that done, and we had to test on dogfood to make sure Soyuz won't blow up on publishing22:36
wgrantThen again, somebody has also done a promotion in the Release pocket after a release, which ends up a little odd...22:36
wgrantYep.22:36
NCommanderwgrant: LP prevents promotion in Release after release22:36
wgrantNCommander: Heh, well, it should.22:37
wgrantBut it doesn't.22:37
wgrantThe new publication will just sit there forever.22:37
wgrant(because the publisher won't touch new records in Release)22:37
NCommanderwgrant: it doesn't?22:37
NCommanderugh22:37
wgrantI need to fix that.22:37
* wgrant -> uni22:39
NCommanderwgrant: cya later22:39
=== jamalta is now known as jamalta-afk
mwhudsongosh lunchtime already23:42
mwhudsonwhere is the day going23:42

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!