/srv/irclogs.ubuntu.com/2012/11/02/#launchpad-dev.txt

StevenKwallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/fetchpage-does-not-send-headers-always/+merge/13263800:43
wallyworld_StevenK: i don't know the code - why can't those 2 classes be change to use getPage()?00:46
StevenKI tried, and Trac broke00:46
wallyworld_it just seems very hacky is all - fetchPage seems to expect a Request and nothing else00:48
wallyworld_so it shouldn't be called with a string00:48
wallyworld_and if it is then that's a mistake to be corrected upstream so to speak00:49
StevenKIt just tosses things into urlopen, and that will deal with either00:49
StevenKIt's just that we want to send headers00:49
StevenKAnd for that, you need a Request00:50
wallyworld_ok, makes more sense now00:50
wallyworld_r=me00:50
StevenKwallyworld_, wgrant: https://code.launchpad.net/~stevenk/launchpad/destroy-distribution-upstreamreport/+merge/13263901:02
StevenKYou two should enjoy this one, it's great.01:02
wgrantThere must be more01:02
wgrantLook harder01:02
StevenKReally now?01:04
wgrantAlso, we need to check with UE before this can go anywhere01:05
StevenKwgrant: Tempted to close bug 89938801:17
_mup_Bug #899388: Some translation statistics are not up to date <lp-translations> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/899388 >01:17
wgrantStevenK: Nope01:18
StevenKIt's still broken?01:18
wgrantThe stats jobs don't do everything01:18
StevenKwgrant: Ah, how do I read https://oops.canonical.com/oops/?oopsid=OOPS-13c0c8bb3601a44992977eaebb7c2d9e ? Since it's a high non-SQL time01:26
wgrantStevenK: You could try getting a profile from qastaging01:29
wgrantIt's probably spending most of the time loading Storm objects01:29
wgrantEnum columns, in particular01:29
StevenK3731696 function calls (3643838 primitive calls) in 28.437 CPU seconds01:44
StevenKwgrant: enumcol calls don't look high to me01:48
StevenK     2043    1.241    0.001    1.392    0.001 store.py:807(_add_to_alive)01:48
StevenK   119112    1.237    0.000    2.744    0.000 {method 'set' of 'storm.variables.Variable' objects}01:48
StevenK     2290    0.907    0.000    5.849    0.003 store.py:680(_load_object)01:48
StevenKLooks like it's loading stacks of objects01:49
wgrantStevenK: Found anything?02:10
StevenKwgrant: Indeed, I found my lunch, and it was good.02:40
wgrantYou also found a way to break buildbot :P02:40
StevenKOh, for the love of ...02:42
* StevenK fixes02:42
wgrantNow to pretend to be a builder...02:47
lifelesswrong name02:49
StevenKCrumbs03:22
StevenKLast failure is test_rendered_query_counts_constant_with_team_memberships03:22
wgrant:(03:23
StevenKThat didn't fail on buildbot, so I'm not sure why testr ran it03:23
StevenKwgrant: Reply from Bryce on the bug03:29
wgrantSo I saw03:29
StevenKI'm not sure how to read that reply.03:34
StevenKIt sounds like he's interested and used to graph the data, but doesn't answer my question on what we can do03:35
StevenKwallyworld_: You'll get a kick out of this: http://www.theregister.co.uk/2012/11/01/samsung_case_apple_told_to_apologise_again/03:43
wallyworld_lol03:43
wallyworld_too bad the us courts are so stupid03:44
wallyworld_i wonder how mych campaign money apple donates03:44
wallyworld_that would have nothing to do with it i'm sure03:44
StevenKwallyworld_: I'm just glad that 1. The UK court forced Apple to apologise. 2. That the UK judge is displeased with their first attempt and has told them they have 24 hours to fix it.03:46
wallyworld_yeah, me too. but i despair that crapple lost the battle here but is winning the war03:46
sinzuiStevenK, Bryce and other upstream stewards want a chart of how many bugs in ubuntu packages are triaged, have an upstream task (a task that we presume is upstream), have a bug watch.03:49
sinzuiStevenK, each number comes from a bug search, so api could produce just the numbers you need.03:49
StevenKsinzui: With no changes needed?03:50
sinzuiStevenK, , but if you are patient to get the page to load, you will see that the top 100 packages are someone's acid-fueled fantasy...pidgin, firefox 3.5, f-spot, banshee, ekiga...03:51
sinzuiStevenK, the page lists 6 x-related packages.03:52
* sinzui gets the example urls for an x package03:52
sinzuiStevenK, http://paste.ubuntu.com/1325544/04:00
StevenKsinzui: So you think we provide enough tools for Bryce to graph without having to screenscrape and we're free to destroy Distribution:+upstreamreport and the horse it rode in on?04:01
sinzuiWe know he cannot screenscrape. It took me 10 minutes to load the page04:01
sinzuiWe should delete the page because we should not be working about ekiga bugs. There is not mention of bugs in zeitgeist which is what unity needs04:02
sinzuiSome one could write a better api script that does the same work for any package, then we let the community choose the packages instead of looking back at Hardy with fondness04:03
wgrantIf we were to try to fix it, we would likely end up having to rewrite it based on a full new set of denormalised data structures04:04
wgrantUbuntu has a few more bugs now than it did in 200804:04
StevenKWell, I'm happy we have the data and if it's up to Ubuntu to write an API script to pull it per package, then that's fine and we can destroy the page.04:08
sinzuiWe can destroy the page. If some says they are using it for a package in Quantal, I will write the script on my Lunch04:09
StevenKsinzui: https://code.launchpad.net/~stevenk/launchpad/destroy-distribution-upstreamreport/+merge/132639 is an MP I prepared earlier04:10
StevenKwgrant said there must be more, but I'm not certain if he was trolling. :-)04:10
sinzuir=me04:13
* sinzui find bed04:14
StevenKsinzui: Night!04:14
StevenKwgrant: Right, I think I'm undistracted enough.04:25
wallyworld_wgrant: if you are interested, this currently seems to work nicely. the garbo job needs work to handle existing records in the cache table and to do bulk updates (it is just designed right now to run against a totally empty cache table to get data in)04:37
wallyworld_https://code.launchpad.net/~wallyworld/launchpad/ppa-packages-timeout-1071581/+merge/13264604:37
wgrantwallyworld_: How does it handle the case where I maintain a package that has been uploaded by two different people?04:37
wallyworld_running it on dev produces the same results as previously as far as i can tell04:37
wgrantAlso, ew04:37
wallyworld_there should be a row for each of those04:38
wgrantSure04:38
wgrantThat's the problem :)04:38
wallyworld_isn't that the case now?04:38
wgrantIt's only meant to show on my +maintained-packages once, even if it was uploaded by multiple people04:38
wgrantToday we filter by maintainer and distinct on (archive, distroseries, sourcepackagename)04:38
wallyworld_oh ok, so just the latest shouyld show?04:38
wgrantbut the cache is distincted on (archive, distroseries, sourcepackagename, creator, maintainer)04:39
wallyworld_sure, that's what i thought we needed04:39
wallyworld_but i can change it04:39
wgrantAlso, the garbo job is very much non-wgrant compliant04:40
wallyworld_actually, maybe it does correctly only have the one row, will have to check04:40
wallyworld_why?04:40
wgrantOne does not simply concatenate arbitrary strings into SQL.04:40
wallyworld_you mean for the job table update?04:41
wgrantYes04:42
wallyworld_well, it's done elsewhere04:42
wallyworld_and it's just for a simple, safe update04:42
wgrantWhere?04:42
StevenKThat is not an argument.04:42
wallyworld_in many other garbo jobs04:43
wgrantI'm going to find that place and delete it :)04:43
wallyworld_have a look04:43
wallyworld_there's lots04:43
StevenKCargo-culting existing bad practise is bad04:43
wallyworld_it's not worth introducing a whole new class04:43
wgrantNot seeing it...04:43
wallyworld_just to update one row in a job table04:43
wgrantHmm?04:43
wgrantUsing store.execute and using string contatenation to create SQL are orthogonal concepts :)04:44
wgrantstore.execute is fine04:44
wgrantString concatenation is not04:44
wallyworld_see OpenIDConsumerAssociationPruner for example04:44
wallyworld_i agree though, this was just quick and dirty to get something going04:46
wallyworld_i could care less right now about the garbo job - it needs work as i said04:46
wallyworld_i'm more concerned with the correctness of the table population04:47
wgrant("couldn't")04:47
wgrantBut everyone needs to not even jump to string concatenation as a first step, ever04:48
wallyworld_i was using colloquial04:48
wgrant, params=('foo', 'bar') is not that hard :)04:48
wallyworld_sigh, it was a poc04:48
wgrantLP PoCs have a habit of making it to production :(04:48
wallyworld_not once i'm finnished with it04:49
wallyworld_i just want to get the data in palce to see if the business logic works04:49
StevenKwgrant: If you're done lambasting wallyworld_, could you help with ++profile++show?04:49
wallyworld_and the important logic is all strormified04:49
wgrantStevenK: What's the issue?04:49
StevenKwallyworld_: You're fun to argue with. When you get frustrated you start typing like sinzui.04:50
wgrantHeh04:50
wallyworld_yes i do.04:50
StevenKwgrant: Milestone:+index again, now I'm now undistracted.04:50
wgrantStevenK: Sure, but what about the profile?04:50
wgrantLike, what do you need help with?04:50
wgrantIf it involves killing dogfood I'm all for it04:50
StevenKwgrant: I'm not sure how to use the profile to point out the terrible bits of code.04:51
wgrantheh04:51
wgranttype 'ubuntu-10' in awesomebar04:51
wgrantfirst result is https://qastaging.launchpad.net/ubuntu/+milestone/ubuntu-10.04.2/++profile++show04:51
StevenKAside from 'all of it'04:51
wgrantThat was months ago04:51
StevenKwgrant: Haha04:51
StevenKI have https://qastaging.launchpad.net/ubuntu/+milestone/ubuntu-10.10/++profile++show/+index loaded04:52
wgrantwallyworld_: So, the only obvious logic concern is the creator vs maintainer thingy04:52
wallyworld_i just added in maintainer to distinct on04:52
wallyworld_i think that wil fix it04:52
wgrantWell04:52
wgrantI'm not sure that a single table can satisfy both04:53
wallyworld_not sure, i'll have to look into it some more04:53
wgrantGiven that the table is meant to satisfy the distinct itself, hmm04:54
wgrantBut it probably can't, because it has to filter on either creator or maintainer04:54
wallyworld_maybe i need 2 tables then04:55
wgrantStevenK: You managed to get the table to not time out, then?04:55
wallyworld_or the record count will be ssmall enough that some live filtering can be done04:55
wgrantwallyworld_: Quite likely. They end up pretty much identical and can be maintained by a live job04:55
wgrants/a live/the same/04:55
StevenKwgrant: Yeah, 10.10 loaded with qas, but ++profile++show's title shows Error: Timeout04:56
wgrantHmm04:57
wgrantSo your initial assessment was pretty accurate04:57
StevenKwgrant: That it's just loading 2,100 odd objects?04:58
wgrantNo, the "all of it" bit04:59
StevenKOh, right.04:59
StevenKYeah, the code looks pretty terrible05:02
wgrantStevenK: http://people.canonical.com/~wgrant/output.png05:19
StevenKWhat's that from?05:20
wgrantgprof2dot05:20
wgrantubuntu-10.04.2, not ubuntu-10.10, but it's slow too05:20
StevenKlaunchpad_dogfood=> SELECT productrelease, COUNT(libraryfile) FROM productreleasefile GROUP BY productrelease HAVING COUNT(libraryfile) > 0 ORDER BY COUNT(libraryfile) DESC LIMIT 1;05:21
StevenK productrelease | count05:21
StevenK----------------+-------05:21
StevenK          32517 |   89905:21
wgrantYeah05:21
StevenKI dispair at our DB05:21
wgrantPRF is a bit crap05:21
StevenKwgrant: It's pretty, but it's a mess of colour :-)05:22
StevenKwgrant: So I do wonder if the mixin's _bugtasks method is too blame for most of the mess.05:27
wgrantWell05:27
wgrantThat's probably just spending time loading lots of objects05:27
StevenKWhich is most of the problem, no?05:28
wgrantSure05:28
StevenKWe only want the title, importance, assignee and status. Can't we just populate a list straight from BTF and skip most of the garbage?05:29
wgrantIf you want to reimplement all of Launchpad, sure05:29
StevenKwgrant: I meant for Milestone:+index specifically, how does your comment follow on?05:31
wgrantStevenK: Milestone:+index is part of Launchpad, and relies on a lot of other parts of Launchpad, all of which rely on having these objects05:31
wgrantWe may be able to optimise the loading of those objects05:31
wgrantThe first step is probably to try to reproduce it locally.05:31
StevenKwgrant: Well, what I was actually suggesting lobomitizing Milestone:+index to not have the bug objects which is pulling in a lot of baggage, and work from a list populated by BTF.05:33
wgrantStevenK: The DB queries aren't the problem (and BTF doesn't have title)05:34
wgrantEliminating the bug objects would work, but it's a lot of work05:34
StevenKHmmm05:35
StevenKwgrant: Did you attempt wallyworld_'s QA?05:48
wgrantI've been at it on and off for the last 5 hours :/05:51
wgrantDF is a little bit slow05:51
StevenKA little? :-P05:51
wgrantI've about to give in and qa-ok it half-way through05:51
StevenKMight be a little late to deplay today, though05:52
wgrantThe bug is fixed, and I don't see how it could have broken the existing functionality05:52
wgrantI can see my correctly defaulted binary in the queue, but acceptance times out...05:52
* wgrant concedes defeat06:11
czajkowskimorning07:35
deryckMorning, all.08:50
czajkowskideryck: ello08:50
=== Ursinha-afk is now known as Ursinha
rick_h_morning deryck11:10
deryckrick_h_, hey, man.11:12
czajkowskirick_h_: what time is it where you are?11:12
rick_h_czajkowski: 7am11:20
rick_h_trying to get back on schedule11:20
czajkowskiwow11:21
deryckczajkowski, rick_h_ never stops. :)11:37
czajkowskifinally choolate and nice fruit with ginger ale are on offer!11:38
czajkowski*chocolate11:39
=== Ursinha is now known as Ursinha-afk
deryckSorry for all the email Orange Squad.  Trying to get cards on board categorized and prioritized.11:53
deryckAssuming you get board email :)11:53
rick_h_deryck: rgr yea noticed11:56
derycklunch time!12:00
abentleyderyck: I don't get board mail, so I'm happy.13:00
deryckabentley, excellent then :)13:00
abentleyderyck: I plan to back-burner 1066905 in favour of 1074139, because the latter can lock users out of their projects, and it should be an easy fix.13:03
deryckabentley, I trust your judgement.  can't look right now, in meeting.13:04
joeyHi, what's the best way over the api to return a list of teams that match a wildcard?16:12
joeye.g. ubuntu-*16:12
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
deryckrick_h_: hey, there.16:51
rick_h_deryck: howdy16:51
deryckrick_h_: see my note on the timeout bug.  Did we change something with profile queries?  Or just now suddenly profile queries are slower for everyone?16:52
deryckor timing out for everyone, rather16:52
sinzuideryck, rick_h_, I am not seeing these profile page timeouts. I suspect I cannot because I am a CA...privacy security checks short circuit for me16:52
rick_h_deryck: not sure, I see the one query that's taking 4s around blueprints16:52
derycksinzui: interesting16:52
rick_h_so looking at where it's at and what it's trying to do16:52
sinzuioh, but I still cannot see private blueprints/bugs/branches that I am not subscribed to, so that does not make sence16:53
deryckrick_h_: yeah, that query is slow, no doubt.  But did we just add the filter?  or is maybe something else also changed recently to make matters worse?16:53
rick_h_deryck: not sure, I've not tracked the bzr history of it atm. I can peek and try to see if there's some recent stuff that hit it16:53
rick_h_was more just trying to understand and figure out where the ineffecient part is16:53
rick_h_it's got somethjing like 5 or 6 subqueries in it atm ugh16:54
deryckrick_h_: yeah, the 4 second query is awful.  fixing that will most likely fix the issue.16:54
deryckrick_h_: it just seems strange that we *suddenly* started OOPSing every user page.  unless we just deployed this query, or something else changed around it.16:55
rick_h_everything else seems pretty light16:55
rick_h_deryck: ok, cool I didn't realize it was everyone hitting this and suddenly. I just assumed it was something for those with private blueprints started seeing recently16:56
deryckrick_h_: no, you don't need private blueprints to cause the problem.  we're querying to make sure you can see your blueprints, whether you have private ones or not.16:57
rick_h_right, my profile page loads fine16:57
deryckhmmm16:57
deryckrick_h_: not for me.16:58
rick_h_deryck: but this is coming from abentley's new assigned_specs_in_progress16:58
rick_h_so I bet that's why it's more recent.16:58
rick_h_that was just something last week that was worked on/added16:58
rick_h_while we were in CPH I believe16:58
deryckrick_h_: ah, right, I see the rev now.  yeah, that makes more sense.16:59
deryckrick_h_: so chat with abentley about it then.16:59
rick_h_will do, thanks for forcing me to look at that closer and connect the dots.16:59
abentleyrick_h_: My profile page is zippy.16:59
deryckrick_h_: np.  hi, abentley16:59
abentleyderyck: Hi.16:59
rick_h_abentley: yea, mine seems to be ok as well, but we're getting a bunch of these16:59
deryckso every profile page OOPes for me.  as does for other people in the bug.17:00
rick_h_but I don't have many BP and none are private17:00
sinzuiWe don't use blueprints, that is why we don't see them17:00
rick_h_right, that seems to fit for me as well. and this is one UGLY generated sql in the oops17:00
abentleyderyck: Can I have an oops-id?17:00
rick_h_https://oops.canonical.com/oops/?oopsid=OOPS-8844d78960633507626e1325e8cad35917:00
rick_h_abentley: ^17:00
deryckabentley: https://oops.canonical.com/oops/?oopsid=OOPS-1390ee2cc5227ea249e6dbdc7c1f79f017:00
deryckheh17:00
deryckOOPS, twice :)17:00
rick_h_you dare race me deryck :P17:01
derycknever :)17:01
rick_h_I'm back on a real keyboard in the office again bwuhahaha17:01
abentleyderyck: Are you free to compare query times?17:01
deryckabentley: sure.17:01
abentleyAw man, even here we don't have the substitutions.17:02
abentleyderyck: I can't see how to find hloeung's db id via the API.  Could you look that up, please?17:08
deryckabentley: https://pastebin.canonical.com/77605/ (private paste, since it's data for a query.)17:11
=== Ursinha is now known as Ursinha-afk
deryckabentley: I need to go down for dinner.  maybe sinzui could run queries for you?17:25
sinzuiI can17:26
abentleysinzui: Thanks.17:26
* sinzui is stuck so needs a diversion17:26
deryckawesome, thanks guy!17:27
deryckguys even17:27
abentleysinzui: Could you please time this against qastaging?  (I'm expecting it to run a long time.)17:29
abentleysinzui: https://pastebin.canonical.com/77606/17:29
sinzuioops, I ran on staging17:31
abentleysinzui: Should be fine.17:31
sinzuiwell both are subsecond to get 2 rowns17:32
sinzuirows17:32
abentleysinzui: That is not good, because it suggests I've done the substitutions wrong.17:32
abentleysinzui: Or else the query is not consistently slow.17:32
sinzuiabentley, this is the query plan form qastaging: https://pastebin.canonical.com/77609/17:33
sinzuiabentley, surely there is more private blueprints in production than qastaging.17:34
sinzuistaging is 2 weeks old, so that would be better17:34
abentleysinzui: You ran it against staging, though.17:35
sinzuiyes, but then I switched to qastaging...17:35
sinzuioh, the query plans are different!17:35
abentleysinzui: You said staging was subsecond.17:35
sinzuithis is staging's explain, and yes, subsecond for both: https://pastebin.canonical.com/77610/17:36
abentleysinzui: I think we need to try it on prod.17:38
sinzuiwe need webops for that17:38
=== yofel_ is now known as yofel
=== udsslayer is now known as shadeslayer
abentleyrick_h_: Do we have a bug for these oopses?18:27
rick_h_abentley: yes, https://bugs.launchpad.net/launchpad/+bug/107423918:28
_mup_Bug #1074239: Timeout error when trying to view any user profile page. <lp-blueprints> <private-projects> <timeout> <Launchpad itself:In Progress by rharding> < https://launchpad.net/bugs/1074239 >18:28
abentleyrick_h_: ty18:30
=== yofel_ is now known as yofel
joeywell nobody liked my last question so how about a different one.. I need help optimizing a short piece of python code that's comparing LP object to LP object22:41
joeyanyone up for that?22:41
lifelessjoey: you know the rule... don't ask to ask ;)22:47
joeyyeah but I'm just too polite22:47
joeygoogle's not been very helpful... I think it's python thing...22:48
joeylifeless: how's the new job?22:48
lifelessjoey: pretty good22:49
lifelessjoey: --big-- company22:49

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!