/srv/irclogs.ubuntu.com/2011/06/17/#launchpad-dev.txt

wgrantlifeless, sinzui: I was planning to do a deployment this morning.00:18
wgrantBut then a certain monstrous completely un-QAable undisable hopeless merge failure buggy criss-cross branch landed.00:19
wgrantWe won't be deploying for a few days :/00:19
=== wallyworld_ changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: wallyworld (*jtv) | Critical bugs: 202 - 0:[######=_]:256
=== wallyworld_ changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 202 - 0:[######=_]:256
wallyworld_wgrant: ^^^^^^ which branch out of interest?00:37
wgrantIt's tempting to roll it back on principle.01:21
wgrantThere are bugs.01:21
wgrantNot fatal if it was flagged.01:21
wgrantBut...01:21
wgrantIt's not.01:21
StevenKWasn't it rolled back once already?01:27
wgrantYes.01:27
wgrantYay01:30
wgrantIt crashes in Firefox 3.5, at least sometimes.01:30
wgrant:/ this is a big UI change.01:32
wgrantNo longer obvious how to subscribe, and you can't tell who is directly subscribed to the bug...01:33
LPCIBotProject parallel-test build #44: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/parallel-test/44/01:45
LPCIBotProject windmill-db-devel build #398: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/398/01:51
lifelesswallyworld_: the bugsubscription one01:51
wallyworld_the 12000 line one i assume01:52
StevenKwgrant: Can haz link to notify() bug?01:53
wgrantStevenK: There doesn't seem to be one. Check #launchpad from 21:48 yesterday.01:54
LPCIBotProject db-devel build #642: STILL FAILING in 5 hr 41 min: https://lpci.wedontsleep.org/job/db-devel/642/02:02
lifelesswgrant: if its buggy, its a rollback02:05
wgrantlifeless: The ordering is apparently arbitrary and the JS crashes when not logged in.02:27
wgrantAnd it completely changes the UI of a feature we announced a week ago :/02:27
lifelesscrashing when not logged in is a pretty big blocker02:29
=== Ursula is now known as Ursinha-afk
wgrantIt's not a regression from what we have now.02:31
wgrantOhhh, yes it is, actually.02:31
wgrantSince at least now the direct subscribers are shown.02:32
lifelesswgrant: crashing is a regression02:33
lifelesswgrant: the UI aspect is deliberate AIUI. IMBW02:33
wgrantlifeless: Deliberate, but untested and a break from what we announced :/02:37
huwshimilifeless: ITMFTFOWTFYW02:50
lifelesshuwshimi: ?02:50
huwshimilifeless: It takes me forever to figure what you write02:51
huwshimilifeless: Sentences like this: "lifeless: wgrant: the UI aspect is deliberate AIUI. IMBW"02:51
lifelesshuwshimi: sorry !02:51
huwshimilifeless: haha, it's ok02:52
huwshimilifeless: Most people on Launchpad do it, I just spend half my time opening a browser and googling what everyone is saying02:52
wgrants/Launchpad/IRC/02:52
huwshimiwgrant: True02:52
=== wgrant changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 205 - 0:[######=_]:256
=== lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 205 - 0:[######=_]:256
LPCIBotProject windmill-devel build #241: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/241/03:21
wgrantStevenK: Any luck with the notifications?03:30
StevenKwgrant: I've been a little distracted, sorry. Still digging.03:31
StevenKwgrant: Unsigned changes file, since it comes from a buildd, so it adds Changed-By.03:35
wgrantStevenK: Ah.03:36
wgrantStevenK: How did this not happen before?03:36
StevenKI suspect the old code only did that if it wasn't a PPA.03:37
wgrantProbably.03:37
wgrantI also suspect we need more unit tests.03:37
StevenKwgrant: I think the bug is firmly in get_recipients(), I was just going to read the old code03:41
StevenKwgrant: Hm, the bug seems present in the old code too03:47
wgrantStevenK: But I know it didn't happen :/03:47
wgrantStevenK: Possibly archiveuploader just didn't notify?03:47
wgrantI'll look.03:47
StevenKBut I didn't change that bit!03:47
StevenKwgrant: What I mean is _getRecipients() always added Changed-By if packageupload.signing_key is None03:48
wgrantYeah.03:48
wgrant        # If this is not a PPA, we also consider maintainer and changed-by.03:51
wgrant        if self.signing_key and not self.isPPA():03:51
wgrantStevenK: From the original code.03:51
StevenKwgrant: But look above that03:52
StevenKwgrant:         debug(packageupload.logger,03:52
StevenK            "Changes file is unsigned, adding changer as recipient")03:52
StevenK        candidate_recipients.append(changer)03:52
wgrantFair point.03:53
wgrantOh.03:53
wgrantsec03:54
wgrantHm.03:54
wgrantThe buildds override Maintainer, but not Changed-By.03:54
wgrantSo Changed-By should be the source Changed-By... hm.03:54
wgrantThis will all have to be rewritten soon anyway :/03:55
wgrant2011-06-17 00:49:35 DEBUG     Subject: [PPA stackapplet-dev-stackapplet] stackapplet_1.5+147~maverick1_source.changes rejected03:56
wgrant2011-06-17 00:49:35 DEBUG     Sender: Ubuntu Installer <archive@ubuntu.com>03:56
wgrant2011-06-17 00:49:35 DEBUG     Recipients: Launchpad Beta Testers <noreply@launchpad.net>03:56
wgrantwhat03:56
StevenKAnother rewrite? :-/03:59
wgrantStevenK: For DDs, yes.03:59
LPCIBotYippie, build fixed!03:59
LPCIBotProject devel build #813: FIXED in 5 hr 24 min: https://lpci.wedontsleep.org/job/devel/813/03:59
wgrantStevenK: Once we're copying between primary archives.03:59
wgrantStevenK: Otherwise Debian people will probably be notified, which would be Bad.03:59
wgrantAnd Ubuntu people would definitely be notified about Linaro builds.04:00
StevenKIndeed04:00
StevenKwgrant: So, shall we just remove the else bit that adds Changed-By for unsigned uploads?04:01
wgrantStevenK: No.04:01
wgrantStevenK: We need to work out what has changed.04:01
StevenKBetween a refactor a month ago, a rewrite 2 weeks ago, and 2 tweaks. :-/04:02
wgrantInteresting. Logs start on the 19th, and the first occurence was the 30th.04:03
wgrantIt was broken then.04:03
StevenKThe 30th of May? That's the rewrite04:04
wgrantSo it was.04:06
wgrantSo.04:06
wgrantI can't see why it wasn't notifying before :/04:06
StevenKWhat do the DEBUG logs show for occurances of this problem prior to the rewrite?04:06
wgrantThere are none.04:06
wgrantBut they only go back 11 days earlier.04:07
StevenKBugger.04:07
wgrantThat exception is somehow not seen at all before then.04:07
wgrant(which is not unthinkable; it also wasn't seen between the 3rd and the 10th of this month)04:07
StevenKYes, they don't happen very often.04:07
wgranthttps://lists.ubuntu.com/archives/ubuntu-reviews/2010-April/000281.html04:09
wgrantThat's a log from a year ago.04:09
wgrantThere are more recent ones, but that came up quickly in Google.04:09
StevenKThat doesn't trip the Changes file is unsigned, adding changer as recipien04:10
StevenKSigh. But you get the idea.04:10
StevenKIn fact, that doesn't send a notification at all.04:11
wgrantI wonder...04:12
wgrantIt's possible that this is unrelated.04:13
wgrantprocess-upload was originally run by b-m, with -M04:13
wgrantSo no mail was sent.04:13
wgrantBut then late last year that changed...04:13
wgrantCould this have been happening for 6 months? Odd that we only heard about it now.04:14
StevenKIndeed.04:14
wgrantIf only we kept logs for a while :/04:16
wgrantWhat we keep is only 13MB :(04:16
wgrantI don't understand why we don't keep pretty much all our logs for just about ever :(04:16
jtvhi wallyworld_, I'm back.  Any reviews to mentor?04:41
wallyworld_jtv: hi. none yet04:43
jtvok04:43
jtvwgrant: is there something similar to the qa-tagging going on with bug statuses?  I notice you've marked a few bugs Fix Released before all their branches had landed.04:45
StevenKwgrant: Since we don't have logs, we have no way to be certain -- but the work that divorced process-upload and b-m could have caused this? So I'm off the hook?05:45
wgrantStevenK: I think so :/05:46
wgrantjtv: Those didn't look like multi-branch bugs, and they weren't landed with [incr], so I didn't notice, sorry.05:46
wallyworld_jtv: i'm off to pick up the kid from school. still no reviews. will check again later06:00
jtvok wallyworld_06:09
jtvwgrant: I just knew there was something I wasn't thinking of… incr.  It's getting altogether a bit finicky for me, this tagged-double-bookkeeping.06:10
wgrantIndeed.06:11
jtvOMG my branch made it in.  Starting at 07:30 pays off.06:11
jtvI think that's about 8K of today's 9K lines landed.  :)06:11
nigelbso, I have the sort bug still pending. where is a good place to add a util function?06:12
LPCIBotProject parallel-test build #45: STILL FAILING in 1 hr 16 min: https://lpci.wedontsleep.org/job/parallel-test/45/06:14
LPCIBotProject windmill-db-devel build #399: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/399/06:19
=== _mup__ is now known as _mup_
StevenKDoes anyone know lib/lp/{bugs,code,soyuz}/notifications/ is used for? All three of them contain a zero-length __init__.py06:42
lifelessstub: hi07:07
stubYo07:07
lifelessstub: I did that lock experiment; the blocked alter table blocked out new readers.07:08
lifelesssession one: begin; set transaction isolation level serializable; select id from bug;07:08
lifelesssession two: being; alter table bug add column foo int;07:08
lifelesssession three: begin; set transaction isolation level serializable; select id from bug;07:08
lifelesstwo and three both block07:09
lifelessdoing a rollbackl in session one - andtwo and three unblock07:09
lifelesswell, once two is committed.07:09
stubok. There might be something more complex going on (transactions working with related tables causing the ALTER TABLE to block).07:11
stubOr not07:11
lifelessstub: staging has displayed one of these typical 'fail to apply' all day07:12
lifelessstub: hloeung and I have been gathering data and poking at it07:12
stubBut so far, I've never had success doing anything that needed to get a lock on Person on production. 'ALTER TABLE BugSummary ADD CONSTRAINT bugsummary__viewed_by__fk FOREIGN KEY (viewed_by) REFERENCES Person' for instance.07:12
lifelessstub: the appservers and scripts were running07:12
stubHas the lag check rung any alarms?07:13
lifelessyeah07:13
stubSo the script is either blocked for some reason (eg. a DB patch trying to grab a lock on the Person table ;) ), or the daemons have fallen over, or replication is broken07:14
stubGood the lag check is alerting us07:14
lifelesshttps://pastebin.canonical.com/48634/07:14
lifelesspending locks07:14
lifelesshttps://pastebin.canonical.com/48634/07:15
lifelessbah07:15
lifelesshttps://pastebin.canonical.com/48632/07:15
stubIt seems to be currently running my bugsummary update. This is the first run on a production like system... the query might just be way slow.07:16
lifelessstub: 5 hours ?07:16
StevenKstub: 2208-00-0.sql is current, or is that the amalgamation of 2207? Happy to wait until you're done digging.07:16
lifelessstub: if its the one I prepared, it runs into a temp table in a few minutes07:16
stubThere is no 2207 in our tree. launchpad-2208-00-0.sql is the current baseline.07:17
StevenKstub: Right, but 2207 used to exist07:19
stublifeless: I didn't notice or decode your one (need to feed that stuff through a formatter).07:19
stubyes07:19
LPCIBotProject windmill-devel build #242: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/242/07:19
StevenKstub: IE, where did 2207-* move to? 2208-00-0, or somewhere else?07:19
stubIt was either moved to 2208 or deleted07:19
stubannotate on 2208 should give you the revno07:20
StevenKI'd like to drop a view from 2208-00-007:20
StevenK(Not yet, mind)07:20
stubThat needs a DB patch. You don't modify the baseline - it is a snapshot of production at a point in time.07:20
StevenKstub: Okay. So for an ec2 run (for instance), the view will get created and then removed?07:21
stubIf you hack the baseline, it will never get applied to production07:21
stubYes07:21
StevenKstub: Okay, thanks.07:21
stubUntil we rollup the next baseline.07:21
stub(probably next month - we are in the 70's already)07:22
StevenKBring on 220907:22
stubAny particular reason? Applying those patches takes about a second.07:23
StevenKNo, just forward progress :-)07:23
stubGoing back to 1 is backwards progress ;)07:23
StevenKHaha07:23
=== almaisan-away is now known as al-maisan
stublifeless: So it is my bugsummary population query running. Possibly an earlier one was slow - I won't get timings until (if) it completes.07:25
lifelessstub: no, it was that one07:26
lifelessjust digging up the query I had07:26
stublifeless: We can either wait or kill it - things should rollback fine at the current point in time.07:26
lifelessstub: I'd like to debug whats up with it07:26
stubIts chewing 100% of a core.07:26
stubthe EXISTS always is dangerous to performance.07:27
stubYour version had that?07:27
stubAnyway - we can fix it directly in the existing patch since it hasn't landed yet...07:27
lifelessyeah07:27
lifelesshttps://bugs.launchpad.net/launchpad/+bug/793848/comments/407:28
_mup_Bug #793848: Distribution:+bugtarget-portlet-bugfilters-stats timeouts <dba> <timeout> <Launchpad itself:Triaged by stub> < https://launchpad.net/bugs/793848 >07:28
lifelesshad my query07:28
stubBut the cpu utilization means it isn't blocked.07:28
lifelessyah07:28
stubMaybe something stoopid like a missing join condition07:28
lifelessso we need to diff the two queries07:28
lifelessstub: it was blocked for a while07:29
lifelessstub: was waiting for a lock on the sl_event07:29
stubSome other process inserting data that needed to be replicated.07:29
lifelessstub: it looked like a slon had started up badly and was deadlocking itself07:32
stublifeless: So how long has this patch been running do you think?07:33
lifelessstub: https://pastebin.canonical.com/48632/07:34
lifelessstub: submitted 11 hours or so ago07:34
lifelesspostgres 32012 99.9  2.2 2280364 726944 ?      Rs   Jun16 567:55  \_ postgres: slony lpmain_staging [local] INSERT07:35
stublifeless: looking at the slony stuff being executed, it was blocked then on the event lock07:36
lifelessstub: so, tha db patch has used 567 cpu minutes07:37
stubOne slony process had been waiting 4 minutes07:37
stubok, and nothing else would have had that process before hand.07:37
lifelessright07:37
stub(won't be the case after we install pgbouncer)07:38
lifelessstub: slonik will be direct won't it ?07:39
stubTrue07:39
stublifeless: So yeah, the queries are different where it matters. I created a new CTE with clause for bugs_fixed_upstream, and that obviously sucks.07:45
lifelessbbs, grabbing dinner07:45
stubYer, so the CTE will be executed once for the query (which is why we normally use them), then every insert is doing a full scan on that set to see if the EXISTS clause works. Your version with it inline will be doing an index lookup.07:48
LPCIBotYippie, build fixed!07:49
LPCIBotProject db-devel build #643: FIXED in 5 hr 47 min: https://lpci.wedontsleep.org/job/db-devel/643/07:49
lifelessstub: yah08:03
lifelessstub: you probably need to merge db-devel and push08:11
lifelessstub: the diff looks horrible08:11
stubWhoops... lp-land already away08:14
lifelessah well08:14
lifelesswe'll see what happens ;)08:14
stubPumped db-devel... must have had an older version?08:15
lifelesssomething :)08:17
stubIt landed (!)08:18
stubI think it is wierd because I ran bzr pump --from db-devel in my second pipe, so it had more db-devel than my first pipe08:20
stubLanded diff is sane anyway08:22
stubSo how to we get this on staging?08:22
lifeless4 hours for it to get through BB08:22
stubI can kill the existing job which should fail safely08:22
lifelessits a bit hacky08:23
lifelessbut I'd kill the existing slonik run08:23
stubGiven it has taken so long so far to apply the patch on the master, I'll assume we can't be arsed waiting for it to complete and run a second time on the slave.08:23
stubyup08:23
lifelessdrio your new file in08:23
lifelessand do the deploy thing yourself08:23
stubselect pg_cancel_backend(32012);08:24
stubThat should have killed the existing run08:25
stublosaping: Can we kick of a staging code update without the code update?08:26
lifelessstub: what do you mean ?08:26
stubAfter modifying the live tree, we don't want to have the scripts blat it08:27
lifelessstub: so, disable the cron job for a few minutes?08:28
lifelessstub: oh, I see where you are going; ignore me08:28
stubI can try running upgrade.py manually live. Not sure exactly what happens :)08:29
lifelessstub: as long as we avoid staging-restore auto running the old patch I'm happy ;)08:46
adeuringgood morning08:52
stubI can't see how to override the code update.08:53
stubapart from directly editing the script08:54
lifeless+1?09:02
lifelessstub: think you've got the patch right now ;>09:22
stubI'm sure there are other exciting ways I can screw up09:22
lifelessWas Not A Challenge! :)09:30
LPCIBotProject windmill-db-devel build #400: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/400/09:35
bigjoolsstub: did you benchmark 2208.75.0 on staging yet?10:00
stubnot the new version, no10:01
bigjoolsrunning for 24h on dogfood so far :/10:01
=== adeuring changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv), adeuring | Critical bugs: 205 - 0:[######=_]:256
rvbawgrant: Hi, I know you're in fire-fighting mode right now ... but if you have time after that (I know it's going to be late for you), I'll be happy if you could take a look at my (never-ending) multi parent thing.10:02
rvbawgrant: I you don't have time, that's really fine and we could talk about it on monday.10:02
lifelessbigjools: yes, kill it10:02
bigjoolslifeless: ?10:03
bigjoolsyou are reverting it?10:03
lifelessno, stub landed a fix10:04
lifelessits in db-devel10:04
bigjoolsgrar10:04
wgrantrvba: I'll try to get to it tonight, but no promises :/10:07
rvbawgrant: that's fine, thanks a lot.10:07
lifelessstub: whats the state of staging, given you're going ?10:10
stubNeeds a losa10:14
lifelessping for one in -ops ? ;)10:15
stubI've patched the tree but not sure how to continue. I could improvise... but I'd rather someone who knows the correct processes does it10:15
stublosaping here from ages ago...10:15
lifelessthey don't watch -dev10:15
lifelesstom requested that we only do such pings in -ops10:15
LPCIBotProject windmill-db-devel build #401: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-db-devel/401/10:16
jmlgood morning10:26
bigjoolsstub: how do I tell the difference between the old and new patch?  I just diffed DF's version against the one in db-devel and it's the same still.10:26
bigjoolsjml: morning10:26
lifelessbigjools: the fixedtasks CTE should be gone10:31
lifelessline 78 of the patch10:31
bigjoolslifeless: 78 for me says "OR ("10:32
lifelessbigjools: did you pull db-stable or db-devel10:32
bigjoolsdevel10:32
lifeless*db-devel*10:32
bigjoolsyes db-devel10:32
lifelesswhat revno do you have?10:32
bigjoolsit's hardly likely to be in devel is it :)10:32
bigjools1067910:33
bigjoolsis the one I am using now, but it says no diff to 1068510:33
wgrantdanilos: I played around with the new subscription listing thing this morning, and noticed a couple of things. The JS crashes for anonymous users, the subscriptions don't seem to be sorted, and I can't work out who is directly subscribed.10:34
lifelessbigjools: I see a huge diff10:34
daniloswgrant, well, everyone except the "Maybe notified" section are direct subscribers10:34
wgrantdanilos: Oh.10:34
wgrantThat's less than obvious.10:35
daniloswgrant, some of them might choose not to receive some emails10:35
wgrantI assumed they were just people who had selective structural subscriptions :(10:35
daniloswgrant, you can have selective direct subscriptions as well10:35
wgrantTrue.10:35
wgrantSo "Always notified" is sort of a misnomer?10:36
daniloswgrant, no, always notified is those who are always notified (i.e. "receive all"10:36
daniloswgrant, see eg. https://bugs.qastaging.launchpad.net/ubuntu/+source/network-manager/+bug/152754 for example of other sections10:36
_mup_Bug #152754: NetworkManager fails and crashes after resume from suspend <network-manager (Ubuntu):Incomplete> < https://launchpad.net/bugs/152754 >10:36
daniloswgrant, (you can see my different subscriptions on different levels)10:37
wgrantdanilos: aaaaaa10:37
wgrantSo "Maybe notified" is all the structsubs? I seeeee.10:37
wgrantHmm.10:37
daniloswgrant, yeah, "maybe" is all struct subs, all dupe subs10:37
daniloswgrant, for your own subscriptions, you are supposed to use "edit bug mail" link instead10:38
bigjoolslifeless: weird, a bzr ancestor diff shows nothing, but bzr missing shows 6 revisions10:38
wgrantdanilos: I've never liked that link... shouldn't it be "Edit subscriptions" or something that doesn't sound like it's editing emails? :/10:38
daniloswgrant, basically, this is something that's hard (and potentially expensive) to display in succint form with enough details (while staying correct)10:38
wgrantBut yeah.10:38
lifelessbigjools: uhm an ancestor diff should show nothing10:39
lifelessbigjools: because your current rev is the common ancestor10:39
wgrantI understand it's hard to calculate, yes. But it's not obvious what it's actually showing.10:39
bigjoolsgah, roight10:39
bigjoolslifeless: I am stupid10:39
lifeless:P10:39
daniloswgrant, yeah, I think the attempt was to clarify that this is about controlling the email you get from that particular bug10:39
lifelessbigjools: didn't you just implement a vcs layer for distros?10:39
daniloswgrant, and I think we just copied whatever was there already on the pillar pages which was "Edit bug mail"10:39
bigjoolsI'd like to think not :)10:40
daniloswgrant, I do see your point, but maybe you are thinking too much in terms of implementation, rather than trusting what the text says?10:40
LPCIBotProject parallel-test build #46: STILL FAILING in 1 hr 13 min: https://lpci.wedontsleep.org/job/parallel-test/46/10:40
daniloswgrant, "Notified of all changes" are, indeed, people/teams who get all the bug mail for this bug10:40
wgrantdanilos: Links like that have always talked about subscriptions before, or "Subscribe to bug mail", not "Edit bug mail"10:40
wgrantSince editing bug mail doesn't really make sense.10:41
lifelessdanilos: wgrant: was saying earlier that the js crashes for anonymous users10:41
daniloswgrant, hey, I am not going to argue over that, I don't like "Edit bug mail" anymore than you do10:41
wgrantHeh10:41
wgrantk10:41
daniloslifeless, yeah, I'm going to look into it10:41
wgrantBut yes, what lifeless said.10:41
daniloswgrant, ack10:41
lifelessdanilos: if it needs fixing we should roll the branch back10:41
wgrant... and land the unrelated pieces separately? :)10:42
lifelessdanilos: we have 30! revisions to deploy already, and more will queue up over the weekend10:42
lifelesswgrant: orthogonal discussion10:42
wgrantlifeless: Not quite... I rolled out up to that rev this morning.10:42
lifelesswgrant: danilos: grah sorry; stale info10:42
wgrantThere's only 8 in the queue ATM10:42
daniloslifeless, if it needs fixing, we'll fix it in the next few hours10:42
daniloslifeless, if subscribers list doesn't render for anonymous users, it's not a big deal since none of the JS works for them anyway10:43
daniloslifeless, iow, that should not block the rollout, but it should be fixed asap10:43
wgrantWhy shouldn't it block the rollout?10:44
daniloslifeless, I was also hoping to get more QA on this and try to deploy it on Monday instead (and any such problems should be fixed by then)10:44
daniloswgrant, why should it? I don't consider it a critical feature, and I value the time it takes to get something landed much more than this particular problem (if it's not as severe, like I said)10:45
wgrantdanilos: Some browser react to JS errors with violent popups.10:45
wgrantAnd we treat JS errors as OOPSes now.10:45
wgrantI think we'd consider it a blocker if the most commonly accessed page on LP OOPSed for anonymous users.10:46
daniloswgrant, right, and fixing the problem is probably going to take 30 mins as soon as I get to it, whereas just doing a revert is going to take me 10-20 mins10:46
wgrantI don't place much value on rushing a 12000 line branch through, TBH.10:46
wgrantIt deserves a lot of caution.10:46
daniloswgrant, I agree, and I want to exercise some10:46
daniloswgrant, but I want to be realistic as well10:47
allenapstub: Do you have some time to help analyse the slow queries in https://lp-oops.canonical.com/oops.py/?oopsid=1993DY5?11:04
stuballenap: Nope. Last minute scramble for a plane.11:05
allenapstub: Okay, have a good trip :)11:05
allenaplifeless: I know it's silly o'clock for you, but do you have some time to help analyse the slow queries in https://lp-oops.canonical.com/oops.py/?oopsid=1993DY5?11:06
lifelessallenap: lets get an analyse for the 3 second one off of prod, after gnuoy finishes fixing staging11:07
allenapOkay, cool. I have it ready on devpad.11:10
bigjoolslifeless: do you know how much quicker that new db patch is?  30 minutes and counting on DF so far.11:44
lifeless157 seconds when it ran on a temp table11:45
lifelessare you sure you got the new one ?11:45
bigjoolsyes11:45
bigjools:)11:45
lifelessallenap: ok so11:45
lifeless6.6 seconds11:46
allenapOuch.11:46
lifelesssee the actual time11:46
lifelesshash join on spr to dsd11:46
lifeless351K rows11:46
danilosadeuring, wallyworld_: hi, I need a review of a short branch (151 line), any takers? :)11:47
lifelessthe stats suggested to postgresql it would see 2.2K rows11:47
adeuringdanilos: I'll look11:47
lifelesscould be a stats issue11:47
danilosadeuring, thanks11:47
allenaplifeless: Can we get stats recalculated online?11:47
lifelessyeah11:48
lifelessanalyze tablename;11:48
lifelessallenap: how many rows are you expecting to return ?11:48
allenaplifeless: Around 1:1 with DistroSeriesDifference rows.11:49
allenapSo around the same as the big IN (5, 7, ...) clause.11:50
=== al-maisan is now known as almaisan-away
lifelessright, so the plan is problematic, or the model is problematic or the query is11:50
allenapHehe.11:50
LPCIBotProject windmill-devel build #243: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/243/11:51
allenaplifeless: Is an analyze cheap enough that we could try that first? DistroSeriesDifference until recently was empty for example, that might be a candidate for refreshing stats?11:51
allenapAlthough that doesn't appear in the bad part of the plan :-/11:52
lifelessyes11:52
lifelessbigjools: in other news, DF sucks.12:00
bigjoolslifeless: tell me something I don't know12:00
lifelessstaging appears to have run the update12:00
bigjoolshow slow?12:00
lifelesstimes will be in the logs12:01
bigjoolswhat, you're making me look myself?  tsk :)12:01
lifeless11pm and I'm sick.... so yes :)12:01
lifelessalso woo12:03
lifeless27ms stats portlets here we come12:03
bigjoolswhich file has the timings?12:04
lifelessthe staging-restore-$date IIRC12:04
bigjools2011-06-17 07:24:24 ERROR   slonik script failed12:04
bigjoolsFri Jun 17 07:24:24 UTC 2011 ERROR: Failed to run upgrade.py12:04
lifelessit won't have rsynced yet so you'll need to pull it over by hand12:04
bigjoolsk12:04
lifelessthats the run we canned12:04
adeuringdanilos: nice small branch, r=me12:07
danilosadeuring, thanks12:08
bigjools lifeless: so it's far easy to look in LaunchpadDatabaseRevision.... ;)12:08
bigjools7 mins on staging it seems12:09
lifelessallenap: so not stats12:10
allenapNo.12:11
lifeless34*1054712:11
lifeless35859812:11
lifelessthat explains the 351K rows I think12:11
lifeless10K spph * 34 archive12:12
allenapI wonder what we can do to narrow the spph set down.12:14
lifelesswell12:14
lifelessits a cross join atm12:14
lifelessso one thing to try is an inner join12:15
lifelessrather than a constrained cross join12:15
allenaplifeless: Okay, I don't know enough to know how to do that, but I can go and find out. In fact, I'd be happy to do that. But if you want to see some swifter progress you'll have to help me out on that one :)12:16
allenaplifeless: Okay, I think I know how.12:18
lifelessquery in ~robertc on carob12:21
allenaplifeless: Thanks :)12:26
lifelessah, I misread12:28
lifelessproblem doing this tired12:28
lifeless                           ->  Index Scan using sourcepackagerelease_pkey on sourcepackagerelease  (cost=0.00..0.87 rows=1 width=8) (actual time=0.012..0.013 rows=1 loops=351423)12:28
lifeless                                 Index Cond: (sourcepackagerelease.id = sourcepackagepublishinghistory.sourcepackagerelease)12:29
lifelessthats the 350K12:29
lifelessspr12:29
lifelessI have to crash12:29
lifelesshowever12:29
lifelesssome thoughts:12:29
lifeless - we know the distroseries, so can constrain the query further12:29
lifeless - there are only 34 primary archives, so we can eliminate the archive join if we want (may not matter)12:29
lifelessand that would let us drop the distroseries join too12:30
wgrantjelmer: Thanks!12:31
allenaplifeless: Thanks for your help :) Have a good weekend.12:31
jelmerwgrant: For what, wasting your morning ? :)12:32
lifelesserm, no first analysis was right I think12:32
lifelesswe're expanding out through archive12:32
lifelessbecause you have this12:32
lifelessconstraints - join - join - join - join - constraints12:32
jelmerwgrant: Sorry about the importd upgrade trouble, I wonder how neither of us had caught this on staging though.12:32
lifelesseach join, even inner ones, will multiply until it can constraint12:33
wgrantjelmer: For fixing that :) (and it was my afternoon/evening, not my morning!)12:33
wgrantjelmer: Not sure... I wonder if it lied about the format.12:33
wgrantPossibly the import was new, but still had the old metadata, or something.12:33
wgrantAnyway, was easily fixed.12:33
LPCIBotProject windmill-devel build #244: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-devel/244/12:33
daniloswgrant, fwiw, the fix for the anonymous users JS is playing in ec212:33
wgrantAlthough it was a major panic moment before I realised there would probably still be a copy on escudero...12:33
wgrantdanilos: Ah, great! That was quick.12:34
daniloswgrant, if I did more integration testing for JS stuff (I did loads of JS unit tests, with the ratio of tests to code being roughly 2:1 in number of lines), I would not have let it happen; unfortunately, I don't and I did12:36
daniloswgrant, so, I hope to start on more windmill tests if they are the way to go for this12:36
lifelessnn, must go12:38
jelmerwgrant: Yeah, I can imagine..12:40
=== bac changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: adeuring, bac | Critical bugs: 205 - 0:[######=_]:256
=== matsubara-afk is now known as matsubara
matsubaramorning13:08
allenapSorry lifeless, had to go. My missus had been badgering me for half an hour already.13:08
LPCIBotProject windmill-devel build #245: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-devel/245/13:16
jmlmatsubara: good morning13:23
matsubarahi jml13:23
danilosmatsubara, good morning as well :)13:23
jmlbigjools: can  we have a call at 4pm to touch base about derivative distros?13:24
matsubarahey danilos13:24
matsubaradanilos, btw, did that branch land?13:24
danilosmatsubara, fwiw, new subscription stuff is up on qastaging, there is a bug with anon users, but the fix is playing in ec2 already13:25
danilosmatsubara, yeah, it did, it should be good to test13:25
matsubaradanilos, cool. I'll give it a try today.13:25
danilosmatsubara, excellent, you can also report any problems you find on bug 772754, I'll be reading it13:25
_mup_Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-needstesting> <story-better-bug-notification> <Launchpad itself:Fix Committed by gary> < https://launchpad.net/bugs/772754 >13:25
matsubarawill do13:26
danilosmatsubara, thanks!13:26
matsubaradanilos, np. Yesterday you mentioned r13248 as the one you were waiting. Was that a typo or that's the one with the subsequent fix playing on ec2?13:27
danilosmatsubara, no, that's the one which includes everything but the fix for anonymous users (basically, there is a JS error for anonymous users), which is in bug 79862213:28
_mup_Bug #798622: JS failure for anonymous users on bug page <story-better-bug-notification> <Launchpad itself:In Progress by danilo> < https://launchpad.net/bugs/798622 >13:28
danilosmatsubara, so, qastaging should be good for QAing for logged in users (I already did some, as did wgrant: he found the anon user problem)13:28
matsubaradanilos, there's no r13248 on qastaging. the one available is r13243 linked to bug #772754 so I assume that's the right number not 13248 :-)13:30
_mup_Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-needstesting> <story-better-bug-notification> <Launchpad itself:Fix Committed by gary> < https://launchpad.net/bugs/772754 >13:30
danilosmatsubara, qastaging for me has "r13251" in the footer13:31
matsubaradanilos, I'm looking at the dashboard. that might be a bit outdated13:31
* danilos looks at his dashboard, and thinks redline at 7000 rpm is too low13:32
matsubaraactually, I'm confusing the qastaging with what's in the dashboard13:32
danilosmatsubara, I am thoroughly confused since I don't know what the dashboard is :)13:32
matsubara13243 is block the rollout and qastaging has 1325113:32
matsubaradanilos, http://lpqateam.canonical.com/devops/13:32
danilosmatsubara, ah, nice13:33
danilosmatsubara, right, so we still haven't marked it as qa-ok because I'd prefer to get a little bit more QA done first13:33
matsubaraI see now, should have looked at the development-stable report. I can see that 13243 was reverted by 13247 and then 13248 landed afterwards13:34
matsubarano worries. I unconfused myself and things should be ok to start now. thanks!13:34
danilosmatsubara, cool, thanks13:35
=== Ursula is now known as Ursinha
LPCIBotProject windmill-devel build #246: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-devel/246/13:56
bigjoolsjml: not feeling too well but if I am here, yes14:38
jmlbigjools: cool. thanks.14:38
LPCIBotProject windmill-devel build #247: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/247/15:05
bachi adeuring -- i've just grabbed wallyworld_'s branch.  you're not already looking at it are you?15:09
adeuringbac: no15:09
LPCIBotProject parallel-test build #47: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/parallel-test/47/15:14
=== jtv1 is now known as jtv
=== almaisan-away is now known as al-maisan
* jml ducking out for a bit, back in ~20m15:54
nigelbbigjools: heh, lol That as a fun commit :-)16:06
nigelb*was16:06
bigjoolsjml: you no wanna call then?16:06
bigjoolsnigelb: the comments are hilarious16:07
flacostejames_w: i again need your help to qa a change, could you confirm that you are able to call setBranch with the UPDATES or RELEASE pocket on lucid/maverick on qastaging? pretty please16:10
jmlbigjools: I do, just need to get some other stuff out of the way first.16:11
LPCIBotProject windmill-db-devel build #402: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/402/16:11
nigelbbigjools: Yup, with new memes!16:13
sinzuijcsackett, mumble?16:17
jcsackettsinzui: sure.16:17
sinzuijcsackett, If you cannot hear me I think I need to restart mumble16:18
jmlbigjools: ok. pie safely consumed, call can proceed16:18
jcsackettsinzui: i heard you. seems you could not hear me. i'm restarting.16:18
bigjoolsthe pie should be on ya face16:18
jmlbigjools: 200 bugs stand between my face and pie.16:19
bigjoolsjml: gimme a couple of mins16:20
jmlalso, a hot steak & stilton pie would be inappropriate for face-planting16:20
sinzuimaybe I can get empiphay to work16:20
jcsackettsinzui: i'm having a connection hiccup.16:22
sinzuiempathy say I have sip, but I cannot use it, maybe you can call me16:23
jcsackettsinzui: i hear you, it seems you cannot hear me. i will try sip.16:26
flacostemaxb: the fix for the importer is deployed to qastaging, i asked james_w to confirm that it works, maybe you can do it also?16:28
maxbflacoste: No, I'm not a core-dev (or any sort of uploader actually)16:30
jcsackettsinzui: lost call. recalling in a moment.16:31
cr3what part of launchpad adds the utc timezone to a datetime between getting it from the database (which always has "without time zone") to the restful interface (which has the timezone as "+00:00", therefore explicitly defined somewhere)?16:31
bigjoolsjml: sorry.  what is your call platform of choice?16:33
jmlbigjools: voip16:33
jmlbigjools: skype for second preference16:34
bigjoolsgah16:34
jml(am I allowed to have a second preference?)16:34
bigjoolspulseaudio keeps usurping Twinkle's alsa connection16:34
bigjoolsand twinkle doesn't know about PA16:35
jmlahh, I see.16:35
bigjoolsI hate PA16:35
jmlpulseaudio is great in natty for me16:35
jmlI guess that's because everything is using it.16:35
bigjoolsI hate it because it limits my options16:36
bigjoolsI am on skype16:36
jmlok16:36
=== al-maisan is now known as almaisan-away
jmlbigjools: have you seen https://dev.launchpad.net/Projects/DerivativeDistributions ?16:43
cr3nevermind folks, I found that it's UtcDateTimeCol that sets the tzinfo. oddly though, that doesn't seem to behave the same as non-sqlobject derived storm variables which only set the tzinfo when writing to the db16:52
LPCIBotProject windmill-devel build #248: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-devel/248/16:56
=== almaisan-away is now known as al-maisan
=== salgado is now known as salgado-lunch
LPCIBotProject windmill-devel build #249: STILL FAILING in 43 min: https://lpci.wedontsleep.org/job/windmill-devel/249/17:44
bigjoolsbye, have a good weekend everyone18:03
=== Ursinha is now known as Ursinha-lunch
=== salgado-lunch is now known as salgado
LPCIBotProject db-devel build #645: FAILURE in 5 hr 35 min: https://lpci.wedontsleep.org/job/db-devel/645/18:47
=== al-maisan is now known as almaisan-away
LPCIBotProject windmill-db-devel build #403: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-db-devel/403/19:31
fjlacostemaxb: james_w confirmed the fix is working, I've changed the Debian distribution maintainers, so everything should be back to normal at the next nodowntime deploy19:44
fjlacostemaxb: probably Sunday (antepodean Monday) as we don't roll-out just before the week-end19:44
maxbAwesome, thanks, so on Monday we should be back in business19:44
fjlacostemaxb: again, i'm very sorry for the breakage19:44
fjlacosteand apologize for it19:44
maxbSometimes regressions happen - thanks for pushing through a fix promptly19:47
fjlacostethanks for your understanding19:52
LPCIBotProject parallel-test build #48: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/parallel-test/48/19:54
fjlacosteanyone knows what LostObjectError: means?20:06
fjlacostei'm writing a view test and getting this20:06
=== Ursinha-lunch is now known as Ursinha
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
fjlacostewow, that's intersting20:17
fjlacosteit happens because there is actually a validation error in the form20:18
fjlacostewhich resets the transaction20:18
fjlacosteand then when then a storm attribute is accessed after the end of transaction20:18
fjlacosteand boom20:18
baci'm trying to export a new class to the api and i'm getting:20:24
bacUnknownEntryAdapter: No IEntry adapter found for IProcessorFamily (web service version: devel).  Encountered as a result of the entry interface <InterfaceClass lp.soyuz.interfaces.archive.IArchiveEntry_devel>, field 'enabled_restricted_families'.20:24
bacnormally that would be due to a circular dependency, but i've handled that (i think)20:24
bacfjlacoste: you now what ^^ could be?20:25
fjlacostebac: that's because the IProcessFamily isn't exported20:25
fjlacosteyou need to import it into lp.soyuz.interfaces.webservice probably20:25
bacfjlacoste: ah, but it is:  http://pastebin.ubuntu.com/628552/20:25
bacer20:26
fjlacosteit's not :-)20:26
fjlacostesee lp.soyuz.interfaces.webservice20:26
fjlacosteyou need to import it in there20:26
fjlacostebecause only definitions in that file are processed for export20:26
bacoh!20:26
bacfjlacoste: you are a super-genius!  thanks!20:29
=== fjlacoste is now known as flacoste
flacosteabentley: is it possible to set the pre-requisite branch from lp-propose?20:55
abentleyflacoste: not directly, but it's done automatically if you use pipelines.20:56
flacostei didn't20:56
flacosteis it possible to set the pre-requisite branch on the created merge proposal?20:56
flacosteor i need to resubmit20:56
abentleyflacoste: You can set the pre-requisite branch on the created merge proposal by re-submitting.20:57
flacostethanks20:58
flacostebac: a review for you pretty please: https://code.launchpad.net/~flacoste/launchpad/bug-781993-2/+merge/6505820:58
flacosteshould be simple20:58
bacflacoste: sure21:04
bacflacoste: can you get rid of the conflicts?21:04
bacflacoste: actually the conflicts don't show up in the MP.  looks good.21:13
flacostethanks for the review bac21:34
sorenHi. The person listed as help contact in #launchpad isn't online and noone seems to respond there, so I'm escalating here. Hope you can forgive me. Uploads to PPA's don't work. I get "Changes file must be signed with a valid GPG signature: Verification failed 3 times: ['General error', 'General error', 'General error'] : Permission denied." Ampelbein in #launchpad pointed me to bug 757248.21:43
_mup_Bug #757248: poppy-sftp's signature checking relies on long-term survival of a directory in /tmp <poppy> <qa-ok> <regression> <Launchpad itself:Fix Released by wallyworld> < https://launchpad.net/bugs/757248 >21:43
soren...which, interestingly, was marked "fixed in stable" exactly one month ago today.21:43
* soren <- saaaad panda21:44
flacostesoren: i asked IS to check what's up21:50
sorenflacoste: ta very much.21:51
flacostesoren: someone internal says that the upload should still be accepted21:51
flacosteit is an avatar of the old bug21:52
lifelesswell21:52
flacostebut it shouldn't reject your upload21:52
lifelessit might be21:52
sorenflacoste: Ah, yes, you seem to be right. I didn't even bother checking. It looked pretty darn fatal to me :)21:53
lifelessit does; we're going to have lots of folk double-upload and panic if this is happening again21:53
flacosteindeed, soren can you file a new bug (one might already exists though)21:54
sorenflacoste: Sure21:55
=== matsubara is now known as matsubara-afk
flacostethanks21:57
sorenhttps://bugs.launchpad.net/launchpad/+bug/79895721:57
_mup_Bug #798957: Uploads are seemingly (but not actually) rejected <Launchpad itself:New> < https://launchpad.net/bugs/798957 >21:57
lifelesssoren: perhaps your gpg key expired recently or something?22:02
LPCIBotProject windmill-devel build #250: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/250/22:05
sorenlifeless: It started happening a couple of hours ago for both the automated openstack package uploader thingamajig and myself.22:06
sorenlifeless: ...and they both never expire. I checked :)22:10
lifelessk22:10
=== Ursinha is now known as Ursinha-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!