[00:18] <wgrant> lifeless, sinzui: I was planning to do a deployment this morning.
[00:19] <wgrant> But then a certain monstrous completely un-QAable undisable hopeless merge failure buggy criss-cross branch landed.
[00:19] <wgrant> We won't be deploying for a few days :/
[00:37] <wallyworld_> wgrant: ^^^^^^ which branch out of interest?
[01:21] <wgrant> It's tempting to roll it back on principle.
[01:21] <wgrant> There are bugs.
[01:21] <wgrant> Not fatal if it was flagged.
[01:21] <wgrant> But...
[01:21] <wgrant> It's not.
[01:27] <StevenK> Wasn't it rolled back once already?
[01:27] <wgrant> Yes.
[01:30] <wgrant> Yay
[01:30] <wgrant> It crashes in Firefox 3.5, at least sometimes.
[01:32] <wgrant> :/ this is a big UI change.
[01:33] <wgrant> No longer obvious how to subscribe, and you can't tell who is directly subscribed to the bug...
[01:45] <LPCIBot> Project parallel-test build #44: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/parallel-test/44/
[01:51] <LPCIBot> Project windmill-db-devel build #398: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/398/
[01:51] <lifeless> wallyworld_: the bugsubscription one
[01:52] <wallyworld_> the 12000 line one i assume
[01:53] <StevenK> wgrant: Can haz link to notify() bug?
[01:54] <wgrant> StevenK: There doesn't seem to be one. Check #launchpad from 21:48 yesterday.
[02:02] <LPCIBot> Project db-devel build #642: STILL FAILING in 5 hr 41 min: https://lpci.wedontsleep.org/job/db-devel/642/
[02:05] <lifeless> wgrant: if its buggy, its a rollback
[02:27] <wgrant> lifeless: The ordering is apparently arbitrary and the JS crashes when not logged in.
[02:27] <wgrant> And it completely changes the UI of a feature we announced a week ago :/
[02:29] <lifeless> crashing when not logged in is a pretty big blocker
[02:31] <wgrant> It's not a regression from what we have now.
[02:31] <wgrant> Ohhh, yes it is, actually.
[02:32] <wgrant> Since at least now the direct subscribers are shown.
[02:33] <lifeless> wgrant: crashing is a regression
[02:33] <lifeless> wgrant: the UI aspect is deliberate AIUI. IMBW
[02:37] <wgrant> lifeless: Deliberate, but untested and a break from what we announced :/
[02:50] <huwshimi> lifeless: ITMFTFOWTFYW
[02:50] <lifeless> huwshimi: ?
[02:51] <huwshimi> lifeless: It takes me forever to figure what you write
[02:51] <huwshimi> lifeless: Sentences like this: "lifeless: wgrant: the UI aspect is deliberate AIUI. IMBW"
[02:51] <lifeless> huwshimi: sorry !
[02:52] <huwshimi> lifeless: haha, it's ok
[02:52] <huwshimi> lifeless: Most people on Launchpad do it, I just spend half my time opening a browser and googling what everyone is saying
[02:52] <wgrant> s/Launchpad/IRC/
[02:52] <huwshimi> wgrant: True
[03:21] <LPCIBot> Project windmill-devel build #241: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/241/
[03:30] <wgrant> StevenK: Any luck with the notifications?
[03:31] <StevenK> wgrant: I've been a little distracted, sorry. Still digging.
[03:35] <StevenK> wgrant: Unsigned changes file, since it comes from a buildd, so it adds Changed-By.
[03:36] <wgrant> StevenK: Ah.
[03:36] <wgrant> StevenK: How did this not happen before?
[03:37] <StevenK> I suspect the old code only did that if it wasn't a PPA.
[03:37] <wgrant> Probably.
[03:37] <wgrant> I also suspect we need more unit tests.
[03:41] <StevenK> wgrant: I think the bug is firmly in get_recipients(), I was just going to read the old code
[03:47] <StevenK> wgrant: Hm, the bug seems present in the old code too
[03:47] <wgrant> StevenK: But I know it didn't happen :/
[03:47] <wgrant> StevenK: Possibly archiveuploader just didn't notify?
[03:47] <wgrant> I'll look.
[03:47] <StevenK> But I didn't change that bit!
[03:48] <StevenK> wgrant: What I mean is _getRecipients() always added Changed-By if packageupload.signing_key is None
[03:48] <wgrant> Yeah.
[03:51] <wgrant>         # If this is not a PPA, we also consider maintainer and changed-by.
[03:51] <wgrant>         if self.signing_key and not self.isPPA():
[03:51] <wgrant> StevenK: From the original code.
[03:52] <StevenK> wgrant: But look above that
[03:52] <StevenK> wgrant:         debug(packageupload.logger,
[03:52] <StevenK>             "Changes file is unsigned, adding changer as recipient")
[03:52] <StevenK>         candidate_recipients.append(changer)
[03:53] <wgrant> Fair point.
[03:53] <wgrant> Oh.
[03:54] <wgrant> sec
[03:54] <wgrant> Hm.
[03:54] <wgrant> The buildds override Maintainer, but not Changed-By.
[03:54] <wgrant> So Changed-By should be the source Changed-By... hm.
[03:55] <wgrant> This will all have to be rewritten soon anyway :/
[03:56] <wgrant> 2011-06-17 00:49:35 DEBUG     Subject: [PPA stackapplet-dev-stackapplet] stackapplet_1.5+147~maverick1_source.changes rejected
[03:56] <wgrant> 2011-06-17 00:49:35 DEBUG     Sender: Ubuntu Installer <archive@ubuntu.com>
[03:56] <wgrant> 2011-06-17 00:49:35 DEBUG     Recipients: Launchpad Beta Testers <noreply@launchpad.net>
[03:56] <wgrant> what
[03:59] <StevenK> Another rewrite? :-/
[03:59] <wgrant> StevenK: For DDs, yes.
[03:59] <LPCIBot> Yippie, build fixed!
[03:59] <LPCIBot> Project devel build #813: FIXED in 5 hr 24 min: https://lpci.wedontsleep.org/job/devel/813/
[03:59] <wgrant> StevenK: Once we're copying between primary archives.
[03:59] <wgrant> StevenK: Otherwise Debian people will probably be notified, which would be Bad.
[04:00] <wgrant> And Ubuntu people would definitely be notified about Linaro builds.
[04:00] <StevenK> Indeed
[04:01] <StevenK> wgrant: So, shall we just remove the else bit that adds Changed-By for unsigned uploads?
[04:01] <wgrant> StevenK: No.
[04:01] <wgrant> StevenK: We need to work out what has changed.
[04:02] <StevenK> Between a refactor a month ago, a rewrite 2 weeks ago, and 2 tweaks. :-/
[04:03] <wgrant> Interesting. Logs start on the 19th, and the first occurence was the 30th.
[04:03] <wgrant> It was broken then.
[04:04] <StevenK> The 30th of May? That's the rewrite
[04:06] <wgrant> So it was.
[04:06] <wgrant> So.
[04:06] <wgrant> I can't see why it wasn't notifying before :/
[04:06] <StevenK> What do the DEBUG logs show for occurances of this problem prior to the rewrite?
[04:06] <wgrant> There are none.
[04:07] <wgrant> But they only go back 11 days earlier.
[04:07] <StevenK> Bugger.
[04:07] <wgrant> That exception is somehow not seen at all before then.
[04:07] <wgrant> (which is not unthinkable; it also wasn't seen between the 3rd and the 10th of this month)
[04:07] <StevenK> Yes, they don't happen very often.
[04:09] <wgrant> https://lists.ubuntu.com/archives/ubuntu-reviews/2010-April/000281.html
[04:09] <wgrant> That's a log from a year ago.
[04:09] <wgrant> There are more recent ones, but that came up quickly in Google.
[04:10] <StevenK> That doesn't trip the Changes file is unsigned, adding changer as recipien
[04:10] <StevenK> Sigh. But you get the idea.
[04:11] <StevenK> In fact, that doesn't send a notification at all.
[04:12] <wgrant> I wonder...
[04:13] <wgrant> It's possible that this is unrelated.
[04:13] <wgrant> process-upload was originally run by b-m, with -M
[04:13] <wgrant> So no mail was sent.
[04:13] <wgrant> But then late last year that changed...
[04:14] <wgrant> Could this have been happening for 6 months? Odd that we only heard about it now.
[04:14] <StevenK> Indeed.
[04:16] <wgrant> If only we kept logs for a while :/
[04:16] <wgrant> What we keep is only 13MB :(
[04:16] <wgrant> I don't understand why we don't keep pretty much all our logs for just about ever :(
[04:41] <jtv> hi wallyworld_, I'm back.  Any reviews to mentor?
[04:43] <wallyworld_> jtv: hi. none yet
[04:43] <jtv> ok
[04:45] <jtv> wgrant: is there something similar to the qa-tagging going on with bug statuses?  I notice you've marked a few bugs Fix Released before all their branches had landed.
[05:45] <StevenK> wgrant: Since we don't have logs, we have no way to be certain -- but the work that divorced process-upload and b-m could have caused this? So I'm off the hook?
[05:46] <wgrant> StevenK: I think so :/
[05:46] <wgrant> jtv: Those didn't look like multi-branch bugs, and they weren't landed with [incr], so I didn't notice, sorry.
[06:00] <wallyworld_> jtv: i'm off to pick up the kid from school. still no reviews. will check again later
[06:09] <jtv> ok wallyworld_
[06:10] <jtv> wgrant: I just knew there was something I wasn't thinking of… incr.  It's getting altogether a bit finicky for me, this tagged-double-bookkeeping.
[06:11] <wgrant> Indeed.
[06:11] <jtv> OMG my branch made it in.  Starting at 07:30 pays off.
[06:11] <jtv> I think that's about 8K of today's 9K lines landed.  :)
[06:12] <nigelb> so, I have the sort bug still pending. where is a good place to add a util function?
[06:14] <LPCIBot> Project parallel-test build #45: STILL FAILING in 1 hr 16 min: https://lpci.wedontsleep.org/job/parallel-test/45/
[06:19] <LPCIBot> Project windmill-db-devel build #399: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/399/
[06:42] <StevenK> Does anyone know lib/lp/{bugs,code,soyuz}/notifications/ is used for? All three of them contain a zero-length __init__.py
[07:07] <lifeless> stub: hi
[07:07] <stub> Yo
[07:08] <lifeless> stub: I did that lock experiment; the blocked alter table blocked out new readers.
[07:08] <lifeless> session one: begin; set transaction isolation level serializable; select id from bug;
[07:08] <lifeless> session two: being; alter table bug add column foo int;
[07:08] <lifeless> session three: begin; set transaction isolation level serializable; select id from bug;
[07:09] <lifeless> two and three both block
[07:09] <lifeless> doing a rollbackl in session one - andtwo and three unblock
[07:09] <lifeless> well, once two is committed.
[07:11] <stub> ok. There might be something more complex going on (transactions working with related tables causing the ALTER TABLE to block).
[07:11] <stub> Or not
[07:12] <lifeless> stub: staging has displayed one of these typical 'fail to apply' all day
[07:12] <lifeless> stub: hloeung and I have been gathering data and poking at it
[07:12] <stub> But so far, I've never had success doing anything that needed to get a lock on Person on production. 'ALTER TABLE BugSummary ADD CONSTRAINT bugsummary__viewed_by__fk FOREIGN KEY (viewed_by) REFERENCES Person' for instance.
[07:12] <lifeless> stub: the appservers and scripts were running
[07:13] <stub> Has the lag check rung any alarms?
[07:13] <lifeless> yeah
[07:14] <stub> So the script is either blocked for some reason (eg. a DB patch trying to grab a lock on the Person table ;) ), or the daemons have fallen over, or replication is broken
[07:14] <stub> Good the lag check is alerting us
[07:14] <lifeless> https://pastebin.canonical.com/48634/
[07:14] <lifeless> pending locks
[07:15] <lifeless> https://pastebin.canonical.com/48634/
[07:15] <lifeless> bah
[07:15] <lifeless> https://pastebin.canonical.com/48632/
[07:16] <stub> It seems to be currently running my bugsummary update. This is the first run on a production like system... the query might just be way slow.
[07:16] <lifeless> stub: 5 hours ?
[07:16] <StevenK> stub: 2208-00-0.sql is current, or is that the amalgamation of 2207? Happy to wait until you're done digging.
[07:16] <lifeless> stub: if its the one I prepared, it runs into a temp table in a few minutes
[07:17] <stub> There is no 2207 in our tree. launchpad-2208-00-0.sql is the current baseline.
[07:19] <StevenK> stub: Right, but 2207 used to exist
[07:19] <stub> lifeless: I didn't notice or decode your one (need to feed that stuff through a formatter).
[07:19] <stub> yes
[07:19] <LPCIBot> Project windmill-devel build #242: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/242/
[07:19] <StevenK> stub: IE, where did 2207-* move to? 2208-00-0, or somewhere else?
[07:19] <stub> It was either moved to 2208 or deleted
[07:20] <stub> annotate on 2208 should give you the revno
[07:20] <StevenK> I'd like to drop a view from 2208-00-0
[07:20] <StevenK> (Not yet, mind)
[07:20] <stub> That needs a DB patch. You don't modify the baseline - it is a snapshot of production at a point in time.
[07:21] <StevenK> stub: Okay. So for an ec2 run (for instance), the view will get created and then removed?
[07:21] <stub> If you hack the baseline, it will never get applied to production
[07:21] <stub> Yes
[07:21] <StevenK> stub: Okay, thanks.
[07:21] <stub> Until we rollup the next baseline.
[07:22] <stub> (probably next month - we are in the 70's already)
[07:22] <StevenK> Bring on 2209
[07:23] <stub> Any particular reason? Applying those patches takes about a second.
[07:23] <StevenK> No, just forward progress :-)
[07:23] <stub> Going back to 1 is backwards progress ;)
[07:23] <StevenK> Haha
[07:25] <stub> lifeless: So it is my bugsummary population query running. Possibly an earlier one was slow - I won't get timings until (if) it completes.
[07:26] <lifeless> stub: no, it was that one
[07:26] <lifeless> just digging up the query I had
[07:26] <stub> lifeless: We can either wait or kill it - things should rollback fine at the current point in time.
[07:26] <lifeless> stub: I'd like to debug whats up with it
[07:26] <stub> Its chewing 100% of a core.
[07:27] <stub> the EXISTS always is dangerous to performance.
[07:27] <stub> Your version had that?
[07:27] <stub> Anyway - we can fix it directly in the existing patch since it hasn't landed yet...
[07:27] <lifeless> yeah
[07:28] <lifeless> https://bugs.launchpad.net/launchpad/+bug/793848/comments/4
[07:28] <_mup_> Bug #793848: Distribution:+bugtarget-portlet-bugfilters-stats timeouts <dba> <timeout> <Launchpad itself:Triaged by stub> < https://launchpad.net/bugs/793848 >
[07:28] <lifeless> had my query
[07:28] <stub> But the cpu utilization means it isn't blocked.
[07:28] <lifeless> yah
[07:28] <stub> Maybe something stoopid like a missing join condition
[07:28] <lifeless> so we need to diff the two queries
[07:29] <lifeless> stub: it was blocked for a while
[07:29] <lifeless> stub: was waiting for a lock on the sl_event
[07:29] <stub> Some other process inserting data that needed to be replicated.
[07:32] <lifeless> stub: it looked like a slon had started up badly and was deadlocking itself
[07:33] <stub> lifeless: So how long has this patch been running do you think?
[07:34] <lifeless> stub: https://pastebin.canonical.com/48632/
[07:34] <lifeless> stub: submitted 11 hours or so ago
[07:35] <lifeless> postgres 32012 99.9  2.2 2280364 726944 ?      Rs   Jun16 567:55  \_ postgres: slony lpmain_staging [local] INSERT
[07:36] <stub> lifeless: looking at the slony stuff being executed, it was blocked then on the event lock
[07:37] <lifeless> stub: so, tha db patch has used 567 cpu minutes
[07:37] <stub> One slony process had been waiting 4 minutes
[07:37] <stub> ok, and nothing else would have had that process before hand.
[07:37] <lifeless> right
[07:38] <stub> (won't be the case after we install pgbouncer)
[07:39] <lifeless> stub: slonik will be direct won't it ?
[07:39] <stub> True
[07:45] <stub> lifeless: So yeah, the queries are different where it matters. I created a new CTE with clause for bugs_fixed_upstream, and that obviously sucks.
[07:45] <lifeless> bbs, grabbing dinner
[07:48] <stub> Yer, so the CTE will be executed once for the query (which is why we normally use them), then every insert is doing a full scan on that set to see if the EXISTS clause works. Your version with it inline will be doing an index lookup.
[07:49] <LPCIBot> Yippie, build fixed!
[07:49] <LPCIBot> Project db-devel build #643: FIXED in 5 hr 47 min: https://lpci.wedontsleep.org/job/db-devel/643/
[08:03] <lifeless> stub: yah
[08:11] <lifeless> stub: you probably need to merge db-devel and push
[08:11] <lifeless> stub: the diff looks horrible
[08:14] <stub> Whoops... lp-land already away
[08:14] <lifeless> ah well
[08:14] <lifeless> we'll see what happens ;)
[08:15] <stub> Pumped db-devel... must have had an older version?
[08:17] <lifeless> something :)
[08:18] <stub> It landed (!)
[08:20] <stub> I think it is wierd because I ran bzr pump --from db-devel in my second pipe, so it had more db-devel than my first pipe
[08:22] <stub> Landed diff is sane anyway
[08:22] <stub> So how to we get this on staging?
[08:22] <lifeless> 4 hours for it to get through BB
[08:22] <stub> I can kill the existing job which should fail safely
[08:23] <lifeless> its a bit hacky
[08:23] <lifeless> but I'd kill the existing slonik run
[08:23] <stub> Given it has taken so long so far to apply the patch on the master, I'll assume we can't be arsed waiting for it to complete and run a second time on the slave.
[08:23] <stub> yup
[08:23] <lifeless> drio your new file in
[08:23] <lifeless> and do the deploy thing yourself
[08:24] <stub> select pg_cancel_backend(32012);
[08:25] <stub> That should have killed the existing run
[08:26] <stub> losaping: Can we kick of a staging code update without the code update?
[08:26] <lifeless> stub: what do you mean ?
[08:27] <stub> After modifying the live tree, we don't want to have the scripts blat it
[08:28] <lifeless> stub: so, disable the cron job for a few minutes?
[08:28] <lifeless> stub: oh, I see where you are going; ignore me
[08:29] <stub> I can try running upgrade.py manually live. Not sure exactly what happens :)
[08:46] <lifeless> stub: as long as we avoid staging-restore auto running the old patch I'm happy ;)
[08:52] <adeuring> good morning
[08:53] <stub> I can't see how to override the code update.
[08:54] <stub> apart from directly editing the script
[09:02] <lifeless> +1?
[09:22] <lifeless> stub: think you've got the patch right now ;>
[09:22] <stub> I'm sure there are other exciting ways I can screw up
[09:30] <lifeless> Was Not A Challenge! :)
[09:35] <LPCIBot> Project windmill-db-devel build #400: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/400/
[10:00] <bigjools> stub: did you benchmark 2208.75.0 on staging yet?
[10:01] <stub> not the new version, no
[10:01] <bigjools> running for 24h on dogfood so far :/
[10:02] <rvba> wgrant: Hi, I know you're in fire-fighting mode right now ... but if you have time after that (I know it's going to be late for you), I'll be happy if you could take a look at my (never-ending) multi parent thing.
[10:02] <rvba> wgrant: I you don't have time, that's really fine and we could talk about it on monday.
[10:02] <lifeless> bigjools: yes, kill it
[10:03] <bigjools> lifeless: ?
[10:03] <bigjools> you are reverting it?
[10:04] <lifeless> no, stub landed a fix
[10:04] <lifeless> its in db-devel
[10:04] <bigjools> grar
[10:07] <wgrant> rvba: I'll try to get to it tonight, but no promises :/
[10:07] <rvba> wgrant: that's fine, thanks a lot.
[10:10] <lifeless> stub: whats the state of staging, given you're going ?
[10:14] <stub> Needs a losa
[10:15] <lifeless> ping for one in -ops ? ;)
[10:15] <stub> I've patched the tree but not sure how to continue. I could improvise... but I'd rather someone who knows the correct processes does it
[10:15] <stub> losaping here from ages ago...
[10:15] <lifeless> they don't watch -dev
[10:15] <lifeless> tom requested that we only do such pings in -ops
[10:16] <LPCIBot> Project windmill-db-devel build #401: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-db-devel/401/
[10:26] <jml> good morning
[10:26] <bigjools> stub: how do I tell the difference between the old and new patch?  I just diffed DF's version against the one in db-devel and it's the same still.
[10:26] <bigjools> jml: morning
[10:31] <lifeless> bigjools: the fixedtasks CTE should be gone
[10:31] <lifeless> line 78 of the patch
[10:32] <bigjools> lifeless: 78 for me says "OR ("
[10:32] <lifeless> bigjools: did you pull db-stable or db-devel
[10:32] <bigjools> devel
[10:32] <lifeless> *db-devel*
[10:32] <bigjools> yes db-devel
[10:32] <lifeless> what revno do you have?
[10:32] <bigjools> it's hardly likely to be in devel is it :)
[10:33] <bigjools> 10679
[10:33] <bigjools> is the one I am using now, but it says no diff to 10685
[10:34] <wgrant> danilos: I played around with the new subscription listing thing this morning, and noticed a couple of things. The JS crashes for anonymous users, the subscriptions don't seem to be sorted, and I can't work out who is directly subscribed.
[10:34] <lifeless> bigjools: I see a huge diff
[10:34] <danilos> wgrant, well, everyone except the "Maybe notified" section are direct subscribers
[10:34] <wgrant> danilos: Oh.
[10:35] <wgrant> That's less than obvious.
[10:35] <danilos> wgrant, some of them might choose not to receive some emails
[10:35] <wgrant> I assumed they were just people who had selective structural subscriptions :(
[10:35] <danilos> wgrant, you can have selective direct subscriptions as well
[10:35] <wgrant> True.
[10:36] <wgrant> So "Always notified" is sort of a misnomer?
[10:36] <danilos> wgrant, no, always notified is those who are always notified (i.e. "receive all"
[10:36] <danilos> wgrant, see eg. https://bugs.qastaging.launchpad.net/ubuntu/+source/network-manager/+bug/152754 for example of other sections
[10:36] <_mup_> Bug #152754: NetworkManager fails and crashes after resume from suspend <network-manager (Ubuntu):Incomplete> < https://launchpad.net/bugs/152754 >
[10:37] <danilos> wgrant, (you can see my different subscriptions on different levels)
[10:37] <wgrant> danilos: aaaaaa
[10:37] <wgrant> So "Maybe notified" is all the structsubs? I seeeee.
[10:37] <wgrant> Hmm.
[10:37] <danilos> wgrant, yeah, "maybe" is all struct subs, all dupe subs
[10:38] <danilos> wgrant, for your own subscriptions, you are supposed to use "edit bug mail" link instead
[10:38] <bigjools> lifeless: weird, a bzr ancestor diff shows nothing, but bzr missing shows 6 revisions
[10:38] <wgrant> danilos: I've never liked that link... shouldn't it be "Edit subscriptions" or something that doesn't sound like it's editing emails? :/
[10:38] <danilos> wgrant, basically, this is something that's hard (and potentially expensive) to display in succint form with enough details (while staying correct)
[10:38] <wgrant> But yeah.
[10:39] <lifeless> bigjools: uhm an ancestor diff should show nothing
[10:39] <lifeless> bigjools: because your current rev is the common ancestor
[10:39] <wgrant> I understand it's hard to calculate, yes. But it's not obvious what it's actually showing.
[10:39] <bigjools> gah, roight
[10:39] <bigjools> lifeless: I am stupid
[10:39] <lifeless> :P
[10:39] <danilos> wgrant, yeah, I think the attempt was to clarify that this is about controlling the email you get from that particular bug
[10:39] <lifeless> bigjools: didn't you just implement a vcs layer for distros?
[10:39] <danilos> wgrant, and I think we just copied whatever was there already on the pillar pages which was "Edit bug mail"
[10:40] <bigjools> I'd like to think not :)
[10:40] <danilos> wgrant, I do see your point, but maybe you are thinking too much in terms of implementation, rather than trusting what the text says?
[10:40] <LPCIBot> Project parallel-test build #46: STILL FAILING in 1 hr 13 min: https://lpci.wedontsleep.org/job/parallel-test/46/
[10:40] <danilos> wgrant, "Notified of all changes" are, indeed, people/teams who get all the bug mail for this bug
[10:40] <wgrant> danilos: Links like that have always talked about subscriptions before, or "Subscribe to bug mail", not "Edit bug mail"
[10:41] <wgrant> Since editing bug mail doesn't really make sense.
[10:41] <lifeless> danilos: wgrant: was saying earlier that the js crashes for anonymous users
[10:41] <danilos> wgrant, hey, I am not going to argue over that, I don't like "Edit bug mail" anymore than you do
[10:41] <wgrant> Heh
[10:41] <wgrant> k
[10:41] <danilos> lifeless, yeah, I'm going to look into it
[10:41] <wgrant> But yes, what lifeless said.
[10:41] <danilos> wgrant, ack
[10:41] <lifeless> danilos: if it needs fixing we should roll the branch back
[10:42] <wgrant> ... and land the unrelated pieces separately? :)
[10:42] <lifeless> danilos: we have 30! revisions to deploy already, and more will queue up over the weekend
[10:42] <lifeless> wgrant: orthogonal discussion
[10:42] <wgrant> lifeless: Not quite... I rolled out up to that rev this morning.
[10:42] <lifeless> wgrant: danilos: grah sorry; stale info
[10:42] <wgrant> There's only 8 in the queue ATM
[10:42] <danilos> lifeless, if it needs fixing, we'll fix it in the next few hours
[10:43] <danilos> lifeless, if subscribers list doesn't render for anonymous users, it's not a big deal since none of the JS works for them anyway
[10:43] <danilos> lifeless, iow, that should not block the rollout, but it should be fixed asap
[10:44] <wgrant> Why shouldn't it block the rollout?
[10:44] <danilos> lifeless, I was also hoping to get more QA on this and try to deploy it on Monday instead (and any such problems should be fixed by then)
[10:45] <danilos> wgrant, why should it? I don't consider it a critical feature, and I value the time it takes to get something landed much more than this particular problem (if it's not as severe, like I said)
[10:45] <wgrant> danilos: Some browser react to JS errors with violent popups.
[10:45] <wgrant> And we treat JS errors as OOPSes now.
[10:46] <wgrant> I think we'd consider it a blocker if the most commonly accessed page on LP OOPSed for anonymous users.
[10:46] <danilos> wgrant, right, and fixing the problem is probably going to take 30 mins as soon as I get to it, whereas just doing a revert is going to take me 10-20 mins
[10:46] <wgrant> I don't place much value on rushing a 12000 line branch through, TBH.
[10:46] <wgrant> It deserves a lot of caution.
[10:46] <danilos> wgrant, I agree, and I want to exercise some
[10:47] <danilos> wgrant, but I want to be realistic as well
[11:04] <allenap> stub: Do you have some time to help analyse the slow queries in https://lp-oops.canonical.com/oops.py/?oopsid=1993DY5?
[11:05] <stub> allenap: Nope. Last minute scramble for a plane.
[11:05] <allenap> stub: Okay, have a good trip :)
[11:06] <allenap> lifeless: I know it's silly o'clock for you, but do you have some time to help analyse the slow queries in https://lp-oops.canonical.com/oops.py/?oopsid=1993DY5?
[11:07] <lifeless> allenap: lets get an analyse for the 3 second one off of prod, after gnuoy finishes fixing staging
[11:10] <allenap> Okay, cool. I have it ready on devpad.
[11:44] <bigjools> lifeless: do you know how much quicker that new db patch is?  30 minutes and counting on DF so far.
[11:45] <lifeless> 157 seconds when it ran on a temp table
[11:45] <lifeless> are you sure you got the new one ?
[11:45] <bigjools> yes
[11:45] <bigjools> :)
[11:45] <lifeless> allenap: ok so
[11:46] <lifeless> 6.6 seconds
[11:46] <allenap> Ouch.
[11:46] <lifeless> see the actual time
[11:46] <lifeless> hash join on spr to dsd
[11:46] <lifeless> 351K rows
[11:47] <danilos> adeuring, wallyworld_: hi, I need a review of a short branch (151 line), any takers? :)
[11:47] <lifeless> the stats suggested to postgresql it would see 2.2K rows
[11:47] <adeuring> danilos: I'll look
[11:47] <lifeless> could be a stats issue
[11:47] <danilos> adeuring, thanks
[11:47] <allenap> lifeless: Can we get stats recalculated online?
[11:48] <lifeless> yeah
[11:48] <lifeless> analyze tablename;
[11:48] <lifeless> allenap: how many rows are you expecting to return ?
[11:49] <allenap> lifeless: Around 1:1 with DistroSeriesDifference rows.
[11:50] <allenap> So around the same as the big IN (5, 7, ...) clause.
[11:50] <lifeless> right, so the plan is problematic, or the model is problematic or the query is
[11:50] <allenap> Hehe.
[11:51] <LPCIBot> Project windmill-devel build #243: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/243/
[11:51] <allenap> lifeless: Is an analyze cheap enough that we could try that first? DistroSeriesDifference until recently was empty for example, that might be a candidate for refreshing stats?
[11:52] <allenap> Although that doesn't appear in the bad part of the plan :-/
[11:52] <lifeless> yes
[12:00] <lifeless> bigjools: in other news, DF sucks.
[12:00] <bigjools> lifeless: tell me something I don't know
[12:00] <lifeless> staging appears to have run the update
[12:00] <bigjools> how slow?
[12:01] <lifeless> times will be in the logs
[12:01] <bigjools> what, you're making me look myself?  tsk :)
[12:01] <lifeless> 11pm and I'm sick.... so yes :)
[12:03] <lifeless> also woo
[12:03] <lifeless> 27ms stats portlets here we come
[12:04] <bigjools> which file has the timings?
[12:04] <lifeless> the staging-restore-$date IIRC
[12:04] <bigjools> 2011-06-17 07:24:24 ERROR   slonik script failed
[12:04] <bigjools> Fri Jun 17 07:24:24 UTC 2011 ERROR: Failed to run upgrade.py
[12:04] <lifeless> it won't have rsynced yet so you'll need to pull it over by hand
[12:04] <bigjools> k
[12:04] <lifeless> thats the run we canned
[12:07] <adeuring> danilos: nice small branch, r=me
[12:08] <danilos> adeuring, thanks
[12:08] <bigjools>  lifeless: so it's far easy to look in LaunchpadDatabaseRevision.... ;)
[12:09] <bigjools> 7 mins on staging it seems
[12:10] <lifeless> allenap: so not stats
[12:11] <allenap> No.
[12:11] <lifeless> 34*10547
[12:11] <lifeless> 358598
[12:11] <lifeless> that explains the 351K rows I think
[12:12] <lifeless> 10K spph * 34 archive
[12:14] <allenap> I wonder what we can do to narrow the spph set down.
[12:14] <lifeless> well
[12:14] <lifeless> its a cross join atm
[12:15] <lifeless> so one thing to try is an inner join
[12:15] <lifeless> rather than a constrained cross join
[12:16] <allenap> lifeless: Okay, I don't know enough to know how to do that, but I can go and find out. In fact, I'd be happy to do that. But if you want to see some swifter progress you'll have to help me out on that one :)
[12:18] <allenap> lifeless: Okay, I think I know how.
[12:21] <lifeless> query in ~robertc on carob
[12:26] <allenap> lifeless: Thanks :)
[12:28] <lifeless> ah, I misread
[12:28] <lifeless> problem doing this tired
[12:28] <lifeless>                            ->  Index Scan using sourcepackagerelease_pkey on sourcepackagerelease  (cost=0.00..0.87 rows=1 width=8) (actual time=0.012..0.013 rows=1 loops=351423)
[12:29] <lifeless>                                  Index Cond: (sourcepackagerelease.id = sourcepackagepublishinghistory.sourcepackagerelease)
[12:29] <lifeless> thats the 350K
[12:29] <lifeless> spr
[12:29] <lifeless> I have to crash
[12:29] <lifeless> however
[12:29] <lifeless> some thoughts:
[12:29] <lifeless>  - we know the distroseries, so can constrain the query further
[12:29] <lifeless>  - there are only 34 primary archives, so we can eliminate the archive join if we want (may not matter)
[12:30] <lifeless> and that would let us drop the distroseries join too
[12:31] <wgrant> jelmer: Thanks!
[12:31] <allenap> lifeless: Thanks for your help :) Have a good weekend.
[12:32] <jelmer> wgrant: For what, wasting your morning ? :)
[12:32] <lifeless> erm, no first analysis was right I think
[12:32] <lifeless> we're expanding out through archive
[12:32] <lifeless> because you have this
[12:32] <lifeless> constraints - join - join - join - join - constraints
[12:32] <jelmer> wgrant: Sorry about the importd upgrade trouble, I wonder how neither of us had caught this on staging though.
[12:33] <lifeless> each join, even inner ones, will multiply until it can constraint
[12:33] <wgrant> jelmer: For fixing that :) (and it was my afternoon/evening, not my morning!)
[12:33] <wgrant> jelmer: Not sure... I wonder if it lied about the format.
[12:33] <wgrant> Possibly the import was new, but still had the old metadata, or something.
[12:33] <wgrant> Anyway, was easily fixed.
[12:33] <LPCIBot> Project windmill-devel build #244: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-devel/244/
[12:33] <danilos> wgrant, fwiw, the fix for the anonymous users JS is playing in ec2
[12:33] <wgrant> Although it was a major panic moment before I realised there would probably still be a copy on escudero...
[12:34] <wgrant> danilos: Ah, great! That was quick.
[12:36] <danilos> wgrant, if I did more integration testing for JS stuff (I did loads of JS unit tests, with the ratio of tests to code being roughly 2:1 in number of lines), I would not have let it happen; unfortunately, I don't and I did
[12:36] <danilos> wgrant, so, I hope to start on more windmill tests if they are the way to go for this
[12:38] <lifeless> nn, must go
[12:40] <jelmer> wgrant: Yeah, I can imagine..
[13:08] <matsubara> morning
[13:08] <allenap> Sorry lifeless, had to go. My missus had been badgering me for half an hour already.
[13:16] <LPCIBot> Project windmill-devel build #245: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/windmill-devel/245/
[13:23] <jml> matsubara: good morning
[13:23] <matsubara> hi jml
[13:23] <danilos> matsubara, good morning as well :)
[13:24] <jml> bigjools: can  we have a call at 4pm to touch base about derivative distros?
[13:24] <matsubara> hey danilos
[13:24] <matsubara> danilos, btw, did that branch land?
[13:25] <danilos> matsubara, fwiw, new subscription stuff is up on qastaging, there is a bug with anon users, but the fix is playing in ec2 already
[13:25] <danilos> matsubara, yeah, it did, it should be good to test
[13:25] <matsubara> danilos, cool. I'll give it a try today.
[13:25] <danilos> matsubara, excellent, you can also report any problems you find on bug 772754, I'll be reading it
[13:25] <_mup_> Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-needstesting> <story-better-bug-notification> <Launchpad itself:Fix Committed by gary> < https://launchpad.net/bugs/772754 >
[13:26] <matsubara> will do
[13:26] <danilos> matsubara, thanks!
[13:27] <matsubara> danilos, np. Yesterday you mentioned r13248 as the one you were waiting. Was that a typo or that's the one with the subsequent fix playing on ec2?
[13:28] <danilos> matsubara, no, that's the one which includes everything but the fix for anonymous users (basically, there is a JS error for anonymous users), which is in bug 798622
[13:28] <_mup_> Bug #798622: JS failure for anonymous users on bug page <story-better-bug-notification> <Launchpad itself:In Progress by danilo> < https://launchpad.net/bugs/798622 >
[13:28] <danilos> matsubara, so, qastaging should be good for QAing for logged in users (I already did some, as did wgrant: he found the anon user problem)
[13:30] <matsubara> danilos, there's no r13248 on qastaging. the one available is r13243 linked to bug #772754 so I assume that's the right number not 13248 :-)
[13:30] <_mup_> Bug #772754: After better-bug-notification changes, list of bug subscribers is confusing <qa-needstesting> <story-better-bug-notification> <Launchpad itself:Fix Committed by gary> < https://launchpad.net/bugs/772754 >
[13:31] <danilos> matsubara, qastaging for me has "r13251" in the footer
[13:31] <matsubara> danilos, I'm looking at the dashboard. that might be a bit outdated
[13:32]  * danilos looks at his dashboard, and thinks redline at 7000 rpm is too low
[13:32] <matsubara> actually, I'm confusing the qastaging with what's in the dashboard
[13:32] <danilos> matsubara, I am thoroughly confused since I don't know what the dashboard is :)
[13:32] <matsubara> 13243 is block the rollout and qastaging has 13251
[13:32] <matsubara> danilos, http://lpqateam.canonical.com/devops/
[13:33] <danilos> matsubara, ah, nice
[13:33] <danilos> matsubara, right, so we still haven't marked it as qa-ok because I'd prefer to get a little bit more QA done first
[13:34] <matsubara> I see now, should have looked at the development-stable report. I can see that 13243 was reverted by 13247 and then 13248 landed afterwards
[13:34] <matsubara> no worries. I unconfused myself and things should be ok to start now. thanks!
[13:35] <danilos> matsubara, cool, thanks
[13:56] <LPCIBot> Project windmill-devel build #246: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-devel/246/
[14:38] <bigjools> jml: not feeling too well but if I am here, yes
[14:38] <jml> bigjools: cool. thanks.
[15:05] <LPCIBot> Project windmill-devel build #247: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/247/
[15:09] <bac> hi adeuring -- i've just grabbed wallyworld_'s branch.  you're not already looking at it are you?
[15:09] <adeuring> bac: no
[15:14] <LPCIBot> Project parallel-test build #47: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/parallel-test/47/
[15:54]  * jml ducking out for a bit, back in ~20m
[16:06] <nigelb> bigjools: heh, lol That as a fun commit :-)
[16:06] <nigelb> *was
[16:06] <bigjools> jml: you no wanna call then?
[16:07] <bigjools> nigelb: the comments are hilarious
[16:10] <flacoste> james_w: i again need your help to qa a change, could you confirm that you are able to call setBranch with the UPDATES or RELEASE pocket on lucid/maverick on qastaging? pretty please
[16:11] <jml> bigjools: I do, just need to get some other stuff out of the way first.
[16:11] <LPCIBot> Project windmill-db-devel build #402: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/402/
[16:13] <nigelb> bigjools: Yup, with new memes!
[16:17] <sinzui> jcsackett, mumble?
[16:17] <jcsackett> sinzui: sure.
[16:18] <sinzui> jcsackett, If you cannot hear me I think I need to restart mumble
[16:18] <jml> bigjools: ok. pie safely consumed, call can proceed
[16:18] <jcsackett> sinzui: i heard you. seems you could not hear me. i'm restarting.
[16:18] <bigjools> the pie should be on ya face
[16:19] <jml> bigjools: 200 bugs stand between my face and pie.
[16:20] <bigjools> jml: gimme a couple of mins
[16:20] <jml> also, a hot steak & stilton pie would be inappropriate for face-planting
[16:20] <sinzui> maybe I can get empiphay to work
[16:22] <jcsackett> sinzui: i'm having a connection hiccup.
[16:23] <sinzui> empathy say I have sip, but I cannot use it, maybe you can call me
[16:26] <jcsackett> sinzui: i hear you, it seems you cannot hear me. i will try sip.
[16:28] <flacoste> maxb: the fix for the importer is deployed to qastaging, i asked james_w to confirm that it works, maybe you can do it also?
[16:30] <maxb> flacoste: No, I'm not a core-dev (or any sort of uploader actually)
[16:31] <jcsackett> sinzui: lost call. recalling in a moment.
[16:31] <cr3> what part of launchpad adds the utc timezone to a datetime between getting it from the database (which always has "without time zone") to the restful interface (which has the timezone as "+00:00", therefore explicitly defined somewhere)?
[16:33] <bigjools> jml: sorry.  what is your call platform of choice?
[16:33] <jml> bigjools: voip
[16:34] <jml> bigjools: skype for second preference
[16:34] <bigjools> gah
[16:34] <jml> (am I allowed to have a second preference?)
[16:34] <bigjools> pulseaudio keeps usurping Twinkle's alsa connection
[16:35] <bigjools> and twinkle doesn't know about PA
[16:35] <jml> ahh, I see.
[16:35] <bigjools> I hate PA
[16:35] <jml> pulseaudio is great in natty for me
[16:35] <jml> I guess that's because everything is using it.
[16:36] <bigjools> I hate it because it limits my options
[16:36] <bigjools> I am on skype
[16:36] <jml> ok
[16:43] <jml> bigjools: have you seen https://dev.launchpad.net/Projects/DerivativeDistributions ?
[16:52] <cr3> nevermind folks, I found that it's UtcDateTimeCol that sets the tzinfo. oddly though, that doesn't seem to behave the same as non-sqlobject derived storm variables which only set the tzinfo when writing to the db
[16:56] <LPCIBot> Project windmill-devel build #248: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-devel/248/
[17:44] <LPCIBot> Project windmill-devel build #249: STILL FAILING in 43 min: https://lpci.wedontsleep.org/job/windmill-devel/249/
[18:03] <bigjools> bye, have a good weekend everyone
[18:47] <LPCIBot> Project db-devel build #645: FAILURE in 5 hr 35 min: https://lpci.wedontsleep.org/job/db-devel/645/
[19:31] <LPCIBot> Project windmill-db-devel build #403: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-db-devel/403/
[19:44] <fjlacoste> maxb: james_w confirmed the fix is working, I've changed the Debian distribution maintainers, so everything should be back to normal at the next nodowntime deploy
[19:44] <fjlacoste> maxb: probably Sunday (antepodean Monday) as we don't roll-out just before the week-end
[19:44] <maxb> Awesome, thanks, so on Monday we should be back in business
[19:44] <fjlacoste> maxb: again, i'm very sorry for the breakage
[19:44] <fjlacoste> and apologize for it
[19:47] <maxb> Sometimes regressions happen - thanks for pushing through a fix promptly
[19:52] <fjlacoste> thanks for your understanding
[19:54] <LPCIBot> Project parallel-test build #48: STILL FAILING in 1 hr 11 min: https://lpci.wedontsleep.org/job/parallel-test/48/
[20:06] <fjlacoste> anyone knows what LostObjectError: means?
[20:06] <fjlacoste> i'm writing a view test and getting this
[20:17] <fjlacoste> wow, that's intersting
[20:18] <fjlacoste> it happens because there is actually a validation error in the form
[20:18] <fjlacoste> which resets the transaction
[20:18] <fjlacoste> and then when then a storm attribute is accessed after the end of transaction
[20:18] <fjlacoste> and boom
[20:24] <bac> i'm trying to export a new class to the api and i'm getting:
[20:24] <bac> UnknownEntryAdapter: No IEntry adapter found for IProcessorFamily (web service version: devel).  Encountered as a result of the entry interface <InterfaceClass lp.soyuz.interfaces.archive.IArchiveEntry_devel>, field 'enabled_restricted_families'.
[20:24] <bac> normally that would be due to a circular dependency, but i've handled that (i think)
[20:25] <bac> fjlacoste: you now what ^^ could be?
[20:25] <fjlacoste> bac: that's because the IProcessFamily isn't exported
[20:25] <fjlacoste> you need to import it into lp.soyuz.interfaces.webservice probably
[20:25] <bac> fjlacoste: ah, but it is:  http://pastebin.ubuntu.com/628552/
[20:26] <bac> er
[20:26] <fjlacoste> it's not :-)
[20:26] <fjlacoste> see lp.soyuz.interfaces.webservice
[20:26] <fjlacoste> you need to import it in there
[20:26] <fjlacoste> because only definitions in that file are processed for export
[20:26] <bac> oh!
[20:29] <bac> fjlacoste: you are a super-genius!  thanks!
[20:55] <flacoste> abentley: is it possible to set the pre-requisite branch from lp-propose?
[20:56] <abentley> flacoste: not directly, but it's done automatically if you use pipelines.
[20:56] <flacoste> i didn't
[20:56] <flacoste> is it possible to set the pre-requisite branch on the created merge proposal?
[20:56] <flacoste> or i need to resubmit
[20:57] <abentley> flacoste: You can set the pre-requisite branch on the created merge proposal by re-submitting.
[20:58] <flacoste> thanks
[20:58] <flacoste> bac: a review for you pretty please: https://code.launchpad.net/~flacoste/launchpad/bug-781993-2/+merge/65058
[20:58] <flacoste> should be simple
[21:04] <bac> flacoste: sure
[21:04] <bac> flacoste: can you get rid of the conflicts?
[21:13] <bac> flacoste: actually the conflicts don't show up in the MP.  looks good.
[21:34] <flacoste> thanks for the review bac
[21:43] <soren> Hi. The person listed as help contact in #launchpad isn't online and noone seems to respond there, so I'm escalating here. Hope you can forgive me. Uploads to PPA's don't work. I get "Changes file must be signed with a valid GPG signature: Verification failed 3 times: ['General error', 'General error', 'General error'] : Permission denied." Ampelbein in #launchpad pointed me to bug 757248.
[21:43] <_mup_> Bug #757248: poppy-sftp's signature checking relies on long-term survival of a directory in /tmp <poppy> <qa-ok> <regression> <Launchpad itself:Fix Released by wallyworld> < https://launchpad.net/bugs/757248 >
[21:43] <soren> ...which, interestingly, was marked "fixed in stable" exactly one month ago today.
[21:44]  * soren <- saaaad panda
[21:50] <flacoste> soren: i asked IS to check what's up
[21:51] <soren> flacoste: ta very much.
[21:51] <flacoste> soren: someone internal says that the upload should still be accepted
[21:52] <flacoste> it is an avatar of the old bug
[21:52] <lifeless> well
[21:52] <flacoste> but it shouldn't reject your upload
[21:52] <lifeless> it might be
[21:53] <soren> flacoste: Ah, yes, you seem to be right. I didn't even bother checking. It looked pretty darn fatal to me :)
[21:53] <lifeless> it does; we're going to have lots of folk double-upload and panic if this is happening again
[21:54] <flacoste> indeed, soren can you file a new bug (one might already exists though)
[21:55] <soren> flacoste: Sure
[21:57] <flacoste> thanks
[21:57] <soren> https://bugs.launchpad.net/launchpad/+bug/798957
[21:57] <_mup_> Bug #798957: Uploads are seemingly (but not actually) rejected <Launchpad itself:New> < https://launchpad.net/bugs/798957 >
[22:02] <lifeless> soren: perhaps your gpg key expired recently or something?
[22:05] <LPCIBot> Project windmill-devel build #250: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/250/
[22:06] <soren> lifeless: It started happening a couple of hours ago for both the automated openstack package uploader thingamajig and myself.
[22:10] <soren> lifeless: ...and they both never expire. I checked :)
[22:10] <lifeless> k