| wallyworld | sinzui: @##@@@# mic acting up again :-( | 00:04 |
|---|---|---|
| poolie | hi all | 00:07 |
| poolie | lifeless, anyone, can you suggest how (if at all) i should qa the librarian-mail-naming change? | 00:07 |
| LPCIBot | Project devel build #788: STILL FAILING in 6 hr 13 min: https://lpci.wedontsleep.org/job/devel/788/ | 00:08 |
| LPCIBot | Project devel build #789: STILL FAILING in 5.6 sec: https://lpci.wedontsleep.org/job/devel/789/ | 00:08 |
| poolie | i guess by sending it mail, syncing the logs, then peeking in there? | 00:08 |
| lifeless | send mail to qas, sget someone with qas db access to check the .raw attribute and follow that back to make sure its accessible in the librarian | 00:08 |
| poolie | by handcrafting a url from the database row? | 00:09 |
| lifeless | more-or-less :) | 00:18 |
| lifeless | rather more than less | 00:18 |
| sinzui | wgrant: wallyworld, StevenK bug 1334, bug 80902 | 00:22 |
| lifeless | not 1337 ? | 00:26 |
| gary_poster | :-P | 00:27 |
| gary_poster | night | 00:27 |
| sinzui | wallyworld: are you saying you fixed bug 86861 | 00:30 |
| _mup_ | Bug #86861: SinglePopupWidget only works with vocabulary registered by name <infrastructure> <lp-foundations> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/86861 > | 00:30 |
| wallyworld | sinzui: it appears so - i would have to read the bug details to be sure | 00:30 |
| poolie | o/ wallyworld | 00:46 |
| wallyworld | ? | 00:47 |
| StevenK | Hm. buildbot has failed to check out devel -- I suspect since it tried during the rollout | 00:58 |
| StevenK | Do I need to force a build to get it to start again? | 00:58 |
| wgrant | StevenK: I already did. | 00:59 |
| wgrant | 17 seconds before you asked, it would seem. | 01:00 |
| StevenK | Haha | 01:00 |
| wgrant | jelmer: Are bzr-svn imports meant to be taking ages now? | 01:30 |
| wgrant | jelmer: Perhaps it is only the first run with the new version, but it is still slightly worrying. | 01:30 |
| wgrant | eg. https://code.launchpad.net/~vcs-imports/ljcode/trunk and https://code.launchpad.net/~vcs-imports/wxwidgets2.6/trunk | 01:30 |
| StevenK | 30 seconds to an hour? Orsum. | 01:32 |
| wgrant | They do seem to eventually finish. | 01:54 |
| wgrant | Let's see if the second run is faster. | 01:55 |
| LPCIBot | Project windmill-devel build #191: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/191/ | 02:04 |
| wgrant | lifeless: Erk. | 02:38 |
| wgrant | https://lp-oops.canonical.com/oops.py/?oopsid=1985CE168 | 02:38 |
| wgrant | Trigger contention on bugsummary? | 02:38 |
| lifeless | I hope not | 02:39 |
| wgrant | As do I, but what else? | 02:39 |
| lifeless | so, a bug on lp is fine | 02:41 |
| lifeless | gcc-linaro is a similar size | 02:42 |
| lifeless | wgrant: its possible; the ubuntu row specifically will be fairly high volume, but 8 seconds -- argh | 02:44 |
| lifeless | so, we need to make write transactions faster | 02:44 |
| lifeless | for bugs | 02:44 |
| lifeless | >< | 02:44 |
| wgrant | s/argh/oh shit shit shit/' | 02:44 |
| lifeless | I will talk with stub this evening | 02:44 |
| wgrant | We'll have to see how bad this gets. | 02:45 |
| lifeless | I think it will survive until then | 02:45 |
| wgrant | But it may be pretty dire. | 02:45 |
| lifeless | we may be have to disable the triggers which can be done pretty quickly | 02:45 |
| lifeless | and then rollback my use of bugsummary. | 02:45 |
| wgrant | 20 / 1 Bug:EntryResource:subscribe | 02:45 |
| lifeless | but I hope not | 02:45 |
| wgrant | That was in half an hour? | 02:45 |
| lifeless | subscribing to public bugs won't lock any rows | 02:46 |
| lifeless | (in theory) | 02:46 |
| lifeless | but we're going on theory to blame bugsummary | 02:46 |
| wgrant | It's a pretty reasonable theory. | 02:46 |
| wgrant | Hm, OK, only 5 subscribe timeouts today so far. | 02:47 |
| lifeless | yesterday - 3 / 0 Bug:EntryResource:newMessage | 02:48 |
| wgrant | May be tolerable for now. | 02:48 |
| lifeless | that new task | 02:49 |
| lifeless | was on a product | 02:49 |
| lifeless | 258 | 02:49 |
| wgrant | Yes. | 02:49 |
| lifeless | which is gcc | 02:50 |
| lifeless | choose-affected-product is 2.16 99th percentile historically | 02:50 |
| wgrant | ... are there enough triggers involved here? | 02:50 |
| lifeless | to get 8 seconds due to contention you'd need simultaneous edits of 4 gcc bugs serialised | 02:51 |
| lifeless | the new task won't affect the ubuntu rows in bugsummary | 02:52 |
| lifeless | now, we *may* have a problem where we don't shortcircuit no-op changes and we need to | 02:52 |
| wgrant | -- Grab a suitable lock before we start calculating bug summary data | 02:53 |
| wgrant | -- to avoid race conditions. This lock allows SELECT but blocks writes. | 02:53 |
| wgrant | LOCK TABLE BugSummary IN ROW EXCLUSIVE MODE; | 02:53 |
| wgrant | bugsummary_tasks deals with a bug, not a bugtask. | 02:54 |
| wgrant | It will be locking all tasks. | 02:54 |
| lifeless | yes | 02:54 |
| lifeless | I was just looking at that | 02:54 |
| wgrant | == we are fucked | 02:54 |
| wgrant | Pretty much | 02:54 |
| lifeless | we're going to have higher than desirable contention on the primary row in Ubuntu | 02:55 |
| wgrant | == 9s insert queries == aaaaaaaaa | 02:55 |
| lifeless | so what we need to do is capture the bug rows (rather than summarise), capture after, take a diff, and then apply the diff | 02:56 |
| wgrant | No, what we need to do is drop the triggers. | 02:56 |
| wgrant | And fix them later. | 02:56 |
| lifeless | this is non trivial to implement | 02:56 |
| lifeless | we need to wait for stub, to determine the right way to disable them with slony | 02:57 |
| lifeless | First thing, ident.ica and irc notices | 02:58 |
| wgrant | 16:55 < stub> wgrant: That one is a 'possibly, but very problematic'. We need to drop and recreate a trigger, which is fast enough if we manage to grab a lock but grabbing the lock is problematic. And then we need to apply a db patch *to the slaves only* during the next rollout that updates the trigger. | 02:58 |
| wgrant | So it looks like it needs a lock and drop on the master. | 02:58 |
| lifeless | well there are two ways | 02:58 |
| lifeless | we can redefine the trigger function | 02:58 |
| lifeless | or we can drop the use of the trigger | 02:58 |
| wgrant | lifeless: Do you know why it uses all the tasks? | 02:59 |
| wgrant | I'm not really keen on reading 500 lines of PL/pgSQL... | 03:00 |
| lifeless | wgrant: | 03:00 |
| lifeless | its pretty simple code | 03:00 |
| lifeless | see bugtask_maintain_bug_summary | 03:00 |
| wgrant | Yes, but it's PL/pgSQL. | 03:00 |
| lifeless | IF TG_OP = 'INSERT' THEN | 03:00 |
| lifeless | IF TG_WHEN = 'BEFORE' THEN | 03:00 |
| lifeless | PERFORM unsummarise_bug(bug_row(NEW.bug)); | 03:00 |
| lifeless | ELSE | 03:00 |
| lifeless | PERFORM summarise_bug(bug_row(NEW.bug)); | 03:00 |
| lifeless | END IF; | 03:00 |
| lifeless | RETURN NEW; | 03:00 |
| lifeless | wgrant: thus its clearer than python :P | 03:00 |
| wgrant | lifeless: Yes, that says how it calls it, but not why. | 03:01 |
| wgrant | Is it because it's lazy and just updates everything? | 03:01 |
| wgrant | By removing the bug from everywhere and then readding it? | 03:01 |
| lifeless | yes, remove the bug from the table, let the task be added, summarise the bug into the table | 03:01 |
| lifeless | 13:56 < lifeless> so what we need to do is capture the bug rows (rather than summarise), capture after, take a diff, and then apply the diff | 03:02 |
| wgrant | Looks somewhat buggy to me. | 03:02 |
| lifeless | if you consider undue contention buggy | 03:02 |
| wgrant | When unsummarising it uses the new rows. | 03:02 |
| lifeless | no | 03:02 |
| wgrant | So I don't think adding a task will actually increment... | 03:02 |
| lifeless | this is an insert | 03:02 |
| lifeless | it only has the NEW bug id | 03:02 |
| wgrant | Oh, this is a before. Right. | 03:03 |
| lifeless | http://www.postgresql.org/docs/8.4/static/trigger-datachanges.html | 03:04 |
| wgrant | So, are we going to wake up stub or hope he arrives in a timely manner or wing it? | 03:04 |
| lifeless | we're not winging it | 03:04 |
| lifeless | its bad but its not disastrous | 03:04 |
| lifeless | (its not too far off disastrous) | 03:05 |
| wgrant | It needs exclusive locks on either side :/ | 03:05 |
| wgrant | It's only not disastrous because most people have EODed already. | 03:06 |
| LPCIBot | Yippie, build fixed! | 03:06 |
| LPCIBot | Project db-devel build #622: FIXED in 6 hr 24 min: https://lpci.wedontsleep.org/job/db-devel/622/ | 03:06 |
| lifeless | no, its not disastrous because it hasn't gone -completely- dead | 03:06 |
| lifeless | what do you mean by exclusive locks either side ? | 03:07 |
| wgrant | Both sides of each bugtask INSERT require an exclusive lock on BugSummary. | 03:07 |
| wgrant | How ugly. | 03:07 |
| lifeless | are you worried about deadlock ? | 03:07 |
| wgrant | Possibly. | 03:08 |
| lifeless | the lock once taken applies through to commit | 03:08 |
| lifeless | its per-row | 03:08 |
| wgrant | LOCK TABLE BugSummary IN ROW EXCLUSIVE MODE; | 03:08 |
| wgrant | That's not per-row. | 03:08 |
| wgrant | Oh. | 03:08 |
| lifeless | ... iz not? | 03:09 |
| wgrant | I missed the "ROW" because the comment is slightly misleading. | 03:09 |
| wgrant | Ahem. | 03:09 |
| wgrant | LOCK TABLE only deals with table-level locks, and so the mode names involving ROW are all misnomers. These mode names should generally be read as indicating the intention of the user to acquire row-level locks within the locked table. Also, ROW EXCLUSIVE mode is a sharable table lock. | 03:10 |
| wgrant | How odd. | 03:10 |
| lifeless | yeah | 03:11 |
| lifeless | I am refreshing this too | 03:11 |
| wgrant | I'm still not sure how this isn't disastrous. | 03:12 |
| lifeless | An exclusive row-level lock on a specific row is automatically acquired when the row is updated or deleted | 03:12 |
| lifeless | its 8am for stub | 03:14 |
| wgrant | Oh, true, it is later than I had thought. | 03:14 |
| lifeless | I'm considering raising the default timeout to 20 seconds | 03:19 |
| lifeless | the contention is self limiting bounded on permitted transaction time | 03:20 |
| lifeless | if the change rate is below some N (unknown) then we won't backoff indefinitely, it will just spike up to some number of seconds at high concurrency changes | 03:21 |
| lifeless | if the change rate is above N, it will cascade and push back past whatever timeout we set | 03:21 |
| wgrant | Yes, that's my worry. | 03:21 |
| lifeless | ok, the lock mode is wrong I think | 03:23 |
| wgrant | 18 subscribe timeouts so far. | 03:23 |
| wgrant | On single-task Ubuntu bugs, too :/ | 03:24 |
| lifeless | oh hell | 03:24 |
| lifeless | subscribe changes the bug last-updated field doesn't it. | 03:25 |
| lifeless | lalalalaalalaala | 03:25 |
| wgrant | Hee hee so it does. | 03:25 |
| wgrant | 00186-09119@SQL-launchpad-main-master INSERT INTO BugSubscription (bug, bug_notification_level, date_created, subscribed_by, person) VALUES (317370, 40, CURRENT_TIMESTAMP AT TIME ZONE 'UTC', 690731, 690731) RETURNING BugSubscription.id | 03:25 |
| wgrant | Hmm. | 03:25 |
| wgrant | But it's a single-task bug. | 03:25 |
| wgrant | That shouldn't be that bad. | 03:25 |
| lifeless | shouldn't matter: | 03:25 |
| lifeless | IF TG_OP = 'UPDATE' THEN | 03:25 |
| lifeless | IF OLD.duplicateof IS DISTINCT FROM NEW.duplicateof | 03:25 |
| lifeless | OR OLD.private IS DISTINCT FROM NEW.private THEN | 03:25 |
| lifeless | and hah - where is status? | 03:26 |
| lifeless | oh, bugsubsctiption might not be fast-pathing | 03:27 |
| lifeless | yes, thats it | 03:28 |
| lifeless | whats the bug #f or this ? | 03:28 |
| wgrant | Bug #794802? | 03:28 |
| _mup_ | Bug #794802: OOPS-1986EA9 trying to add 'linux' task to a bug <regression> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/794802 > | 03:28 |
| lifeless | no answer on either phone | 03:31 |
| lifeless | trying phone again | 03:45 |
| lifeless | I have a fixed bugsubscription trigger | 03:45 |
| lifeless | which is likely to be a rather huge component. | 03:46 |
| wgrant | It was updating even for public bugs? | 03:46 |
| lifeless | yes | 03:46 |
| lifeless | [un]summarise doesn't know that it could skip for subscription triggered summarisation | 03:48 |
| wgrant | stub! | 03:51 |
| hloeung | he's on | 03:51 |
| StevenK | I think that is what wgrant was commenting on | 03:52 |
| hloeung | yeah, I'm just saying.... | 03:52 |
| wgrant | lifeless: Around? | 03:52 |
| lifeless | stub: hi | 03:52 |
| stub | yo | 03:52 |
| lifeless | stub: can I nab you for a moderately urgent voice call about fallot from bugsummary? | 03:52 |
| stub | k | 03:52 |
| lifeless | stub: (and sorry for repeatedly trying your mobile, all will become clear) | 03:53 |
| lifeless | https://bugs.launchpad.net/launchpad/+bug/794802 and the topic in -ops | 03:53 |
| _mup_ | Bug #794802: many bug activities timing out due to contention on bugsummary <regression> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/794802 > | 03:53 |
| lifeless | https://code.launchpad.net/~lifeless/launchpad/bug-794802 | 03:53 |
| lifeless | oh, and http://www.postgresql.org/docs/8.4/static/explicit-locking.html#LOCKING-ROWS is going to be referenced | 03:54 |
| lifeless | stub: skype or your mobile? | 03:54 |
| lifeless | ah, skype. :) | 03:54 |
| StevenK | lifeless: Is there going to be a test? | 03:54 |
| lifeless | stub: http://bazaar.launchpad.net/~lifeless/launchpad/bug-794802/revision/13176 | 04:02 |
| lifeless | stuart and I are talking through it | 04:04 |
| lifeless | the table level lock is deliberate due to edge cases | 04:04 |
| lifeless | we'll need to consider that in future | 04:04 |
| lifeless | we are doing 0.19 busummary updates / second | 04:04 |
| * StevenK suspects the topic here is out of date | 04:05 | |
| lifeless | this may be contention (5 second transactions) or it may be that we're mostly filtered through | 04:05 |
| lifeless | we're updating the bugsubscription trigger | 04:06 |
| lifeless | stub: http://pastebin.com/RZi7gfkq | 04:07 |
| lifeless | ok, stuart has applied the new subscription filter | 04:11 |
| lifeless | wgrant: whats the oops frequency looking like ? | 04:29 |
| wgrant | lifeless: Only 5 subscription timeouts in the last hour. | 04:29 |
| poolie | lifeless: one idea for lp people in Dublin is to to a bit of work towards turning loggerhead into a service, and then making it be hooked into the main lp web app | 04:30 |
| lifeless | poolie: its already a service | 04:30 |
| lifeless | poolie: I think hooking it in better and expanding its web service to be more useful to LP would be great | 04:31 |
| poolie | well, "used as a service" | 04:31 |
| poolie | is it already? | 04:31 |
| poolie | do you think this would be realistic to stab at in a one week sprint? | 04:31 |
| lifeless | its json api is used to show diffs / revs when yo uclick on expanders | 04:31 |
| lifeless | poolie: I wouldn't want the lp app servers doing callouts to loggerhead today: | 04:32 |
| lifeless | - we are not in position to parallelise | 04:32 |
| lifeless | - without parallelism it will make the appservers tiiiime out | 04:32 |
| lifeless | wgrant: what was the last one ? | 04:32 |
| lifeless | poolie: and loggerhead isn't consistently fast yet, and we don't have reliable timeout reports for it yet | 04:33 |
| poolie | ok, so the only path would be to have the browser pull data from it directly? | 04:33 |
| lifeless | right, which it does now but from a different domain | 04:34 |
| lifeless | so calculating the right url to pass and putting that in the lp pages, and making those urls available under the main hostnames to avoid SSL - thats tractable | 04:34 |
| wgrant | lifeless: 2011-06-09T03:01:31.659696+00:00 | 04:34 |
| lifeless | wgrant: thanks | 04:34 |
| poolie | "under the main hostname" meaning having apach proxy them or something? | 04:38 |
| cody-somerville | Why is the Ubuntu branches celebrity being removed? | 04:39 |
| wgrant | cody-somerville: Ubuntu's owners and uploaders are fulfilling the role instead. | 04:39 |
| cody-somerville | LP #524173 - There is a need for a 'bot' to have write access to the branches but not upload permissions. Would it make sense to create a 'bot account' and make that a celebrity? | 04:40 |
| wgrant | There will be no more celebrities. | 04:40 |
| poolie | so iow if we did this, we would model it by having an acl-type slot on the ubuntu distribution for "people who can write to branches but not upload"? | 04:41 |
| cody-somerville | So whats the recommended way for a process like package-import to get elevated privileges? | 04:41 |
| wgrant | cody-somerville: Give it upload privs, I suppose. | 04:41 |
| wgrant | cody-somerville: It will effectively have them anyway. | 04:41 |
| poolie | that was francis's approach | 04:41 |
| cody-somerville | wgrant, How? | 04:42 |
| wgrant | cody-somerville: How what? | 04:42 |
| cody-somerville | wgrant, That is, how will it effectively have upload permissions if it has write permissions to the branch? | 04:42 |
| wgrant | cody-somerville: BFBIP | 04:42 |
| wgrant | If you can alter the branch, you can compromise it. | 04:43 |
| wgrant | The next person to use it will get your exploit into the primary archive. | 04:43 |
| lifeless | stub: https://bugs.launchpad.net/ubuntu/+bugtarget-portlet-tags-content | 04:44 |
| === stub1 is now known as stub | ||
| lifeless | stub: https://bugs.launchpad.net/ubuntu/+bugtarget-portlet-tags-content | 04:45 |
| cody-somerville | wgrant, lots of teams in Debian maintain packages in branches. Folks can have write privs to the branch but not upload. Its a legitimate use case by its self. The fact that the upload could potentially not review changes others have made before uploading does not mean having write privs is the same as having upload privs. | 04:45 |
| wgrant | cody-somerville: Launchpad official branches operate under the rule that upload permissions == edit permissions | 04:46 |
| wgrant | To change that would be a redesign. | 04:46 |
| poolie | i'll propose this on the tb | 04:46 |
| lifeless | ok, we're rolling forward | 04:47 |
| StevenK | poolie: To the TB, during their meeting, or on the TB's mailing list? | 04:47 |
| poolie | i think moving from james to a role account would be a step forward | 04:47 |
| poolie | on the list | 04:47 |
| lifeless | poolie: yes, apache rewrite rules ftw | 04:48 |
| cody-somerville | I imagine The Developer Membership Board is the appropriate body to approach as the TB has delegated controlling upload permissions to that body. | 04:48 |
| StevenK | However, the TB has not delegated the branch permission, so I think it should go before the TB, not the DMB. | 04:49 |
| lifeless | poolie: what are you proposing to the tb ? | 04:49 |
| persia | Well, LP is kinda special: if LP decides to implement an internal role uploader, it doesn't really match the Ubuntu processes. | 04:49 |
| wgrant | persia: package-import isn't part of LP. | 04:49 |
| poolie | to change the package importer from impersonating james to having its own account, per bug 524173 | 04:49 |
| _mup_ | Bug #524173: package-import uses james_w credentials <Ubuntu Distributed Development:Triaged by mbp> < https://launchpad.net/bugs/524173 > | 04:49 |
| persia | LP has more opportunity to be malicious with Ubuntu contents than any uploader. | 04:50 |
| persia | wgrant, Should it be? | 04:50 |
| cody-somerville | persia, +1 | 04:50 |
| wgrant | persia: Depends on your definition of "part of LP". | 04:50 |
| lifeless | so, I would love to participate in this | 04:50 |
| lifeless | but I have a critical regression to fix. | 04:50 |
| lifeless | This has been discussed with at least various TB members already | 04:50 |
| poolie | you can reply to my mail or on the bug | 04:50 |
| poolie | it will not be settled today | 04:50 |
| persia | I'd consider a role account running services in a LOSA-controlled environment to be "part of LP" for the purposes of this discussion. | 04:51 |
| poolie | cjw at least has commented there | 04:51 |
| poolie | i just want to get it unblocked | 04:51 |
| lifeless | at the moment you have access to operate as james_w | 04:51 |
| lifeless | so you can be as malicious as you like | 04:51 |
| poolie | so, generally, lp will perhaps move into less-trusted interacting services | 04:51 |
| lifeless | a dedicated account will mitigate that | 04:51 |
| lifeless | by making it clear who uploaded etc | 04:52 |
| lifeless | TB can easily have a bot that watches this account and makes sure it has no gpg key :) | 04:52 |
| cody-somerville | What about the principle of least privilege? | 04:52 |
| persia | cody-somerville, See comment #8 on the bug: there isn't really any semantic difference between commit-to-branch and upload. | 04:53 |
| poolie | it's a good principle | 04:53 |
| lifeless | within the year push to the branch will do the build | 04:53 |
| poolie | this moves us closer towards it | 04:53 |
| lifeless | so the principle will say 'this is the least privilege' | 04:53 |
| lifeless | (modulo various details about how BFBIP all works) | 04:53 |
| cody-somerville | if the role account is a member of Ubuntu Core Developers team then it still has tons of permissions it does not legitimately require | 04:53 |
| StevenK | Agreed. | 04:54 |
| poolie | so, | 04:54 |
| lifeless | we can always add a dedicated role in future if desired... but note that *right now* its running as jamesw | 04:54 |
| persia | cody-somerville, Easier is to have the role account just have upload to everything, rather than making it a core-dev. | 04:54 |
| lifeless | so it has those permissions,. | 04:54 |
| lifeless | Moving to a dedicated losa administered account is an improvement. | 04:54 |
| lifeless | you're welcome to argue that more improvement is needed. But that is a separate discussion. | 04:54 |
| lifeless | nothing is *regressing* | 04:54 |
| lifeless | (and I am sure that me, poolie, wgrant etc are all open to such an argument) | 04:55 |
| poolie | i completely agree | 04:55 |
| poolie | this is a step forward and not a step back | 04:55 |
| persia | And there are more steps, but they need to be thought about more, and not having thought about them yet shouldn't block this. | 04:56 |
| cody-somerville | Who would have access to the account? | 04:57 |
| poolie | canonical staff who maintain the importer | 04:58 |
| lifeless | (that is, the people that currently have access to james_w's account) | 04:58 |
| cody-somerville | If you say only LOSAs then I'd be alot happier about this | 04:58 |
| cody-somerville | (as they can do anything they want already) | 04:58 |
| poolie | right | 04:58 |
| poolie | that is not true at the moment but it will become so | 04:58 |
| lifeless | cody-somerville: there is an RT to move it to losa-only. | 04:58 |
| poolie | there is an RT asking for it in the queeu | 04:58 |
| wgrant | Eventually it should only be the LOSAs, yes. | 04:58 |
| lifeless | cody-somerville: its also in-progress. | 04:58 |
| wgrant | And the role account should only have ArchivePermissions for the primary archive, not any others. | 04:59 |
| poolie | http://pastebin.ubuntu.com/622270/ | 05:08 |
| poolie | ^ draft email; comments welcome | 05:09 |
| persia | poolie, Enough of TB has read access to RT that it may be worth mentioning the ticket number. | 05:10 |
| hloeung | ^ great, more mail to ignore ;-) | 05:10 |
| poolie | :) | 05:10 |
| poolie | ok | 05:10 |
| poolie | good idea persia | 05:10 |
| persia | poolie, Also, it's probably worth phrasing the request differently. The TB acts as a deliberative body, but is intentionally not expected to be a bottleneck. The expectation is that people do stuff, and the TB provides observance and guidance. | 05:11 |
| poolie | so "i plan to do this next week"? | 05:11 |
| persia | I'd suggest either announcing to the TB that LP plans to do this, and asking for feedback, *OR* structuring it as a request for the TB to grant the appropriate permissions to the robot account. | 05:12 |
| persia | In either case, I recommend cc: bryceh as the representative stakeholder | 05:13 |
| poolie | ok, ka-thunk | 05:15 |
| === jtv is now known as jtv-eat | ||
| === mwhudson_ is now known as mwhudson | ||
| LPCIBot | Project devel build #790: STILL FAILING in 5 hr 56 min: https://lpci.wedontsleep.org/job/devel/790/ | 06:12 |
| StevenK | Hmmmm | 06:13 |
| lifeless | and it passes tests. --woot-- | 06:26 |
| lifeless | maybe not all :P | 06:26 |
| lifeless | wgrant: lp:~lifeless/launchpad/bug-794802 | 06:31 |
| lifeless | if you're interested | 06:31 |
| === almaisan-away is now known as al-maisan | ||
| === al-maisan is now known as almaisan-away | ||
| LPCIBot | Project windmill-devel build #192: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/192/ | 07:48 |
| lifeless | wgrant: AWOL for ~ 40; if stub turns up point him at his email. | 08:08 |
| wgrant | lifeless: Sure. | 08:08 |
| adeuring | good morning | 08:15 |
| wgrant | lifeless: FWIW we're still seeing some +choose-affected-product timeouts, but no subscribe ones. | 08:21 |
| wgrant | And lots of BugTask:EntryResource timeouts now :/ | 08:22 |
| wgrant | Mostly status changes on Ubuntu bugs. | 08:22 |
| wgrant | Particularly linux. | 08:22 |
| wgrant | Almost entirely on a single bug. Perhaps lots of retries. | 08:23 |
| wgrant | Tag changes taking 5s... | 08:26 |
| === wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:214 - 0:[######=_]:256 | ||
| LPCIBot | Project windmill-devel build #193: STILL FAILING in 41 min: https://lpci.wedontsleep.org/job/windmill-devel/193/ | 08:30 |
| lifeless | wgrant: yeah | 08:38 |
| lifeless | (back) | 08:39 |
| === wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs:209 - 0:[######=_]:256 | ||
| lifeless | \o/ | 08:44 |
| spiv | Yay! | 08:44 |
| wgrant | Still well above where we were three weeks ago :( | 08:44 |
| lifeless | need to stop adding bugs | 08:45 |
| wgrant | We've only had like one new critical today. | 08:45 |
| lifeless | 3 ? | 08:46 |
| lifeless | bugsummary | 08:46 |
| wgrant | Did I miss some? | 08:46 |
| lifeless | francis escalated the accessibility-in-bugtask bug | 08:46 |
| lifeless | and I think there was another | 08:46 |
| wgrant | :( | 08:46 |
| lifeless | certainly 2 | 08:47 |
| lifeless | another day w/no microservice :( | 08:48 |
| lifeless | wgrant: thanks | 08:52 |
| wgrant | lifeless: Huh? | 08:52 |
| lifeless | closing off the bugs | 08:52 |
| wgrant | Oh, right. | 08:52 |
| wgrant | I left the subs ones open for Yellow to close or not. | 08:53 |
| lifeless | https://bugs.launchpad.net/ubuntu/+bugtarget-portlet-tags-content is still fast. | 09:09 |
| lifeless | I'm having to pinch myself :) | 09:09 |
| wgrant | Yes, but it made everything else slow :( | 09:10 |
| lifeless | gotta pick your battles | 09:10 |
| lifeless | we should chang that to json | 09:10 |
| wgrant | We should change a lot of things to JSON. | 09:11 |
| lifeless | yes | 09:11 |
| lifeless | this is one of them | 09:11 |
| === almaisan-away is now known as al-maisan | ||
| mrevell | Morning | 09:20 |
| * jelmer waves | 09:21 | |
| bigjools | morning mrevell, jelmer | 09:21 |
| jelmer | hi bigjools | 09:22 |
| jelmer | wgrant: it's odd, some bzr-svn imports seem a lot slower; locally I don't see that effect though | 09:23 |
| wgrant | jelmer: Some got fast again. | 09:23 |
| wgrant | After a few tries. | 09:23 |
| wgrant | I retried one 4 times. | 09:24 |
| jelmer | wgrant: which one? | 09:24 |
| wgrant | First was 40 minutes, second 25ish, third 10, fourth 1. | 09:24 |
| wgrant | I can't remember... chromium crashed. | 09:24 |
| wgrant | Let me see if I can find it in history. | 09:24 |
| lifeless | ahh alpha software | 09:24 |
| jelmer | wgrant: it looks to me like it's all bzr-svn imports that are affected - have you seen any bzr-git or bzr-hg imports becoming slower? | 09:25 |
| wgrant | jelmer: https://code.launchpad.net/~vcs-imports/flylinkdc/trunk | 09:25 |
| wgrant | jelmer: No, only bzr-svn. | 09:25 |
| wgrant | jelmer: ANd only bzr-svn has been eating swap. | 09:25 |
| wgrant | AFAIK | 09:25 |
| jelmer | ah, so it's a memory thing? | 09:26 |
| wgrant | Well, pear was swapping heavily. | 09:26 |
| wgrant | One import eating 40% of the RAM. | 09:26 |
| lifeless | \o/ | 09:26 |
| wgrant | But it had vanished before the next ps. | 09:26 |
| wgrant | So we don't know which it is. | 09:26 |
| lifeless | I thought we had ulimit on it | 09:26 |
| jelmer | even with some swapping, it seems like there shouldn't be a 10 second vs 1 hour difference | 09:32 |
| jelmer | wgrant: how did you get to the ps output, losa interaction? | 09:42 |
| wgrant | jelmer: Yes, LOSA. | 09:47 |
| jelmer | wgrant: ahh, I think it's got to do with the fix for bug 309682 | 10:06 |
| _mup_ | Bug #309682: tags are copied but their revisions may not be <gnome> <udd> <Bazaar:Fix Released by spiv> <Bazaar Git Plugin:Fix Released by jelmer> <Bazaar Hg Plugin:Triaged by jelmer> <Bazaar Subversion Plugin:Fix Committed by jelmer> < https://launchpad.net/bugs/309682 > | 10:06 |
| wgrant | jelmer: It looks rather like that, yes. | 10:09 |
| jelmer | the fact that they become increasingly faster (rather than just being slow once) is really weird though | 10:09 |
| wgrant | jelmer: It's not going to try to pull all the revs mentioned by tags in the whole repo, is it? :) | 10:09 |
| wgrant | Yes... | 10:09 |
| jelmer | wgrant, I'm wondering if we should make that an optional thing, there are situations in which it's not correct and causes a lot of extra data | 10:11 |
| wgrant | jelmer: Possibly. Anyway, it should settle down eventually, I guess. | 10:25 |
| LPCIBot | Project windmill-devel build #194: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill-devel/194/ | 10:36 |
| === al-maisan is now known as almaisan-away | ||
| === Daviey is now known as Da | ||
| === Da is now known as Daviey | ||
| bigjools | is there any reason why a "bzr pull" would update loads of files and then finish with "bzr: ERROR: [Errno 13] Permission denied" ? | 11:36 |
| LPCIBot | Yippie, build fixed! | 11:37 |
| LPCIBot | Project devel build #791: FIXED in 5 hr 25 min: https://lpci.wedontsleep.org/job/devel/791/ | 11:37 |
| LPCIBot | Project parallel-test build #22: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/parallel-test/22/ | 11:47 |
| * henninge lunches | 12:03 | |
| LPCIBot | Project windmill-db-devel build #375: STILL FAILING in 1 hr 16 min: https://lpci.wedontsleep.org/job/windmill-db-devel/375/ | 12:12 |
| === almaisan-away is now known as al-maisan | ||
| StevenK | bigjools: O hai. You haz many lots QA to do. | 12:45 |
| bigjools | me personally? | 12:46 |
| StevenK | bigjools: Yes, four revisions are marked as assigned to you on the deployment report. | 12:46 |
| bigjools | StevenK: and they were all submitted with --no-qa, | 12:47 |
| StevenK | I see. | 12:47 |
| StevenK | rvba: O hai. You haz many lots QA to do. | 12:47 |
| rvba | StevenK: indeed. | 12:58 |
| rvba | StevenK: please don't forget to take a look at the multiple parents init stuff when you get a chance :) | 13:00 |
| LPCIBot | Project windmill-devel build #195: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/195/ | 13:07 |
| jml | lifeless: it might have to be next week. sorry. | 13:31 |
| === matsubara-afk is now known as matsubara | ||
| deryck | Morning, all. | 13:59 |
| jcsackett | morning folks. | 14:11 |
| bigjools | morning | 14:11 |
| deryck | Morning, jcsackett | 14:12 |
| jcsackett | heya henninge: did the branch you merged my stuff into land? | 14:20 |
| jml | jelmer: hi | 14:20 |
| huwshimi | jml: Hey | 14:21 |
| jml | huwshimi: hello | 14:21 |
| jml | huwshimi: let's have a call. | 14:21 |
| huwshimi | jml: Sure | 14:21 |
| jelmer | jml: hi | 14:21 |
| jml | jelmer: was going to talk to you about BFBIP, but need to talk to huwshimi first. | 14:21 |
| jelmer | jml: ah, ok. you know where to find me :) | 14:22 |
| === Ursula is now known as Ursinha | ||
| === mrevell is now known as mrevell-lunch | ||
| henninge | jcsackett: yes, it did, your branches say so, too. ;-) | 14:33 |
| henninge | jcsackett: it's on the next buildbot ride | 14:34 |
| jcsackett | henninge: cool. do you know if qa-tagger will be able to keep track of this, or should i keep an eye on your branch to qa my related bugs? | 14:35 |
| henninge | jcsackett: I linked my branch to bug 787595, too, so it should update it. | 14:35 |
| _mup_ | Bug #787595: person picker could have a link to choose myself when I am a valid choice <disclosure> <person-picker> <qa-untestable> <Launchpad itself:In Progress by jcsackett> < https://launchpad.net/bugs/787595 > | 14:35 |
| jcsackett | ah, cool. thanks, henninge. | 14:36 |
| henninge | np | 14:36 |
| jcsackett | and sorry for the confusion regarding my branch names. | 14:36 |
| === mrevell-lunch is now known as mrevell | ||
| LPCIBot | Project windmill-devel build #196: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill-devel/196/ | 15:21 |
| === al-maisan is now known as almaisan-away | ||
| jcsackett | deryck: in YUI widgets the "initializer" method is the right place to parse and deal with passed in config stuff, right? | 15:45 |
| deryck | jcsackett, yup | 15:46 |
| jcsackett | cool. wanted to make sure i wasn't embarking on something goofy (as has so always been the case). :-P | 15:46 |
| deryck | jcsackett, if you need to, of course. the widget infrastructure does all of the default values and changing values based on passed in config for you. | 15:47 |
| deryck | jcsackett, but if you need something based on the value of two config options, for example, then the initializer would be the place to do that. | 15:47 |
| jcsackett | deryck: yeah, this is dealing with non-default config options. | 15:47 |
| deryck | right | 15:48 |
| deryck | yup, you're on the right track | 15:48 |
| jcsackett | cool. thanks, deryck. | 15:48 |
| deryck | np | 15:48 |
| sinzui | henninge: test_picker_displays_empty_list does not pass in devel | 15:54 |
| sinzui | I am debugging this now. | 15:54 |
| henninge | sinzui: oh | 15:55 |
| henninge | sinzui: what do you mean? | 15:55 |
| sinzui | The test fails in my browser and in the YUI test layer I just fixed | 15:56 |
| henninge | ah | 15:56 |
| henninge | strange, it passed for me | 15:56 |
| sinzui | We expect an empty string, but get undefined | 15:56 |
| sinzui | henninge: does it still pass in devel for you? I think this may be a merge compliation | 15:57 |
| henninge | let me try | 15:57 |
| sinzui | henninge: I tested with fireforx and chromium | 15:58 |
| sinzui | The test runner will use webkit | 15:58 |
| henninge | sinzui: all tests pass | 16:00 |
| sinzui | how many tests passed? | 16:00 |
| sinzui | I have 9 pass and 1 fail | 16:00 |
| henninge | 10 pass | 16:00 |
| sinzui | have you pulled devel in that last hour? | 16:01 |
| henninge | sinzui: what does your line 22 to 25 of lib/lp/app/javascript/widgets.js look like? | 16:01 |
| henninge | I just pulled right now | 16:01 |
| sinzui | henninge: It looked like the line in the diff | 16:03 |
| sinzui | if (data.title === undefined) { | 16:03 |
| sinzui | // Display an empty element if data is empty. | 16:03 |
| sinzui | return li_title; | 16:03 |
| henninge | ok | 16:03 |
| sinzui | } | 16:03 |
| henninge | that should pass | 16:04 |
| henninge | the failure you described looked like those lines were missing. | 16:04 |
| LPCIBot | Project parallel-test build #23: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/parallel-test/23/ | 16:04 |
| henninge | sinzui: 10 pass, both in firefox and chromium. | 16:05 |
| sinzui | I can see the for the innerHTML for yui3-picker-results, yet the innerTEXT is undefined | 16:05 |
| sinzui | oh. | 16:05 |
| sinzui | maybe *I* need to rebuild | 16:05 |
| henninge | yes, this must be something to do with your local setup | 16:06 |
| sinzui | deryck: I see that tests spend more than 1/5 of the time setting individual YUI test layers for each app. Do we really want a BugsYUITestLayer if we can run all the tests in 2 minutes | 16:09 |
| deryck | sinzui, no, we don't need app specific yui layers. | 16:09 |
| deryck | app-specific anything is old school anyway ;) | 16:09 |
| sinzui | I will remove them. | 16:10 |
| deryck | especially with yui tests it makes less sense anyway.... say I change the picker.... I need to know all uses of the picker are safe, not just one app's version. | 16:10 |
| sinzui | henninge: I think the merge is the issue. Your block of code is never entered. I need to review the test setup I think. | 16:14 |
| LPCIBot | Project windmill-db-devel build #376: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/376/ | 16:27 |
| LPCIBot | Project db-devel build #623: FAILURE in 5 hr 57 min: https://lpci.wedontsleep.org/job/db-devel/623/ | 16:34 |
| === matsubara is now known as matsubara-lunch | ||
| jml | sorry it's been such a brief day folks, but I have to go. back tomorrow for Action Friday. | 17:07 |
| LPCIBot | Project windmill-devel build #197: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-devel/197/ | 17:11 |
| === deryck is now known as deryck[lunch] | ||
| === salgado is now known as salgado-lunch | ||
| sinzui | I need help with an ec2 image. I am following the steps in https://dev.launchpad.net/EC2Test/Image | 17:27 |
| sinzui | I have uploaded the image and I see it in S3. I do not know how to make it public | 17:27 |
| bigjools | sinzui: is that a new thing? I don't remember having to do that | 17:28 |
| sinzui | I added my name to lib/devscripts/ec2test/account.py, but I do not see my ami listed. I assumed maybe that was because it is private. | 17:31 |
| bigjools | maybe | 17:31 |
| bigjools | you probably need the AWS console | 17:31 |
| bigjools | right click the AMI and select "edit permissions" | 17:32 |
| bigjools | I have a public/private radio selection there | 17:32 |
| sinzui | bigjools: I do not see that. I see a bucket in s3 with with image name, but it does not say it is an ami | 17:33 |
| bigjools | sinzui: click on "AMIs" on the left | 17:34 |
| sinzui | My browser says there is no "AMI" on the page :( | 17:34 |
| bigjools | https://console.aws.amazon.com/ec2/home?region=us-east-1#s=Images | 17:34 |
| sinzui | bigjools: thank you. aws started in s3, not ec2 | 17:35 |
| bigjools | heh | 17:35 |
| bigjools | it's a wall of jargon on that page | 17:35 |
| gary_poster | hi abentley. Without having to do any research, can you confirm that we do not support importing private github branches? If so, please do. | 17:39 |
| abentley | gary_poster: I can't be 100% sure without research, but I think it's very unlikely. | 17:54 |
| gary_poster | ok, thanks abentley | 17:54 |
| === deryck[lunch] is now known as deryck | ||
| === salgado-lunch is now known as salgado | ||
| === Ursinha is now known as Ursinha-lunch | ||
| === matsubara-lunch is now known as matsubara | ||
| lifeless | morning y'all | 19:14 |
| cody-somerville | There seems like there used to always be an on call reviewer but now every time I've checked the topic its just '-'. : ( | 19:56 |
| lifeless | what do you need reviewed? | 19:57 |
| cody-somerville | https://code.launchpad.net/~timrchavez/launchpad/set_ppa_private_from_api_724740-2/+merge/63950 | 19:58 |
| cody-somerville | lifeless, I notice that the associated bug (LP #724740) is listing the superseded MP instead of the new one. Is there something that needs to be done to update that or is that a bug? | 19:59 |
| _mup_ | Bug #724740: setting a ppa private cannot be done over the API <api> <oem-services> <ppa> <Launchpad itself:In Progress by timrchavez> < https://launchpad.net/bugs/724740 > | 19:59 |
| lifeless | cody-somerville: file a bug, include a screenshot and the url to the old mp and the new mp | 19:59 |
| lifeless | that should get reviewed today, unf mid-crisis still | 19:59 |
| cody-somerville | and of course for some reason it now shows the new mp, lol. page must have been cached in my browser. | 20:01 |
| jcsackett | cody-somerville: I am OCR, it would seem my morning topic change didn't take. | 20:02 |
| === jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett | Critical bugs:209 - 0:[######=_]:256 | ||
| jcsackett | cody-somerville: i'm looking at it now, and sorry about the topic confusion. | 20:05 |
| cody-somerville | jcsackett, No problem! :) Kudos! | 20:11 |
| === elmo__ is now known as k | ||
| lifeless | k: special? | 20:21 |
| === k is now known as Guest82647 | ||
| === Guest82647 is now known as elmo | ||
| === Ursinha-lunch is now known as Ursinha | ||
| nigelb | Argh. | 20:25 |
| nigelb | Just found a bug in LP. | 20:25 |
| nigelb | Well, possible bug. | 20:25 |
| lifeless | no, really? | 20:25 |
| nigelb | lifeless: I headdesked for an hour trying to figure this out :-) | 20:26 |
| nigelb | https://launchpad.net/~ubuntumembers/+members | 20:26 |
| nigelb | There are 521 direct members of the "Ubuntu Members" team, and 695 people | 20:26 |
| nigelb | the 695 is not entirely true. | 20:26 |
| nigelb | So, if you have 10 teams | 20:26 |
| lifeless | teams are members too | 20:27 |
| nigelb | if counts the unique members in those 10 teams | 20:27 |
| nigelb | lifeless: ah. | 20:27 |
| lifeless | this could be presented better | 20:27 |
| nigelb | Yes | 20:27 |
| lifeless | and a case can be made that a team shouldn't count as a member | 20:27 |
| nigelb | exactly | 20:27 |
| nigelb | I was trying to figure out why my script wasn't catching all the members | 20:27 |
| lifeless | however our db doesn't know the difference today in teamparticipation | 20:28 |
| lifeless | so its not cheap to do differently | 20:28 |
| nigelb | So, doing this would be expensive. | 20:28 |
| lifeless | on your script shouldn't be iterating members | 20:28 |
| nigelb | Right. | 20:28 |
| lifeless | it should be interating participants | 20:28 |
| LPCIBot | Project parallel-test build #24: STILL FAILING in 1 hr 15 min: https://lpci.wedontsleep.org/job/parallel-test/24/ | 20:28 |
| nigelb | members was my use case. I wanted to get a list of all the members including sub teams. | 20:28 |
| nigelb | I can just exclude the teams out of it and I could get a clean list. | 20:29 |
| nigelb | or does participants do that for me? | 20:29 |
| lifeless | https://launchpad.net/api/1.0/~launchpad/participants | 20:30 |
| lifeless | [yes] | 20:30 |
| lifeless | hi stub | 20:30 |
| SpamapS | still fighting the perf regression? | 20:31 |
| stub | yo | 20:31 |
| lifeless | SpamapS: yeah | 20:31 |
| lifeless | SpamapS: did you see atimeout ? | 20:31 |
| SpamapS | It renders the sru-accept script we use in ubuntu-archive-tools unusable | 20:31 |
| lifeless | what does the script do | 20:32 |
| nigelb | lifeless: "members = team.participants" heh, I was already using participants :) | 20:32 |
| lifeless | nigelb: then that is recursing into subteams for you | 20:32 |
| nigelb | yup \m/ | 20:32 |
| lifeless | nigelb: transitive closure, but I think it excludes teams. IMBW, check the Person interface and source ;) | 20:32 |
| lifeless | stub: I have some feedback on the -journal case | 20:32 |
| SpamapS | lifeless: looks through the tasks of given bug #'s for the given package name/series, and marks it to Fix Committed, then tags the bug verification-needed and comments | 20:33 |
| lifeless | stub: and am looking into the test failure | 20:33 |
| nigelb | lifeless: Thanks! | 20:33 |
| stub | cool. It didn't seem obvious, but looks legit. | 20:33 |
| SpamapS | lifeless: I'm not sure which operation is timing out | 20:33 |
| stub | Only seems the one failure though, but I couldn't see what is different between the bugtag case and others | 20:33 |
| lifeless | stub: is it safe to chang the python stuff around person merge without adding the references metadata to the db ? | 20:33 |
| stub | lifeless: yes, that is fine | 20:34 |
| SpamapS | lifeless: but I have the HTML of the failure | 20:34 |
| lifeless | stub: oh, bugsummary.txt is blowing up for me just now | 20:34 |
| lifeless | SpamapS: got the oops ? | 20:34 |
| stub | lifeless: I think it is one failure (off by one, as if a bugtag didn't get created), and fallout from that | 20:34 |
| SpamapS | OOPS-1986AO121 | 20:35 |
| lifeless | stub: I'm concerned that we're going to journal a complete-remove and complete-add to the journal even when nothing interesting changed - 50% of the rows or more could be noise | 20:35 |
| stub | lifeless: My assumption is that just spitting inserts to the journal is faster than trying to do updates and avoids the locking issues. | 20:36 |
| SpamapS | and OOPS-1986DY97 | 20:36 |
| lifeless | stub: the temp journal won't have locking issues | 20:36 |
| stub | lifeless: Using the temp file seems a good idea though | 20:36 |
| stub | temp table I mean | 20:36 |
| lifeless | stub: ok, I have a patch that does that, let me commit and push for your hilarity | 20:36 |
| SpamapS | lifeless: only look into it if it helps with the solution. If not, I can manually accept bugs until the problem is solved. :) | 20:37 |
| stub | btw. there are no mixed case tablenames - FooBar is cast to lowercase. "FooBar" is a mixed case table name. | 20:37 |
| * lifeless headdesks | 20:37 | |
| lifeless | stub: lp:~lifeless/launchpad/bug-794802 | 20:38 |
| lifeless | SpamapS: it hasn't synced yet; did you get a backtrace in the html ? | 20:39 |
| lifeless | stub: there are two reasons I can see slow queries - contention, or we're just adding too much work | 20:40 |
| lifeless | stub: we're kindof betting its contention | 20:40 |
| stub | the queries I've checked are only attempting to update a handful of rows, so I'm betting on contention | 20:40 |
| lifeless | stub: argh, *really* not awake | 20:40 |
| SpamapS | lifeless: no sorry just the oops number. | 20:40 |
| lifeless | stub: yeah, if its slow on the db, its contention | 20:41 |
| lifeless | (now that the seq scans are gone) | 20:41 |
| lifeless | stub: they are, aren't they? | 20:41 |
| lifeless | stub: as in the slow occurences you see when explained show index use ? | 20:41 |
| stub | My scenario is some dude triaging a big and clicking three of four things - change status, target, add a comment. Each is a separate ajax request being launched around the same time and wanting to lock the same rows in BugSummary. | 20:41 |
| benji | jcsackett: thanks for the review | 20:42 |
| lifeless | stub: so status, target would serialise normally | 20:42 |
| jcsackett | benji: you're welcome. | 20:42 |
| lifeless | stub: (same row in bug) | 20:42 |
| stub | Yes. We have fixed index use. Queries are fast, except when they are real slow. Looking at the plans they seem fine. | 20:42 |
| lifeless | stub: tag + status would serialise on bugsummary | 20:42 |
| lifeless | stub: as would tag + target | 20:42 |
| lifeless | commenting won't change the aggregates so won't lock anything | 20:43 |
| stub | So if one takes 4 seconds, the other is blocked for 4 seconds before it gets to issue its query. | 20:43 |
| lifeless | well, before the flush gets the row its trying to update | 20:43 |
| stub | And this blocking is happening inside the trigger, so counts as SQL evaluation time. | 20:43 |
| lifeless | yeah | 20:43 |
| lifeless | so this failure | 20:44 |
| lifeless | we're looking for tag is null | 20:44 |
| stub | Best theory I've come up with anyway :) | 20:44 |
| lifeless | (line 136 of the doctest) | 20:44 |
| lifeless | 3 new bugs should result in an increase of 3 in the tag=null rows | 20:45 |
| lifeless | but we're seeing a total of three | 20:45 |
| lifeless | if I comment out line 128 I see the same thing | 20:46 |
| stub | So given how similar the current branch is to the previous, wtf is is failing now and not before? | 20:47 |
| lifeless | I'm trying with the rollup() call commented out | 20:47 |
| stub | I guess it is in the combinedbug view... maybe rollup | 20:48 |
| lifeless | which results in the right raw data | 20:48 |
| lifeless | so the journal is capturing everything | 20:48 |
| lifeless | the rollup appears to be discarding a row | 20:48 |
| === benji is now known as Guest88900 | ||
| lifeless | I'm going to toss a few sample bugs into launchpad_dev and look at the rollup by hand | 20:50 |
| === benji___ is now known as benji | ||
| lifeless | yeah its messed up I think | 20:55 |
| lifeless | id | sum | product | productseries | distribution | distroseries | sourcepackagename | viewed_by | tag | status | milestone | 20:55 |
| lifeless | ----+-----+---------+---------------+--------------+--------------+-------------------+-----------+-----+--------+----------- | 20:55 |
| lifeless | | 2 | 21 | | | | | | | 10 | | 20:55 |
| lifeless | | 1 | 21 | | | | | | moo | 10 | | 20:55 |
| lifeless | (2 rows) | 20:55 |
| lifeless | launchpad_dev=# select * from bugsummaryjournal; | 20:55 |
| lifeless | id | count | product | productseries | distribution | distroseries | sourcepackagename | viewed_by | tag | status | milestone | 20:55 |
| lifeless | ----+-------+---------+---------------+--------------+--------------+-------------------+-----------+-----+--------+----------- | 20:55 |
| lifeless | 1 | 1 | 21 | | | | | | | 10 | | 20:55 |
| lifeless | 2 | 1 | 21 | | | | | | | 10 | | 20:55 |
| lifeless | 3 | -1 | 21 | | | | | | | 10 | | 20:55 |
| lifeless | 4 | 1 | 21 | | | | | | moo | 10 | | 20:55 |
| lifeless | 5 | 1 | 21 | | | | | | | 10 | | 20:55 |
| lifeless | (5 rows) | 20:55 |
| lifeless | no, actually, that sums to 3 | 20:55 |
| lifeless | and we get three | 20:55 |
| lifeless | though adding the tag should really have only journalld one row | 20:56 |
| lifeless | but the rollout output was | 20:57 |
| lifeless | 65 | 1 | 21 | | | | | | | 10 | | 20:57 |
| lifeless | 66 | 1 | 21 | | | | | | moo | 10 | | 20:57 |
| lifeless | which is clearly wrong | 20:57 |
| stub | because we expand the tag to the null and the tag - two rows. | 20:57 |
| lifeless | stub: no | 20:57 |
| lifeless | because _dec and _inc only update by 1 | 20:57 |
| lifeless | they don't use the vector | 20:57 |
| lifeless | stub: UPDATE BugSummary SET count = count - 1 | 20:58 |
| lifeless | easy fix | 20:58 |
| stub | Yer | 20:58 |
| stub | I see. count might be 2 or -2, but we dec or inc by one only | 20:59 |
| stub | 'Cause I was a smart arse and wanted to minimize updates :) | 20:59 |
| stub | You fixing on your branch? | 20:59 |
| lifeless | yes | 20:59 |
| lifeless | test running now | 20:59 |
| lifeless | worked for the doctest | 21:01 |
| lifeless | pushing now | 21:02 |
| lifeless | we're running on the temp journal atm right ? | 21:02 |
| lifeless | stub: :!bzr push | 21:02 |
| lifeless | Using saved push location: lp:~lifeless/launchpad/bug-794802 | 21:02 |
| lifeless | Pushed up to revision 13189. | 21:02 |
| stub | Nothing in test_bugsummary is invoking rollup, so all those tests passed. | 21:02 |
| lifeless | thank mercy for doctests :P | 21:02 |
| stub | lifeless: Yes. The first MP is what is running right now, with patches -1 and -2 | 21:02 |
| lifeless | so I think we're busy corrupting by off-by-ones just now | 21:03 |
| lifeless | anyhow, my branch is pushed | 21:03 |
| stub | So in patch-2208-75-0.sql we should delete * from bugsummary and reinitialize using the SQL from 63-0 | 21:04 |
| lifeless | http://bazaar.launchpad.net/~lifeless/launchpad/bug-794802/revision/13188 and http://bazaar.launchpad.net/~lifeless/launchpad/bug-794802/revision/13189 are my incremental changes over your code | 21:04 |
| lifeless | stub: nah - we've got those new dimensions to add | 21:04 |
| lifeless | stub: do the reinit in that patch | 21:04 |
| stub | lifeless: The bug is in rollup, and that isn't what is running atm. So data should be fine. | 21:05 |
| lifeless | stub: no, the bug was in _inc and _dec, which the temp journal code uses | 21:06 |
| stub | oh... your journal has the same problem | 21:06 |
| stub | yer. | 21:06 |
| lifeless | so how should we proceed; its late for you I know. | 21:06 |
| stub | lifeless: ahh... but is there anyway to get anything but -1 or +1 in the count atm? | 21:06 |
| lifeless | stub: there is | 21:07 |
| lifeless | distro bug tasks and bugs with tags | 21:07 |
| lifeless | erm, scratch tags. | 21:07 |
| lifeless | distro source package bug tasks | 21:07 |
| stub | invoking multiple triggers in one transaction | 21:07 |
| lifeless | actually no I think its fine | 21:07 |
| lifeless | because each _row_ changed flushes separately to the table | 21:08 |
| lifeless | that that had damn well better be unary or we are so screwed. | 21:08 |
| stub | yes, I think that is right. | 21:08 |
| stub | which is why the doctest used to pass with the same bug | 21:09 |
| lifeless | yer | 21:09 |
| stub | lifeless: So count is positive when we call _dec? Yay. | 21:10 |
| lifeless | *blink* | 21:10 |
| stub | + UPDATE BugSummary SET count = count - $1.count | 21:10 |
| lifeless | can't be | 21:10 |
| lifeless | thats a thinko - and uncovered by tests | 21:11 |
| lifeless | we only call _dec from rollup now | 21:11 |
| stub | lifeless: Might want to invoke rollout in the test_bugsummary helper just after the flush and invalidate, like the doctest. | 21:11 |
| stub | it has much more coverage | 21:12 |
| lifeless | ok | 21:12 |
| lifeless | perhaps we should run it twice | 21:12 |
| lifeless | once without and once with | 21:12 |
| lifeless | that can wait | 21:12 |
| stub | I'm more interested in seeing if there are other lurking edge cases | 21:13 |
| lifeless | running all the tests with the rollup call added and the buggy _dec | 21:14 |
| lifeless | I want to see if we have coverage | 21:14 |
| stub | So in theory, something should fail now or the laws of mathematics are being warped. | 21:15 |
| lifeless | non euclidean geometry FTW | 21:15 |
| lifeless | SpamapS: your oops is | 21:15 |
| lifeless | SELECT bug_summary_dec( $1 )"\\nPL/pgSQL function "bug_summary_flush_temp_journal" line 9 at PERFORM\\nSQL statement "SELECT bug_summary_flush_temp_journal | 21:15 |
| lifeless | SpamapS: e.g. the contention we're working on | 21:16 |
| lifeless | yeah, we get fail | 21:18 |
| lifeless | stub: all tests pass with that patch | 21:20 |
| stub | woot | 21:20 |
| stub | Push again? | 21:20 |
| lifeless | 13191 available at your local bzr server. | 21:21 |
| === almaisan-away is now known as al-maisan | ||
| lifeless | stub: I'm going to fry up some brekky | 21:25 |
| stub | k | 21:25 |
| lifeless | stub: while you review | 21:25 |
| stub | Yup | 21:25 |
| stub | Don't worry about me, I'm already stuff full of dead pig. | 21:26 |
| lifeless | yum | 21:30 |
| lifeless | stub: so, this safe to progress? | 21:31 |
| lifeless | the tag portlets will freeze until we get the new view code deployed | 21:32 |
| lifeless | but thats fine IMO | 21:32 |
| stub | You cook too fast | 21:32 |
| lifeless | they are for most projects pretty static | 21:32 |
| lifeless | <- optimiser | 21:32 |
| stub | I thought views on this were still behind a feature flag? | 21:32 |
| lifeless | no | 21:33 |
| flacoste | bac: now that we have IServiceUsage, shouldn't we kill ILaunchpadUsage? | 21:33 |
| lifeless | stub: but there is only one using it | 21:33 |
| lifeless | stub: https://bugs.launchpad.net/ubuntu/+bugtarget-portlet-tags-content | 21:33 |
| bac | flacoste: looking | 21:34 |
| lifeless | stub: I'm going to delete patch 75 from the branch, so that we can land on devel | 21:34 |
| stub | lifeless: DELETE FROM bugsummary_temp_journal; is better spelt TRUNCATE bugsummary_temp_journal (but only with temporary tables until we modernize our slony) | 21:34 |
| stub | lifeless: Sure. | 21:34 |
| jcsackett | bac: flacoste: i seem to recall there being a uses_malone use case that buggered our desire to completely kill ILaunchpadUsage | 21:34 |
| jcsackett | i may be confusing things though. | 21:34 |
| flacoste | jcsackett: we could have trim the interface though to only keep the uses_malone attribute in that case? | 21:36 |
| lifeless | 13192 pushed (just deletes that patch) | 21:36 |
| jcsackett | flacoste: if i'm remembering correctly, yes. | 21:36 |
| bac | flacoste, jcsackett: i have the same recollection. they certainly look like they could be combined | 21:36 |
| stub | lifeless: Seems fine. Extra points if you switch to the truncate for probably unnoticable performance boost, and subclass the tests in test_bugsummary so they get run twice, once with rollup, once without. | 21:36 |
| flacoste | bac, jcsackett: anyway, just wondered when I saw both, thanks for the clarifications | 21:36 |
| lifeless | stub: s/subclass/parameterise :P | 21:36 |
| flacoste | might yak shave it at some point | 21:36 |
| jcsackett | flacoste: probably worth filing a bug against it. | 21:37 |
| * flacoste only files yak shaving bug when actually doing the shave... | 21:37 | |
| lifeless | stub: I would like to do that separately as a tech debt issue, because we've run them both with and without | 21:37 |
| jcsackett | flacoste: seems like a reasonable policy. :-) | 21:37 |
| lifeless | stub: I will change to the truncate right now | 21:37 |
| stub | I'd just put the rollup into a method, subclass, and override it with a noop. But whatever. | 21:37 |
| stub | cool. | 21:37 |
| lifeless | stub: well theres a bunch of small classes | 21:38 |
| stub | I thought all the tests were on one class? never mind then. | 21:38 |
| lifeless | stub: so all of them need to subclass too - testscenarios is designed to multiply it out | 21:38 |
| lifeless | ahm you are right | 21:38 |
| lifeless | they are | 21:38 |
| lifeless | so its easy and I'll do that | 21:38 |
| stub | So... lets see if I can get these patches applied live to staging. | 21:40 |
| stub | I guess qastaging first since it isn't replicated, then confirm I can add the table to the existing replication set on staging, then push the big red button | 21:40 |
| lifeless | those changes are done, testing | 21:41 |
| lifeless | \o/ big red buttons | 21:41 |
| stub | Yell when you have pushed so I don't get confused versions of stuff on things | 21:41 |
| * stub loves technical jargon like 'stuff' and 'things' | 21:42 | |
| lifeless | sure thing | 21:43 |
| stub | lifeless: production is blocked until the new replica is built, unless I kill it (which is a pita to clean up) | 21:44 |
| stub | but we can get staging etc. done | 21:44 |
| lifeless | so I believe there is a dsd script to run thats also blocked | 21:44 |
| === al-maisan is now known as almaisan-away | ||
| lifeless | perhaps we should kill it, run that scrpit, do this on prod, and then start the replica over | 21:45 |
| stub | yes, but it was declared non urgent | 21:45 |
| lifeless | right, it isn't urgent, but this is | 21:45 |
| lifeless | stub: and see -ops, the script may be urgent | 21:46 |
| lifeless | ok tests passed | 21:46 |
| lifeless | 13193 pushing | 21:46 |
| lifeless | statik: I think your cal update to weekly went awol :) | 21:46 |
| lifeless | statik: pushed | 21:46 |
| lifeless | bah stub: pushed | 21:47 |
| sinzui | jcsackett: do you have time to mumble about yui tests? | 22:08 |
| jcsackett | sinzui: sure. | 22:08 |
| jcsackett | i am in mumble now. | 22:09 |
| LPCIBot | Yippie, build fixed! | 22:16 |
| LPCIBot | Project db-devel build #624: FIXED in 5 hr 42 min: https://lpci.wedontsleep.org/job/db-devel/624/ | 22:16 |
| jcsackett | sinzui: you broke up on me. | 22:33 |
| LPCIBot | Project devel build #793: FAILURE in 5 hr 32 min: https://lpci.wedontsleep.org/job/devel/793/ | 22:40 |
| === Ursinha is now known as Ursula | ||
| === matsubara is now known as matsubara-afk | ||
| === Ursula is now known as Ursinha | ||
| poolie | hi all | 23:31 |
| lifeless | SpamapS: please try your script now | 23:42 |
| SpamapS | lifeless: good timing I delayed an accept just in case you fixed things. :) | 23:44 |
| lifeless | SpamapS: hows it looking ? | 23:52 |
| SpamapS | lifeless: unfortunately, the queue is full of stuff to reject today.. I haven't found something to accept just yet.. moment. | 23:52 |
| SpamapS | Ok running now | 23:57 |
| SpamapS | \o/ | 23:57 |
| SpamapS | lifeless: fixed. :) | 23:57 |
| lifeless | woot | 23:58 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!