[04:44] timeouts are looking goo [04:44] d === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === michaelh1|away is now known as michaelh1 === wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 237 - 0:[######=_]:256 [21:51] aieee http://pgbouncer.projects.postgresql.org/doc/faq.html#_how_to_upgrade_pgbouncer_without_dropping_connections [21:52] Impressive. [21:53] looks like a race still exists [21:54] Oh? [21:54] SUSPEND will stop it accepting connections. [21:55] SHOW FDS will transfer the sockets, including anything waiting on the listener. [21:58] yes, and somsone connecting between suspend and the new process binding will get a rejection [21:59] unless the listening socket is handed over [21:59] it might do that [21:59] That's what I assumed. [21:59] [which you said ... :P] [21:59] Otherwise it would be pretty useless. [22:00] And pretty flawed. [22:01] lifeless: Found any issues with batchnav/ [22:02] none [22:02] abel found that glitch by code inspection [22:02] Great. [22:02] Argh. [22:02] People hardcoding Ubuntu. [22:02] grar [22:02] url hacking can create it, but I don't think regular use will - at least not in any volume [22:02] wgrant: where? [22:03] http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13446 [22:03] The idea is flawed, and it hardcodes Ubuntu too :( [22:06] We really need to SOA Soyuz :( [22:06] a funding project for it is in the pipeline [22:06] Ah, excellent. [22:10] lifeless: Shall we qa-ok bug #809719, then? [22:10] <_mup_> Bug #809719: current lazr.batchnavigator version requires high OFFSET queries < https://launchpad.net/bugs/809719 > [22:10] what can possibly go wrong [22:10] [yes] [22:11] Now we need to check every page with an expander. [22:17] Hmm. [22:17] Bad, but very minor so far. [22:18] Expanded suggestions on +filebug don't have a height limit on the sprite, so the sprite below (the branch icon) leaks onto the page. [22:18] usable ? [22:18] Yes. [22:18] So I'm not going to qa-bad it yet. [22:18] I'm inclined to tolerate that w/ a regression bug [22:19] Yup. [22:20] Ah, it actually used an before. [22:28] Fail. [22:28] The regression fix did not fix the regression. [22:30] Oh. [22:30] It didn't actually land. [22:30] https://code.launchpad.net/~danilo/launchpad/fix-806925 [22:31] Looks like it was thrown at ec2, though :/ [22:31] lifeless: lp-land and pray? Or roll the whole lot back again? [22:32] We are almost up to 50 revs :/ [22:33] so the regression fix was landed separately to relanding what was rolled back ? [22:34] * lifeless wants to understand a little more what went on [22:34] Well, there were two revs doing the same sort of thing. I rolled back one, which turned out to be the wrong one, despite modifying the relevant file. It was relanded once the fix was written, but apparently the fix was not landed. [22:34] I assumed they landed together. [22:34] Only noticing now that it's not actually fixed. [22:37] so [22:37] we're 12 hours till danilo [22:37] 5 hours to find out what went wrong in ec2 [22:38] ~1 hour to fix, another 5 to bb [22:38] so 11 if we try to fix ourselves [22:38] vs 5 if we land a revert [22:39] We really need to fix those LXC hangs. [22:39] we can expect about 3 revs to land if we were to ec2 it and have it land ok [22:40] wgrant: the sudo thing? [22:40] lifeless: No, what I believe to be a reliable LibrarianLayer setup hang on one of the 6 workers. [22:41] I haven't seen that [22:41] Have you run the whole suite parallel? [22:41] anyhow, I'm +1 on a revert [22:41] wgrant: yes [22:41] HMm. [22:42] wgrant: other than the sudo thing, it seemed ok in terms of not hanging [22:42] I need more RAM. [22:42] wgrant: it helps; I have 16GB :P [22:49] * wgrant reverts. [23:27] wallyworld_: Morning [23:27] hello [23:27] wallyworld_: Any idea what creates that horrible broken red picker error box? [23:28] Ah, activator.renderFailure, it seems. [23:28] Kill. [23:28] i looked a little at some point in the past and sort of found some generic error handling stuff but that was while ago [23:29] activator.xxx sounds right [23:29] it's pretty ugly [23:29] at least it's out of lazr-js now and we can easily fix it :-) [23:29] It includes the error page's HTML in a

.
[23:29]  Yeah, pretty ugly.
[23:29]  reminds me, i need to delete the lazr-js eggs
[23:29]  I did that on Saturday.
[23:29]  Cleaned out download-cache.
[23:29]  oh cool
[23:30]  Except for a couple of things that are still needed by deployed code.
[23:32]  2 revs to go.
[23:34]  One of them seems to be designed to time out.
[23:34]  wgrant: i was just about to qa my bug, you beat me to it. did you login as a non-priviledged user to test. if so, is there a generic user that we use for such things on qas?
[23:34]  is there a setup.py setup() parameter for test only requirements ?
[23:35]  lifeless: Sort of. Check txlongpoll
[23:35]  lifeless: You declare it in extras_require.
[23:35]  lifeless: Not sure how to install them without buildout (where you say, eg., 'txlongpoll [test]').
[23:35]  wallyworld_: I have a few test SSO accounts. I don't know of any generic one.
[23:36]  ok. thanks
[23:36]  LOSA ping
[23:37]  Woah
[23:37]  impressive.
[23:37]  It doesn't time out.
[23:37]  It hung chromium.
[23:37]  But it didn't time out.
[23:37]  garh
[23:37]  hth does anyone use bootstrap.py
[23:37]  (getBranchTips with no since date, so it returned the tip for every Ubuntu branch)
[23:37]  Oh?
[23:37]  It is a terrible terrible thing, but it works.
[23:38]  :!./bootstrap.py
[23:38]  Downloading http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
[23:38]  While:
[23:38]    Initializing.
[23:38]  Ah, yes.
[23:38]  Error: The specified download cache:
[23:38]  '/home/robertc/source/launchpad/python-pgbouncer/working/download-cache'
[23:38]  Doesn't exist.
[23:38]  make
[23:38]  Most people use a global download-cache.
[23:38]  yes
[23:38]  But the makefile will create download-cache for you.
[23:38]  bringing up a new project
[23:38]  Length: 31270702 (30M) [application/json]
[23:38]  Saving to: 'ubuntu?ws.op=getBranchTips'
[23:39]  I am *really* impressed that it rendered.
[23:39]  me too
[23:39]  its a new page
[23:39]  Getting like 700KB/s, too.
[23:39]  it should have a 5 second hard timeout
[23:39]  More than I've ever got from the DC.
[23:39]  I geuss it comrpesses.
[23:39]  can you get such a timeout put on it please?
[23:39]  Not sure if we can.
[23:40]  its a api call right?
[23:40]  it will have a page id
[23:40]  Yes. I forget how pageids work there.
[23:40]  Oh, right, a third bit.
[23:40]  Distribution:EntryResource:getBranchTips or so?
[23:40]  * wgrant checks.
[23:40]  wgrant: hi
[23:40]  Distribution:EntryResource:#getBranchTips I think
[23:41]  hloeung: Sorry, thought qastaging had blown up, but it survived.
[23:41]  lifeless: Yeah.
[23:41]  lifeless: Is that your +1 for the flag change?
[23:41]  yes
[23:41]  you don't need one on qastaging
[23:41]  (a +1 that is)
[23:42]  hloeung: Could you please add the feature flag 'hard_timeout pageid:Distribution:EntryResource:#getBranchTips 5 5000' to qastaging?
[23:43]  wgrant: ok done
[23:43]  Oh, last_scanned_id is stored in branch. That's cheating.
[23:43]  hloeung: Thanks.
[23:44]  lifeless: Still renders after the second try!
[23:46]  neato
[23:48]  I bet it fails the '99% < 1 second' test, but reliably under 5s is sufficient to make me happy.
[23:48]  :!bin/buildout
[23:48]  Develop: '/home/robertc/source/launchpad/python-pgbouncer/working/.'
[23:48]  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'extra_requires'
[23:49]    warnings.warn(msg)
[23:49]  lifeless: extras_require
[23:49]  (yay)
[23:50]  so how do you trigger the test deps ?
[23:51]  With buildout you say 'eggs = some.package [test]'.
[23:51]  In the section that builds the testrunner.
[23:51]  Or, potentially, in the main section.
[23:52]  You can also do 'easy_install txlongpoll[test]', I believe.
[23:53]  Not sure if setup.py develop supports it.
[23:53]  [scripts]\neggs=pgbouncer[test]
[23:53]  seems to trigger it
[23:53]  Yeah.
[23:53]  now I need to find out why van.pg isn't findable
[23:53]  pgbouncer egg?
[23:53]  new fixture
[23:54]  What does it do that van.pg doesn't?
[23:54]  pgbouncer
[23:54]  Why do we want pgbouncer?
[23:54]  fastdowntime
[23:54]  less idle connections on db servers
[23:54]  No, why do we want it in the test suite?
[23:54]  fastdowntime behaviour checking
[23:55]  It's not sufficiently transparent?
[23:55]  So we must introduce a new layer of slowness, pain, death and destruction into the web of trouble that is our postgres fixture already? :(
[23:55]  no
[23:55]  we need 3-4 tests that use this
[23:55]  Ah
[23:56]  With a better name than python-pgbouncer, sounds good.
[23:57]  suggestions welcome; I already discounted pgbouncefixture
[23:57]  Why?
[23:57]  pgbouncer is a tad generic.
[23:58]  it is
[23:58]  but I can see us wanting a fixture, a config generator, and some glue logic to rewrite user configs and so forth
[23:59]  I have reservations around polluting the global namespace like this.
[23:59]  It's alright if it's under lazr, where it can only kill us...
[23:59]  I looked for existing modules