[00:05] sinzui: the +subscriptions mp is ready for rereview now - https://code.edge.launchpad.net/~brian-murray/launchpad/limited-subscriptions-page/+merge/35177 [00:14] I'm so behind on email. [00:15] jml: i suggest putting a small weight on your 'y' key [00:15] mwhudson: I much prefer '[' [00:16] mwhudson: even so, I've still got 50+ threads marked as "actually, I should read this" and another 20+ as "should answer" [00:16] ah ok, that's a harder problem [00:17] yes. [00:17] I think the only answer is to never concentrate ever again [00:18] (unless an email demands that I do so) [00:29] jml: Remove the label from all the emails. Important things that you should have paid attention to will either (a) get taken care of naturally or (b) spiral out of control and explode. [00:29] jml: It'll be easy to see which things need attention. :) [00:30] jml: I've been enjoying the labelling system you described to me... I find it really hard to be disciplined to actually do the "and get back to deal with labelled items" part, though. [00:31] Much of the time I procrastinate and then, when I get back to the items, they're no longer relevant. It's quite an effective technique. I'm very efficient. [00:36] lunch calls [00:46] buildd-manager seems broken. [00:47] Can someone check its logs? [00:47] Did the CP end up happening? [00:53] StevenK: ^^ [00:53] thumper: i have a deal for you [00:55] thumper: get someone from your team to review and sponsor jam's lp-serve branch, and then he can move on to commit to stacked branches [01:12] sinzui: Does the move of tales to lp.app imply that we're not going to split it? [01:12] no [01:12] Ah, good. [01:13] 25% belongs in lp.app, 50% registry [01:15] poolie: ok [01:15] poolie: deal [01:20] thumper, skype? [01:20] rockstar: sure, let me finish my roll first [01:25] rockstar: ok, now [01:31] sinzui: Are you sure that users should be granted access to see *everything*? Even embargoed security bugs? [01:32] wgrant, I am not sure [01:37] In the context of an already private project? [01:37] / distro [01:37] Right. [01:38] lifeless I think the issue can be generalised to all projects and distros. [01:38] If I grant view on a public project with private bugs and branches I assume most cases I mean to let the user see everything [01:39] but in the case of security (which is security contact, not bug supervisor) we may want to require subscription [02:47] thumper: ping [02:50] wallyworld__, I understand you're wanting to do some YUI stuff. [02:51] rockstar: yeah. do you want to take a peek at https://code.edge.launchpad.net/~wallyworld/launchpad/improved-broken-link-handling [02:51] and tell me i'm doing it all wrong [02:51] we can skype after you've had a look? [02:52] wallyworld__, okay, I'll take a look. I was actually going to say "we can skype tomorrow and chat about it" [02:53] rockstar: tomorrow is fine [02:53] wallyworld__, as it stands, this is my wife's night off this week, and if I don't watch Madmen with her, I will probably find the locks are changed the next time I leave the house. [02:53] rockstar: np. we can chat tomorrow. i forgot it was way past eod for you, sorry [02:54] wallyworld__, my EOD is kinda fuzzy some days. :) [02:54] rockstar: me too :-) i was up late last night figuring out some of the YUI stuff that i'm playing with now [02:55] hopefully i'm not totally on the wrong track. :-) [02:55] wallyworld__, I'm still trying to find the right track with YUI. :) [02:56] well, i just tried to see how other stuff had been done and go with the flow. anyway, have a good time watching madmen. talk tomorrow [02:56] mwhudson: Does Shotwell realise that it doesn't own your photos? [02:56] wgrant: what do you mean? [02:57] mwhudson: F-Spot likes to copy the photos into its local library and store all metadata somewhere other than the EXIF tags. [02:57] Which makes it impossible to use across multiple machines. [02:57] wgrant: i think it's better about the copying (when you import it asks if you want to copy them) [02:57] wgrant: dunno about tags [03:00] wallyworld__: hey [03:02] thumper: just wanted to skype at some point about the new work. when you are free [03:02] wallyworld__: I'm never free :) [03:02] cheap maybe, but not free [03:02] wallyworld__: let me get a small bit of work done [03:02] then we can talk [03:02] ack. i'll have lunch [03:07] #bshellz [03:08] ProgrammingError: operator does not exist: text = bytea [03:08] has anyone been fixing these on trunk yet? [03:08] no [03:08] thumper, bug 631010 [03:08] <_mup_> Bug #631010: ProgrammingError: operator does not exist: text = bytea [03:08] thumper, working on it right now [03:08] that reminds me to talk to doko or simila rabout it [03:08] is that the package downgrade hack? [03:08] yes [03:09] * thumper sighs [03:09] mars: if the fix is 'scatter u around the place' then its going to make the code base really ugly. [03:09] mars: so please don't do that ;) [03:09] lifeless, https://bugs.edge.launchpad.net/launchpad-foundations/+bug/631010/comments/4 [03:09] <_mup_> Bug #631010: ProgrammingError: operator does not exist: text = bytea [03:10] mars: do you mean to say you're putting the older version in the ppa ? [03:10] mars: if so cool, but can you please file a bug on maverick at the same time? [03:10] mars: do you have the downgrade command handy? [03:11] thumper, no, sorry, in the thread, look for "aptitude hold python-psycopg2", maybe "apt-get install python-psycopg2=2.0.13-2ubuntu2" [03:11] just a guess [03:13] lifeless, what do you mean "file a bug against maverick"? Like bug 585704? [03:13] <_mup_> Bug #585704: Storm test suite fails when using psycopg2 2.2 [03:13] no [03:13] I mean that psycopg2 in maverick is buggy [03:14] this change (presumably from upstream) is -not safe- to make in python 2.x modules. [03:14] python 2.x is hopeless confused about bytes vs text, and I can pretty much guarantee that every python webapp on pgsql will be fucked by maverick. [03:14] lifeless, may I suggest you file the bug then, if you know the implications and the people involved? [03:14] I don't know the people. [03:14] but sure [03:16] OTOH we could just fix LP. [03:17] sudo dpkg -i --force-downgrade python-psycopg2_2.0.13-2ubuntu2_amd64.deb [03:17] ah [03:17] that's what I did (for future reference) [03:17] wgrant: it will, I guarantee, bit us for months. [03:17] wgrant: its unbelievably hard to be 'correct' in this regard in python 2. [03:17] even if you just use the core built in things like 'Exception' [03:18] lifeless: It is hard, but Storm already forces us to be pretty much correct. [03:18] There are only a few places where we're not. [03:18] wgrant: really? [03:18] so does anyone know if simply copying the package from Lucid to Maverick in the LP PPA would work? [03:18] wgrant: so < 2 hours work? [03:18] lifeless: Storm will refuse str vs unicode mismatches. [03:18] lifeless: So it's only where we're using string-based SQL that there are problems. [03:18] wgrant: thats only one form of incorrectness [03:19] wgrant: if its so transparent, presumably storm can fix it transparently? [03:19] lifeless: Storm is pedantic in the way you say psycopg2 shouldn't be. [03:19] And has been since day 1. [03:19] wgrant, maybe you know if the package can simply be copied? [03:20] wgrant: then how can this issue exist? we pass 99% of everything through storm [03:20] lifeless: Pillar name retrievals don't go through Storm. [03:20] mars: if wgrant is correct just fixing LP would be better? [03:20] eg. getUtility(IDistributionSet)['ubuntu'] doesn't. [03:20] wgrant: they don't? [03:20] Because of the pillar name mess. [03:20] what does it go through ? [03:21] lifeless, I think henning looked at that, but he "stopped doing this when I hit the first decoding errors. :(" [03:21] He was putting u'' or unicode() around the query parameters in the test suite [03:21] lifeless: IPillarNameSet, which constructs parameterised SQL manually. [03:21] wgrant: but that still goes through storm [03:22] we ask storm to run it and storm asks psycopg to run it [03:22] Right. [03:22] so I repeat, why can't storm fix this? [03:22] does "fix released locking" mean that once a bug is fixreleased, it can't leave that state? [03:22] actually lets back off [03:22] poolie: I think so [03:22] Because Storm is even more pedantic than psycopg2. It would go against its normal policies to fix it. [03:22] wgrant: We have two competing theories. [03:22] wgrant: one is that its shallow. [03:23] wgrant: the other is that its complex and contains surprises. [03:23] mars: You should be able to get away with copying the Lucid *primary archive* version of psycopg2 to Maverick in the PPA. [03:23] wgrant: The data we have supporting it being shallow is nonexistent. [03:23] wgrant: its speculation based on the changes storm needed itself to work. [03:23] wgrant, ok, that would solve the Py2.5 dep problem by being for Py2.6. I'll give it a shot [03:24] I was able to fix lots of it just by hacking the celebrity descriptors to unicode() their names. [03:24] mars: We shouldn't have psycopg2 for Lucid in the PPA at all. [03:24] wgrant: the data we have suggesting it being complex is - decode errors being thrown when putting u before strings was attempted. [03:24] plus how freaking hard it was to get bzr roughly sane, and the testtools stack similarly. [03:25] wgrant: I humbly submit that you're being optimistic, unless you have more data you haven't presented. [03:25] The thing is that LP is already very close, because Storm and Zope force it to be. [03:25] poolie, I wouldn't be surprised. There must be a fair bit of thought behind a request like that. [03:25] wgrant: but they don't [03:25] Foo.name=='thing' <- clearly not 'correct' [03:25] Foo.name=u'thing' <- clearly ugly. Vive la Python tres. [03:26] Well, let's get Zope running on 3.2 :P [03:26] wgrant: and 'thing' works at the moment in lucid. [03:26] wgrant: whyever its not working in maverick is a regression in the stack. [03:26] in python 3 it would be correct, and historically in python 2 it is also correct. [03:27] Has anybody talked to the psycopg2 folks about this? [03:29] meh, mere facts. [03:29] https://bugs.edge.launchpad.net/ubuntu/+source/psycopg2/+bug/650777 [03:29] <_mup_> Bug #650777: operator does not exist: text = bytea

[03:31] http://comments.gmane.org/gmane.comp.python.storm/1256 [03:31] its also partly storm [03:31] treating str as a bytea [03:32] which is arguable either way as 'correct' in Python2 - but clearly isn't pleasant to interoperate with. [03:34] is there any systematic preference towards having one feature rules page that appears either editable or not, vs having a separate edit page to which only some people have access? [03:34] launchpad tends towards the latter but the former seems to have less dry [03:34] less repetition, i mean [03:37] Can't open average time db /var/debbuild/avg-build-times [03:37] Can't open average space db /var/debbuild/avg-build-space [03:37] mtaylor: That's normal. [03:46] wgrant: then why did this fail [03:46] wgrant: http://launchpadlibrarian.net/56710656/buildlog_ubuntu-lucid-amd64.drizzle_2010.09.1797-1.1~lucid01_BUILDING.txt.gz [03:47] That looks like an upload failure. [03:47] Can you link me to the build? [03:48] Ah, there. [03:48] wgrant: ah yes [03:48] -> http://launchpadlibrarian.net/56710652/57D507LpzoCnzjO9QXav5M64BhB.txt (duplicate key value violates unique constraint "binarypackagerelease_binarypackagename_key" [03:48] Yes. [03:48] Which means a bug has resurfaced. [03:48] Sigh. [03:48] Your build is fine; just ignore the failure. [03:49] It already finished once, then magically got illegally retried. [03:53] ok. cool [03:53] well, glad I could help find a bug :) [03:55] poolie: I don't care either way. [03:56] mtaylor: With any luck it will be fixed after tonight's cherrypick. [03:56] w00t [03:56] It replaces what I suspect to be the problematic code, for others reasons. [04:30] lifeless: Ping [04:32] hi [04:33] lifeless: Re: your mail about poppy, is there anything you need me to fix? [04:34] it was meant to educate [04:34] I knew you and jml collaborated on that stuff [04:34] and this got through your pair programming sessions. [04:35] lifeless: I'm happy for education and the chance to clean up after my mistakes [04:36] StevenK: my 'test' branch has fixes. [04:37] StevenK: I appreciate the offer to clean up, but there's no need to handoff here. [04:41] I'll be back around later [05:18] lifeless: Are you hacking layers today? [05:33] stub: lp:~lifeless/launchpad/test has layer stuff in it and is in ec2land atm [05:33] stub: got a request? [05:33] I didn't want to conflict. [05:34] if you check that you don't conflict with my branch, it will be fine. [05:34] what are you doing ? [05:34] I think I might have pushed too early - pqm is currently in process of landing by branch, and I pushed new revisions. Have I borked it or will it only merge revisions pushed when it started processing? [05:35] Making the thread check more robust - there is a codehosting test that occasionally fails, so I'm giving garbage threads a few seconds to shut down and gc.collect() to help them along [05:36] Uncommit and push overwrite before PQM finishes processing? [05:37] Doesn't the signed request include a rev ID? [05:38] No it doesn't [05:38] lifeless: ^^ [05:39] Well. That's a bit strange. [05:39] * stub tries the push overwrite before it is too late [05:39] bah - something else has it locked [05:39] it will only merge what it found when it started. [05:40] ok. So I can get this changed reviewed and check for conflicts :) [05:40] stub: any reason we can't just remove the thread check? what does it actually help us with? Or make it like the bzr one (reports at the end on tests that leaked) [05:41] lifeless: We can drop the thread check if we don't care about tests leaving threads lying around. I've mentioned that option when that has come up in the past, but people have always opted for fixing the source of the issue. [05:41] stub: I quite like what bzr does: [05:41] - gathers threads [05:41] - doesn't fail, but reports on the tests that leaked. [05:41] its a mix of both worlds. [05:42] I very much like fixing the source of issues, but a thread not being gone by the time a test finishes isn't prima facie an issue [05:42] bbs [05:42] A test can turn the fail into a warn if it wants, or we could set it globally by changing the flag in BaseLayer. [05:50] any guesses why my validate method is getting an empty 'data' dict? [05:52] i guess the field is not correctly seen as a widget [06:02] jtv i realized i can use validate() to give a better syntax error message [06:02] poolie: yes, that's a good place for it. [06:02] stub: so rather than adding multi second delays [06:02] stub: lets make it a warning [06:03] stub: globally [06:03] lifeless: Tests rarely hit this - we will be delaying maybe one in 10 test runs a few seconds [06:03] poolie: I see no excuse for the field not to be in the data dict… does the form use GET instead of POST or something? [06:03] I'd like to land it as is because the revision did get through PQM :-/ [06:04] stub: thats fine, but perhaps you could follow up with more stuff? [06:05] So the sleep is still valuable as it is 1) unreasonable to expect tests to wait on every thread spawed, particularly if they were not responsible for spawning it 2) sometimes threads can take a while to shut down. [06:05] Are both those points reasonable? [06:05] mmm [06:06] so we had a period in bzr where we tried a few different things [06:07] We can of course make the sleep less - maybe 100 checks at 0.1 second intervals rather than 10 at 1 second? [06:08] If it is just the thread didn't get allocated enough cpu to shutdown, it is likely sorted in milliseconds. [06:08] so, to cleanup threads, they *have* to be joined [06:09] jtv i've solved it now [06:09] poolie: what was it? [06:10] i had handcrafted html for the textarea and i should have let launchpad generate it [06:10] well, zope [06:10] then the field name would have matched what it expected [06:10] ah ah ah [06:10] So your data wasn't empty—you just weren't looking for the right key! [06:11] lifeless: I looked for a way to do that and couldn't find it. A second check now and it looks like I *can* iterate over the new threads and join them with a timeout. [06:12] stub: well, more importantly, I think this invalidates 1) [06:12] test code is either directly responsible for spawning the thread, or its using an api which must have a facility to join its own threads [06:13] Yes. I think we can make BaseLayer wait for the threads to terminate for a period of time. I think it is worth failing the test if they don't die, as it is a leak. [06:13] or its something like a thread pool which reasonably wants to stay active. [06:13] jtv, it was empty, but that's because zope code doesn't pass through unrecognized input parametres [06:14] my god it's a bit disgusting how many http requests there are per page in test mode [06:14] We have a flag to turn the fail into a warn. I think it is worth keeping it as a fail until we actually have threads that should remain active across threads. [06:14] i realize in production some things are rolled up into larger resources [06:14] if we want to catch incidental spawned threads that leak, we either need an API to say that some threads *are* expected, or we have to ensure that all tests clean up everytime [06:15] poolie: a lot of those will be for icing that isn't served by LP [06:15] setting up a thread pool is both reasonably cheap and also reasonably expressed as a fixture for sharing across tests. [06:15] so I think we can say 1) it is reasonable to require tests to join all threads they cause (in)directly to be spawned. [06:15] poolie: but it'd be great if we could reduce the requests and db queries we see on dev systems, yes [06:16] stub: with that assertion, 2) becomes irrelevant, [06:16] stub: what do you think? [06:17] Yes, 2 becomes irrelevant. [06:18] 1) is still relevant though, as threads could be spawned by third party libraries like bzrlib. [06:18] does my argument for revising (1) make sense to you? [06:18] No. [06:18] If bzrlib starts spawning a new thread where previously it didn't, it is unreasonable to update all the LP tests to check for this new thread to terminate. [06:19] or zope or psycopg or whatever [06:19] But a test fail is a test fail. [06:19] poolie: perhaps you can help me with the traceback you see near the end of this "make harness" session: https://pastebin.canonical.com/37820/ [06:19] stub: I mean, I agree, thats why I was suggesting warning only initially; [06:20] The layer will wait for the thread to terminate if it takes time (the spurious failures I'm trying to address), and if it doesn't terminate in a few seconds, then it fails. [06:20] poolie: that's on the staging db. Why the lp-internal URL scheme, and why does convert_path_to_url choke on it? [06:20] warnings just get ignored, and we are left with a test isolation leak [06:21] I think warnings that are very clear are less likely to be ignored [06:22] 'test foo leaked 4 tests' is pretty clear. [06:22] stub: waiting for e.g. a thread pool thread to terminate is likely to just timeout, only a few threads are ones that will naturally go away. [06:23] So what is the advantage of warning? I see absolutely none, except a potential problem has leaked into the codebase. Its not like we expect this to happen - we have never had a thread leak that wasn't a) spurious b) an error [06:23] stub: I don't think its a strong argument for the layer waiting at all given that being able to wait successfully is itself a special case. [06:23] (a or b) [06:23] jtv, this is just spm randomly typing stuff? [06:23] jtv, that transport is registered by the lp-serve plugin [06:24] sigh, openid grrr grr grr grr [06:24] so for some reason that's not loaded in this process [06:24] poolie: no, this is spm pasting a script I wrote up for this purpose into make harness. [06:24] either [06:24] 1- you didn't ask for plugins to be loaded in this interpreter [06:24] 2- that plugin wasn't found, probably because it wasn't on your path [06:24] 3- other [06:25] stub: so there are three approaches: a) wait and fail; b) warn [no point waiting if it can't fail], c) fail [make it the tests responsibility] [06:27] For b), we should wait. We don't want spurious warnings as people just filter them out as noise. [06:27] stub: I don't see the value of waiting [06:27] poolie: well I didn't ask for them to be loaded in this script, but it's not unthinkable that the branch scanner (which goes through similar steps) doesn't either. How do I ask for that? [06:28] stub: we've spent enough time on it - do what makes sense to you. [06:28] stub: I can say that I wouldn't wait :) [06:28] lifeless: warn and not wait is the solution everyone has rejected so far. [06:32] jtv bzrlib.plugin.load_plugins [06:32] jtv importing lp.codehosting seems to do that as a side effect [06:33] poolie: yup, see it in the __init__ [06:33] .py [06:33] So that's my script, not the branch scanner code [06:33] Thanks! [06:33] try running bzrlib.plugin.plugins() [06:34] poolie: locally that gives me a bunch, including lpserve. [06:34] k [06:34] perhaps the paths are being set up wrong so it's not loaded [06:35] poolie: well in that script I think I simply didn't do anything that loads it. Trying a corrected version now. [06:39] stub: sure, thats doesn't mean its wrong :P [06:39] stub: anyhow, as I say, do what makes sense to you [07:23] lifeless: What does the bad-commit-11566 tag do? [07:24] hi jtv? all ok? [07:24] hi poolie! No reply from spm yet [07:28] wgrant: https://dev.launchpad.net/QAProcessContinuousRollouts [07:33] StevenK: I would like an answer on my concurrent poppy question [07:34] lifeless: I've had a few stack dumps since then, can you please re-ask? [07:35] StevenK: look in the mailing list :) [07:36] lifeless: And no, we don't [07:37] lifeless: Both cocoplum and germanium run the FTP and SFTP servers [07:37] And I have no idea about how either of those services would cope or behave in a load-balancing situation [07:38] Morning ppl [07:39] StevenK: we don't what ? [07:40] lifeless: We don't run a single copy of both the FTP and SFTP servers. [07:42] for ppas, how many copies do we run ? [07:43] One, on germanium [07:43] But cocoplum's works perfectly well. [07:44] lifeless: Notice I'm not disagreeing it's a SPOF, I'm disagreeing that we only run one copy. [07:44] StevenK: I don't know what you're saying at all [07:44] StevenK: so i'm trying to pin it down [07:44] wgrant: will a ppa upload to *either* work for PPAs' ? [07:44] lifeless: Yes. [07:45] so all we hve to do is shove ha proxy in front and make all the dns names point at the same place? [07:45] That will then cause havoc [07:45] I'd prefer to resolve a germanium SPOF in a way that doesn't involve cocoplum. [07:45] what havoc? [07:45] Since people download and upload from ppa.launchpad.net [07:45] let me explain what I need. [07:45] I need: [07:45] lifeless: Does haproxy do FTP and SFTP? [07:46] - to know whether the software is safe to run concurrently for a given 'queue' (or whatever abstraction it uses - sounds like it is) [07:46] poppy can safely run concurrently, yes. [07:46] - to provide a detailed request for config changes to the losas to eliminate the current downtime during upgrades. [07:46] It's not safe to run multiple upload processors, but we do anyway. [07:46] upload processors are less concerning [07:47] because its not listening on the network [07:47] Right. [07:47] so [07:47] But we can't safely run multiple upload processors from different hosts. [07:48] so, which machine is the ppa one [07:48] germanium [07:48] and what is cocoplum known as? [07:48] cocoplum is the other instance -- it is ftpmaster. [07:48] does ftpmaster provide an HTTPS interface? [07:48] and HTTP ? [07:48] No [07:48] Not externally. [07:48] so, how does this sound. [07:48] HTTP only, and only internally [07:49] is that http apache? [07:49] Yes [07:49] here is what I propose [07:49] ha proxy with ppa and ftpmaster pointing at it [07:50] ha proxy directs http requests to germanium [07:50] ha proxy or germanium's apache sends ftpmaster internal requests to cocoplum [07:50] ha proxy directs ftp and sftp to either machine depending on whats running [07:50] This sounds like two SPOFs. [07:51] Also, will FTP be much of a fan of being run through haproxy? [07:51] wgrant: SEP [07:51] Or SFTP, for that matter [07:51] SFTP should be fine. [07:51] It's a single TCP stream. [07:51] so, elmo has said there is a tool (it might not be haproxy) for doing this stuff. [07:51] I'm worried about host keys [07:51] if there isn't we'll have to go find or write one. [07:52] StevenK: ok, thats a good point. We can do one of two things. [07:52] StevenK: They'd have to be the same on both. [07:52] Easy enough. [07:52] we can say 'these should really have been one service', announce and JFDI it. [07:52] I'm also worried about user-confusion [07:53] lifeless: Why should ftpmaster.internal requests go through germanium? [07:53] or we can say 'these are meant to be different', and we then bring up new instances of these services on other machines and tell the twisted daemon to use an appropriate host key. [07:53] StevenK: getting rid of downtime can go a long way :) [07:53] note that openssh host keys are not involved here [07:53] because its twisted doing ssh [07:53] lifeless: The two hostnames used for FTP and SFTP are upload.ubuntu.com and ppa.launchpad.net. [07:54] wgrant: so what are the two spofs you see? [07:54] wgrant: I'm suggesting only that ftpmaster things on the hostname that poppy is on get redirected. [07:54] wgrant: if there is no http/https on 'upload.ubuntu.com' then its irrelevant [07:55] lifeless: ftpmaster.internal should still point at cocoplum. upload.ubuntu.com and ppa.launchpad.net will point at haproxy. [07:55] yes [07:55] haproxy forwards HTTP(S) to germanium, and sends ftp/sftp to germanium or cocoplum. [07:55] yes [07:56]