[00:43] desperately seeking susan^Wreviewer [00:54] checkwatches.base needs expurgation of its oops stuff. [01:06] wgrant: StevenK: I need a hand, make jsbuild is failing, and I have no idea why ;) [01:06] I'm guessing one of you ran into that recently [01:06] wallyworld: or ^ [01:07] lifeless: what's the issue? [01:07] no rule to make target 'lib/canonical/launchpad/icing/yui/yui/yui.js' needed by .../launchpad.js [01:08] hmmm. haven't seen anything like that in a while. last time that sort of thing happened, a make clean got it going again [01:08] trying that, thanks [01:08] it sounds like the yui symlinks got messed up [01:08] mwhudson: can we nuke vostok-archive ? [01:09] lifeless: yeah, i reckon so [01:09] nuke all of vostok if it's getting in the way any i guess [01:09] wallyworld: thank, that works (which means we have a bug in our dep rules) [01:09] mwhudson: it just seems stubby to me [01:10] mwhudson: as in, an abandoned experiment [01:10] yep [01:10] (hey at least i managed to clean up the publisher code in the rest of lp a bit when i added it ...) [01:10] \o/ [01:11] and I'm back to nuking getLastOops calls [01:11] what are those calls being replaced with? [01:11] self.oopses[-1] [01:11] ok [01:12] or direct subscription in doctests [01:12] why ? [01:12] I mean, if you're hacking in that area, we can collaborate [01:13] I will broadcast to the list once I've sorted it all out [01:13] i was just curious [01:13] kk :) [01:13] so getLastOopsReport is not isolated between tests [01:13] so test A can write an oops test B sees [01:13] and we've had this happening [01:14] its also not threadsafe [01:14] (test thread A can write a report test thread B thinks is its) [01:14] yeah, i recall some issues from a while ago in this area, can't remember the details [01:48] hey, don't know if someone reported this already, but it seems launchpad is blocking new packages to be both published and built today [01:48] https://launchpad.net/~linaro-maintainers/+archive/overlay/+packages [01:49] I copied a few packages from other series today, and also pushed new ones, but they are all waiting for hours already [01:49] Yes, we're debugging an issue with the PPA publisher [01:50] StevenK: ok, great [01:50] yeah, the issue is just at the publisher, just saw that the packages I pushed today were all built fine [01:50] but they're now all locked at the publisher [01:55] hah [01:55] mailman doc tests are not being run [01:56] I thought that was by design? [01:57] for our monkey patches? no [02:21] wow [02:21] test_uploadProcessor is full of pain [02:41] lifeless: i'm having a weird feature flag issue that has got me stumped, can you spare a minute to help a poor lost soul? [02:42] sure [02:42] if i uncomment the commented out line, it works: https://pastebin.canonical.com/54429/ [02:43] so something is messing with the feature flag infrastructure [02:43] to add a bit of content - i am expecting delete_allowed to be true [02:44] and whats it set to in the test ? [02:44] flags = {u"disclosure.delete_bugtask.enabled": u"on"} [02:44] there are other tests which all work [02:44] but for this test, the flag cannot be found [02:45] the rule :) [02:45] the flag is always found, but may evaluation to None [02:45] what does getFeatureFlag( [02:45] 'disclosure.delete_bugtask.enabled') [02:45] evaluate to ? [02:45] oh, and are you sure its reaching your code? [02:45] I assume the authorization cache is being populated by the owner check. [02:45] none i think, i'll check [02:45] zope security caches [02:46] * lifeless bites back a biting commentary on using caches to solve architectural problems [02:46] (and yes, I'm aware of CPU layer-N caches) [02:47] here's a test which works for example: i have a security adaptor and it is getting to that point ok [02:47] oops [02:47] paste error [02:48] https://pastebin.canonical.com/54430/ [02:48] and to answer your question, getFeatureFlag('xxxx') evaluates to None [02:48] I didn't think flags worked inside the security adapter. [02:49] inside the security adaptor, unless i uncomment that line [02:49] StevenK: they appear to work, at least for the tests i have which pass :-) [02:49] StevenK: No reason they shouldn't. [02:49] Apart from caching issues like this :) [02:49] We'll see if they're relevant soon... [02:50] not sure if it's a caching issue per se [02:50] It very probably is. [02:50] wallyworld: I'd like to see the following: a print / pdb session in the security adapter showing the feature controller and the evaluation of the flag, and if using print, prints before and after the call so we can eliminate other entries into the codepath [02:51] wgrant: Where is your obsolete-distroseries fix at? [02:51] StevenK: I should finish that off today, good point. [02:51] lifeless: any particular attributes of the feature controller? [02:52] nope [02:53] * wallyworld starts gather data [03:02] mwhudson: *cough* beautifulsoup on codeimport creation forms. [03:02] wgrant: heh heh heh [03:02] is that still there? [03:02] I dare not check. [03:02] lib/lp/code/browser/codeimport.py:from BeautifulSoup import BeautifulSoup [03:02] lib/lp/code/browser/codeimport.py: soup = BeautifulSoup(self.widgets['rcs_type']()) [03:02] Yes [03:02] wgrant: beatifulsoup > re.compile(r'(?<=class=["\'])(.*)(?=["\'])') though [03:03] True. [03:03] wgrant: i think the beautifulsoup thing falls into my bucket of "automatic form generation is a crock of **** and you shouldn't use it ever" [03:09] mwhudson: Somewhat like ORMs that lazy-load, automatic form generation makes small things easy but inevitably screws you over completely in expensive ways. [03:11] lifeless: here's some printed data. the act of putting in the code to print the data made the test pass. https://pastebin.canonical.com/54431/ [03:13] 15:52 < lifeless> nope [03:13] 15:53 * wallyworld starts gather data [03:13] before my adsl stopped [03:14] lifeless: here's some printed data. the act of putting in the code to print the data made the test pass. https://pastebin.canonical.com/54431/ [03:14] * StevenK starts a fund to buy lifeless some better Internets [03:14] take out a hit on telecom [03:14] Haha [03:14] the first printout is just before the check_permission call [03:15] wallyworld: I wanted the controller itself :) [03:15] ah [03:15] wallyworld: so I could see if it was falling back to a different object [03:15] * wallyworld tries again [03:15] e.g. due to the participation / interaction being futzed or something weird [03:15] lifeless: My DSL has been connected since the end of Aug [03:15] interesting data point that the debug fixed it [03:15] So 45 days or so [03:16] lifeless: I don't recall you having issues when you were in Epping [03:17] StevenK: indeed [03:17] StevenK: thus, telecom. [03:17] I will ring and rant soon [03:18] Do you hold out much hope? [03:19] lifeless: before and inside the call, the feature controller is [03:19] it fails without all the other print statements [03:20] i'll see if i can find which print statement makes it work [03:22] StevenK: if we can id the issue, yes [03:22] thats mainly dependent on getting through first level 'technical' support [03:23] Oh, absolutely [03:24] which the rant is all about [03:25] lifeless: so calling features.getAllFlags() before the check_permission call makes it work (as well as bug.default_bugtask) [03:25] and not calling either of those 2 things makes it fail [03:25] jtv: There's a regression fix for cocoplum that I'd like to deploy tonight, and it's stuck behind your translations-export-to-branch fix. Are you likely to have QA for that done in the next 4 or so hours? [03:26] wgrant: I think I will, but it won't be much sooner. [03:27] OK. I may have to cowboy it anyway, since there's an existing possibly unclobberable cowboy there. [03:32] lifeless: narrowed it down to feature_controller.rule_source.getAllRulesAsDict() - so it seems the StormFeatureRuleSource() content is getting clobbered somehow? [03:35] I wonder if you can't trigger the first feature rule lookup from within a security adapter because they don't nest? [speculation] [03:38] not sure [03:38] but it all seems rather fragile at the moment [03:39] wgrant: what do you think about my speculation ? [03:39] lifeless: What don't nest? [03:40] wgrant: I'm trying to come up with a story that would explain wallyworlds symptoms [03:41] weird thing is that bug.default_bugtask also makes it work [03:41] you probably need to step through with pdb [03:42] yeah, started to do that. lots of api calls to look at [03:42] zope is phat [03:42] at least it's not something dumb i'm doing wrong (hopefully) [03:43] seems like a genuine problem with the infrastructure [03:46] lifeless: Are you sure the mailman test bug is actually a bug? We deliberately don't run MailmanLayer by default, because it's crap. [03:50] wgrant: we have tests that exist; they should be run, or not exist. [03:51] wgrant: otherwise they -will- bitrot and -will- just accumulate debt [03:51] Delete them, then [03:51] I found a bug that they were not being run, but it was fix released years ago indicating they were meant to run again. [03:51] StevenK: not without chatting to curtis I think [03:52] I wish I had more knowledge so we could delete lib/mailman [04:08] s/I/LP/ s/more knowledge/some architecture/ [04:08] Harsh [04:10] 6 uses of getLastOops [04:17] mwhudson: hey, around ? [04:17] mwhudson: have a quickie on codehosting [04:18] lifeless: yep [04:18] mwhudson: make_error_utility appends the pid to the oops prefix [04:18] Haahahah [04:18] is this for any reason *other* than oops sucking at concurrency [04:18] I think we established that there isn't. [04:18] lifeless: no, i'm pretty sure that's only to avoid races [04:18] It's there to avoid concurrency issues, and to confuse the shit out of everyone. [04:18] mwhudson: as the processes are ephemeral, I'm checking theres not need to keep that [04:18] mwhudson: great, deleted. [04:19] wgrant: if you want confusing [04:19] grep for setOopsToken and note the EMAIL usage [04:19] mwhudson: I'm assuming pullerworker is the same ? [04:19] Huh. Handy. [04:19] lifeless: yes [04:20] wgrant: do you happen to know if thats semantic or just crazy [04:20] lifeless: I assume crazy. [04:20] But don't know for sure. [04:21] Delete it and see if matsubara complains? :) [04:25] aieee @ available_oops_prefixes [04:25] Delete [04:27] oh man [04:27] I hope its not punning that with concurrency limiting [04:36] lol [04:38] lifeless: I think you may have projectegg set badly in python-oops-tools [04:38] It currently tries to use oops-tools.settings as the settings module. [04:38] Which obviously isn't going to work :) [04:44] lifeless: i've found the problem. the setAllRules() method on StormFeatureRuleSource needs a store.flush(). Or else the rules passed into the fixture setup are not written to tge db because check_permission() has a @block_implicit_flushes decorator [04:44] Hahaha [04:45] and those other things which trigger the test tp pass must have done so because they caused a flush [04:45] wgrant: thats fixed by my branch [04:45] wgrant: you're welcome to review it if you want [04:48] omg === wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 262 [04:49] wgrant: Hm? [04:49] The number went down :) [04:49] Significantly. [04:49] By 7 [04:49] It was actually 272 this morning. [04:50] Oh, so 10 [04:55] 26 days to go!? [04:56] Heh [04:57] mwhudson: You tell funny jokes [04:57] I think I have to file one, anyway [04:58] I can't see DSP:+questions in our bugs [05:01] wgrant: is it the ppa publisher that's backed up? [05:01] The queries are quick, so it's unlikely to time out much. [05:01] mwhudson: Was, but yes. [05:01] mwhudson: It's been fixed for a few hours. [05:02] And now even has scriptmonitor running on it. [05:02] wgrant: ok, how often does it run when it's not backed up? [05:02] Every 5 minutes. [05:02] But often only every 10. [05:02] Because it's crap. [05:02] * mwhudson has a vague memory of */20 [05:02] hah [05:02] ok [05:02] It was */20 long ago [05:36] wgrant: you've been spelunking a lot; whats the fastest way to tell if script X has an oops config === almaisan-away is now known as al-maisan [06:02] wgrant: you've been spelunking a lot; whats the fastest way to tell if script X has an oops config [06:03] lifeless: Somewhat disturbingly, despite porting dozens of scripts around to LaunchpadScript and rewriting its internals, I've not run into that bit of code. [06:03] I want to check the setOopsToken('EMAIL') thing is safe when gone, if you see what i mean [06:05] Oh, that's lovely. [06:05] Scripts normally just call globalErrorUtility.configure('something') themselves. [06:12] +214 [06:12] -687 [06:12] 1874 lines of diff [06:12] and we're not done yet [06:12] + it would be freakishly hard to make this separate branches [06:13] I pity the fool^Wreviewer [06:14] StevenK: oops -> critical [06:14] StevenK: also, we don't use confirmed :) === al-maisan is now known as almaisan-away [06:50] poolie: hi, can we talk about your pending writes branch briefly ? [06:50] sure! [06:50] here, or phone? [06:50] either, whats your pref ? [06:51] let's start here [06:51] so, the bug (as I read it) is that when someone pushes twice in quick succession, we don't update the merge diff properly [06:51] there are a few different orders to the race condition [06:51] sometimes we generate the error and update proplerl [06:51] i think that's how you reach it yes [06:51] y [06:52] yes it seems so [06:52] sometimes we generate the error and don't update properly [06:52] that may be possible [06:52] I'm worried that you're papering over the issue [06:52] i can understand that concern [06:52] however [06:52] i think there are really two bugs [06:53] 1- "sometimes mp diffs are not generated if the branch is repeatedly written to" [06:53] 2- "launchpad sends pointless spam" [06:53] i'm trying to fix 2 [06:53] i'm not sure if 1 actually exists [06:53] I'm sure it does [06:53] per the analysis in comment #2 [06:55] i thought that perhaps the completion of the second write would cause a new job to be generated [06:55] perhaps there is some ordering where that doesn't happen [06:55] can only have one job outstanding for the branch [06:55] at any rate i don't see how leaving bug 2 open helps us fix bug 1 [06:55] at the moment we don't even log when this occurs! [06:55] so if the first job hasn't finished erroring before the second job is created, the second job isn't made and the first job just fails. [06:56] poolie: we don't generate an OOPS ? [06:56] no [06:56] ok [06:56] it is telling only the users who can't do anything about it [06:56] unless the idea is to annoy them (me) into fixing the whole bug :) [06:56] which is a valid, though risky, strategy [06:57] so, I think bug 1, which is the bug your branch purports to be about, is about fixing the race condition [06:57] <_mup_> Bug #1: Microsoft has a majority market share bah, 1- [06:57] :) [06:57] my mp is only about suppressing the mail [06:57] i get annoyed by the mail but i never see a missing diff [06:57] And also, issue 1- is the only one where the user has no control over the situation [06:57] because they can delete/filter the mail? [06:58] poolie: because they can push content into the branch, or delete the mp if they had done something crazy [06:58] I admire your desire to stop sending spam, but I don't think, except for case 1-, that these branch mails -are- spam [06:58] and case 1- has an analysis of the race condition, just needs coding [06:58] lots of people seem to disagree :) [06:59] poolie: not on that bug [06:59] :( [06:59] poolie: in general, 'lp sends too much mail', sure : but telling you something you requested fails is useful [06:59] it doesn't tell you what failed [06:59] as james said "I don't really know what it means, which merge proposal it is referring to, or what [06:59] I can do about it, so I don't know why I got an email about it." [07:00] lp really should not be sending that [07:00] it's different to bugmail [07:00] so, the other bug, which I've unduped, is about the lack of context [07:00] which one? [07:00] fixing that will address some of james_w's mystery around the mail [07:00] bug 640882 [07:00] <_mup_> Bug #640882: " Launchpad error while generating the diff for a merge proposal" mails don't indicate branch

< https://launchpad.net/bugs/640882 > [07:00] ok [07:00] i'll just drop it [07:01] i'm sad because i was trying to make this a little less crap and i feel like it's being held hostage to fixing the whole thing [07:01] not sending pointless mail to people is a step forward [07:01] recording when something goes wrong is a step forward [07:01] I'd love to see improvements here, I don't think masking the issue is one; fixing the issue (which should be ~ as simple as self.suspend(5 minutes) or something) would be [07:01] poolie: I agree that not sending pointless mail is a step forward; and recording when it goes wrong is a step forward. [07:01] how is this masking it? [07:02] poolie: My understanding was that you were going to squelch the email for this case, and that that was the sum of the branch [07:02] and i was going to log that it failed [07:03] ok, if just doing self.suspend(5 minutes) is enough, i'll try that [07:03] I'm handwaving [07:03] Hi [07:04] poolie: jobs have a defer-for-a-bit system, I don't know the details. [07:04] i don't feel you and aaron are taking into account the actual user data here [07:04] poolie: I think for the pending-writes case, logging and not emailing is fine; I agree with Aaron that the other cases are different enough not to change. [07:04] nobody is saying "i'm glad lp told me about this" or "that explains why my thing had no diff" [07:04] poolie: I think fixing the issue, logging and not emailing is even better [07:05] I feel like you are saying 'not getting email is more important than the system working' [07:05] I know thats not what you mean [07:05] but it kindof feels that way [07:05] mm === jam1 is now known as jam [07:06] I think you mean 'not sending email in this case is better even if its not fixed', and I've acked that - twice I think - above [07:06] pending writes shouldn't be categorised as a user error [07:06] mm [07:06] i get more annoyance from lp spam than i do from diffs being missing [07:07] in that sense it's more important [07:07] and, generally, there are always going to be some errors, and i think handling them gracefully is important [07:07] I get annoyance from devs having to spend time tracking down, *again*, a self inflicted case of user confusion [07:07] :) [07:07] why 'self inflicted'? [07:07] (self inflicted by us developers) [07:08] oh i see [07:08] poolie: because we created a system with a race condition, classified it as user error, and tada [07:08] yep [07:08] and, i think, did not look at the actual mail that was sent [07:08] this needs two changes: unclassify it as user error, and fix the race condition [07:08] yep [07:08] and yes, the lack of context in the mail is the icing on the cake [07:08] if the race is as simple as just rescheduling the job i can do that [07:09] I think that if the branch has pending writes, the job should just wait for it [07:09] indefinitely [07:10] i guess the 'lack of context' bug can then apply to other mail sent about branches, if any [07:10] poolie: there are, IIRC, 3 other cases for MP's where the same template is used for the email [07:10] i agree, though i think the "users need to trust whether lp is working" argument applies equally there [07:10] poolie: cases which this bugfix won't impact [07:10] poolie: well, pushing up an empty branch and proposing it for merge, *is* a user error [07:11] if you get one of those mails, I think its helpful (if it told you the branch :)) [07:11] yes, bug 640882 will be irrelevant to the specific case it complains about, but relevant to things like empty branches [07:11] <_mup_> Bug #640882: " Launchpad error while generating the diff for a merge proposal" mails don't indicate branch

< https://launchpad.net/bugs/640882 > [07:11] ok [07:12] so [07:12] this would have been a lot easier if someone had just said "why don't you just call self.suspend and that will probably fix it" [07:12] in the first place [07:13] poolie: that would have been nice [07:13] understand I'm handwaving, bug there is something like that there :) [07:13] and I'll be happy (tomorrow) to go spelunking with you looking for it [07:14] it's probably fairly obvious on the base class [07:14] so then no oops, just deferral [07:14] and maybe a log message [07:15] yah [07:21] wgrant: Q/A for those codehosting translations-export bugs is done. Go ahead. [07:21] jtv: Thanks. [07:22] lifeless: most of the bugs have all the issues of "no context" and "shouldn't get this mail anyhow" tangled together [07:22] please don't undupe them all [07:22] poolie: there were two that were previously a unit, and you'd moved to the other bug, I was just restoring the, [07:25] poolie: (i.e. I've no more tweaking planned on these bugs) [07:26] lifeless: The bug I filed is not the cause of the OOPS -- that is already filed. [07:26] lifeless: I used High since the bug is *shown* in the OOPS, but isn't the cause. [07:26] StevenK: ah, that wasn't clear to me. Sorry for creating noise. [07:26] lifeless: Should I set it back to High, then? [07:27] StevenK: up to you; lazy evaluation and timeouts can be nonobvious - we may well have timeouts due to that bug anyhow [07:27] (it is a timeout isn't it ?) [07:27] lifeless: Yes, but the timeout is due to the direction=backward madness [07:28] ah right, a clear cause :) [07:28] jtv: Bug 375013 is not marked OK, but 812500 is [07:28] <_mup_> Bug #375013: Cannot commit directly to a stacked branch

< https://launchpad.net/bugs/375013 > [07:28] StevenK: I guess one of my changes didn't come through. Hang on. [07:29] lifeless: i think the other thing here is https://bugs.launchpad.net/launchpad/+bug/483945 [07:29] <_mup_> Bug #483945: No way to ask Launchpad to refresh a stale diff

< https://launchpad.net/bugs/483945 > [07:29] to give people a way tor ecover [07:29] poolie: that would be nice [07:29] StevenK: actually, it did come through. So the deployment report simply hasn't picked it up yet. [07:30] StevenK: the one that's not marked OK yet is the one I updated last, IIRC. Here's hoping this is not a problem with multiple bugtasks. [07:53] Can I get a review? https://code.launchpad.net/~stevenk/launchpad/dsp-questions-statement-death/+merge/79519 [08:04]