lifelessdesperately seeking susan^Wreviewer00:43
lifelesscheckwatches.base needs expurgation of its oops stuff.00:54
lifelesswgrant: StevenK: I need a hand, make jsbuild is failing, and I have no idea why ;)01:06
lifelessI'm guessing one of you ran into that recently01:06
lifelesswallyworld: or ^01:06
wallyworldlifeless: what's the issue?01:07
lifelessno rule to make target 'lib/canonical/launchpad/icing/yui/yui/yui.js' needed by .../launchpad.js01:07
wallyworldhmmm. haven't seen anything like that in a while. last time that sort of thing happened, a make clean got it going again01:08
lifelesstrying that, thanks01:08
wallyworldit sounds like the yui symlinks got messed up01:08
lifelessmwhudson: can we nuke vostok-archive ?01:08
mwhudsonlifeless: yeah, i reckon so01:09
mwhudsonnuke all of vostok if it's getting in the way any i guess01:09
lifelesswallyworld: thank, that works (which means we have a bug in our dep rules)01:09
lifelessmwhudson: it just seems stubby to me01:09
lifelessmwhudson: as in, an abandoned experiment01:10
mwhudson(hey at least i managed to clean up the publisher code in the rest of lp a bit when i added it ...)01:10
lifelessand I'm back to nuking getLastOops calls01:11
wallyworldwhat are those calls being replaced with?01:11
lifelessor direct subscription in doctests01:12
lifelesswhy ?01:12
lifelessI mean, if you're hacking in that area, we can collaborate01:12
lifelessI will broadcast to the list once I've sorted it all out01:13
wallyworldi was just curious01:13
lifelesskk :)01:13
lifelessso getLastOopsReport is not isolated between tests01:13
lifelessso test A can write an oops test B sees01:13
lifelessand we've had this happening01:13
lifelessits also not threadsafe01:14
lifeless(test thread A can write a report test thread B thinks is its)01:14
wallyworldyeah, i recall some issues from a while ago in this area, can't remember the details01:14
rsalvetihey, don't know if someone reported this already, but it seems launchpad is blocking new packages to be both published and built today01:48
rsalvetiI copied a few packages from other series today, and also pushed new ones, but they are all waiting for hours already01:49
StevenKYes, we're debugging an issue with the PPA publisher01:49
rsalvetiStevenK: ok, great01:50
rsalvetiyeah, the issue is just at the publisher, just saw that the packages I pushed today were all built fine01:50
rsalvetibut they're now all locked at the publisher01:50
lifelessmailman doc tests are not being run01:55
StevenKI thought that was by design?01:56
lifelessfor our monkey patches? no01:57
lifelesstest_uploadProcessor is full of pain02:21
wallyworldlifeless: i'm having a weird feature flag issue that has got me stumped, can you spare a minute to help a poor lost soul?02:41
wallyworldif i uncomment the commented out line, it works: https://pastebin.canonical.com/54429/02:42
wallyworldso something is messing with the feature flag infrastructure02:43
wallyworldto add a bit of content - i am expecting delete_allowed to be true02:43
lifelessand whats it set to in the test ?02:44
wallyworldflags = {u"disclosure.delete_bugtask.enabled": u"on"}02:44
wallyworldthere are other tests which all work02:44
wallyworldbut for this test, the flag cannot be found02:44
lifelessthe rule :)02:45
lifelessthe flag is always found, but may evaluation to None02:45
lifelesswhat does getFeatureFlag(02:45
lifeless            'disclosure.delete_bugtask.enabled')02:45
lifelessevaluate to ?02:45
lifelessoh, and are you sure its reaching your code?02:45
wgrantI assume the authorization cache is being populated by the owner check.02:45
wallyworldnone i think, i'll check02:45
lifelesszope security caches02:45
* lifeless bites back a biting commentary on using caches to solve architectural problems02:46
lifeless(and yes, I'm aware of CPU layer-N caches)02:46
wallyworldhere's a test which works for example: i have a security adaptor and it is getting to that point ok02:47
wallyworldpaste error02:47
wallyworldand to answer your question, getFeatureFlag('xxxx') evaluates to None02:48
StevenKI didn't think flags worked inside the security adapter.02:48
wallyworldinside the security adaptor, unless i uncomment that line02:49
wallyworldStevenK: they appear to work, at least for the tests i have which pass :-)02:49
wgrantStevenK: No reason they shouldn't.02:49
wgrantApart from caching issues like this :)02:49
wgrantWe'll see if they're relevant soon...02:49
wallyworldnot sure if it's a caching issue per se02:50
wgrantIt very probably is.02:50
lifelesswallyworld: I'd like to see the following: a print / pdb session in the security adapter showing the feature controller and the evaluation of the flag, and if using print, prints before and after the call so we can eliminate other entries into the codepath02:50
StevenKwgrant: Where is your obsolete-distroseries fix at?02:51
wgrantStevenK: I should finish that off today, good point.02:51
wallyworldlifeless: any particular attributes of the feature controller?02:51
* wallyworld starts gather data02:53
wgrantmwhudson: *cough* beautifulsoup on codeimport creation forms.03:02
mwhudsonwgrant: heh heh heh03:02
mwhudsonis that still there?03:02
wgrantI dare not check.03:02
wgrantlib/lp/code/browser/codeimport.py:from BeautifulSoup import BeautifulSoup03:02
wgrantlib/lp/code/browser/codeimport.py:        soup = BeautifulSoup(self.widgets['rcs_type']())03:02
mwhudsonwgrant: beatifulsoup > re.compile(r'(?<=class=["\'])(.*)(?=["\'])') though03:02
mwhudsonwgrant: i think the beautifulsoup thing falls into my bucket of "automatic form generation is a crock of **** and you shouldn't use it ever"03:03
wgrantmwhudson: Somewhat like ORMs that lazy-load, automatic form generation makes small things easy but inevitably screws you over completely in expensive ways.03:09
wallyworldlifeless: here's some printed data. the act of putting in the code to print the data made the test pass. https://pastebin.canonical.com/54431/03:11
lifeless15:52 < lifeless> nope03:13
lifeless15:53  * wallyworld starts gather data03:13
lifelessbefore my adsl stopped03:13
wallyworldlifeless: here's some printed data. the act of putting in the code to print the data made the test pass. https://pastebin.canonical.com/54431/03:14
* StevenK starts a fund to buy lifeless some better Internets03:14
lifelesstake out a hit on telecom03:14
wallyworldthe first printout is just before the check_permission call03:14
lifelesswallyworld: I wanted the controller itself :)03:15
lifelesswallyworld: so I could see if it was falling back to a different object03:15
* wallyworld tries again03:15
lifelesse.g. due to the participation / interaction being futzed or something weird03:15
StevenKlifeless: My DSL has been connected since the end of Aug03:15
lifelessinteresting data point that the debug fixed it03:15
StevenKSo 45 days or so03:15
StevenKlifeless: I don't recall you having issues when you were in Epping03:16
lifelessStevenK: indeed03:17
lifelessStevenK: thus, telecom.03:17
lifelessI will ring and rant soon03:17
StevenKDo you hold out much hope?03:18
wallyworldlifeless: before and inside the call, the feature controller is <lp.services.features.flags.FeatureController object at 0xf7d4b10>03:19
wallyworldit fails without all the other print statements03:19
wallyworldi'll see if i can find which print statement makes it work03:20
lifelessStevenK: if we can id the issue, yes03:22
lifelessthats mainly dependent on getting through first level 'technical' support03:22
StevenKOh, absolutely03:23
lifelesswhich the rant is all about03:24
wallyworldlifeless: so calling features.getAllFlags() before the check_permission call makes it work (as well as bug.default_bugtask)03:25
wallyworldand not calling either of those 2 things makes it fail03:25
wgrantjtv: There's a regression fix for cocoplum that I'd like to deploy tonight, and it's stuck behind your translations-export-to-branch fix. Are you likely to have QA for that done in the next 4 or so hours?03:25
jtvwgrant: I think I will, but it won't be much sooner.03:26
wgrantOK. I may have to cowboy it anyway, since there's an existing possibly unclobberable cowboy there.03:27
wallyworldlifeless: narrowed it down to feature_controller.rule_source.getAllRulesAsDict() - so it seems the StormFeatureRuleSource() content is getting clobbered somehow?03:32
lifelessI wonder if you can't trigger the first feature rule lookup from within a security adapter because they don't nest? [speculation]03:35
wallyworldnot sure03:38
wallyworldbut it all seems rather fragile at the moment03:38
lifelesswgrant: what do you think about my speculation ?03:39
wgrantlifeless: What don't nest?03:39
lifelesswgrant: I'm trying to come up with a story that would explain wallyworlds symptoms03:40
wallyworldweird thing is that bug.default_bugtask also makes it work03:41
lifelessyou probably need to step through with pdb03:41
wallyworldyeah, started to do that. lots of api calls to look at03:42
lifelesszope is phat03:42
wallyworldat least it's not something dumb i'm doing wrong (hopefully)03:42
wallyworldseems like a genuine problem with the infrastructure03:43
wgrantlifeless: Are you sure the mailman test bug is actually a bug? We deliberately don't run MailmanLayer by default, because it's crap.03:46
lifelesswgrant: we have tests that exist; they should be run, or not exist.03:50
lifelesswgrant: otherwise they -will- bitrot and -will- just accumulate debt03:51
StevenKDelete them, then03:51
lifelessI found a bug that they were not being run, but it was fix released years ago indicating they were meant to run again.03:51
lifelessStevenK: not without chatting to curtis I think03:51
StevenKI wish I had more knowledge so we could delete lib/mailman03:52
wgrants/I/LP/ s/more knowledge/some architecture/04:08
lifeless6 uses of getLastOops04:10
lifelessmwhudson: hey, around ?04:17
lifelessmwhudson: have a quickie on codehosting04:17
mwhudsonlifeless: yep04:18
lifelessmwhudson: make_error_utility appends the pid to the oops prefix04:18
lifelessis this for any reason *other* than oops sucking at concurrency04:18
wgrantI think we established that there isn't.04:18
mwhudsonlifeless: no, i'm pretty sure that's only to avoid races04:18
wgrantIt's there to avoid concurrency issues, and to confuse the shit out of everyone.04:18
lifelessmwhudson: as the processes are ephemeral, I'm checking theres not need to keep that04:18
lifelessmwhudson: great, deleted.04:18
lifelesswgrant: if you want confusing04:19
lifelessgrep for setOopsToken and note the EMAIL usage04:19
lifelessmwhudson: I'm assuming pullerworker is the same ?04:19
wgrantHuh. Handy.04:19
mwhudsonlifeless: yes04:19
lifelesswgrant: do you happen to know if thats semantic or just crazy04:20
wgrantlifeless: I assume crazy.04:20
wgrantBut don't know for sure.04:20
wgrantDelete it and see if matsubara complains? :)04:21
lifelessaieee @ available_oops_prefixes04:25
lifelessoh man04:27
lifelessI hope its not punning that with concurrency limiting04:27
wgrantlifeless: I think you may have projectegg set badly in python-oops-tools04:38
wgrantIt currently tries to use oops-tools.settings as the settings module.04:38
wgrantWhich obviously isn't going to work :)04:38
wallyworld_lifeless: i've found the problem. the setAllRules() method on StormFeatureRuleSource needs a store.flush(). Or else the rules passed into the fixture setup are not written to tge db because check_permission() has a @block_implicit_flushes decorator04:44
wallyworld_and those other things which trigger the test tp pass must have done so because they caused a flush04:45
lifelesswgrant: thats fixed by my branch04:45
lifelesswgrant: you're welcome to review it if you want04:45
=== wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 262
StevenKwgrant: Hm?04:49
wgrantThe number went down :)04:49
StevenKBy 704:49
wgrantIt was actually 272 this morning.04:49
StevenKOh, so 1004:50
mwhudson26 days to go!?04:55
StevenKmwhudson: You tell funny jokes04:57
StevenKI think I have to file one, anyway04:57
StevenKI can't see DSP:+questions in our bugs04:58
mwhudsonwgrant: is it the ppa publisher that's backed up?05:01
wgrantThe queries are quick, so it's unlikely to time out much.05:01
wgrantmwhudson: Was, but yes.05:01
wgrantmwhudson: It's been fixed for a few hours.05:01
wgrantAnd now even has scriptmonitor running on it.05:02
mwhudsonwgrant: ok, how often does it run when it's not backed up?05:02
wgrantEvery 5 minutes.05:02
wgrantBut often only every 10.05:02
wgrantBecause it's crap.05:02
* mwhudson has a vague memory of */2005:02
wgrantIt was */20 long ago05:02
lifelesswgrant: you've been spelunking a lot; whats the fastest way to tell if script X has an oops config05:36
=== almaisan-away is now known as al-maisan
lifelesswgrant: you've been spelunking a lot; whats the fastest way to tell if script X has an oops config06:02
wgrantlifeless: Somewhat disturbingly, despite porting dozens of scripts around to LaunchpadScript and rewriting its internals, I've not run into that bit of code.06:03
lifelessI want to check the setOopsToken('EMAIL') thing is safe when gone, if you see what i mean06:03
wgrantOh, that's lovely.06:05
wgrantScripts normally just call globalErrorUtility.configure('something') themselves.06:05
lifeless1874 lines of diff06:12
lifelessand we're not done yet06:12
lifeless+ it would be freakishly hard to make this separate branches06:12
lifelessI pity the fool^Wreviewer06:13
lifelessStevenK: oops -> critical06:14
lifelessStevenK: also, we don't use confirmed :)06:14
=== al-maisan is now known as almaisan-away
lifelesspoolie: hi, can we talk about your pending writes branch briefly ?06:50
pooliehere, or phone?06:50
lifelesseither, whats your pref ?06:50
poolielet's start here06:51
lifelessso, the bug (as I read it) is that when someone pushes twice in quick succession, we don't update the merge diff properly06:51
lifelessthere are a few different orders to the race condition06:51
lifelesssometimes we generate the error and update proplerl06:51
pooliei think that's how you reach it yes06:51
poolieyes it seems so06:52
lifelesssometimes we generate the error and don't update properly06:52
pooliethat may be possible06:52
lifelessI'm worried that you're papering over the issue06:52
pooliei can understand that concern06:52
pooliei think there are really two bugs06:52
poolie1- "sometimes mp diffs are not generated if the branch is repeatedly written to"06:53
poolie2- "launchpad sends pointless spam"06:53
pooliei'm trying to fix 206:53
pooliei'm not sure if 1 actually exists06:53
lifelessI'm sure it does06:53
lifelessper the analysis in comment #206:53
pooliei thought that perhaps the completion of the second write would cause a new job to be generated06:55
poolieperhaps there is some ordering where that doesn't happen06:55
lifelesscan only have one job outstanding for the branch06:55
poolieat any rate i don't see how leaving bug 2 open helps us fix bug 106:55
poolieat the moment we don't even log when this occurs!06:55
lifelessso if the first job hasn't finished erroring before the second job is created, the second job isn't made and the first job just fails.06:55
lifelesspoolie: we don't generate an OOPS ?06:56
poolieit is telling only the users who can't do anything about it06:56
poolieunless the idea is to annoy them (me) into fixing the whole bug :)06:56
pooliewhich is a valid, though risky, strategy06:56
lifelessso, I think bug 1, which is the bug your branch purports to be about, is about fixing the race condition06:57
_mup_Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Confirmed for compscibuntu-bugs> <dylan.NET.Reflection:Invalid> <dylan.NET:Invalid> <EasyPeasy Overview:Invalid by ramvi> <GenOS:In Progress by gen-os> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <LibreOffice:In Progress by bjoern-michaelsen> <Linux:New> <Linux Mint:In Progress> <The Linux OS P06:57
lifelessbah, 1-06:57
pooliemy mp is only about suppressing the mail06:57
pooliei get annoyed by the mail but i never see a missing diff06:57
lifelessAnd also, issue 1- is the only one where the user has no control over the situation06:57
pooliebecause they can delete/filter the mail?06:57
lifelesspoolie: because they can push content into the branch, or delete the mp if they had done something crazy06:58
lifelessI admire your desire to stop sending spam, but I don't think, except for case 1-, that these branch mails -are- spam06:58
lifelessand case 1- has an analysis of the race condition, just needs coding06:58
poolielots of people seem to disagree :)06:58
lifelesspoolie: not on that bug06:59
lifelesspoolie: in general, 'lp sends too much mail', sure : but telling you something you requested fails is useful06:59
poolieit doesn't tell you what failed06:59
poolieas james said "I don't really know what it means, which merge proposal it is referring to, or what06:59
poolieI can do about it, so I don't know why I got an email about it."06:59
poolielp really should not be sending that07:00
poolieit's different to bugmail07:00
lifelessso, the other bug, which I've unduped, is about the lack of context07:00
pooliewhich one?07:00
lifelessfixing that will address some of james_w's mystery around the mail07:00
lifelessbug 64088207:00
_mup_Bug #640882: " Launchpad error while generating the diff for a merge proposal" mails don't indicate branch <code-review> <email> <lp-code> <Launchpad itself:Triaged> < https://launchpad.net/bugs/640882 >07:00
pooliei'll just drop it07:00
pooliei'm sad because i was trying to make this a little less crap and i feel like it's being held hostage to fixing the whole thing07:01
poolienot sending pointless mail to people is a step forward07:01
poolierecording when something goes wrong is a step forward07:01
lifelessI'd love to see improvements here, I don't think masking the issue is one; fixing the issue (which should be ~ as simple as self.suspend(5 minutes) or something) would be07:01
lifelesspoolie: I agree that not sending pointless mail is a step forward; and recording when it goes wrong is a step forward.07:01
pooliehow is this masking it?07:01
lifelesspoolie: My understanding was that you were going to squelch the email for this case, and that that was the sum of the branch07:02
poolieand i was going to log that it failed07:02
poolieok, if just doing self.suspend(5 minutes) is enough, i'll try that07:03
lifelessI'm handwaving07:03
lifelesspoolie: jobs have a defer-for-a-bit system, I don't know the details.07:04
pooliei don't feel you and aaron are taking into account the actual user data here07:04
lifelesspoolie: I think for the pending-writes case, logging and not emailing is fine; I agree with Aaron that the other cases are different enough not to change.07:04
poolienobody is saying "i'm glad lp told me about this" or "that explains why my thing had no diff"07:04
lifelesspoolie: I think fixing the issue, logging and not emailing is even better07:04
lifelessI feel like you are saying 'not getting email is more important than the system working'07:05
lifelessI know thats not what you mean07:05
lifelessbut it kindof feels that way07:05
=== jam1 is now known as jam
lifelessI think you mean 'not sending email in this case is better even if its not fixed', and I've acked that - twice I think - above07:06
lifelesspending writes shouldn't be categorised as a user error07:06
pooliei get more annoyance from lp spam than i do from diffs being missing07:06
pooliein that sense it's more important07:07
poolieand, generally, there are always going to be some errors, and i think handling them gracefully is important07:07
lifelessI get annoyance from devs having to spend time tracking down, *again*, a self inflicted case of user confusion07:07
pooliewhy 'self inflicted'?07:07
lifeless(self inflicted by us developers)07:07
poolieoh i see07:08
lifelesspoolie: because we created a system with a race condition, classified it as user error, and tada07:08
poolieand, i think, did not look at the actual mail that was sent07:08
lifelessthis needs two changes: unclassify it as user error, and fix the race condition07:08
lifelessand yes, the lack of context in the mail is the icing on the cake07:08
poolieif the race is as simple as just rescheduling the job i can do that07:08
lifelessI think that if the branch has pending writes, the job should just wait for it07:09
pooliei guess the 'lack of context' bug can then apply to other mail sent about branches, if any07:10
lifelesspoolie: there are, IIRC, 3 other cases for MP's where the same template is used for the email07:10
pooliei agree, though i think the "users need to trust whether lp is working" argument applies equally there07:10
lifelesspoolie: cases which this bugfix won't impact07:10
lifelesspoolie: well, pushing up an empty branch and proposing it for merge, *is* a user error07:10
lifelessif you get one of those mails, I think its helpful (if it told you the branch :))07:11
poolieyes, bug 640882 will be irrelevant to the specific case it complains about, but relevant to things like empty branches07:11
_mup_Bug #640882: " Launchpad error while generating the diff for a merge proposal" mails don't indicate branch <code-review> <email> <lp-code> <Launchpad itself:Triaged> < https://launchpad.net/bugs/640882 >07:11
pooliethis would have been a lot easier if someone had just said "why don't you just call self.suspend and that will probably fix it"07:12
pooliein the first place07:12
lifelesspoolie: that would have been nice07:13
lifelessunderstand I'm handwaving, bug there is something like that there :)07:13
lifelessand I'll be happy (tomorrow) to go spelunking with you looking for it07:13
poolieit's probably fairly obvious on the base class07:14
poolieso then no oops, just deferral07:14
poolieand maybe a log message07:14
jtvwgrant: Q/A for those codehosting translations-export bugs is done.  Go ahead.07:21
wgrantjtv: Thanks.07:21
poolielifeless: most of the bugs have all the issues of "no context" and "shouldn't get this mail anyhow" tangled together07:22
poolieplease don't undupe them all07:22
lifelesspoolie: there were two that were previously a unit, and you'd moved to the other bug, I was just restoring the,07:22
lifelesspoolie: (i.e. I've no more tweaking planned on these bugs)07:25
StevenKlifeless: The bug I filed is not the cause of the OOPS -- that is already filed.07:26
StevenKlifeless: I used High since the bug is *shown* in the OOPS, but isn't the cause.07:26
lifelessStevenK: ah, that wasn't clear to me. Sorry for creating noise.07:26
StevenKlifeless: Should I set it back to High, then?07:26
lifelessStevenK: up to you; lazy evaluation and timeouts can be nonobvious - we may well have timeouts due to that bug anyhow07:27
lifeless(it is a timeout isn't it ?)07:27
StevenKlifeless: Yes, but the timeout is due to the direction=backward madness07:27
lifelessah right, a clear cause :)07:28
StevenKjtv: Bug 375013 is not marked OK, but 812500 is07:28
_mup_Bug #375013: Cannot commit directly to a stacked branch <commit> <launchpad> <lp-translations> <qa-ok> <rodeo2011> <stacking> <Bazaar:Fix Released by jameinel> <bzr-builder:Fix Released by jelmer> <Launchpad itself:Fix Committed by jtv> < https://launchpad.net/bugs/375013 >07:28
jtvStevenK: I guess one of my changes didn't come through.  Hang on.07:28
poolielifeless: i think the other thing here is https://bugs.launchpad.net/launchpad/+bug/48394507:29
_mup_Bug #483945: No way to ask Launchpad to refresh a stale diff <code-review> <lp-code> <mp-preview-diff> <openstack> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/483945 >07:29
poolieto give people a way tor ecover07:29
lifelesspoolie: that would be nice07:29
jtvStevenK: actually, it did come through.  So the deployment report simply hasn't picked it up yet.07:29
jtvStevenK: the one that's not marked OK yet is the one I updated last, IIRC.  Here's hoping this is not a problem with multiple bugtasks.07:30
StevenKCan I get a review? https://code.launchpad.net/~stevenk/launchpad/dsp-questions-statement-death/+merge/7951907:53
adeuringgoood mornin08:04
StevenKadeuring: Hi, I know it's not your OCR day, but would you mind a small (+22/-5) review? https://code.launchpad.net/~stevenk/launchpad/dsp-questions-statement-death/+merge/7951908:07
lifelesswgrant: hi08:07
lifelesswgrant: https://code.launchpad.net/~lifeless/python-oops-tools/amqp/+merge/7950508:07
lifelessadeuring: StevenK: I've reviewed that branch08:07
lifelesswgrant: baaah08:07
StevenKOh, have you?08:07
lifelessStevenK: https://code.launchpad.net/~lifeless/python-oops-tools/amqp/+merge/7950508:07
* StevenK refreshes08:07
StevenKlifeless: Haha08:08
StevenKlifeless: Fine, I'll look at your MP then. :-P08:08
StevenKlifeless: If you look at the OOPS attached to the bug it did 1,200 SPN queries08:09
adeuringStevenK: anyway, a nice branch!08:10
lifelessStevenK: yes, crazy shit man08:10
StevenKadeuring: :-)08:11
StevenKlifeless: r=me08:14
stubHow do I set the submit branch?08:15
StevenKbzr push --remember08:16
wgrantThat's the push branch.08:16
wgrantwgrant@lucid-test-lp:~/launchpad/lp-branches/bug-876171$ grep -A1 lp-branches.$ ~/.bazaar/locations.conf08:17
wgrantsubmit_branch = bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel08:17
lifelessthe default submit branch is the parent branch isn't it ?08:17
wgrantI think that applies in some places but not others.08:17
stubI gotta weird one from an old branch and nothing in locations.conf08:17
wgrantstub: .bzr/branch/branch.conf?08:18
stub  submit branch: bzr+ssh://bazaar.launchpad.net/%2Bbranch/losa-db-scripts/08:18
* stub looks08:18
=== almaisan-away is now known as al-maisan
* StevenK peers at the e-mail he just got from PQM08:19
stubyer - in there.08:19
wgrantStevenK: Did lifeless set the wrong default reviewer?08:19
StevenKPQMException: 'Failed to verify signature: gpgv exited with error code 208:19
StevenKI don't think I caused that08:20
* wgrant looks.08:20
wgrantYou sent email to it.08:20
* wgrant fixes.08:20
StevenKOh, the project is broken.08:20
wgrantThe branch, specifically.08:21
nigelbMorning everyone08:52
nigelbOn second thoughts, evening :)08:52
rvbalifeless: Hi Rob, may I email you with the details of a test isolation failure I'm fighting with?  Maybe you'll have an idea on what's going on.08:59
rvbaThanks :)09:02
rvbaThis is on a branch that is really not urgent.09:02
lifeless\o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/09:07
lifeless(I have a loopback test grabbing oopses off of amqp into TestCase.oopses09:07
wgrantoh, nice.09:08
lifelessthe end is in sight09:08
lifelessrvba: I will look tomorrow09:11
rvbalifeless: Thank you.09:11
lifelessrvba: if you want to look in the interim, I'd question the state that rabbit is in when the test starts09:11
rvbalifeless: hum, right, I'll have a look.09:12
lifelessrvba: I haven't looked at your failures yet but09:13
lifelessI note that 'session' is a thread local09:13
lifelessrvba: and they are generally trouble09:14
lifelessrvba: in particular, I don't see anythin in RabbitSession to handle rabbit going down09:14
nigelbbigjools: lol @ cricket :D09:14
lifelessrvba: currently, as I read the code, the first reset of rabbit will break all LP appservers trying to use rabbit09:15
lifelessbigjools: ^ you might like to file a bug on that, for fixing before we go love.09:15
bigjoolsnigelb: meh09:15
lifelessbigjools: (or tell me I'm wrong :))09:15
rvbalifeless: that's the reason why we wanted to isolation rabbit's failures as much as possible.09:16
bigjoolsnigelb: it's probably the dodgy food upsetting them09:16
lifelessrvba: sure, but we need to handle things like rabbit being reset09:16
lifelessrvba: the problem is that RabbitSession assumes it *never* goes away, but the Unreliable variant assumes that *any error doesn't matter*09:17
lifelessrvba: there is a middle ground09:17
lifelesswhere an EPIPE is handled by reconnecting, but other errors cause a failure09:17
rvbalifeless: true, but since we don't know where this middle ground is we wanted to use the Unreliable variant and watch what was happening (hence the agressive oops logging).09:18
rvbalifeless: that's the purpose of the branch in question.09:18
lifelessok, so I can give a -little- guidance (I'm a novice here too :P) about the middle ground09:19
lifelessthere are two broad cases I see09:19
lifelessfirstly if rabbit goes away and comes back, the first transaction *after* it comes back it should start working09:20
lifelessthis is testable (see the oops_amqp tests for inspiration)09:20
lifelesssecondly, if rabbit goes away and comes back, messages queue for send-at-end during the current transaction should also be sent09:20
lifeless(also testable :P) - only if it goes and stays gone, should we be going into zomg mode09:21
lifelessrvba: I think aggressive logging and oopsing is great as well09:21
lifelessrvba: these two things just seem like high frequency cases we can anticipate09:21
rvbalifeless: makes sense.09:21
lifelesswgrant: btw the rabbit-management package should really put the cli on sbin09:23
rvbalifeless: I'll file bugs about these two cases.09:23
lifelessrvba: thanks!09:23
rvbalifeless: thank *you* ;)09:24
lifelessrvba: I have a minor tweak to rabbit.py in lp:~lifeless/launchpad-useoops - dunno when the branch will land, but thought a headsup might be useful09:24
rvbaOkay, I'll have a look.09:24
wgrantlifeless: Probably, yeah.09:25
lifelessfor now, -> bed09:25
wgrantlifeless: File a bug against the PPA oh wait :)09:25
rvbaNight lifeless.09:25
lifelesstomorrow I shall wire up subprocesses sending to amqp09:25
lifelesscall sync() on the 7 uses of getLastOopsReport09:25
lifelessand purge purge purge purge purge purge09:26
mrevellHey henninge, how's it going?11:00
henningeHi mrevell!11:00
henningemrevell: Going well, thanks!11:01
mrevellGood to hear :)11:01
henningemrevell: I have been looking for a place to find out who was hired to replace me. Wasn't there a squad listing on the dev or help wiki?11:04
mrevellhenninge, That person hasn't joined yet.11:04
henningehard to replace me, I know ...11:04
mrevellhenninge, As for the page... let me look11:04
mrevellheh, true true :)11:04
mrevellhenninge, There's this: https://dev.launchpad.net/Squads11:05
henningeoh, that simple11:05
henningestupid moin, searching for "squads" does not yield that page.11:06
henningeah, case-sensitive search11:06
henningemrevell: thanks! ;)11:06
mrevellhenninge, Heh. We really need to sort out the dev wiki. We have a new Usability and Communications Specialist joining soon who will help us with that.11:07
=== al-maisan is now known as almaisan-away
=== benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: benji | Critical bugs: 262
=== almaisan-away is now known as al-maisan
=== stub1 is now known as stub
deryckMorning, everyone.13:02
abentleybenji: could you please review https://code.launchpad.net/~abentley/launchpad/mustache-bugs/+merge/79441 ?13:16
benjiabentley: gladly13:16
bigjoolsanyone else notice that DatabaseLayer is massively slower to start up these days?13:22
deryckadeuring, ping for standup.13:31
adeuringderyck: ok, soory13:31
abentleyderyck: https://code.launchpad.net/~abentley/launchpad/mustache-bugs/+merge/7944113:35
deryckabentley, got it, thanks.13:51
benjiabentley: the branch looks good; I did note one small thing I think you'll want to change14:24
abentleybenji: what's that?14:24
benjiabentley: I suspect you want to remove the "<em>Server-side mustache</em>" and "<em>Client-side mustache</em>" bits as well as making the client/server rendering conditional on the feature flag.14:24
abentleybenji:   The client-server rendering is already conditional on the feature flag.  I don't want to remove the multiple copies yet, because I need to be able to QA all the renderings.14:26
abentleybenji: See line 450 of the patch for where the rendering is made conditional.14:27
benjiabentley: right, but it renders both versions; does the "normal" version get rendered if the feature flag is off?14:29
abentleybenji: Yes, the normal version gets rendered regardless of the feature flag status.14:29
benjiif so, I guess the feature flag is (at least at this moment) more about being able to do in-production QA and not about being able to really turn the feature on and off14:29
abentleybenji: Right, the feature is only for the team developing it at the moment.14:30
benjiabentley: k; I might have made that conditional on a query string instead of a feature flag, but I've heard that some people aren't me ;)14:31
abentleybenji: This is part of an actual feature that is planned to take 4-6 weeks to complete, and the flag will prevent work on that feature from affecting normal users while we are working on it.14:32
=== matsubara is now known as matsubara-lunch
sinzuigary_poster, bigjools: can either of you allocate someone to fix bug 87013014:36
_mup_Bug #870130: shortlist error requesting recipe build <easy> <oops> <recipe> <Launchpad itself:Triaged> < https://launchpad.net/bugs/870130 >14:36
jelmerhi benji, can I add another branch to your queue?14:36
benjijelmer: certainly14:36
bigjoolssinzui: I can14:37
gary_posterbigjools, if someone on your side could grab it...thank you bigjools :-) .14:37
=== al-maisan is now known as almaisan-away
gmbCan someone who knows something about the lazr.restfulclient .deb confirm or invalidate this bug: https://bugs.launchpad.net/lazr.restfulclient/+bug/87644514:39
_mup_Bug #876445: missing lazr.uri dependency in the debian/ubuntu package <lazr.restfulclient:New> < https://launchpad.net/bugs/876445 >14:39
gmb(Or tell me how to)14:39
benjigmb: I don't know anything about the .deb but if it doesn't include a dependency on lazr.uri, then it should14:41
gmbbenji: Agreed, but I don't know where to look to check. I could grab the source package I suppose...14:42
gmbbenji: And whaddaya know, it doesn't have that dependency.14:43
jelmerbenji: Thanks - the MP is @ https://code.launchpad.net/~jelmer/launchpad/stacked-code-imports-newer-bzr/+merge/7953814:43
rvbabenji: Hi, could you please review this js branch? https://code.launchpad.net/~rvb/launchpad/bug-869221-archindepwidget/+merge/7956114:44
benjirvba: sure14:44
gmbbenji: See, I like this arrangement, other people get to do the thinking and I look like I've done some actual work.14:44
gmbThanks :)14:44
rvbabenji: Thank you.14:44
benjigmb: I do too, I just have to type and other people do the work14:45
gmbbenji: It's symbiotic development at its best.14:46
benjigmb: I'm reminded of this xkcd: http://xkcd.com/722/14:46
gmbHeh :)14:46
sinzuijcsackett, do you have time to mumble in the new purple channel14:51
* jcsackett goes to find the new purple channel.14:52
gary_postersinzui, hi.  Thank you for the html5browser review and help.  On a slightly related subject, Francis wanted me to offer my services to your squad to help you with the new yuixhr tests, if that is of value.  They should be of particular interest to feature squads.  I'm happy to help however you think might be appropriate, including phone calls and/or more/better documentation.14:57
gary_posterOnce the html5browser changes are live, I can (try to re-) land some simplifications and improvements to the yuixhr usage.  After that would probably be the right time to really dig in.14:57
gary_poster(That's all I had to say; just making the offer.)14:57
sinzuigary_poster, otp14:58
nigelbsinzui: You guys did take the name I suggested!15:29
nigelb(I suggested Teal Assasins) :P15:30
sinzuinigelb, Then my thanks go to you and wallyworld_ who contributed the colour15:30
nigelbI'm still laughing uncontrollably :P15:31
=== matsubara-lunch is now known as matsubara
flacostesinzui: shouldn,t it be Aubergine Assassins ;-)16:02
sinzuigary_poster, bigjools: We have a spam project in the review list given to ~registry. The registrant must be suspended too. There may be other spams projects I did not see in my commercial view.16:02
gary_postersinzui, bigjools, I'll take it16:03
nigelbflacoste: heh16:03
bigjoolsDammit I wanted to rename to purple!16:03
gary_posterI'll start my CHR run in a few, and include that in it16:03
sinzuiflacoste, I pondered that. I didn't want people thinking we we will fix every Canonical stakeholder issue16:03
sinzuiWe can fix community bugs too16:04
nigelbflacoste: if they were named aubergine, you could sacrifice that squad to stakeholders :P16:08
abentleybigjools: You can rename to aubergine.16:09
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
bigjoolsI shall think of better16:09
sinzuibigjools, Teal is available now16:12
bigjoolsflacoste: oooo get you16:12
bigjoolssinzui: lol16:13
bigjoolsTaupe Twits16:13
nigelbbigjools: *cough* cricket :P16:14
nigelbbigjools: You can still take "Men in blue"16:14
bigjoolsnigelb: how many games did you win over here all summer, again?16:14
=== beuno is now known as beuno-lunch
mrevellNight all17:10
lifelesssinzui: you might like to rename your lp team17:42
lifelessfrom 'green'17:42
sinzuilifeless, I renamed the teal team before the annoucement17:43
sinzuiis there a green team?17:43
lifelesssinzui: there is a wiki link17:43
lifelessI saw it in one of the pages you edited17:43
sinzuiI will hunt those down17:44
=== beuno-lunch is now known as beuno
lifelessflacoste: yo18:37
flacostehi lifeless18:41
flacostelifeless: do you want to talk now?18:42
lifelessthat would be cool :)18:42
* deryck goes offline for long late lunch, back later19:20
micahgbenji: got a minute to chat about bug 876594?  I think this has the potential for ill will and is a regression in this case19:45
_mup_Bug #876594: rejected builds for synced packages send mail to Debian maintainer <derivation> <Launchpad itself:Triaged> < https://launchpad.net/bugs/876594 >19:45
benjimicahg: sure, I'm glad to discuss it.  (Thanks for letting me know it's a regression, I'll update the bug.)19:46
micahgbenji: I think it is, cjwatson could probably verify19:46
cjwatsonmicahg: sounds like it, yes.  We should not be mailing Debian maintainers for activity in Ubuntu just because their name was on the package we synced20:03
cjwatson(to see why not, imagine the full set of Debian derivatives)20:03
micahgbenji: yeah ^^, so what you did was fine, thanks20:04
benjimicahg: my pleasure20:05
micahgcjwatson: thanks20:05
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
=== benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 26
huwshimiMorning all22:06
=== lifeless changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 26
=== StevenK changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 273
elmoare you guys ever going to implement the critical/high split?  the number in /topic is depressing me and I'm not even on the LP team22:33
StevenKelmo: I hope so. Everytime I look through the Critical list, I get depressed.22:33
=== sinzui changed the topic of #launchpad-dev to: Performance Tuesday! | https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 273
lifelesselmo: what critical/high split ?22:37
elmolifeless: I thought there was discussion on splitting the 273 criticals into critical/high, with critical being reserved for the genuinely holy-crap-wtf crticials22:38
lifelesselmo: there is discussion, there isn't resolution22:39
lifelesselmo: we're analysing *why* we are generating so many new criticals22:40
lifelesselmo: thats a crucial step in addressing the issue, because we're fairly good at closing them.22:40
elmolifeless: *shrug* k - just seems crazy to me - you de facto can't have 273 criticals, some of those must be more critical than others22:42
elmoi.e. the ones you're working on now and next22:42
elmorather - you de facto can't have 273 criticals which are actually critical by the definition most people understand 'critical' to mean22:43
lifelesselmo: so we have a policy that says things that are broken are critical, plus things stakeholders have escalted are critical22:43
lifeless-all- the criticals meet one of those definitions22:43
lifelessyes its a problem, yes folk can't be working on 273 things in parallel, but the root issue here is that we are breaking things faster than we're fixing them22:44
lifelessif we shuffle all our highs to medium/low criticals to high and then pick a subset of brokenness to treat as critical, the problem doesn't get any better22:46
elmoI'm not saying that's not an issue22:46
elmobut I think it's orthogonal to the issue I'm unsolicitedly whining about22:46
elmoyou could still figure out the breaking faster than fixing thing, if stuff got shuffled, e.g. by treating escalations as high22:47
lifelesspolitically, we want to work on escalations before we've fixed all thats broken22:47
lifeless(and escalation are ~5% of maintenance work atm, IIRC)22:47
elmook, sure22:48
elmoI guess what I'm saying is - all the other stuff aside, having non-now/next items in critical strikes me as insane, and doing it over such a long period of time has to be demotiviational22:49
elmobut *shrug*, that's just MO22:49
elmoI'll shut up and go back to my own criticals ;-)22:50
lifelessso, what is non-now/next in the critical bucket ?22:50
elmoI'm talking non-now/next in the kanban sense22:51
elmoat least AIUI22:51
elmowhich means you can't possibly have a capacity that would allow you to fit all 273 into now + next22:52
elmowhich means you've got X on the go and X that you'll be doing next22:52
elmoand that X+X << 27322:52
lifelessright, kanban wise if each engineer has a WIP limit of 2, now+next = 10*2*2 -> 40 items22:52
elmowhere did 10 come from?22:53
lifelessbut that aspect of kanban w.r.t. software development is primarily about directed-effort, this is category..22:53
lifelesssquad size 5, maintenance squad count 222:53
lifelessso squads * members * WIP * (now + next=2)22:55
StevenKEr, we have 4 squads?22:56
lifeless2 on maintenance22:56
lifelessfeature squads are not working from the critical queue22:56
StevenK[09:53] < lifeless> squad size 5, maintenance squad count 222:56
lifelessStevenK: yes?22:57
StevenKOh, the number of people in the squad22:57
* StevenK goes back to breaking the archive.22:57
jelmerhah, there is my punishment for deprecating Branch.revision_history23:02
jelmerlots of things in Launchpad seem to rely on it23:02
wallyworld_thumper: that utf-8 branch mail patch is deployed. can you let me know if you see any issues come up23:03
lifelessjelmer: also pqm23:04
thumperwallyworld_: awesome, thanks23:04
lifelessjelmer: would love a patch for that too23:04
wallyworld_thumper: np. and thanks for not mentioning the rugby :-(23:04
thumperwallyworld_: np23:04
lifelesselmo: when we restart rabbit in the dc, how long does it take to come back up?23:11
lifelesselmo: also, when we restart a server of the type its going to be on, how long would it be down for?23:11
elmolifeless: I start to get worried if a DL380 takes > 100s to come back23:12
lifelesselmo: I'm writing some code that I'd like to be resilient to rabbit getting maintained, but to not assume it will always come good eventually23:12
lifelesselmo: so I'd like to have a timeout on its retry-connections duration, after which it will bail out and exit23:12
lifelesselmo: does this sound reasonable /23:12
lifeless(its the suck OOPSes from AMQP and push them into the oops-tools database code)23:12
elmolifeless: restarting rabbit on a random staging box took ~4s23:13
lifelessok, so ignoring truely exception things like box rebuilds, where we'd reconfigure to a different rabbit anyway, 2 minutes as a timeout seems sufficient ?23:13
lifelesselmo: and does this approach sound ok from your ops perspective, or would you rather just fail-hard, or never-fail policies ?23:14
elmolifeless: is this oops, or in general?23:14
lifelessthis specific code is in oops_amqp so yes, oops specific23:14
lifelessfor the LP appservers we have clear transaction boundaries we can retry on, and disable rabbit if its not up at the start of the transaction23:15
lifelessI guess the txlongpoll service has/needs something similar to this as well - i haven't audit its code.23:15
lifelesswgrant: what does the txlongpoll twistd daemon do if rabbit is away / goes away ?23:15
elmolifeless: does this mean that an app server could stall for up to 2m trying to talk to rabbit?23:16
lifelesselmo: no23:16
lifelesselmo: for oopses, the sender fails fast23:16
lifelesselmo: this is for the slave, the oops-tools side of it23:16
wgrantlifeless: Not sure.23:17
lifelesselmo: which has nothing tickling it to say 'try again now' because its only interface is rabbit pushing stuff to it23:17
elmolifeless: so the thing trying to consume off the queue?23:17
elmohow would a 'never-fail' policy work?23:17
elmoinfinite retry?23:17
lifelesswhich makes me nervous :)23:18
elmook, I think I prefer the 2m-timeout or fail-hard23:18
elmoI don't mind which23:18
elmoI guess 2m-timeout is a little more forgiving23:18
lifelessjust means a little more self healing23:18
lifelessone less zomg chase after a reboot of rabbit23:18
elmoself-healing would be nice23:18
elmobecause apparently I just broke U1 staging23:18
elmowith my test restart of rabbit23:18
lifelessits bug filing time :)23:19
elmoso, if we could avoid that same brain damage in LP, I'd appreciate it23:19
lifelesswe filed bugs about this for the new stuff last night, when I read the current code23:19
lifelessas I say, I haven't checked txlongpoll yet23:19
lifelessits probably ok, if its written naively, it will only connect when a js session connects, and not share channels/connections, which scales poorly but would make it robust around this23:20

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!